You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi all, I would like to know what "do_sample" does in the generation settings and why memory consumption increases after turning it off. With do_sample turned off it is almost impossible to generate on 12gb video memory and llama-13b-4bit. I get oom:
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 12.00 GiB total capacity; 11.15 GiB already allocated; 0 bytes free; 11.24 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
What does "do_sample" do?
Hi all, I would like to know what "do_sample" does in the generation settings and why memory consumption increases after turning it off. With do_sample turned off it is almost impossible to generate on 12gb video memory and llama-13b-4bit. I get oom:
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 12.00 GiB total capacity; 11.15 GiB already allocated; 0 bytes free; 11.24 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONFBeta Was this translation helpful? Give feedback.
All reactions