Can someone create a 4 dummies WSL install?

[bbecausereasonss]

Tried installing, and then went onto the 4bit installation, only to receive one error after the next. No torch, no cuda. I'm lost.

[ye7iaserag]

I have a docker image that I run over WSL, would that be beneficial for you?

[bbecausereasonss]

That would be great! Sent from Mobile.

…

On Thu, Mar 16, 2023, 11:40 PM ye7iaserag ***@***.***> wrote: I have a docker image that I run over WSL, would that be beneficial for you? — Reply to this email directly, view it on GitHub <#373 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE2TIUEPVWKEI5GU5DTLE5TW4PMKBANCNFSM6AAAAAAV6AR5JM> . You are receiving this because you authored the thread.Message ID: <oobabooga/text-generation-webui/repo-discussions/373/comments/5341166@ github.com>

[ye7iaserag]

Yes and you get the local repo from nvidia developer site
https://developer.download.nvidia.com/compute/cuda/11.7.1/local_installers/cuda-repo-debian11-11-7-local_11.7.1-515.65.01-1_amd64.deb

[DeathwingTheBoss]

@ye7iaserag Any ideas as to why docker is killing the container while loading the model?

[ye7iaserag]

If you are using deepspeed then yes, I don't think it works on docker or WSL
Else you might not have enough memory
What's ur PC specs? and what command are you using in the entry point? (you have to uncomment one to start else it will finish excuting and close)

[DeathwingTheBoss]

I have managed to debug and fix it. WSL by default, limits the amount of RAM Docker can use (since it's basically WSL) and if it runs out of RAM while loading the model (llama 30b, in my case) it just dies.

By editing the .wslconfig and increasing WSL's allowed RAM to >26GB, I was able to solve this problem. But this does mean you would need ~32GB RAM. Maybe SWAP is possible, but I don't know.

[ye7iaserag]

llama 30b 4bit does not work on my side since it tries to load more than the 12GB vram I have, offloading to cpu not working for me not sure why
And I do not remember having to edit .wslconfig
You can use nvme offloading but beware it will kill it at some point, it's not mentioned around a lot but too many write operations to an ssd will kill it after sometime