Did any one succeed on getting OPT-13B-Erebus to work on 3080ti?

[ye7iaserag]

I'm using python server.py --auto-devices --cai-chat --load-in-8bit --listen --listen-port=8888
But I'm getting

 Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit
the quantized model. If you have set a value for `max_memory` you should increase that. To have
 an idea of the modules that are set on the CPU or RAM you can print model.hf_device_map.

I have 12GB of VRAM and 64GB or RAM, and I think those should load the model correctly with --auto-devices
Is there something wrong that I'm doing or is it impossible to get it to run with this hardware?

[oobabooga]

The memory usage of OPT-13B-Erebus with --load-in-8bit for me is 14131MiB.

I am not sure if it is possible to mix offloading with 8bit (I think not), so the best currently available option would be to use deepspeed: https://github.com/oobabooga/text-generation-webui/wiki/DeepSpeed or simply --auto-devices with --gpu-memory 10 or similar.

[ye7iaserag]

deepspeed crashes on startup when loading the pygmalion 6B model, which works normally without deepspeed

[ye7iaserag]

also does using a sharded model help with offloading to cpu?

[oobabooga]

also does using a sharded model help with offloading to cpu?

I am not sure, but possibly. It helps with reducing the RAM usage while loading a model to the GPU.

[ye7iaserag]

Anyone trying this, I got it working on a system with 3080ti with 64GB ram using the following command
python server.py --auto-devices --cai-chat --gpu-memory 10 --bf16 --listen --listen-port=8888 --model=OPT-13B-Erebus
Also I do not think --auto-devices works with --load-in-8bit, they are mutually exclusive