Skip to content

Latest commit

 

History

History
41 lines (30 loc) · 685 Bytes

Deploy.md

File metadata and controls

41 lines (30 loc) · 685 Bytes

Using GPU for TEI

sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo systemctl restart docker

With GPUs

docker run --rm --gpus all -d nvidia/cuda:11.8.0-base nvidia-smi

docker-compose --profile local-embedding up -d

Without GPUs (OpenAI embedding)

docker-compose up -d

Setup guide

If you plan to not use local embedding, set LOCAL_EMBEDDING = false in .env file

Setup for deploy

  • Make run.sh file executable
chmod +x run.sh
  • For CPU (Using OpenAI Embedding)
./run.sh --openai True
  • For GPU (Self-hosted Embedding Server)
./run.sh local-embedding --local True