Ollama Local Large Language Model (LLM) CPU & GPU Compatible

[krazor94]

Hey everyone!

I’m looking for testers for a module I’ve been working on. It integrates with player bots using a simple LLM setup powered by Ollama, designed to run on a local machine. Right now, it’s using a 1B model, a 3b or 8b is preferable (If you have access to a GPU)

The cool thing is that the 1B model is currently running on CPU, so no GPU is required to use the module. I’m running it on a 6-core server with 24GB of RAM, and it performs well. For reference, the 1B model requires about 8–10GB of memory to run.

The goal is to make the bot responses feel more human-like.

The configuration offers a lot of flexibility. You can:

Adjust how many player bots respond to each input. (Currently, 2–3 bots respond, with a 75% chance of replying and a 25% chance of ignoring.)
Customize the personalities of the bots to make their responses more varied and unique.
If no response is generated, the system falls back to the player bots’ prebuilt responses.
If you’re interested in testing it out, you can download the module from my GitLab server here:
https://gitlab.realsoftgames.win/krazor/mod_llm_chat

Let me know if you have any feedback or ideas to improve it!

also dont forget to install ollama and install the default model with

ollama pull krith/meta-llama-3.2-1b-instruct-uncensored:IQ3_M

Self Hosted Gitlab

Want to test the module running on my WOTLK Server?

Website Registration

Game version: 3.3.5a

set realmlist realsoftgames.ddns.net

[blinkysc]

Dude super cool! I’ll make an account today

Edit: Tried just now but worldserver might be down it seems

[blinkysc]

Maybe auth still pointing to a private address?

[krazor94]

Hmm possible i need to go through and double check off memory its pointing at 0.0.0.0

[blinkysc]

Oh auth should probably point to your dns address or whatever external facing address.

[krazor94]

Hmm interesting 0.0.0.0 allows all connections
I'll change it to my external and test again after work and update you. This worked fine last time off memory when I was testing on my hotspot on my laptop to connect to my server. But again I appreciate you bringing this to my attention as well I'll post back when it's fixed =)

[krazor94]

Sorry it took some time to get it up and running i was working on getting responses a lot better and work got in the way. my appologies but its up and running now. I had a few hicups where running the LLM on the same server as the world server was a really stupid idea lol.

but its essentially still running on the same machine just in 2 seperate Vms the LLM has 6 cores dedicated still only running on CPU too =)

[krazor94]

a bit of a progress update for providing more context to the conversation including Faction, Race, Class, Level, Spec of player and Responding AI, this influences the response based on the chosen personality. for example if its a rude/joking demenor it could poke fun at you being a lower level then the Bot or differences in class, race/Faction

some bugs to iron out. but it takes some data into perspective when responding to requests to give some context about Location, race, class, faction etc

[krazor94]

Gota Add more Context and test some different models out as well as shortening the messages. but its looking good so far =)

[krazor94]

getting a little bit better with context and responses

Scenario 1 - Level/Faction Difference:

Player (Level 30 Human Warrior, Alliance): "lol nice gear noob"
Bot (Level 80 Orc Warrior, Horde): "talking big for a lowbie alliance pup. come back when you've at least hit northrend, maybe you'll have better gear than my bank alt"

Scenario 2 - Combat Status:

Bot (Level 80 Tauren Druid, In Combat at 25% HP): "bit busy trying not to die rn mate, got 3 mobs on me and health's looking rough. ask again in a min if i survive this pull"```

Scenario 3 - Guild/Target Context:

Player (Level 80 Blood Elf Paladin, ): "nice tanking in that last raid"
Bot (Level 80 Tauren Warrior, , Targeting: Raid Boss): "thanks mate! your heals weren't bad either. and should run more raids together. but rn we should prob focus on this boss before the raid leader has our heads lol"

Scenario 4 - Mixed Context:

Player (Level 80 Human Mage, ): "get out of our territory horde scum"
Bot (Level 80 Undead Warlock, , In Combat): "aww look, a trying to act tough. coughs in plague would love to stay and educate you about true power but I'm a bit busy melting your friends right now. 75% hp and barely breaking a sweat, want to be next?"