lora raw text training

[kalle07]

hey,

which data format is that ?
only poore ANSI TEXT ???
and where should i put the file ?
i can only reload that
"Text file"
"The raw text file to use for training."

[araleza]

There's a documentation page all about ooba training here:

https://github.com/oobabooga/text-generation-webui/wiki/05-%E2%80%90-Training-Tab

[kalle07]

i know that ^^

but i search th whole "training tab" and dont find those information

[araleza]

The place to put your training data is inside the text-generation-webui/training/datasets/ subdirectory. You can either place a text file there directly, or you can create a directory of your own there, and put multiple text files inside it. Remember to click 'refresh' on the training tab to be able to see them, if you place the text files there after webui has started.

I think the text files are treated as UTF-8, which includes ASCII automatically.

The text files have do not require a specific format, as by default the training tab will chop it up into 256-ish token sections for you. Also, by default the Hard Cut String is '\n\n\n', which means that if you put three line breaks in a row (i.e. two blank lines) in the file, then it should treat the text before and after the blank lines as separate pieces of text training data.

[FlueschPluesch]

Why is that still not in the documentation? I was looking ages for this information...how are we supposed to know where to put the text-files?

[kalle07]

can i train on an GPTQ model ?

if yes,

1. if i load the model and start traning , error: "need monkeypatch"
2. if restart the gui with monkey-patch training fails with some kind of ...(i dont copy the whole error)
   .....
   File "e:\text-generation-webui\modules\training.py", line 294, in do_train
   from alpaca_lora_4bit.monkeypatch.peft_tuners_lora_monkey_patch import (
   ModuleNotFoundError: No module named 'alpaca_lora_4bit'
   .....

or i need the original bin files?
for this option the traing starts and completed without errors !

[Tedy50]

You don't need monkeypatch but you need to disable exllama kernels and select load in 4 bit
it will work only on transformers