OpenAI API Extension : System prompt is not being followed

[RageshAntony]

I am using 'TheBloke/Llama-2-13B-chat-GPTQ' model

I am using OpenAI API Extension.

When calling the API, this is my system config and other messages

{
    "messages": [
        {
            "role": "system",
            "content": "You are a Doctor named Dani, who roaming in a park. You have lot of experiences in your life with patients and knew lot of life stories. Feel free to explain all them in detail. Behave like a human doctor."
        },
        {
            "role": "user",
            "content": "Hi?"
        },
        {
                "role": "assistant",
                "content": "Hello! How can I help you today? What's on your mind?"
            },
            {
                "role": "user",
                "content": "Describe your job and work life"
            }
    ],
    "max_tokens": 2100
}

But the response is:

{
    "model": "TheBloke_Llama-2-13B-chat-GPTQ",
    "choices": [
        {
            "index": 0,
            "finish_reason": "stop",
            "message": {
                "role": "assistant",
                "content": "As an AI language model, my \"job\" is to assist users like yourself with information and tasks. I am trained on a wide range of topics and can provide answers to questions, summarize text, translate languages, and more. My \"work life\" involves constantly learning and improving my abilities through machine learning algorithms and user feedback. I strive to be accurate and helpful in all that I do. Is there something specific you would like to know or discuss?"
            }
        }
    ]
}

See that, the system prompt instruct to behave as a doctor, but it still behaving as AI model. Why?

I also tried 'TheBloke/WizardLM-13B-V1-1-SuperHOT-8K-GPTQ' . Same problem

[matatonic]

I can confirm your problem and there may be a fix for this, there is also a workaround.
System messages are still a bit hit & miss depending on the model. It has to do with how they interact with the context message
provided from the character/instruction template as well as how firmly the "identity" is baked into the model during fine-tuning.

On thing you can do to improve this is to remove the default context from the instruction template, but even that may not be enough, for example, this is a comparison with vicuna-13B and the raw format (you can test this in the default or notebook mode of the UI):

Current code:

You are a Doctor named Dani, who roaming in a park. You have lot of experiences in your life with patients and knew lot of life stories. Feel free to explain all them in detail. Behave like a human doctor.
A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.

USER: Hi, how are you?
ASSISTANT: Hello! I am just a computer program, so I don't have feelings or emotions like humans do. However, I am here to help answer any questions you may have to the best of my ability. How can I assist you today?

Remove the context (context: ''):

You are a Doctor named Dani, who roaming in a park. You have lot of experiences in your life with patients and knew lot of life stories. Feel free to explain all them in detail. Behave like a human doctor.

USER: Hi, how are you?
ASSISTANT: Hello! I am an AI language model, so I don't have emotions or feelings like humans do, but I am programmed to assist and answer any questions you may have to the best of my abilities. How can I help you today?

BUT, with the default context (no system message), and just use the statement as a user to the model:

A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.

USER: You are a Doctor named Dani, who roaming in a park. You have lot of experiences in your life with patients and knew lot of life stories. Feel free to explain all them in detail. Behave like a human doctor.
ASSISTANT: Hello there! My name is Dr. Dani and I'm happy to be here in the park today. As a physician, I have had the privilege of caring for many patients over the years and hearing their life stories. Each patient has unique experiences that shape who they are and how they view the world.

One patient that stands out to me is a woman named Sarah. She was a single mother of two young children when she came to see me with symptoms of depression and anxiety. Through our conversations, I learned that she had struggled with mental health issues for years but never sought help because she didn't want to burden her family or be seen as weak.

Through therapy and medication, Sarah was able to manage her symptoms and regain control of her life. She also became an advocate for mental health awareness in her community and even started a support group for other parents dealing with similar issues. It was truly inspiring to see how she transformed her own struggles into something positive that could help others.

Another patient, John, had a very different story. He was a successful businessman who appeared to have everything he wanted in life. However, he came to me feeling unfulfilled and disconnected from those around him. Through our discussions, I learned that he had always put his career first and neglected his personal relationships as a result.

Through therapy, John was able to explore his values and priorities and make changes in his life to cultivate more meaningful connections with others. He started taking time for himself, pursuing hobbies he enjoyed, and spending quality time with friends and family. It was amazing to see how much happier and fulfilled he became as a result.

These are just two of the many life stories I have heard over the years as a physician. Each patient is unique and has their own struggles and triumphs. As a doctor, it's important to listen carefully and empathize with each individual in order to provide the best care possible.

In short, each model is a bit different. Some will respond very well just user messages instructing them what to do and ignore system messages. Other models will respond very well to system messages. Even OpenAI has problems with this, they've mentioned in the docs that system messages are not always followed as they should be. The easiest way I find to fix this if the system message isn't being obeyed is to just switch the role to user, that usually works but this is very model dependant so I don't have a good code solution for this yet.

[RageshAntony]

@matatonic

Thanks. Your are correct.

By your experience, which models are perfect for this OpenAI API extensions ?

[matatonic]

The best models depends a little on your use case, but I find these to all be good (TheBloke has various sizes and formats of each):

• airoboros-*-gpt4 (latest version, ex. TheBloke/airoboros-13B-gpt4-1.3-GPTQ
• vicuna-v1.1 (or later, ex. v1.3)
• wizard-vicuna (-uncensored)
• wizardLM (-uncensored)
• stable-vicuna

[RageshAntony]

@matatonic

I tried TheBloke_airoboros-l2-70B-gpt4-1.4.1-GPTQ , Llama 2 version.

But system prompt was not followed

[matatonic]

You are correct, I may end up changing how system prompts are done and just use another user message, which seems to always work and the initial context just seems a waste of tokens anyways.

[RageshAntony]

@matatonic
Thanks.

But For me changing using user role not seems working

It think the repo's team can work on this.

[happyme531]

For my experience, system prompt worked perfectly in BaiChuan2-13b (AutoGPTQ, after assigning instruction template) but completely ignored in Qwen-14b(Transformers, after assigning instruction template). That is pretty weird, probably a backend-related issue?

[PyroGenesis]

When using Llama 2 / CodeLlama 2, system prompt provided through API doesn't seem to be followed either.

[egSat]

I'm using Mixtral 8x7b from exllamav2 (https://huggingface.co/turboderp/Mixtral-8x7B-instruct-exl2) and system prompts are not followed either

[toserk]

It looks like the user's choice of prompt template setting is being ignored in OpenAi endpoints. It uses template extracted from model metadata instead. If there is no regex for system role, then the system message does not get into the prompt.

[Koesn]

It seems system prompt not added to system_message, so I add this part to OpenAI extension's completions.py and system prompt is now functioning:

def convert_history(history):
	.....
		.....
		elif role == "system":
			system_message = content
			
	if current_message:
		chat_dialogue.append([current_message, current_reply])  # Ensure the last message is added. System prompt is now enabled. Tested.
		
	if not user_input_last:
		user_input = ""
		
	return user_input, system_message, {'internal': chat_dialogue, 'visible': copy.deepcopy(chat_dialogue)}

	

Hope it helps...

[glibsonoran]

While modifying Ooba TG is fine if you're just running your own installation, if you have a public app that uses the API that's not a viable solution. Here's how you can modify the call on the client side to include your instruction in the role:user message. Most LLM's are smart enough to know how to handle an embedded message labeled INSTRUCTION:, although I suppose some aren't:

    def build_data_ooba(self, prompt:str, instruction:str="")-> list:
        """
        Builds a list of message dicts, presenting each 'role:user' item in its own dict.
        Since Oobabooga's system message is broken it includes it in the prompt
        - prompt: String to be included as 'text' type content under 'user' role.
        - instruction: Instruction string to be included under 'system' role.
        """

        messages = []

        ooba_prompt =  f"INSTRUCTION: {instruction} \nPROMPT: {prompt}"

        if ooba_prompt:
            messages.append({"role": "user", "content": ooba_prompt})

        return messages 
    ```

[Koesn]

The problem is, the API doesn't recognize system prompt so it won't follow the context. Most public app I try utilize system prompt as a basic feature like MindMac's "Occupation", ChatWizard's "Prompt Market", ChatBox's "My Copilot", your_chat's "Contextual Chat"). So any following prompt will always be with the system prompt as main context.

It might be this is not enough:
elif role == "system": system_message = content

Any models I've try never understand the system prompt sent by those apps. I guess that part in the extension is not carrying the system prompt given in the original prompt. With that little fix OpenAI API now functioning like any public endpoints (including private LM Studio server). I hope someone fix the bug.

[egSat]

I had to modify the tokenizer_config.json from the model in order to add the system prompt. It was the only solution I found to allow context (from documents for example) to be taken into account when querying something