[Bug]: LLMSingleSelector - Got invalid JSON object

[tbuziak]

Bug Description

I created a RouterQueryEngine with the LLMMultiSelector, and a few query_engine_tools. When querying, I get the following error. The engine either parses or builds the list of choices incorrectly.
With PydanticMultiSelector and Groq(), the error does not occur. I need to use the LLM model with Ollama(), and I must use LLMSingleSelector.

query_tools = []
for collection, description in collections.items():
    index: VectorStoreIndex = indexes[collection]
    query_engine = index.as_query_engine(
        vector_store_query_mode="hybrid",
        response_mode="compact",
        verbose=True,
        text_qa_template=text_qa_template,
        refine_template=refine_template,
        similarity_top_k=similarity_top_k,
        sparse_top_k=similarity_top_k,
        node_postprocessors=[
            models.rerank,
            SimilarityPostprocessor(similarity_cutoff=similarity_cutoff),
            ExpandContextPreprocessor(expand_context_by_lines=expand),
            LongContextReorder(),
        ],
    )
    query_tools.append(
        QueryEngineTool.from_defaults(
            query_engine=query_engine,
            name=collection,
            description=description
        )
    )  
query_engine = RouterQueryEngine(
    selector=LLMSingleSelector.from_defaults(),
    query_engine_tools=query_tools,
    verbose=True,
)

Version

0.12.8

Steps to Reproduce

The error occurs if query_engine_tools contains more than one choice and the user's question pertains to the second or subsequent choice. If the question pertains to the first choice, the error does not occur.

Relevant Logs/Tracbacks

ERROR:root:Got invalid JSON object. Error: Expecting property name enclosed in double quotes: line 2 column 6 (char 7) while constructing a mapping
  in "<unicode string>", line 2, column 5:
        {{
        ^
found unhashable key
  in "<unicode string>", line 2, column 6:
        {{
         ^. Got JSON string: [
    {{
        choice: 3,
        reason: "xxxxx x xxxx."
    }},
    {{
        choice: 1,
        reason: "cccc ccccc cccc"
    }}
]

[dosubot]

Hey there, @tbuziak! I'm here to help you with any bugs, questions, or contributions you have. Let's squash this bug together!

I found a similar closed issue that might be relevant to your problem: [Bug]: SelectionOutputParser incorrectly parsing Router Query Engine. This issue was resolved by a pull request that updated the SelectionOutputParser to handle JSON objects and arrays correctly. You can view the issue here.

In your case, the error message indicates that the JSON object is not being constructed correctly, specifically with unhashable keys and missing double quotes around property names. This suggests that the JSON output from the LLMSingleSelector might not be formatted properly. You might want to check if there are any updates or patches related to JSON handling in the LLMSingleSelector since version 0.12.8, or consider reviewing the JSON construction logic in your implementation.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[logan-markewich]

Yea, and single selector can only select one choice. Use a multil selector if you expect user inputs to select multiple choices

[tbuziak]

Yea, and single selector can only select one choice. Use a multil selector if you expect user inputs to select multiple choices

The same error occurs with LLMMultiSelector.

[tbuziak]

In llama_index/llama-index-core/llama_index/core/output_parsers/selection.py, the escape_curly_braces function replaces the characters { } with double {{ }}. Without _escape_curly_braces, the selector works correctly.

def _escape_curly_braces(input_string: str) -> str:
# Replace '{' with '{{' and '}' with '}}' to escape curly braces
return input_string.replace("{", "{{").replace("}", "}}")