How to pass input data through agent chain?

[LoopKarma]

Hello I've searched and tried most of related examples from docs but no luck so far, maybe you can give me a hand here.
Question is simple: the input request data has item of items that should be used down the agent chain pipeline, how to do this?
Example:

def _save_to_db(input_params: dict, config: dict):
    # company = ????
    for branch_info in input_params['return_values']['branches']:
        insert_branch_info(company, branch_info)

agent = (
        RunnableParallel({
            "input": lambda x: _extract_url_info(x['url'], x['html_containers']),
            "schema": lambda x: convert_pydantic_to_openai_function(Response),
            "company": lambda x: x['company_name'],
            "agent_scratchpad": lambda x: format_to_openai_function_messages(
                x["intermediate_steps"]
            ),
        })
        | prompt
        | llm # model run with functions calls
        | create_parse_to_class(Response.__name__) # converts str to json dict 
        | RunnableLambda(_save_to_db)
)

I want to pass company value from user request down to the very last step of the chain. Is there a way to do this?
And I believe it's worth to describe in the documentation or to create an example for this case.
Thank you very much

[eyurtsev]

Hi @LoopKarma,

Take a look at this example: https://github.com/langchain-ai/langserve/blob/main/examples/passthrough_dict/server.py

It's somewhat awkward, we'll be adding some new LCEL primitives to make this easier.

If you want to try out part of the beta API for LCEL:

https://github.com/langchain-
ai/langchain/blob/65091ebe50d840a45d3eba463ddbf38039e48eb0/libs/core/langchain_core/beta/runnables/context.py#L278

And some examples here:

https://github.com/langchain-ai/langchain/blob/48559643320e9777ca67599671d479ecb6255f0d/libs/core/tests/unit_tests/runnables/test_context.py#L32

We'll welcome any feedback.

[mesutgurlek]

My problem was not about the passing partial_variables, instead it is about defining PromptTemplate with configurable variable in a RunnableLambda. Let me explain the case:

• I have PromptTemplates that have configurable_fields, which are partial_variables and prompt.
• I do not want to take these variables from the client through a rest request to LangServe endpoint, but I want to update the Prompts in the runtime with a trigger. So, I write some custom config override logic inside per_req_config_modifier.
• However, per_req_config_modifier expects that overridden configurable variables should exist in the chain. I was already defined these PromptTemplates with configurable fields in the chain.
• Then I realized that, when an object with configurable_field is defined inside the RunnableLambda or custom Tool class, then configurable fields are not passed as config keys in the per_req_config_modifier function.

I had some key takeaways today, and appreciate if you have some details or additions:
There are 2 different parameter passing strategies: inputs and configs:
1 - If I want to use an input parameter in the arbitrary components of the chain, then whole chain components should be wrapped by RunnablePassthrough, to pass parameter all the way down from the start of the chain to end. It would be more flexible if the Context implementation is provided (not in Beta I mean)

2 - Parameters might be defined as configs in the request, corresponding chain objects might be initialized with configurable_fields. In that case, I am able to modify the chain object attributes in the runtime, using both config parameters of the request or per_req_config_modifier function. per_req_config_modifier intercepts with the request before passing into chain and provides capability to modify configurable variables. However, There are some problems:
a) Configurable fields are exposed to swagger documentation, even though you want to keep it internal.
b) Objects with configurables_fields, should not be located in RunnableLambda or Custom Tool kind of custom runnables, since fields could not passed as config keys into per_req_config_modifier. And in my case, I want to intercept with request, update the variables in the runtime and pass to the chain.

[eyurtsev]

If you have unit tests for your app, I wouldn't be worried about using the context runnable. Things are marked as beta to mean that we might change the API based on user feedback, but with unit tests you'll be able to catch that immediately.

[eyurtsev]

Have to run right now, but will try to respond to the other question when time allows. It'd help if you could include a short snippet of code, I could try to re-write it for you. I get the gist of what you're saying and one solution is to implement a custom runnable or to use a runnable lambda that wrap the necessary code and instantiate an appropriate runnable at run time based on the request.

[mesutgurlek]

Thanks for the advice about Context, definitely will take into consideration.

My other case is like below:

# -------------- Inside get_prompts()
 qa_prompt_template = PromptTemplate(input_variables=["question"],
                                                       template=prompts.qa_prompt).configurable_fields(
        template=ConfigurableField(
            id="qa_prompt",
            name="Question Answer Prompt",
            description="Prompt to generate appropriate response for the question."
        ),
    )

# --------------- Custom Tool

class AnswerGeneratorTool(BaseTool):
    name = ""
    description = ""

    def _run(self, question, context, **kwargs):
        settings = get_settings() # Gets cached settings
        prompts = get_prompts() # Gets cached prompts

        # LLM
        qa_llm = ChatOpenAI(temperature=settings.qa_llm_temp, model=settings.qa_llm_name)

        qa_response = (prompts.qa_prompt_template
                           | cypher_llm
                           | StrOutputParser()
                           )

        return qa_response.invoke({'question': question, 'context': context})

# ---------------  Main chain

chain = (RunnablePassthrough(context=retriever())
              | AnswerGeneratorTool())

# ---------------  Langserve server.py

async def per_req_config_modifier(config: Dict, request: Request) -> Dict:
   # Clear cache and reload prompts so that the configurable prompt can be updated in the runtime without re-initializing PromptTemplate object.
    prompts = get_prompts()

    config["configurable"]["qa_prompt"] = prompts.qa_prompt
    return config

This raises error since inside the per_req_config_modifier it cannot find qa_prompt key because we have defined prompt in the chain in a custom tool.

[eyurtsev]

It might be easier to use the APIHandler directly if you want to have more dynamic behavior based on the request see: https://github.com/langchain-ai/langserve/blob/main/examples/api_handler_examples/server.py#L114

You'll need to register the endpoints yourself, but you'll get more flexibility and can instantiate the runnables yourself.

The alternative is that you need to figure out how to phrase the entire thing as a runnable so that the configuration can bubble up.