astream doesn't show chunks, but astream_log does. why? #395

seanmavley · 2024-01-13T16:29:07Z

seanmavley
Jan 13, 2024

Solution: Works as expected in LangChain 0.1+

I'm using Django with the https://github.com/em1208/adrf package for async

from adrf.views import APIView as AsyncAPIView

class LlmAPIView(AsyncAPIView):

    async def post(self, request, *args, **kwargs):
          payload = request.data.get('payload')

          from langserve import RemoteRunnable
          llm_stream= RemoteRunnable("http://localhost:8005/edu/")
          
          async for chunk in llm_stream.astream(payload):
              print(chunk)

          return Response({}, status=status.HTTP_200_OK)

The above simply prints the entire response once to the console.

However, when I do async for chunk in llm_stream.astream_log(payload): I get a response as this in console:

View the log

RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama',
'value': {'end_time': None,
'final_output': None,
'id': 'dc070017-7812-48b8-90c2-4704363539a1',
'metadata': {'__langserve_endpoint': 'stream_log',
'__langserve_version': '0.0.36',
'__useragent': 'python-httpx/0.26.0'},
'name': 'ChatOllama',
'start_time': '2024-01-13T16:01:25.604',
'streamed_output_str': [],
'tags': ['seq:step:3'],
'type': 'llm'}})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': 'To'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' answer'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' the'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' question'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' about'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' mol'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': 'ality'})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': ','})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' it'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' is'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' important'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' to'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' define'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' it'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' first'})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': '.'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' Mol'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': 'ality'})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': ' ('})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': 'm'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': 'oles'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' per'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' kil'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': 'ogram'})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': ')'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' is'})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': ' a'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' unit'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' of'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' concentration'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' that'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' represents'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' the'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' amount'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' of'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' sol'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': 'ute'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' particles'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' present'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' in'})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': ' a'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' solution'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' per'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' unit'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' mass'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' of'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' sol'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': 'vent'})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': '.'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' The'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' formula'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' for'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' calculating'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' mol'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': 'ality'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' is'})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': ':'})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': '\n'})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': '\n'})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': 'M'})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': 'ol'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': 'ality'})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': ' ='})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': ' ('})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': 'm'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': 'oles'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' of'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' sol'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': 'ute'})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': ' /'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' kg'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' of'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' sol'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': 'vent'})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': ')'})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': '\n'})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': '\n'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': 'For'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' example'})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': ','})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' if'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' we'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' have'})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': ' '})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': '1'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' mole'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' of'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' sol'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': 'ute'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' dissol'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': 'ved'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' in'})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': ' '})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': '1'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' kil'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': 'ogram'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' of'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' sol'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': 'vent'})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': ','})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' the'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' mol'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': 'ality'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' would'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' be'})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': ':'})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': '\n'})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': '\n'})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': 'M'})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': 'ol'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': 'ality'})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': ' ='})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': ' ('})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': '1'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' mol'})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': ' /'})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': ' '})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': '1'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' kg'})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': ')'})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': ' ='})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': ' '})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': '1'})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': ' M'})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': '\n'})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': '\n'})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': 'It'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' is'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' important'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' to'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' remember'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' that'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' the'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' units'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' used'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' for'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' mol'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': 'ality'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' must'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' be'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' consistent'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' throughout'})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': ' a'})
RunLogPatch({'op': 'add',
'path': '/logs/ChatOllama/streamed_output_str/-',
'value': ' problem'})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': '.'})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': '\n'})
RunLogPatch({'op': 'add', 'path': '/logs/ChatOllama/streamed_output_str/-', 'value': ''})

I can see the streaming happening in the logs somewhere of the astream_log yet astream doesn't print to console one by one, as indicated in the example here: https://github.com/langchain-ai/langserve/blob/main/examples/conversational_retrieval_chain/client.ipynb

What am I doing wrong?

According to the docs, using the astream should get me something like this

Answered by eyurtsev

Jan 13, 2024

So all good? 😁

View full answer

eyurtsev · 2024-01-13T19:01:06Z

eyurtsev
Jan 13, 2024
Maintainer

What do you see if you use remote runnable in async with astream but running it outside of django?

3 replies

seanmavley Jan 13, 2024
Author

Ha! I upgraded to LangChain 0.1, and the astream is working as expected, both standalone, and in the django.

The standalone look like this

import asyncio
from langserve import RemoteRunnable
remote_runnable = RemoteRunnable("http://localhost:8005/edu/")

payload = {
    "prompt": [
        {
            "name": "general",
            "prompt": "<|user|> You are a helpful assistant. Answer questions using the following context. Context is as follows:\n\n{context}\n\n. Chat history is as follows:\n\n{chat_history}\n\n {question} <|assistant|> \n"
        }
    ],
    "question": "hi how are you?",
    "chat_history": []
}

async def process_chunks():
    async for chunk in remote_runnable.astream_log(payload):
        print(chunk)

asyncio.run(process_chunks())

eyurtsev Jan 13, 2024
Maintainer

So all good? 😁

Answer selected by seanmavley

seanmavley Jan 13, 2024
Author

Yep, all good. Thanks.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

astream doesn't show chunks, but astream_log does. why? #395

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 3 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

astream doesn't show chunks, but astream_log does. why? #395

seanmavley Jan 13, 2024

Replies: 1 comment · 3 replies

eyurtsev Jan 13, 2024 Maintainer

seanmavley Jan 13, 2024 Author

eyurtsev Jan 13, 2024 Maintainer

seanmavley Jan 13, 2024 Author

seanmavley
Jan 13, 2024

Replies: 1 comment 3 replies

eyurtsev
Jan 13, 2024
Maintainer

seanmavley Jan 13, 2024
Author

eyurtsev Jan 13, 2024
Maintainer

seanmavley Jan 13, 2024
Author