Python Multiprocessing

[raeldor]

We have some python code that pulls cubes into our data mart. We have it running so each cube pull is a new thread using 'multiprocessing.Process'. However, it seems on the actual pull of data using execute_view_dataframe that once inside TM1py it's as if the other threads are not continuing until the pull is complete. Is this possible? Or maybe the print statements I've added are not being printed at the exact time of issuance?

[MariusWirtz]

You can upgrade to the branch #935
@vmitsenko just implemented this new function execute_mdx_dataframe_async that allows you to pass a list of MDX queries.
Then they are executed in parallel, and the results and concatenated in one data frame..

Does that solve your problem?

[raeldor]

I think that might be a little different. I've already implemented the threading in my code, so I'm not looking for that functionality from TM1PY. Rather, I have multiple pulls from execute_view_dataframe running concurrently in separate threads Shouldn't they be able to pull the data concurrently? One shouldn't block the other, right?

[MariusWirtz]

Exactly. They shouldn't block each other.

Do you use ArcForTM1? Can you see the sessions/threads in the TM1 server? Can you check if the MDX queries are (b)locking each other when they are executed?

[raeldor]

Let me check that, thank you. Also, I notice the print statement has a flush=True, which forces it to print at that time rather than potentially later, so I'll check that out too in case I'm chasing a red herring. I'll report back. Thanks!

[raeldor]

So, it looks like print without the 'flush=True' was not printing straight away and not giving true results. It does seem to be threading. Having said that, I'm now interested in the '_async' version of the execute_dataframe proc. Is there an async version of execute_view_dataframe?

[MariusWirtz]

@vmitsenko, what do you think?

@raeldor
As a temporary solution, you could try this:

with TM1Service(**local_params) as tm1:
    view_names = ["view1", "view2", "view3", "view4"]

    mdx_queries = [
        tm1.views.get("Sales", view_name=view_name, private=False).mdx
        for view_name
        in view_names]

    df = tm1.cells.execute_mdx_dataframe_async(mdx_queries)

    print(df.shape)