JSON Error with use_blob, and Server Memory Leakage Retrieving Views

[raeldor]

I am seeing some very strange behavior in the latest TM1PY. We were previously on a forked version we created a couple of years ago with the stream_decode parameter (https://github.com/raeldor/tm1py@feature/stream-decode#egg=TM1py). When I upgrade to the latest version I get the error 'Object of type type is not JSON serializable' when using the use_blob=true parameter on tm1.cubes.cells.execute_view_dataframe(). Without it, the error does not occur. I tried using mdx = tm1.views.get() combined with tm1.cells.execute_mdx_dataframe(), which stops the error from occurring. Strange.

Anyway, either way we are seeing huge memory leakage ON THE SERVER when retrieving views. We pull large views into a data mart and using this method were losing about 20GB of RAM every 12 hours. I've attached a screenshot. The 'X' shows where we restart instances and the arrows show the memory leakage. On the final restart we switched out code back to our old fork and the memory leakage stops.

The fact that the switching of code back to an old forked version of TM1PY solves the server memory leakage is a sign something fishy is going on. I'll try and run some tests later on the forked version without using the stream_decode parameter.

EDIT: Even removing the stream_decode flag it continues in a straight line. So something in the latest TM1PY appears to be causing memory leakage on the server compared to when we originally forked.

[rclapp]

That makes sense the stream parsing option was designed to reduce python memory consumption. That feature is still available and is called “use_iterative_json” From: Ray Price ***@***.***> Reply-To: cubewise-code/tm1py ***@***.***> Date: Tuesday, July 25, 2023 at 1:19 AM To: cubewise-code/tm1py ***@***.***> Cc: Subscribed ***@***.***> Subject: [cubewise-code/tm1py] JSON Error with use_blob, and Server Memory Leakage Retrieving Views (Discussion #943) I am seeing some very strange behavior in the latest TM1PY. We were previously on a forked version we created a couple of years ago with the stream_decode parameter ***@***.***/stream-decode#egg=TM1py). When I upgrade to the latest version I get the error 'Object of type type is not JSON serializable' when using the use_blob=true parameter on tm1.cubes.cells.execute_view_dataframe(). Without it, the error does not occur. I tried using mdx = tm1.views.get() combined with tm1.cells.execute_mdx_dataframe(), which stops the error from occurring. Strange. Anyway, either way we are seeing huge memory leakage ON THE SERVER when retrieving views. We pull large views into a data mart and using this method were losing about 20GB of RAM every 12 hours. I've attached a screenshot. The 'X' shows where we restart instances and the arrows show the memory leakage. On the final restart we switched out code back to our old fork and the memory leakage stops. The fact that the switching of code back to an old forked version of TM1PY solves the server memory leakage is a sign something fishy is going on. I'll try and run some tests later without using the stream_decode parameter. [image]<https://user-images.githubusercontent.com/1768444/255796399-e72d9d06-6652-4762-b9df-2ec790ae78a4.png> — Reply to this email directly, view it on GitHub<#943>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AEK7GZUUAITDCAQWXRVRSQTXR5JK7ANCNFSM6AAAAAA2WQKUF4>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[raeldor]

Sorry, I think I didn't explain well. With any of the flags either on or off, the new version of TM1PY is leaking memory on the server. It's only when we go back to the old version it stops.

[raeldor]

I think I'm going to open this as an issue. It seems very clear that the change of TM1PY version is the only change that caused this memory leakage. Something doesn't seem right here.