Python Marc automatization

Hello Michael Mojžiš,

I hope you're doing well.

I wanted to address the memory usage problem you've encountered with your Marc Mentat script. The issue was due to processing each data point individually, which led to high memory consumption.The script was encountering memory limitations due to the way data points were being sent to Mentat. Each point was processed individually, leading to a high number of API calls and increased memory consumption.

To resolve this, you can try and update the script to handle data in batches. This means that instead of sending each point one by one, the script now collects a batch of points and sends them together. This reduces the overhead from multiple API calls and helps maintain efficient memory usage.

Let me know how it works.

Regards,

Amol Kumbhar

Hello Michael Mojžiš,

I hope you're doing well.

I wanted to address the memory usage problem you've encountered with your Marc Mentat script. The issue was due to processing each data point individually, which led to high memory consumption.The script was encountering memory limitations due to the way data points were being sent to Mentat. Each point was processed individually, leading to a high number of API calls and increased memory consumption.

To resolve this, you can try and update the script to handle data in batches. This means that instead of sending each point one by one, the script now collects a batch of points and sends them together. This reduces the overhead from multiple API calls and helps maintain efficient memory usage.

Let me know how it works.

Regards,

Amol Kumbhar

Hello Amol Kumbhal,

I have tried the script with the batch function several times, but the errors I mentioned earlier have not disappeared—the batch function did not help.

I read that when I send data manually (meaning that I copy them from Excel using Ctrl+C and paste them with Ctrl+V via Edit Table → Add, then press Enter), this method is optimized. When the data is being inserted, I can see that memory usage fluctuates, rising and falling, which indicates internal memory is being freed.

However, when I run a script that inserts the data automatically, it is not optimized. The GUI immediately freezes, and I can only see from the command line that the tables are being completed (because in the script, I included a command to print which table is being modified and then another message confirming the table is complete once points are added).

As I mentioned before, the memory usage keeps increasing without fluctuation, whether the batch function is used or not (I tried assigning different values to this function, ranging from 100 to 20,000). The memory keeps rising without fluctuation, and it always reaches around 4,200 MB, at which point the point insertion process seems to freeze and drastically slow down.

There is no mention of this issue in the manual, nor in the Python documentation for Marc Mentat, and I don’t know how to solve it. I am sending you the latest update with the batch function in the attachment.