Python Marc automatization

Question

Hello, I'd like to ask the following: I created a Python script for automatically entering values into tables in Marc Mentat from a CSV file. This CSV file contains 500 columns, and each column has exactly 50,000 rows of values. I only import the columns into Marc Mentat. Every time I run my Python script, the memory usage of MentatOGL.exe in the Task Manager starts increasing rapidly (about 1 MB/s during the input process). I start at around 2900 MB, and once it reaches approximately 4220 MB (which is around the 17th table), it stops adding values and the memory growth slows down from 1 MB/s to about 0.1 MB/s, effectively causing it to freeze. Can you advise me on why this is happening and how to fix it? 
 I am attaching my Python script with the modified table. mentat_script.rar

Amol Kumbhar · Accepted Answer

Hello Michael Mojži&scaron;, 
 I understand you&rsquo;re having some trouble with memory issues and the Mentat GUI freezing up. Your script's way of handling large data batches and frequently restarting connections might be causing these hiccups. 
 In the updated approach, we start by reading the CSV file line by line and converting each value carefully. Skipping over the header and refining the data before using it is a good practice&mdash;it ensures you're only dealing with clean data. If there's any problematic data, we just print a warning and skip it, so we can keep things moving smoothly. 
 For each set of data, we create a new table in Mentat, setting it up with "temperature" and "time" variables. By adding data points directly to the table, we manage memory usage much better. We also keep a single, stable connection to Mentat throughout, which helps avoid the issues caused by opening and closing connections repeatedly. As we go through each table, we make sure to fit the data properly.

I can&rsquo;t share the exact script with you directly, but I hope this explanation helps you get things sorted. 
 Regards, 
 Amol Kumbhar

Michael Mojžiš · Answer

Hello, Amol Kumbhar. So, my conclusion is: In the Marc Mentat 2010 version, my script works perfectly, flawlessly, and without any issues. However, in the 2020 version, I am experiencing the errors I mentioned in previous messages, and sending the data in batches does not help. 
 My conclusion is that the backend of the older Marc Mentat version was better suited for inserting points than the newer versions. In any case, thank you, Amol Kumbhar, for your help.

Python Marc automatization

Top Replies