Improvement of allocation of big python list to custom message data field

edit

Short explanation:

I am trying to use ROS2 to publish sensor data that I have stored on disk that I "acquired" from a simulator. The sensor data frame is in the form of a protobuf RepeatedScalarContainer. I have no problem in parsing and deserializing the data from protobuf (.bin). I already cast this data into a python list (let's call it sensor_data_list) and this works just fine and it is not my problem.

The problem is when I try to allocate a sensor dataframe (a quite big list of size ~4 million elements which is equivalent to 18 MB, when I use sys.getsizeof()) into a field of a custom message. I think that this is not a ROS2 problem but rather how I handle data in python. However, I thought that someone in the ROS community might have had the same problem or some similar experience.

Detailed explanation:

I needed to create my own sensor custom data to handle a specific sensor data (no, I could not use already available sensor data msgs available in ROS, but this is not the point of my question anyway). Let's call this custome message sensor_data_custom_msg.

My data handling workflow in my ROS2 sensor node is the following:

1. deserialize protobuf sensor data and preload it into a big python list. This is done at the beginning of the node initialization on purpose since we don't care about "spending/wasting" some time in the initialization phase of the node.
2. create the actual sensor publisher node by passing to it the preloaded data in previous point
   in the timer_callback of the node I initialize a sensor_data_custom_msg and I pass to the callback the preloaded data.
3. The problem is at this step: At each time step what I would like to do is to fill the sensor_data_custom_msg field data with a specific element of the list sensor_data_list to then publish my sensor_data_custom_msg filled with sensor data stored on disk. This operation however is taking a lot of time (~0.4 seconds). Therefore, from here I started investigating a bit the problem. What I found is:

If I print the length and type of the data I am trying to fill the sensor_data_custom_msg.data I get respectively: length: 4608000; type: <class 'list'>. You can then understand that my sensor data is quite big and here comes my question: is there a better way to deal with such big lists in python? Can I optimize or change the way I allocate such big chunk of data in python?

Another thing that I would like to point out is about computational resources. I have a rather powerful machine. I am working on ubuntu 20.04 (pop_os) and I have an i9 processor with 32GB of RAM. I am using ROS2 inside a Docker container and the container is not the problem. Indeed, if you are asking yourself: "does the container have enough resources from the host computer?" I think that the answer is yes. Indeed ... (more)