Improvement of allocation of big python list to custom message data field
Short explanation:
I am trying to use ROS2 to publish sensor data that I have stored on disk that I "acquired" from a simulator. The sensor data frame is in the form of a protobuf RepeatedScalarContainer. I have no problem in parsing and deserializing the data from protobuf (.bin). I already cast this data into a python list (let's call it sensor_data_list
) and this works just fine and it is not my problem.
The problem is when I try to allocate a sensor dataframe (a quite big list of size ~4 million elements which is equivalent to 18 MB, when I use sys.getsizeof()
) into a field of a custom message. I think that this is not a ROS2 problem but rather how I handle data in python. However, I thought that someone in the ROS community might have had the same problem or some similar experience.
Detailed explanation:
I needed to create my own sensor custom data to handle a specific sensor data (no, I could not use already available sensor data msgs available in ROS, but this is not the point of my question anyway). Let's call this custome message sensor_data_custom_msg
.
My data handling workflow in my ROS2 sensor node is the following:
- deserialize protobuf sensor data and preload it into a big python list. This is done at the beginning of the node initialization on purpose since we don't care about "spending/wasting" some time in the initialization phase of the node.
- create the actual sensor publisher node by passing to it the preloaded data in previous point
in thetimer_callback
of the node I initialize asensor_data_custom_msg
and I pass to the callback the preloaded data. - The problem is at this step: At each time step what I would like to do is to fill the
sensor_data_custom_msg
fielddata
with a specific element of the listsensor_data_list
to then publish mysensor_data_custom_msg
filled with sensor data stored on disk. This operation however is taking a lot of time (~0.4 seconds). Therefore, from here I started investigating a bit the problem. What I found is:
If I print the length and type of the data I am trying to fill the sensor_data_custom_msg.data
I get respectively:
length: 4608000; type: <class 'list'>
. You can then understand that my sensor data is quite big and here comes my question: is there a better way to deal with such big lists in python? Can I optimize or change the way I allocate such big chunk of data in python?
Another thing that I would like to point out is about computational resources. I have a rather powerful machine. I am working on ubuntu 20.04 (pop_os) and I have an i9 processor with 32GB of RAM. I am using ROS2 inside a Docker container and the container is not the problem. Indeed, if you are asking yourself: "does the container have enough resources from the host computer?" I think that the answer is yes. Indeed ...