Ask Your Question
2

What is more efficient: writing to disk or message passing a huge amount of data as an array between a Python node and CPP node?

asked 2016-10-28 16:59:07 -0500

updated 2016-11-01 15:03:00 -0500

Does anyone have experience with the following scenario:

A Python node that needs to pass a large amount of float values (an array of images) as a multidimensional numpy array to a CPP node.

The dilemma: What is better (in terms of memory or speed):

a) To pass the data as a serialized float32[] message array between these nodes?

b) To write the data to a binary file and just communicate the file's location as a string msg?

edit retag flag offensive close merge delete

Comments

Can you quantify huge?

gvdhoorn gravatar image gvdhoorn  ( 2016-10-29 04:04:11 -0500 )edit

1 Answer

Sort by ยป oldest newest most voted
3

answered 2016-10-28 18:44:14 -0500

ahendrix gravatar image

Sending data as a message (ie memory copies) is significantly faster than writing it to disk.

Have a look at these latency numbers for a comparison of the speed of different operations, and remember that ROS message passing (network traffic) between different nodes on the same computer is actually just a memory copy, and never uses the network hardware.

edit flag offensive delete link more

Comments

My message size is an array (i.e. float32[] of about 648*480*300 or 93312000 elements or 355 MB size). I noticed that writing/reading the array to/from disk takes less time than passing it via a ROS Service. Anyways to speed up the message passing? It takes 5 seconds now to receive the msg.

ubuntuslave gravatar image ubuntuslave  ( 2017-02-14 13:43:16 -0500 )edit

I would like to note that this slow performance happens even when the service client is set to be persistent.

ubuntuslave gravatar image ubuntuslave  ( 2017-02-14 14:40:13 -0500 )edit

I'd expect python would take a while to serialize python datatypes, but I wouldn't think it would take 5 seconds to serialize that much data using the optimized numpy methods.

ahendrix gravatar image ahendrix  ( 2017-02-15 14:09:42 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2016-10-28 16:59:07 -0500

Seen: 536 times

Last updated: Nov 01 '16