Passing large data messages

asked 2011-03-08 14:04:29 -0500

Ivan Dryanovski

4954 ●52 ●70 ●86 http://robotics.ccny.cuny.edu/

updated 2011-03-14 11:51:58 -0500

kwc
12244 ●54 ●90 ●132 http://kwc.org/

Hi,

I would like some suggestions or guidelines for the following design problem. I have a ROS node which maintains a fairly large data structure (a 3D map). I would like to publish it efficiently, so I can create independent packages that visualize or process the data. I can publish ROS messages as shared pointers using the nodelet architecture, but that still leaves the problem of converting the map structure to a ROS message. Translating the map structure to a ROS message is undesirable for two reasons

I need to spend time copying data out of the structure and into the ROS message
The native map is built using some data structures not available to the ROS message, which can only use vectors.

An alternative I can think of is to directly maintain the map data in my program as a ROS message, so no translation would be required. However, again I have the problem that the ROS messages only allow for vectors, so I'd be restricted in how I design the structure.

Is my reasoning correct? Does anyone have a good solution to this problem?

edit retag flag offensive close merge delete

add a comment

3

answered 2011-03-13 20:57:57 -0500

Daniel Stonier
3145 ●24 ●51 ●64

We had a similar discussin in-house at our company a while ago.

It is possible to adapt your own c++ types to the pub/sub mechanisms without having to use an ros msg. See here: MessageSerialisationAndAdaptingTypes. I haven't tried this yet, but we've talked about giving it a whirl on the next stage of one of our projects.

There is one further gotcha with regards to sharing large data types. So long as your subscribers don't need to modify the map structure, you can create all your packages as nodelets and pass around your adapted c++ type via pub/sub without doing any copying. However, if your subscribers need to modify the map, then you need to find another solution as subscribers will only ever get a pointer to a const version of the object.

There was a discussion on the mailing list about that, where it was pointed out that this was to help maintain a contract between publisher/subscriber, to make sure that other parts of your program (which, being very modularised, may be very unknown) do not cause problems by modifying your underlying data.

The best way we figured we might avoid copies on a low perf cpu with a map that we wanted modifiable among several modules might be to create them all as libraries, or ros libplugins, tied together by a single manager which stores the map. The manager would then pass a reference to the map to each module at init() time.

edit flag offensive delete link

Comments

snorri, this is probably what I'm looking for - I'll take a look into it. I don't plan on having clients make changes to the map, so that won't be an issue.

Ivan Dryanovski ( 2011-03-14 02:13:37 -0500 )edit

I believe this solution would work for you. All you need to do is be able to define a message into which you could serialize the data, and write the serializer. Then you can publish with the native data type, and if it's local to the node with the same datatype it will be passed by pointer.

tfoote ( 2011-04-18 20:08:12 -0500 )edit

If it is not subscribed with the custom type it will be serialized and sent as a message. And you will also need to write a deserializer routine. An example of doing the PointCloud datatype in pcl_ros http://www.ros.org/wiki/pcl_ros. Note that after passing the data it cannot be mutated.

tfoote ( 2011-04-18 20:11:00 -0500 )edit

So if you truely need the optimization of being able to maintain one map and have many accessors it's likely that using an abstraction like message passing is inappropriate for your design.

tfoote ( 2011-04-18 20:12:14 -0500 )edit

add a comment

2

answered 2011-03-08 14:21:21 -0500

joq
25443 ●181 ●256 ●408

You are correct. Every message-based system deals with similar problems.

One option: publish just the changes in a message and let your subscribers keep their own copy of the map up to date. Obviously, that only helps if a lot of the data do not change every cycle.

edit flag offensive delete link

add a comment

Passing large data messages

2 Answers

Comments

Question Tools

Stats

Related questions

Passing large data messages edit

2 Answers

Comments

Question Tools

Stats

Related questions

Passing large data messages