Ask Your Question
1

Ros Master-Slave Network is slow because of large amount of network overhead

asked 2020-04-28 18:43:46 -0500

darshb34 gravatar image

updated 2020-05-02 01:16:45 -0500

I am trying to setup a multi-realsense d430 camera setup (4 cams on Jetson tx2 + 2 cams on Pi 4 = 6 cams total). I setup Jetson TX2 as Master and Pi 4 as Slave.

Environment: Ros Melodic, Ubuntu 18.04 Arm 64 bit.

In the first experiment the pi and tx2 were connected via WIFI. Rostopic hz /camera1/depth/color/points on SLAVE gives me 2 Hz while it is 15 Hz on MASTER.

In second experiment I connected the two using ethernet cable as assigned ip using (sudo ip ad add 10.2.2.250/24 dev eth0). Rostopic hz /camera1/depth/color/points on SLAVE gives me 6hz - 12 Hz while it is 15 Hz on MASTER.

Edit: what is the best setup for interfacing 6 realsense cams in ros?

-----> Each camera takes up around 150mb/s for all the topics it publishes

-----> Jetson tx2 cpu utilization hits 100% for all the four cores after 3 cameras are launched. Any more cameras launched after that results in degradation of the publish rate of the previously launched cams.

 /points topic requires 64 mb/s bandwidth

edit retag flag offensive close merge delete

Comments

Ros Master-Slave Network is slow because of large amount of network overhead

this seems like you've already figured out what your problem is. If you believe it's due to network overhead, then it would good if you could explain how you arrived at that conclusion.

gvdhoorn gravatar image gvdhoorn  ( 2020-04-29 04:13:13 -0500 )edit

I have updated the question with more details on the experiment.

darshb34 gravatar image darshb34  ( 2020-04-29 19:27:28 -0500 )edit
1

Streaming data from the realsense over wifi is generally a bad idea. It is well known that the realsense pumps an insane amount of data out from its USB3 port (images, pointclouds and IMU data) and WiFi is not well suited for it. As you showed, wired connections are more efficient. Make sure they are 1Gbps bandwidth rated cables (Cat E). Even over 1 camera directly plugged, it sometimes struggled to get 15fps as the realsense driver takes 1 core of my CPU.

hashirzahir gravatar image hashirzahir  ( 2020-04-29 21:54:02 -0500 )edit
1

Hi do you mind updating how the 150mb/s bandwidth was derived? I could see various 40mb/s topics in your terminal screenshot, but it is not clear which topics they were referring to. I'm guessing its mostly pointcloud2? For your real use case what topics would you need to use from each of the 6 cameras? The ROS network is setup such that the data will not be transferred over the network unless someone is "subscribing" to the data, in this case your rostopic bw command was subscribing. Otherwise, if you were to only subscribe to image feed without pointcloud, I'm guessing the network usage may not be so high. You can also configure the realsense-ros driver to not publish certain topics. That reduces the CPU usage.

hashirzahir gravatar image hashirzahir  ( 2020-05-02 01:36:24 -0500 )edit

the 150mb/s was obtained by adding bw of /depth_img + /infra1_img + /pointcloud .

Yes I modified the driver to not published compressed, theora and compressedDepth, and infra2 image topic.

I am using two topics pointcloud and infra1 image.

darshb34 gravatar image darshb34  ( 2020-05-13 22:48:42 -0500 )edit

1 Answer

Sort by ยป oldest newest most voted
2

answered 2020-04-30 02:28:22 -0500

gvdhoorn gravatar image

updated 2020-04-30 03:46:16 -0500

In the first experiment the pi and tx2 were connected via WIFI. Rostopic hz /camera1/depth/color/points on SLAVE gives me 2 Hz while it is 15 Hz on MASTER.

I would not use any wireless links for something like this. They're too low bandwidth.

In second experiment I connected the two using ethernet cable as assigned ip using (sudo ip ad add 10.2.2.250/24 dev eth0). Rostopic hz /camera1/depth/color/points on SLAVE gives me 6hz - 12 Hz while it is 15 Hz on MASTER.

A gigabit link provides at most a throughput of 125 MB/s. At most, as it is the theoretical limit of the transport. More often than not it won't get there.

I'm not sure what the datarate of the points topic of a realsense is, but if you'd get 6 to 12 Hz, it's probably on the order of 10 to 15MB/sec, which seems like it could be correct.

Running 2 cameras without any compression will not make things any faster.

If with "network overhead" you meant the fact that you're transferring the data over a network (ie: just that fact), then it's indeed true that this will place a bottleneck in your system.

Traditionally though, the term "overhead" is used to mean "additional traffic not directly used to transmit the payload itself" (so: headers, signalling, etc). I'm not sure that would be the main cause of the reduced data rates here. My first though would be you're simply saturating the available bandwidth.

Perhaps try a rostopic bw /camera1/depth/color/points on your master and post those results here.

what is the best setup for interfacing 6 realsense cams in ros?

I'm not convinced your problem is with ROS necessarily: saturating a specific link with traffic will become a problem in many (all?) cases, no matter which communication infrastructure you use. How the saturation is dealt with, that would seem to be specific to whatever you're using.


Edit: something to look into may be compressed_depth_image_transport. It trades spending CPU time for network bandwidth. Instead of sending over entire pointclouds, you would use depth images, compress and then decompress them.

For PointCloud2 there has been some discussion (such as Compressed PointCloud2? on ROS Discourse), and some people have implemented custom solutions (such as paplhjak/point_cloud_transport mentioned in ros-perception/perception_pcl#152).

You could see whether using those helps with the bandwidth problem.

edit flag offensive delete link more

Comments

Thank you for the help. I checked rostopic bw /points with one camera running, and I got 64mb/s for just one topic being transported on the TCP by the roslaunch file. Following your explanation I have concluded that,

my problem is first because of lack of CPU power on TX2 (which involves 1.) ros scripts sourcing data from USB and 2.) ros scripts publishing that data on TCP )

second, the ethernet cable needs to be CAT E and ethernet port should be GigaByte Enabled (1000 mb/s)

darshb34 gravatar image darshb34  ( 2020-05-02 01:23:50 -0500 )edit
1

the ethernet cable needs to be CAT E and ethernet port should be GigaByte Enabled (1000 mb/s)

It's actually Gigabit ethernet. Maximum theoretical transfer rate is 125 megabyte/sec. Not 1000 megabyte/sec.

There is 10 Gigabit ethernet, but your Jetson nor the RPi support that.

gvdhoorn gravatar image gvdhoorn  ( 2020-05-02 06:51:32 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

2 followers

Stats

Asked: 2020-04-28 18:43:46 -0500

Seen: 70 times

Last updated: May 02