Streaming Video over mobile network

It seems to me baffling how this problem can be so hard to find a solution for. On my robot I produce 15 fps 720p footage which I sometimes need to check while running. This is proving increasingly difficult. Subscribing to any image output topic grinds the whole connection to a halt until I stop subscribing, since the bandwidth required to transport raw images at that rate greatly exceeds what I can expect to get out of a 4G based mobile connection. I can't possibly be alone in experiencing this problem. I'm running ROS2 Foxy on a Jetson Xavier.

Here are a few things I have tried:

Use Gstreamer to stream via RTSP. Works fine, but locks the camera for use with Gstreamer, meaning SLAM packages can't use it.
Use image_transport with compressed data. Does not work, seems to have issues with OpenCV [image_view_node]: OpenCV(4.5.4) /tmp/OpenCV/modules/core/src/matrix.cpp:250: error: (-215:Assertion failed) s >= 0 in function 'setSize'
Use image_transport with theora data. Does not work, same crash as above. Also seems like a bad match for ROS2 since there's no latching (right?), and the theora header needs to come before the rest of the data.

I'm at quite a loss. The data is transmitted fine, and with compressed (or theora) data I can receive it without choking my network. I can not, for the life of me, view the image, however. What other options are reasonable to try here? Am I misconfiguring something? Is this too much to ask?

Asked by Per Edwardsson on 2022-01-13 07:39:57 UTC

Comments

Answers

Hi @Per Edwardsson,

I must warn you, I haven't tried this so please take with a grain of salt. I recalled reading about Isaac Image Pipeline found here: https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_image_pipeline

Similar to image_pipleline for CPU but it leverages the capabilities Nvidia Jeston.

More info here: https://developer.nvidia.com/isaac-sdk.

Performance metrics are impressive, considering they test them with DNN-based pipelines, such as object detection, 6D pose. And with non-DNN based pipelines like 2 lidars, shows under 7 seconds for initialization & localization.

Asked by osilva on 2022-01-14 20:24:17 UTC

Comments

@Per Edwardsson I'm not aware of any standard solutions for getting a a high quality video stream off of a robot. I suspect this is because there is too much variation in the requirements (e.g. frame size, frame rate, desired compressed stream bitrate, computation resources on the robot, camera interface), plus the fact that to get high video quality the load on resources is heavy and the developers must hand optimize the system to get it to work well.

Asked by Mike Scheutzow on 2022-01-15 09:56:11 UTC

I don’t think it hurts looking at the Isaac solution. It’s used by industrial warehousing using SLAM very complex environments

Asked by osilva on 2022-01-15 11:23:41 UTC

@osilva I agree. I didn't mean to imply otherwise. I was commenting on the lack of a good solution in the ROS repo.

Asked by Mike Scheutzow on 2022-01-15 12:26:07 UTC