Revision history [back]

I have developed a setup that does exactly this last year. If you have the calibration information for your RGB camera and you know the relative transform between this and the sensor frame of the DUO MC then this is straightforward using OpenCV.

First you need to transform your point cloud so it is in the optical coordinate frame of your RGB camera, then use the OpenCV function cv::projectPoints which will project these 3D points into 2D pixel locations within the image of the RGB camera.

When I implemented this it was actually simpler than I was expecting this part of the project to be, you may have to filter the resulting 2D points since some may be outside of the image frame or even NAN in some cases. There will also be cases where some points may be occuluded by other parts of the scene resulting in the wrong color being projected onto them. This problem is non-trivial to fix but is farily minor in more scenes if the baseline between the two sensors is small.

Hope this helps.