Merge two or more depth images

edit

Strictly speaking what you're trying to do is not possible. If you convert the depth maps to point clouds then you could easily combine them to a super point cloud, but returning to a depth map representation has a few problems.

The problem is that a depth map represents a special sub-set of point clouds, they are both structured and projectable. Structured means that the points are arranged in a 2D grid pattern from the point of view of the sensor, and projectable means that no points occlude any other points.

If your two depth sensors have a parallax baseline between them, then the combined point cloud may loose the projectable property, meaning that it cannot be represented as a depth map without some data loss. The data that would be lost are the parts of the point cloud which are occluded by other points.

However if you're okay with this data loss then converting back to a depth map could be the solution. You'll need to bear in mind that the combined point cloud will no longer be structured, so you'll have to project each point back into camera space to work out which pixel of the new depth map it corresponds to.

Hope this helps.

UPDATE:

I don't know of any ROS nodes that can do the whole thing out of the box. You could use the depth_image_proc package to convert the two depth images into point clouds. You could then make your own node that transformed these two clouds into the same coordinate system and created a super-set of the points in them both.

To convert this super-set back into a point cloud you could follow the final answer on this stackoverflow question. The user shows some example code for projecting points back into camera space using a simple pinhole camera model. For this to do what you want you'll need to define 'virtual' camera intrinsic and extrinsic parameters such that its FOV covers both of the cameras you're trying to merge. Then use these parameters to project the points to image space.

In order to deal with the data loss you'll need to use a z-buffering type approach, where you only add a new point into a pixel if it's closer than the value currently there.

Hope this makes sense, I'm guessing this has turned out more complicated than you were expecting!

You could use depth_image_proc register on one rgbd camera to transform it into the frame (and resolution and instrinsics) of the other rgbd camera (register calls the destination CameraInfo rgb/camera_info), or transform both rgbd cameras into a third frame and combine there.

Then there needs to be a nodelet to take two depth images in the same frame and with the same resolution and intrinsics/distortion (and same timestamp or else additional error) and combine each pixel of each source- pass through the closer valid depth value of the two would make sense, or average or something else. I don't think that nodelet exists anywhere open source (but also haven't looked much), but it wouldn't be hard to adapt the code of another nodelet in depth_image_proc to do it (and ideally upstream it into depth_image_proc).

depth_image_proc/register isn't currently doing the right thing with synchronization and timestamps of the two camera_info inputs so the results will be sloppy, be less correct when the cameras or what they are looking at are moving fast relative to frame rate, unless they are time synchronized at acquisition. It needs to use the destination camera_info to drive the output timestamp, and do a 'full' transform where if transforms the input depth image at the time of the input depth image into the destination frame at the destination camera_info timestamp, using a third frame as a fixed reference (e.g. map or odom).

If the two camera frames aren't that close to each other you may need https://github.com/ros-perception/ima....

probably you could use laser assembler for the task with map as the fixed frame.