Ask Your Question
0

Merge two or more depth images

asked 2018-11-05 09:42:12 -0500

Hendrik Wiese gravatar image

updated 2018-11-06 06:08:15 -0500

I would like to merge multiple depth images from at least two RGBD cameras. The poses of the cameras are known as well as their intrinsic parameters. So I would assume that it is a task that could be done in realtime when no feature extraction is necessary.

My current idea is to simply morph/project the image of camera B on the image plane of camera A by an affine transformation. But that leaves out the actual depth values. I'd have to transform them as well, wouldn't I?

New idea: I could also turn the depth images into point clouds, merge them and convert them back to a new depth image. I could imagine that there's some way using depth_image_proc but I'm not sure.

Is there already something that could do this transformation for me? Maybe in OpenCV or even in ROS itself, based on TF frames and the camerainfo messages for the involved cameras?

ROS version is Kinetic Kame on Ubuntu 16.04 LTS

edit retag flag offensive close merge delete

2 Answers

Sort by ยป oldest newest most voted
0

answered 2018-11-06 06:30:15 -0500

updated 2018-11-06 09:06:42 -0500

Strictly speaking what you're trying to do is not possible. If you convert the depth maps to point clouds then you could easily combine them to a super point cloud, but returning to a depth map representation has a few problems.

The problem is that a depth map represents a special sub-set of point clouds, they are both structured and projectable. Structured means that the points are arranged in a 2D grid pattern from the point of view of the sensor, and projectable means that no points occlude any other points.

If your two depth sensors have a parallax baseline between them, then the combined point cloud may loose the projectable property, meaning that it cannot be represented as a depth map without some data loss. The data that would be lost are the parts of the point cloud which are occluded by other points.

However if you're okay with this data loss then converting back to a depth map could be the solution. You'll need to bear in mind that the combined point cloud will no longer be structured, so you'll have to project each point back into camera space to work out which pixel of the new depth map it corresponds to.

Hope this helps.

UPDATE:

I don't know of any ROS nodes that can do the whole thing out of the box. You could use the depth_image_proc package to convert the two depth images into point clouds. You could then make your own node that transformed these two clouds into the same coordinate system and created a super-set of the points in them both.

To convert this super-set back into a point cloud you could follow the final answer on this stackoverflow question. The user shows some example code for projecting points back into camera space using a simple pinhole camera model. For this to do what you want you'll need to define 'virtual' camera intrinsic and extrinsic parameters such that its FOV covers both of the cameras you're trying to merge. Then use these parameters to project the points to image space.

In order to deal with the data loss you'll need to use a z-buffering type approach, where you only add a new point into a pixel if it's closer than the value currently there.

Hope this makes sense, I'm guessing this has turned out more complicated than you were expecting!

edit flag offensive delete link more

Comments

The two cameras are close together, just at an angle. The idea is to increase the FOV of a single depth sensors by combining it with others of the same type. I'll be okay with data loss if I get back a valid point for each pixel. Are there already nodes/nodelets available that can do this job?

Hendrik Wiese gravatar image Hendrik Wiese  ( 2018-11-06 06:45:46 -0500 )edit

Not that I know of. I've updated my answer with some pointers to make your own though.

PeteBlackerThe3rd gravatar image PeteBlackerThe3rd  ( 2018-11-06 09:07:02 -0500 )edit

At a first glance this totally makes sense, yes. I'll see how I can make use of your pointers. Thank you very much!

Hendrik Wiese gravatar image Hendrik Wiese  ( 2018-11-06 09:18:51 -0500 )edit

That pinhole camera approach works like a charm. I've written two nodelets for this purpose. One that merges two point clouds based on the known relative poses of the cameras and the other rendering a depth image from the new point cloud based on a pinhole camera model. I'll put them on github.

Hendrik Wiese gravatar image Hendrik Wiese  ( 2018-11-09 10:45:18 -0500 )edit

Great news, I'll have a look when you put them up.

PeteBlackerThe3rd gravatar image PeteBlackerThe3rd  ( 2018-11-09 12:32:56 -0500 )edit
0

answered 2018-11-05 13:13:29 -0500

harsha336 gravatar image

probably you could use laser assebler for the task with map as the fixed frame.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

2 followers

Stats

Asked: 2018-11-05 09:42:12 -0500

Seen: 420 times

Last updated: Nov 06 '18