Order of point clouds to use while merging (ICP)

asked 2018-07-16 06:13:11 -0500

Mikhail K. gravatar image

updated 2018-08-07 06:45:04 -0500

My team and I have built the following setup (view from above). It's a 3x3x3m aluminum grid with 4 Kinects placed at height 2.1~2.2m and are facing downwards (at 20~30 degrees). The angle between the neighbors is 90 degrees. At this point I need to merge point clouds from all these cameras (camera calibration, manual transformations have been applied). The question is: do I merge them to one single camera one by one, or follow clockwise/counterclockwise direction, or there is a better way to merge them?

image description

UPDATE: if anyone is interested, I calibrated 3 cameras to 1, then followed the procedure of merging the clouds that showed the least error during the calibration process.

edit retag flag offensive close merge delete


I don't understand the problem if the depth cameras are calibrated and their extrinsic poses are known then you don't need to use ICP at all! You can simply transform the points into the same reference frame and add them to a single output cloud. What am I missing?

PeteBlackerThe3rd gravatar image PeteBlackerThe3rd  ( 2018-08-07 07:10:25 -0500 )edit

@PeteBlackerThe3rd calibration of 4 cameras at this angle turned out to be a serious problem. A couple of calibration toolboxes (MATLAB, OpenCV) gave insane errors. I had to manually align the clouds in CloudCompare software and apply ICP afterwards

Mikhail K. gravatar image Mikhail K.  ( 2018-08-07 07:14:02 -0500 )edit

So you're using ICP to calibrate the poses of the cameras? I would expect a poor overlap given the positions of the sensors. Have you tried using a known shape such as a large cube for calibration and registering to the known shape?

PeteBlackerThe3rd gravatar image PeteBlackerThe3rd  ( 2018-08-07 07:26:10 -0500 )edit

I've calibrated using multiple matching points via CloudCompare and the error is now around 1-2 cm, which is more than enough for my project. As for ICP, I'm not using PCL's ICP, I've ported non-rigid one from cilantro and it works great even for rapidly moving hands. PCL's ICP performed poorly

Mikhail K. gravatar image Mikhail K.  ( 2018-08-07 22:33:10 -0500 )edit

Interesting. If you're scanning rapidly moving objects then there will probably by a small difference in time between each scan because the sensors aren't synchronised. This would explain why non rigid ICP is producing better results.

PeteBlackerThe3rd gravatar image PeteBlackerThe3rd  ( 2018-08-08 13:30:59 -0500 )edit

I use rosbag to record data and then I additionally apply message_filters::sync_policies::ApproximateTime, which decreases the sync error to about 1 ms.

Mikhail K. gravatar image Mikhail K.  ( 2018-08-08 22:25:12 -0500 )edit

How can a message filter decrease the sync error? If they are captured at different times then the temporal offset is fixed.

PeteBlackerThe3rd gravatar image PeteBlackerThe3rd  ( 2018-08-09 02:15:29 -0500 )edit

Timestamps of messages from 4 cameras in my rosbag recordings can differ up to 2-3 ms before applying the filter, so they are not 100% synchronized.

Mikhail K. gravatar image Mikhail K.  ( 2018-08-09 02:24:12 -0500 )edit