Order of point clouds to use while merging (ICP)
My team and I have built the following setup (view from above). It's a 3x3x3m aluminum grid with 4 Kinects placed at height 2.1~2.2m and are facing downwards (at 20~30 degrees). The angle between the neighbors is 90 degrees. At this point I need to merge point clouds from all these cameras (camera calibration, manual transformations have been applied). The question is: do I merge them to one single camera one by one, or follow clockwise/counterclockwise direction, or there is a better way to merge them?
UPDATE: if anyone is interested, I calibrated 3 cameras to 1, then followed the procedure of merging the clouds that showed the least error during the calibration process.
I don't understand the problem if the depth cameras are calibrated and their extrinsic poses are known then you don't need to use ICP at all! You can simply transform the points into the same reference frame and add them to a single output cloud. What am I missing?
@PeteBlackerThe3rd calibration of 4 cameras at this angle turned out to be a serious problem. A couple of calibration toolboxes (MATLAB, OpenCV) gave insane errors. I had to manually align the clouds in CloudCompare software and apply ICP afterwards
So you're using ICP to calibrate the poses of the cameras? I would expect a poor overlap given the positions of the sensors. Have you tried using a known shape such as a large cube for calibration and registering to the known shape?
I've calibrated using multiple matching points via CloudCompare and the error is now around 1-2 cm, which is more than enough for my project. As for ICP, I'm not using PCL's ICP, I've ported non-rigid one from cilantro and it works great even for rapidly moving hands. PCL's ICP performed poorly
Interesting. If you're scanning rapidly moving objects then there will probably by a small difference in time between each scan because the sensors aren't synchronised. This would explain why non rigid ICP is producing better results.
I use rosbag to record data and then I additionally apply message_filters::sync_policies::ApproximateTime, which decreases the sync error to about 1 ms.
How can a message filter decrease the sync error? If they are captured at different times then the temporal offset is fixed.
Timestamps of messages from 4 cameras in my rosbag recordings can differ up to 2-3 ms before applying the filter, so they are not 100% synchronized.