Mapping between pointcloud and image

edit

Hi everyone, For the past few days I have been working to use pointcloud information and the corresponding RGB image of the scene so that I can get a 1 to 1 mapping between objects present in both. I really could not figure it out. I had this idea(below), please share ur views about modifying or completely changing it.

The aim is to mark objects in both (pointcloud and corresponding RGB image of scene) with a box of same color for same objects.

First I thought of extracting the shape features of objects from the edges. For similar objects the shape features are going to remain more or less the same(based on orientation). Edge detection is possible only in a 2D image which I obviously did for the RGB image.

How to do edge detection in a pointcloud which is 3D representation of the scene? In a 3D pointcloud viewed in the same direction as the image is facing will definitely have objects whose shape or boundary look similar. Yes if you look at pointcloud at a different angle than the view of image ,the objects may appear to be of different shapes( eg, bus moving forward infront of you in RGB image(2D) will look squarer but when viewed sideways in pointcloud(3D) looks rectangular)

So I thought of converting the 3D pointcloud into a 2D image in the same viewing direction of RGB image(so that edges could be found). Is it really possible to represent the 3D pointcloud as a 2D image essentially depicting the surfaces and objects present. If yes, how to do it? All these pictures of pointcloud which I see in the Internet are like 3D objects with various shapes in a 2D plane. Are these 2D or a snapshot from the 3D viewer of various softwares??? Eagerly awaiting for ur ideas and views about this problem.

I went through the documentation in http://www.ros.org/wiki/pcl_ros/Tutorials/CloudToImage (Tutorials/CloudToImage) for converting pointcloud to image.I used the node "convert_pointcloud_to_image" But when i tried it i get an error :-

Error in converting cloud to image message: No rgb field!!

In the above are they trying to represent 3D world in a 2D image(that is what i need) or are they just trying change its type to that of sensor_msgs/Image??

I dont know if my approach is correct but please pour in your thoughts about doing this pointcloud to objects in image mapping in some other way if its better or simpler than my approach. Will be truly grateful to you.

Thank you,

Karthik