Projecting image coordinates into world coordinates
Hi
I'm using a Creative Senz3D for human-robot interaction perception tasks. I want to project an image point (u,v) into world coordinates, but where there is no corresponding point in the point cloud. E.g. the sensed pixel may be further away than the range of the depth sensor, but not the color camera.
So I'm using these equations, where DEPTH is the estimated depth of the point.
x = DEPTH * (u - cx) / fx
y = DEPTH * (v - cy) / fy
z = DEPTH
But I'm not sure whether I should be using the unrectifed image or the rectified image? And whether I should be using should be using cx, cy, fx and fy from K or P in CameraInfo.
I'm guessing that if I use the rectified image then the image points will be distorted away from their corresponding depth points... And then if I did cloud.at(u,v) on a point that did have a depth value, then it might not actually find the point.
Thanks
Jamie
Hi! I'm corot, I did the patch for adding RGB info to the pointcloud. Hope people with better understanding will give you a good answer... because all I can tell u is that "it seems to work" as I did. But be aware that you don't have a rectified image within the softkinetic driver!
Hey Corot! Thanks for your answer. Yeah I was using the equations that you would use to rectify an image (I think) by using the variables from P in camera info, but it always seemed like the transformed (u,v) points didn't match up correctly with the point cloud.
So I'm going to try just with K which has the variables for the raw images that the softkinetic driver provides...