ROS Answers: Open Source Q&A Forum - RSS feedhttps://answers.ros.org/questions/Open source question and answer forum written in Python and DjangoenROS Answers is licensed under Creative Commons Attribution 3.0Fri, 08 Feb 2019 08:09:38 -06002D image point to 3Dhttps://answers.ros.org/question/214116/2d-image-point-to-3d/ Hey!
I have a setup with a bottom camera (simple webcam pointed at the ground). Along with the camera is an IMU, which means that I have the attitude of the camera. I also have an estimation of the distance from the camera to the ground.
Since I'm detecting some target on the ground (I'm able to detect and get their position on the image using openCV), I would like to extract their position on the world frame. However I'm lost on how to do it.
Are there any ROS package that implement this? How should I do it?
Thanks in advance!Mon, 20 Jul 2015 05:34:50 -0500https://answers.ros.org/question/214116/2d-image-point-to-3d/Answer by Airuno2L for <p>Hey!</p>
<p>I have a setup with a bottom camera (simple webcam pointed at the ground). Along with the camera is an IMU, which means that I have the attitude of the camera. I also have an estimation of the distance from the camera to the ground. </p>
<p>Since I'm detecting some target on the ground (I'm able to detect and get their position on the image using openCV), I would like to extract their position on the world frame. However I'm lost on how to do it.</p>
<p>Are there any ROS package that implement this? How should I do it?</p>
<p>Thanks in advance!</p>
https://answers.ros.org/question/214116/2d-image-point-to-3d/?answer=214209#post-id-214209It sounds like you have the 3D point that describes the location of the camera, and since you have the IMU with it you know it's orientation as well. Since you also know the estimated location of the ground plane this can be treated as a ray-plane intersection problem, which is a common problem in computer graphics that solves the x,y,z intersection point of a ray and a plane.
The line that starts from the camera and points through the ground is the ray, and the ground or course is the plane. Since the object you're selecting/detecting from the camera image is not always in the center pixel, you'll need to add a pan and tilt angle to the ray depending on which pixel the center of the object corresponds to, the [image_geometry](http://wiki.ros.org/image_geometry) package has tools to help do just that.
As mig mentioned, you'll want to use tf to help keep track of all the transformation frames, and a urdf can make this even easier.
Once you know the ray, google "line-plane intersection". There is even a [Wikipedia](https://en.wikipedia.org/wiki/Line%E2%80%93plane_intersection) article about it to get started. You might even get lucky searching for "how to do line-plane intersection in c++" or python or however you want to do it, and you might find some ready to use code. Tue, 21 Jul 2015 07:21:08 -0500https://answers.ros.org/question/214116/2d-image-point-to-3d/?answer=214209#post-id-214209Comment by nwanda for <p>It sounds like you have the 3D point that describes the location of the camera, and since you have the IMU with it you know it's orientation as well. Since you also know the estimated location of the ground plane this can be treated as a ray-plane intersection problem, which is a common problem in computer graphics that solves the x,y,z intersection point of a ray and a plane. </p>
<p>The line that starts from the camera and points through the ground is the ray, and the ground or course is the plane. Since the object you're selecting/detecting from the camera image is not always in the center pixel, you'll need to add a pan and tilt angle to the ray depending on which pixel the center of the object corresponds to, the <a href="http://wiki.ros.org/image_geometry">image_geometry</a> package has tools to help do just that. </p>
<p>As mig mentioned, you'll want to use tf to help keep track of all the transformation frames, and a urdf can make this even easier. </p>
<p>Once you know the ray, google "line-plane intersection". There is even a <a href="https://en.wikipedia.org/wiki/Line%E2%80%93plane_intersection">Wikipedia</a> article about it to get started. You might even get lucky searching for "how to do line-plane intersection in c++" or python or however you want to do it, and you might find some ready to use code. </p>
https://answers.ros.org/question/214116/2d-image-point-to-3d/?comment=214295#post-id-214295After reading about image_geometry, I think I know how to proceed. Using `projectPixelTo3dRay` I'm able to obtain the ray and after the intersection with the ground plane I should be getting a `x,y,z` for the target. right? Btw, which library is usually used for linear algebra computation on ros?Wed, 22 Jul 2015 05:26:58 -0500https://answers.ros.org/question/214116/2d-image-point-to-3d/?comment=214295#post-id-214295Comment by Airuno2L for <p>It sounds like you have the 3D point that describes the location of the camera, and since you have the IMU with it you know it's orientation as well. Since you also know the estimated location of the ground plane this can be treated as a ray-plane intersection problem, which is a common problem in computer graphics that solves the x,y,z intersection point of a ray and a plane. </p>
<p>The line that starts from the camera and points through the ground is the ray, and the ground or course is the plane. Since the object you're selecting/detecting from the camera image is not always in the center pixel, you'll need to add a pan and tilt angle to the ray depending on which pixel the center of the object corresponds to, the <a href="http://wiki.ros.org/image_geometry">image_geometry</a> package has tools to help do just that. </p>
<p>As mig mentioned, you'll want to use tf to help keep track of all the transformation frames, and a urdf can make this even easier. </p>
<p>Once you know the ray, google "line-plane intersection". There is even a <a href="https://en.wikipedia.org/wiki/Line%E2%80%93plane_intersection">Wikipedia</a> article about it to get started. You might even get lucky searching for "how to do line-plane intersection in c++" or python or however you want to do it, and you might find some ready to use code. </p>
https://answers.ros.org/question/214116/2d-image-point-to-3d/?comment=214319#post-id-214319I'm not sure what people do when using C++, Looking at that tutorial TF::Transform is just a [btTransform](http://www.bulletphysics.com/Bullet/BulletFull/classbtTransform.html) from bullet, but there is nothing stopping you from using other ways. [Eigen](http://eigen.tuxfamily.org/) is good to.Wed, 22 Jul 2015 09:12:45 -0500https://answers.ros.org/question/214116/2d-image-point-to-3d/?comment=214319#post-id-214319Comment by nwanda for <p>It sounds like you have the 3D point that describes the location of the camera, and since you have the IMU with it you know it's orientation as well. Since you also know the estimated location of the ground plane this can be treated as a ray-plane intersection problem, which is a common problem in computer graphics that solves the x,y,z intersection point of a ray and a plane. </p>
<p>The line that starts from the camera and points through the ground is the ray, and the ground or course is the plane. Since the object you're selecting/detecting from the camera image is not always in the center pixel, you'll need to add a pan and tilt angle to the ray depending on which pixel the center of the object corresponds to, the <a href="http://wiki.ros.org/image_geometry">image_geometry</a> package has tools to help do just that. </p>
<p>As mig mentioned, you'll want to use tf to help keep track of all the transformation frames, and a urdf can make this even easier. </p>
<p>Once you know the ray, google "line-plane intersection". There is even a <a href="https://en.wikipedia.org/wiki/Line%E2%80%93plane_intersection">Wikipedia</a> article about it to get started. You might even get lucky searching for "how to do line-plane intersection in c++" or python or however you want to do it, and you might find some ready to use code. </p>
https://answers.ros.org/question/214116/2d-image-point-to-3d/?comment=214327#post-id-214327Thanks for all the help! I will look into the eigen library.Wed, 22 Jul 2015 11:26:03 -0500https://answers.ros.org/question/214116/2d-image-point-to-3d/?comment=214327#post-id-214327Comment by Airuno2L for <p>It sounds like you have the 3D point that describes the location of the camera, and since you have the IMU with it you know it's orientation as well. Since you also know the estimated location of the ground plane this can be treated as a ray-plane intersection problem, which is a common problem in computer graphics that solves the x,y,z intersection point of a ray and a plane. </p>
<p>The line that starts from the camera and points through the ground is the ray, and the ground or course is the plane. Since the object you're selecting/detecting from the camera image is not always in the center pixel, you'll need to add a pan and tilt angle to the ray depending on which pixel the center of the object corresponds to, the <a href="http://wiki.ros.org/image_geometry">image_geometry</a> package has tools to help do just that. </p>
<p>As mig mentioned, you'll want to use tf to help keep track of all the transformation frames, and a urdf can make this even easier. </p>
<p>Once you know the ray, google "line-plane intersection". There is even a <a href="https://en.wikipedia.org/wiki/Line%E2%80%93plane_intersection">Wikipedia</a> article about it to get started. You might even get lucky searching for "how to do line-plane intersection in c++" or python or however you want to do it, and you might find some ready to use code. </p>
https://answers.ros.org/question/214116/2d-image-point-to-3d/?comment=214318#post-id-214318That's exactly right. The tf library has some built in tools for linear algebra. Here is a [small tutorial](http://wiki.ros.org/tf/Overview/Transformations) that shows python use. TF actually uses [transformations.py](http://code.google.com/p/ros-geometry/source/browse/tf/src/tf/transformations.py)Wed, 22 Jul 2015 09:06:30 -0500https://answers.ros.org/question/214116/2d-image-point-to-3d/?comment=214318#post-id-214318Comment by mgruhler for <p>It sounds like you have the 3D point that describes the location of the camera, and since you have the IMU with it you know it's orientation as well. Since you also know the estimated location of the ground plane this can be treated as a ray-plane intersection problem, which is a common problem in computer graphics that solves the x,y,z intersection point of a ray and a plane. </p>
<p>The line that starts from the camera and points through the ground is the ray, and the ground or course is the plane. Since the object you're selecting/detecting from the camera image is not always in the center pixel, you'll need to add a pan and tilt angle to the ray depending on which pixel the center of the object corresponds to, the <a href="http://wiki.ros.org/image_geometry">image_geometry</a> package has tools to help do just that. </p>
<p>As mig mentioned, you'll want to use tf to help keep track of all the transformation frames, and a urdf can make this even easier. </p>
<p>Once you know the ray, google "line-plane intersection". There is even a <a href="https://en.wikipedia.org/wiki/Line%E2%80%93plane_intersection">Wikipedia</a> article about it to get started. You might even get lucky searching for "how to do line-plane intersection in c++" or python or however you want to do it, and you might find some ready to use code. </p>
https://answers.ros.org/question/214116/2d-image-point-to-3d/?comment=214212#post-id-214212good points there!Tue, 21 Jul 2015 07:50:50 -0500https://answers.ros.org/question/214116/2d-image-point-to-3d/?comment=214212#post-id-214212Comment by nwanda for <p>It sounds like you have the 3D point that describes the location of the camera, and since you have the IMU with it you know it's orientation as well. Since you also know the estimated location of the ground plane this can be treated as a ray-plane intersection problem, which is a common problem in computer graphics that solves the x,y,z intersection point of a ray and a plane. </p>
<p>The line that starts from the camera and points through the ground is the ray, and the ground or course is the plane. Since the object you're selecting/detecting from the camera image is not always in the center pixel, you'll need to add a pan and tilt angle to the ray depending on which pixel the center of the object corresponds to, the <a href="http://wiki.ros.org/image_geometry">image_geometry</a> package has tools to help do just that. </p>
<p>As mig mentioned, you'll want to use tf to help keep track of all the transformation frames, and a urdf can make this even easier. </p>
<p>Once you know the ray, google "line-plane intersection". There is even a <a href="https://en.wikipedia.org/wiki/Line%E2%80%93plane_intersection">Wikipedia</a> article about it to get started. You might even get lucky searching for "how to do line-plane intersection in c++" or python or however you want to do it, and you might find some ready to use code. </p>
https://answers.ros.org/question/214116/2d-image-point-to-3d/?comment=214294#post-id-214294That's actually what I had in mind, using the plane z=0 (the ground). However I'm unsure how to obtain the ray, since I only have a 3x3 matrix of intrinsic parameters (camera matrix). And how should I get the scale factor? I'm a little bit lost.Wed, 22 Jul 2015 05:10:11 -0500https://answers.ros.org/question/214116/2d-image-point-to-3d/?comment=214294#post-id-214294Answer by mgruhler for <p>Hey!</p>
<p>I have a setup with a bottom camera (simple webcam pointed at the ground). Along with the camera is an IMU, which means that I have the attitude of the camera. I also have an estimation of the distance from the camera to the ground. </p>
<p>Since I'm detecting some target on the ground (I'm able to detect and get their position on the image using openCV), I would like to extract their position on the world frame. However I'm lost on how to do it.</p>
<p>Are there any ROS package that implement this? How should I do it?</p>
<p>Thanks in advance!</p>
https://answers.ros.org/question/214116/2d-image-point-to-3d/?answer=214174#post-id-214174You're saying you have the position of the camera in 3D already, right?
Then, you would have to write this yourself, but this is fairly easy with the [`tf` library](http://wiki.ros.org/tf) (best read through the documentation and the tutorials). However, you need to have your camera and IMU set up correctly in the `urdf`.
With the position of the target in the camera frame (i.e. `x,y,z` with respect to the camera), you can call `tranformPoint` (see [documentation here](http://wiki.ros.org/tf/Overview/Using%20Published%20Transforms#transformDATA_Methods)) to transform the Point from the camera frame into any other frame you have avaiable (e.g. the `base_link` frame of your robot or any `map` or `world` frame you have set up).Tue, 21 Jul 2015 01:07:08 -0500https://answers.ros.org/question/214116/2d-image-point-to-3d/?answer=214174#post-id-214174Comment by mgruhler for <p>You're saying you have the position of the camera in 3D already, right?</p>
<p>Then, you would have to write this yourself, but this is fairly easy with the <a href="http://wiki.ros.org/tf"><code>tf</code> library</a> (best read through the documentation and the tutorials). However, you need to have your camera and IMU set up correctly in the <code>urdf</code>.
With the position of the target in the camera frame (i.e. <code>x,y,z</code> with respect to the camera), you can call <code>tranformPoint</code> (see <a href="http://wiki.ros.org/tf/Overview/Using%20Published%20Transforms#transformDATA_Methods">documentation here</a>) to transform the Point from the camera frame into any other frame you have avaiable (e.g. the <code>base_link</code> frame of your robot or any <code>map</code> or <code>world</code> frame you have set up).</p>
https://answers.ros.org/question/214116/2d-image-point-to-3d/?comment=214206#post-id-214206Static tf is fine. Just assumed you'd have a robot... You say you have an estimate of the distance from the camera to the ground --> `z`.
Otherwise, from a monocular camera you cannot tell the distance to an object (as long as you don't know the exact parameters of the the object and estimate it).Tue, 21 Jul 2015 06:58:04 -0500https://answers.ros.org/question/214116/2d-image-point-to-3d/?comment=214206#post-id-214206Comment by nwanda for <p>You're saying you have the position of the camera in 3D already, right?</p>
<p>Then, you would have to write this yourself, but this is fairly easy with the <a href="http://wiki.ros.org/tf"><code>tf</code> library</a> (best read through the documentation and the tutorials). However, you need to have your camera and IMU set up correctly in the <code>urdf</code>.
With the position of the target in the camera frame (i.e. <code>x,y,z</code> with respect to the camera), you can call <code>tranformPoint</code> (see <a href="http://wiki.ros.org/tf/Overview/Using%20Published%20Transforms#transformDATA_Methods">documentation here</a>) to transform the Point from the camera frame into any other frame you have avaiable (e.g. the <code>base_link</code> frame of your robot or any <code>map</code> or <code>world</code> frame you have set up).</p>
https://answers.ros.org/question/214116/2d-image-point-to-3d/?comment=214203#post-id-214203Do i need to use the `urdf` model? Is it not enough to define a static tf between the camera and IMU?
How should I get the `x,y,z` with respect to the camera? I only have the `x,y` in the image, and `camera_calibration` only outputs the intrinsic parameters (from what I can tell).Tue, 21 Jul 2015 06:49:06 -0500https://answers.ros.org/question/214116/2d-image-point-to-3d/?comment=214203#post-id-214203Comment by Jägermeister for <p>You're saying you have the position of the camera in 3D already, right?</p>
<p>Then, you would have to write this yourself, but this is fairly easy with the <a href="http://wiki.ros.org/tf"><code>tf</code> library</a> (best read through the documentation and the tutorials). However, you need to have your camera and IMU set up correctly in the <code>urdf</code>.
With the position of the target in the camera frame (i.e. <code>x,y,z</code> with respect to the camera), you can call <code>tranformPoint</code> (see <a href="http://wiki.ros.org/tf/Overview/Using%20Published%20Transforms#transformDATA_Methods">documentation here</a>) to transform the Point from the camera frame into any other frame you have avaiable (e.g. the <code>base_link</code> frame of your robot or any <code>map</code> or <code>world</code> frame you have set up).</p>
https://answers.ros.org/question/214116/2d-image-point-to-3d/?comment=315109#post-id-315109I couldn't find any relevant tutorial to this question over there, could you pinpoint exactly which one is referring to 2d-3d coordinate conversion?Fri, 08 Feb 2019 08:09:38 -0600https://answers.ros.org/question/214116/2d-image-point-to-3d/?comment=315109#post-id-315109