Calibrating Kinect with surroundings

edit

To expand on rbtying's answer:

In ROS, transformations between different coordinate frames, such as the optical frame of the Kinect and the base of the robot arm are handled by the tf library. You just have to specify the transform between those two frames and then can use the tf library to compute the transformations between different frames.

Long answer:

First create an URDF file of your arm, upload it to the parameter server (param name "robot_description"), have some node publishing the joint_states of the current arm configuration and run the robot_state_publisher to integrate URDF and joint states into tf messages. Now you should be able to visualize your arm state in RViz.

Next, you have to specify the relative position of your Kinect to the arm. Assuming that this is a fixed transform, you can use the static_transform_publisher. For example, let's take a look at kinect_camera/launch/kinect_with_tf.launch:

<launch>
  <node pkg="tf" type="static_transform_publisher" name="world_to_kinect_rgb" 
    args="0 0 1.10 -1.57 0 -1.57 /world /kinect_rgb 10" />
  <node pkg="tf" type="static_transform_publisher" name="kinect_rgb_to_kinect_depth" 
    args="-0.03 0.0 0.0 0.0 0.0 0.0 /kinect_rgb /kinect_depth 10" />

  <include file="$(find kinect_camera)/launch/kinect_node.launch" />
</launch>

The first node specification gives the transform between the /world frame and the /kinect_rgb frame. Here, you have to replace /world by the base link of your robot arm, let's say /arm_base_link, if that is the name given in your URDF model. To find the correct parameters for that transform, you can use the tf_publisher_gui instead of the static_transform_publisher. Now you should be able to visualize both your robot model and the Kinect point cloud together in RViz. Fiddle around with the parameters to get your robot model lined up with the actual arm visible in the point cloud.

Now, whenever you have a point in the optical frame of the Kinect, you can use tf to just transform that point into the base frame of the robot arm.

Try making a URDF model and joint-publisher for your robot arm - the tf library will then take care of whatever transformations you need.

URDF tutorials

So I'm not personally familiar with this package and I don't know how applicable/reusable it might be on your problem, but it might be a good starting point: pr2_calibration.

You are describing a very good approach in your question: put markers on the end-effector of the robot, and observe those markers with the kinect camera. It sounds like you have some difficulties with the code to recognize the markers, but that does not make the approach less valuable. We apply the exact same approach when calibrating the PR2 cameras relative to the PR2 arm. We use checkerboard as markers; opencv has good methods to compute the pose of a checkerboard relative to a camera frame.