How to do simple pick motion with robot arm and 3d Camera?
The project I'm currently working on : Control a Techman Robot arm to grab a hanging apple (the robot needs to know the x,y,z of the apple and move to the location)
My progress : I added flange, tool, and camera model to urdf and the respective tf. I can control the real robot arm through moveit by dragging the blue ball on the moveit rviz interface.
My plan and problems :
- I want to control the robot arm with python or cpp code
- I want to know the x,y,z of the apple with my 3D Camera either by using yolov5 or other image processing software , but I only need the camera pixel value of x,y,z (can I write this function and controlling robot arm in a single python or cpp file?)
I want to convert and send the camera pixel value x,y,z to the robot arm (this is where I'm clueless what to do, I see many people says tf or tf2 can convert it, but how? the urdf tf only know the center of the camera model, not the exact position of the lens)
Version and Platform : Ubuntu 20.04 , ROS noetic , Camera : Intel Realsense D455
Techman Robot code link : https://github.com/TechmanRobotInc/tm...
I know it's a bit much but I really need help here, I'm stuck.
Thanks in advance !
The camera must be publishing data with respect to a camera related frame of reference like camera_link. You don't need to worry about lens reference frame since the given camera link in the urdf is well measured. You just have to transform data from camera frame to robot frame like base_link frame and eventually end effector frame.
Also as far as getting the object's x,y,z is concerned use some computer vision algorithm to detect the centroid of the detected apple (x,y) and using this calculate the y value since corresponding to a (x,y) you will have a unique z data from your 3d camera.
@Anubhav Singh , Thanks for the reply ! ,
(more)can you give me a simple example of "The camera must be publishing data with respect to a camera related frame of reference like camera_link" ? I'm quite new to ROS and don't know how to implement it, is it by writing a python publisher? how to write the code ? can you give me a simple example ? or do you know any tutorial about this ?
and for "You don't need to worry about lens reference frame since the given camera link in the urdf is well measured" , the camera_link is not from the company, I added it myself and the tf is on the center of the camera , is it still well measured ?
Lastly "You just have to transform data from camera frame to robot frame like base_link frame and eventually end effector frame." Should I write a code for this? or ...
rostopic list
. you can find all the relevant camera links.For the last part you can use YOLO to get the centroid of the detected object and using that (x,y) get the z from the camera data. Create a message type of
geometry_msgs/Pose
(Quaternion being 0,0,0,1). send this pose to relevant ROS functions likemove_group.computeCartesianPath()
to complete your task.1&2. Yes I have downloaded all the stuff and I can run it on rviz and I tried the examples, but can I run the topics and get the value in python code?
Yes I decided where I mount the camera, it's mounted on the robot arm and I described it already in the urdf file and the tf is on the center back of the camera (not the lens) and the joint between camera and flange is fixed.
Sorry if it's not clear, this is not a question, I just want to tell you that I already have the x,y coordinate value (for example 300,200 that I got from other image processing software)
yes something like this, although in your case since you are getting only x,y,z values, so your quat woud be 0,0,0,1
Today I made the following python code and succeeded in controlling the robot arm (I don't write import due to limited comment space) :
....