Ask Your Question
0

CV_Bridge converts nan to black values when using toImageMsg()

asked 2019-09-02 06:37:48 -0500

acp gravatar image

Dear People.

I have managed to implement the node that takes the /zed/depth/depth_registered and publish /zed/depth/depth_filtered. This node mainly takes the featureless areas and assigns nan to each pixel value.

You can see the pictures, and the title of the pic says the output.

image description

image description

I just have one problem, when I set nan values to the featureless areas in the depth image and then I convert the image from opencv image to ros with the function toImageMsg()

Using this line of the code Depth_Filter::image_pub_.publish(depth_filtered_.toImageMsg());

The cv_bridge converts all nan values to black. I need to have the image with the nan values in the ros topic.

Do you know something about that or how to solve that issue.

Regards and looking forward to your replay.

edit retag flag offensive close merge delete

Comments

The NANs are being lost when they're converted to integers during image transport. You'd need a floating point image format to be able to transmit this information within an image. I don't know if this is currently possible with the image encoders supported.

Could you use an out of range value such as zero instead of NAN? That would make your scheme work with standard image formats.

PeteBlackerThe3rd gravatar imagePeteBlackerThe3rd ( 2019-09-02 07:17:16 -0500 )edit

Hi, thank you for your replay. I do indeed need the ros topic image with nan values.This is the piece of code of the node. Do you have a clue how to solve this issue :) thank you.

     double dNaN = std::numeric_limits<double>::quiet_NaN();
    cv::Mat myMat = Mat(376, 672, CV_32FC1, cv::Scalar(dNaN));

    for(int i = 0; i < nRows; ++i){
     for (int j = 0; j < nCols; ++j){ 
       if(img.at<uchar>(i,j) != 0)
       myMat.at<float>(i,j) = cv_ptr_depth_->image.at<float>(i,j);}}

    cv_bridge::CvImage depth_filtered_;
       depth_filtered_.header.frame_id = depth_msg_->header.frame_id;
       depth_filtered_.header.stamp    = depth_msg_->header.stamp;
     depth_filtered_.encoding        = depth_msg_->encoding;
    depth_filtered_.image           = myMat;
Depth_Filter::image_pub_.publish(depth_filtered_.toImageMsg());
acp gravatar imageacp ( 2019-09-02 07:59:53 -0500 )edit

I'm confused as to why you need to transmit NANs. The code above is simply converting zeros to NANs, and using a 32bit float representation to store values between 1-255.

Very few image encodings are designed to store floating point values. TIFF is the only one I can think of, and I don't think it's supported by the ROS image messaging system.

PeteBlackerThe3rd gravatar imagePeteBlackerThe3rd ( 2019-09-02 09:21:44 -0500 )edit

Okay, thanks for the update, the code make more sense.

Fundamentally your problem is trying to transfer float type images via ROS. If you're transmitting depth images, then int16 is the common choice. It's got enough range you can encode with mm depth precision, and there's generally enough redundancy to define several special values so you can encode you're NAN elements.

PeteBlackerThe3rd gravatar imagePeteBlackerThe3rd ( 2019-09-02 10:41:13 -0500 )edit

Hi, thank you for the replay. You mean change the line: myMat.at<float>(i,j) = cv_ptr_depth_->image.at<float>(i,j); to <uint16_t>(i,j) = cv_ptr_depth_->image.at<uint16_t>(i,j);

By the way looking at the source code of cv_bridge.cpp

  • if (source->encoding == enc::TYPE_32FC1) {

            654             if (std::isnan(float_value)) {
            655               img_scaled->image.at<cv::Vec3b>(j, i) = cv::Vec3b(0, 0, 0);
    

Or how can I encode the nan values?? :) thank you

acp gravatar imageacp ( 2019-09-03 00:15:57 -0500 )edit

1 Answer

Sort by ยป oldest newest most voted
0

answered 2019-09-03 05:07:58 -0500

updated 2019-09-03 07:05:21 -0500

The easiest way of achieving what you want is to use a 16 bit integer image format. These are commonly used for depth images because standard image codecs support this format, and there is enough range to usefully describe the depth information.

You will probably need to check that this solution will work for the type of depth sensor you're using.

If a measurement of mm is used then an unsigned 16 bit integer can represent a range of depths from zero to 65.5 meters. All depths sensors I've ever seen have a maximum depth lower than this as well as a minimum depth somewhere between 30cm and a meter. This is why a depth of zero is used as a special value meaning no depth data available. You could use the same technique and use a depth of 1mm (again outside the range of values from the sensor) to encode a NAN value.

The code do this will look something like this. You'll need to convert the float depth value (which by convention should be in meters, but you'll need to check this) into mm. This code assumes that cv_ptr_depth_ is a pointer to an open CV Mat using single channel 32 bit float encoding in meters.

cv::Mat myMat = Mat(376, 672, CV_16UC1, cv::Scalar(1));

    for(int i = 0; i < nRows; ++i){
     for (int j = 0; j < nCols; ++j){ 
       if(img.at<uchar>(i,j) != 0)
         myMat.at<unsigned short>(i,j) = cv_ptr_depth_->image.at<float>(i,j) * 100.0;
       }
     }

    cv_bridge::CvImage depth_filtered_;
       depth_filtered_.header.frame_id = depth_msg_->header.frame_id;
       depth_filtered_.header.stamp = depth_msg_->header.stamp;
     depth_filtered_.encoding = "mono16";
    depth_filtered_.image = myMat;
Depth_Filter::image_pub_.publish(depth_filtered_.toImageMsg());

You'll have to do the reverse of this if you need to process it as a float image with NANs on the other side, but this way it is compatible with ROS image messages and can be viewed in RVIZ and other tools.

Hope this helps.

edit flag offensive delete link more

Comments

Hi. thank you for your replay. I have tried the changes you suggested but I get an error: CV_1UC1 so I used CV_8UC1. Then It compiles, but when I run the node it dies. I was thinking that since cv_bridge converts nan values into black, I could use just another node where I subscribe to /zed/dept/depth_filtered and there I assign nan to the black values and then republish it without of cv_bridge. Can I do that?

/*The push_back does not assign values to depth_filtered_ */ and How can assign nan values

void depthCallback(const sensor_msgs::Image::ConstPtr& depth_msg_) {

sensor_msgs::Image depth_filtered_;
int data_size_ = depth_msg_->data.size();
depth_filtered_.header.frame_id = depth_msg_->header.frame_id;
depth_filtered_.header.stamp    = depth_msg_->header.stamp;
depth_filtered_.data.resize(depth_msg_->data.size());

for(int i = 0; i < depth_msg_->data.size(); i++){
   depth_filtered_.data.push_back(depth_msg_->data[i]);}
acp gravatar imageacp ( 2019-09-03 06:48:04 -0500 )edit

My mistake, it was meant to be CV_16UC1 I've updated it now, that should stop it crashing!

You really want to avoid trying to publish images with float encoding (required for NANs) It's rarely needed, badly supported and inefficient.

PeteBlackerThe3rd gravatar imagePeteBlackerThe3rd ( 2019-09-03 07:09:50 -0500 )edit

Hi, once again, thank you. Yea it runs now and it does not crash :) The question now how can I recover the float image with NANs values. I have tried to make a node that reads the image (see code above) but I am struggling to assign values to sensor_msgs::Image depth_filtered_;

acp gravatar imageacp ( 2019-09-04 00:56:58 -0500 )edit

You won't be able to assign floating point values to the sensor_msgs::Image it doesn't support them, you'll need to make an OpenCV Mat and assign them to that.

PeteBlackerThe3rd gravatar imagePeteBlackerThe3rd ( 2019-09-04 04:43:34 -0500 )edit

Hi, and thank you again,

So in the end of the day I will have the same problem, right? If convert to opencv and then I assign nan values and then when I convert it to ros, cv_bridge will convert the nan values to black color. In this context, how can I have a ros image with nan values? In the code you proposed me, the nan values where encoded in ones, right? Now, how can I recover those values to nan. So I can have a ros image with nan values.

Thank you

acp gravatar imageacp ( 2019-09-05 01:33:25 -0500 )edit

You have to perform the same operation in reverse when converting your int 16 image (in mm) back to a float image (in meters). If the input value is one then set the output float value to NaN otherwise set the output float value to input / 100.

In short you can't have a ROS image message with actual NaN values, but you can encode the same information into an int 16 image. You just have to add an extra encoding and decoding step.

PeteBlackerThe3rd gravatar imagePeteBlackerThe3rd ( 2019-09-05 03:09:46 -0500 )edit

Hi, thank you for your replay. I am sorry to say, But I am a bit confuse. Is it possible for you to go a bit deep on how to implement things :). I mean, I do understand that I need to go back or in other words to do an inverse conversion. However, the question is where and how I do that inverse conversion. I mean, So far I have a node that reads the zed depth image and then it creates the following matrix cv::Mat myMat = Mat(376, 672, CV_16UC1, cv::Scalar(1)). In this matrix, ones that correspond to a nan remain unchanged (I guess this is the encoding :)) and then it multiplies by 100 the rest of the values. Then I convert the opencv image to ros image and publish it. I am right?. Then how I shall do the inverse. I mean shall I create ...(more)

acp gravatar imageacp ( 2019-09-06 00:40:09 -0500 )edit

To avoid an X Y problem, can we take a step back and find out what is it you're trying to achieve exactly? Why do you need to identify the featureless areas of your depth images, and what is that information being used for downstream in your system?

PeteBlackerThe3rd gravatar imagePeteBlackerThe3rd ( 2019-09-06 04:39:39 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2019-09-02 06:37:48 -0500

Seen: 26 times

Last updated: Sep 03