ROS Resources: Documentation | Support | Discussion Forum | Index | Service Status | ros @ Robotics Stack Exchange
Ask Your Question
0

interpolate function in util2d -- what is special about the way bilinear interpolation is implemented?

asked 2022-05-19 07:48:36 -0500

TurBot gravatar image

updated 2022-09-05 14:05:38 -0500

matlabbe gravatar image

.

 cv::Mat interpolate(const cv::Mat & image, int factor, float depthErrorRatio)
 {
         //UASSERT_MSG(factor >= 1, uFormat("factor=%d", factor).c_str());
         cv::Mat out;
         if(!image.empty())
          {
                  if(factor > 1)
                  {
                          if((image.type() == CV_32FC1 || image.type()==CV_16UC1))
                          {
                                  //UASSERT(depthErrorRatio>0.0f);
                                  out = cv::Mat::zeros(image.rows*factor, image.cols*factor, image.type());
                                  for(int j=0; j<out.rows; j+=factor)
                                  {
                                          for(int i=0; i<out.cols; i+=factor)
                                          {
                                                  if(i>0 && j>0)
                                                  {
                                                          float dTopLeft;
                                                          float dTopRight;
                                                          float dBottomLeft;
                                                          float dBottomRight;
                                                          if(image.type() == CV_32FC1)
                                                          {
                                                                  dTopLeft = image.at<float>(j/factor-1, i/factor-1);
                                                                  dTopRight = image.at<float>(j/factor-1, i/factor);
                                                                  dBottomLeft = image.at<float>(j/factor, i/factor-1);
                                                                  dBottomRight = image.at<float>(j/factor, i/factor);
                                                          }
                                                          else
                                                          {
                                                                  dTopLeft = image.at<unsigned short>(j/factor-1, i/factor-1);
                                                                  dTopRight = image.at<unsigned short>(j/factor-1, i/factor);
                                                                  dBottomLeft = image.at<unsigned short>(j/factor, i/factor-1);
                                                                  dBottomRight = image.at<unsigned short>(j/factor, i/factor);
                                                          }

                                                          if(dTopLeft>0 && dTopRight>0 && dBottomLeft>0 && dBottomRight > 0)
                                                          {
                                                                  float depthError = depthErrorRatio*(dTopLeft+dTopRight+dBottomLeft+dBottomRight)/4.0f;
                                                                  if(fabs(dTopLeft-dTopRight) <= depthError &&
                                                                     fabs(dTopLeft-dBottomLeft) <= depthError &&
                                                                     fabs(dTopLeft-dBottomRight) <= depthError)
                                                                  { std::cout << "Manual interpolation, depthError = "<< depthError << std::endl;
                                                                          // bilinear interpolation
                                                                          // do first and last rows then columns
                                                                          float slopeTop = (dTopRight-dTopLeft)/float(factor);
                                                                          float slopeBottom = (dBottomRight-dBottomLeft)/float(factor);
                                                                          if(image.type() == CV_32FC1)
                                                                          {
                                                                                  for(int z=i-factor; z<=i; ++z)
                                                                                  {
                                                                                          out.at<float>(j-factor, z) = dTopLeft+(slopeTop*float(z-(i-factor)));
                                                                                          out.at<float>(j, z) = dBottomLeft+(slopeBottom*float(z-(i-factor)));
                                                                                  }
                                                                          }
                                                                          else
                                                                          {
                                                                                  for(int z=i-factor; z<=i; ++z)
                                                                                  {
                                                                                          out.at<unsigned short>(j-factor, z) = (unsigned short)(dTopLeft+(slopeTop*float(z-(i-factor))));
                                                                                          out.at<unsigned short>(j, z) = (unsigned short)(dBottomLeft+(slopeBottom*float(z-(i-factor))));
                                                                                  }
                                                                          }

                                                                          // fill the columns
                                                                          if(image.type() == CV_32FC1)
                                                                          {
                                                                                  for(int z=i-factor; z<=i; ++z)
                                                                                  {
                                                                                          float top = out.at<float>(j-factor, z);
                                                                                          float bottom = out.at<float>(j, z);
                                                                                          float slope = (bottom-top)/float(factor);
                                                                                          for(int d=j-factor+1; d<j; ++d)
                                                                                          {
                                                                                                  out.at<float>(d, z) = top+(slope*float(d-(j-factor)));
                                                                                          }
                                                                                  }
                                                                          }
                                                                          else
                                                                          {
                                                                                  for(int z=i-factor; z<=i; ++z)
                                                                                  {
                                                                                          float top = out.at<unsigned short>(j-factor, z);
                                                                                          float bottom = out.at<unsigned short>(j, z);
                                                                                          float slope = (bottom-top)/float(factor);
                                                                                          for(int d=j-factor+1; d<j; ++d)
                                                                                          {
                                                                                                  out.at<unsigned short>(d, z) = (unsigned short)(top+(slope*float(d-(j-factor))));
                                                                                          }
                                                                                  }
                                                                          }
                                                                  }
                                                          }
                                                  }
                                          }
                                  }
                          }
                          else
                          {
                                  cv::resize(image, out, cv::Size(), float(factor), float(factor));
                          }
                  }
                  else
                  {
                          out = image;
                  }
          }
          return out;
  }

Why, in this function of util2d in rtabmap is the bilinear interpolation implemented manually? I understand that one would wish to check for the depth error ratio but why isn't this checked separately and then the cv::resize used for interpolation as a standard, and not only in case the error ratio is too large or the image of a different type than the two mentioned?

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
1

answered 2022-09-05 14:25:34 -0500

matlabbe gravatar image

That function is used only for depth images. For RGB/Mono, the cv::resize() function is used directly. For the depth case, we should not interpolate between pixels with depth difference above depthErrorRatio. For example, there is a pixel at 6 meters, the next one is at 50 meters, if you upsample by 4, you would get new pixels at 17, 28 and 38 meters between them, which is wrong. However, if there is a pixel at 49 meters and next one at 50 meters, you would accept interpolation 49.25, 49.50 and 49.75 meters.

In general, interpolating depth image is bad. It is used mainly in rtabmap just for the masking visual feature (opencv feature mask) which doesn't need to be super accurate, and never used to actually compute depth of a visual feature.

edit flag offensive delete link more

Question Tools

2 followers

Stats

Asked: 2022-05-19 07:48:36 -0500

Seen: 43 times

Last updated: Sep 05 '22