Environment

• HW: Tutlebot3 Waffle with an INTEL NUC
  
• Ubuntu 20.04
  
• ROS2 Foxy
  
• ROS 2 TensorFlow (Alsora) ros2-tensorflow
• Vision_msgs/msg define at vision_msgs
• C++

Goal

I am subscribing to a message produced by tfdetectionpy server from the ros2-tensorflow package. I am using the following callback which I wrote as a quick proof of concept to test if my callback worked and yielded a valid message.

void detections_callback(const vision_msgs::msg::Detection2DArray::SharedPtr detections) const
{
  RCLCPP_INFO(this->get_logger(), "---- ");

  // A quick object detection santiy check 
  RCLCPP_INFO(this->get_logger(), "Detections time stamp seconds %i",
              detections->header.stamp.sec);

  // Undefined behavior when array bounds are out of range
  for (int objectCount=0; objectCount<5; objectCount++)     
  {
    RCLCPP_INFO(this->get_logger(), "  - Object %i", objectCount);
    RCLCPP_INFO(this->get_logger(), "       id %i %i %i",
                 detections->detections[objectCount].results[0].id[0],
                 detections->detections[objectCount].results[0].id[1],
                 detections->detections[objectCount].results[0].id[2]
                 );
     RCLCPP_INFO(this->get_logger(), "       score %f",
                detections->detections[objectCount].results[0].score);
   }
}

This is its output.

[INFO] [1644012380.017523928] [Image_Subscriber_OOP_Test]: Detections time stamp seconds 1644012379
[INFO] [1644012380.017541734] [Image_Subscriber_OOP_Test]:   - Object 0
[INFO] [1644012380.017557160] [Image_Subscriber_OOP_Test]:        id 56 52 0
[INFO] [1644012380.017571761] [Image_Subscriber_OOP_Test]:        score 0.716635
[INFO] [1644012380.017589226] [Image_Subscriber_OOP_Test]:   - Object 1
[INFO] [1644012380.017607646] [Image_Subscriber_OOP_Test]:        id 56 52 0
[INFO] [1644012380.017621308] [Image_Subscriber_OOP_Test]:        score 0.682781
[INFO] [1644012380.017635471] [Image_Subscriber_OOP_Test]:   - Object 2
[INFO] [1644012380.017648167] [Image_Subscriber_OOP_Test]:        id 55 50 0
[INFO] [1644012380.017660931] [Image_Subscriber_OOP_Test]:        score 0.607861
[INFO] [1644012380.017674397] [Image_Subscriber_OOP_Test]:   - Object 3
[INFO] [1644012380.017686806] [Image_Subscriber_OOP_Test]:        id 49 0 59
[INFO] [1644012380.017699744] [Image_Subscriber_OOP_Test]:        score 0.530216

The output is as expected. It displayed four detected objects:

 id 84 (Hex ASCII 56 52) "book"
 id 84 (Hex ASCII 56 52) "book"
 id 72 (Hex ASCII 55 50) "tv"
 id 1  (Hex ASCII 49)    "person"

See coco_labels

This matches what I see in the detections_image I displayed with showimage. This means that my subscriber is working.

Question 1

I want to clean up my code by parsing the Detection2DArray message. I want to write things similar to

vision_msgs::msg::Detection2D detections2d = detections2dArray->detections;
vision_msgs::msg::BoundingBox2D boundingBox = detections2d->bbox;

I can't get my types to match. I get the following kinds of errors. How do I fix it.

/home/eepp/eepp_ws/src/eepp_tf_detections/src/detections_list.cpp: In member function ‘void MinimalImageSubscriber::detections_callback(vision_msgs::msg::Detection2DArray_<std::allocator<void> >::SharedPtr) const’:
/home/eepp/eepp_ws/src/eepp_tf_detections/src/detections_list.cpp:56:69: error: conversion from ‘vision_msgs::msg::Detection2DArray_<std::allocator<void> >::_detections_type’ {aka ‘std::vector<vision_msgs::msg::Detection2D_<std::allocator<void> >, std::allocator<vision_msgs::msg::Detection2D_<std::allocator<void> > > >’} to non-scalar type ‘vision_msgs::msg::Detection2D’ {aka ‘vision_msgs::msg::Detection2D_<std::allocator<void> >’} requested
   56 |     vision_msgs::msg::Detection2D detections2d = detections2dArray->detections;
      |                                                  ~~~~~~~~~~~~~~~~~~~^~~~~~~~~~
make[2]: *** [CMakeFiles/detections_list.dir/build.make:63: CMakeFiles/detections_list.dir/src/detections_list.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:78: CMakeFiles/detections_list.dir/all] Error 2
make: *** [Makefile:141: all] Error 2
---
Failed   <<< eepp_tf_detections [4.33s, exited with code 2]

Looking at the message definitions in vision_msgs I get a data structure that I think should look something like

Detection2DArray
  Header header
  Detection2D[] detections
    Header header
    ObjectHypothesisWithPose[] results
      int64 id
      float64 score
      PoseWithCovariance post
    BoundingBox2D bbox
      Pose2D center
      float64 size_x
      float65 size_y
    Image source_image

// the above may contain interpretation and typing errors

Question 2

How do I find the end of each object detection and each set of image detections. The loop I wrote is fine for proof-of-concept but the program exits prematurely. C++ behavior is undefined in this situation. However, I should be able to detect the end of objects and sets of objects because the following gives me expected output and continues until I kill it.

ros2 topic echo detections

header:
stamp:
  sec: 1643753786
  nanosec: 607326389
frame_id: ''
detections:
- header:
    stamp:
      sec: 1643753786
      nanosec: 607326389
    frame_id: ''
  results:
  - id: '1'
    score: 0.5610617995262146
    pose:
      pose:
        position:
          x: 0.0
          y: 0.0
          z: 0.0
       orientation:
          x: 0.0
          y: 0.0
          z: 0.0
          w: 1.0
      covariance:
    - 0.0
    - 0.0  

// lots more

# A list of 2D detections, for a multi-object 2D detector.

std_msgs/Header header

# A list of the detected proposals. A multi-proposal detector might generate
#   this list with many candidate detections generated from a single input.
Detection2D[] detections

# Defines a 2D detection result.
#
# This is similar to a 2D classification, but includes position information,
#   allowing a classification result for a specific crop or image point to
#   to be located in the larger image.

Header header

# Class probabilities
ObjectHypothesisWithPose[] results

# 2D bounding box surrounding the object.
BoundingBox2D bbox

# The 2D data that generated these results (i.e. region proposal cropped out of
#   the image). Not required for all use cases, so it may be empty.
sensor_msgs/Image source_img

 void detections_callback(const vision_msgs::msg::Detection2DArray::SharedPtr detections2dArray) const
  {
    int isDone = false;
    int detectionsCount = 0;

    RCLCPP_INFO(this->get_logger(), "---- ");
    RCLCPP_INFO(this->get_logger(), "Object detection message");

    try
    {
      while (not isDone)
      {
        vision_msgs::msg::Detection2D aDetection = detections2dArray->detections[detectionsCount++];
        RCLCPP_INFO(this->get_logger(), " ");

        // Id and confidence
        vision_msgs::msg::ObjectHypothesisWithPose result = aDetection.results[0];
        int64_t id = stoi(result.id);
        float score = result.score;
        RCLCPP_INFO(this->get_logger(), "   id    %i", id);
        RCLCPP_INFO(this->get_logger(), "   score %f", score);

        // Bounding boxes
        vision_msgs::msg::BoundingBox2D boundingBox2d = aDetection.bbox;
        geometry_msgs::msg::Pose2D center = boundingBox2d.center;
        RCLCPP_INFO(this->get_logger(), "   position (%f, %f)", center.x, center.y);
        RCLCPP_INFO(this->get_logger(), "   size %f x %f", boundingBox2d.size_x, boundingBox2d.size_y);
       }
    }

    // Exception handling for array out of bounds behavior is undefined. The behavior in
    // this example may not be the same in other C++ environments. Behavior was susceptible to
    //   https://stackoverflow.com/questions/1239938/accessing-an-array-out-of-bounds-gives-no-error-why
    // Behavior is consistent enough in my environment to explore message parsing.
    catch (exception& e)
    {
      RCLCPP_INFO(this->get_logger(), "Exception %s", e.what());
      isDone = true;
    }
  }

// Create a vector object that contains 10 elements.
vector<double> v;
for (int i = 0; i < 10; ++i) {v.push_back(i + 0.14159);}
// Range-based for loop to iterate through the vector, observing in-place.
for( const auto &j : v ) {cout << j << " ";}