I'd like to implement support for RegionOfInterest for my camera driver but I'm not sure how to do it correctly. I'd like image_proc to be able to subscribe to the roi image and operate correctly, though it's not clear what correct operation would look like.

There is some helpful description at http://www.ros.org/reps/rep-0104.html#guidelines-for-camera-drivers 'Raw and Rectified ROI' section but it isn't always clear where responsibility lies for implementing a feature, and what numbers should be published in CameraInfo for use by image_proc.

If dorectify is true, does that mean the associated roi is in rectified coordinates? Should I calculate a much bigger raw roi to send to the camera using the bounding box of the distorted desired roi boundary to make sure there are no black pixels when the undistort happens later, but then publish the original desired roi on the camerainfo topic for use by imageproc which will cause imageproc to select a window within the undistorted image that has no border? But then how would imageproc know where to place the distorted image in the distorted coordinate frame? It seems like there isn't enough information present (unless imageproc is running it's own distort/undistort to get a mapping).

Right now I'm trying to set the camerainfo roi to be the distorted roi sent to the camera, but for some reason I have to divide the camerainfo roi x and y offset by 2 otherwise the image will get increasingly offscreen for larger x and y offset (could be a bug on my part), and I'm still not eliminating the undistort border- that could also be me improperly generating the distorted roi, but I still suspect I'm publishing the wrong thing to camera_info.

The way I like to imagine the process is that imageproc would create a large black image the size of the sensor width and height, place the subwindow image from the camera into it using distorted coordinates, then perform an undistort on the whole image, then window out a new roi image where the user originally requested and publish that. But again it seems like two sets of xy offsets have to change hands from camera driver to imageproc to accomplish that- or is the responsibility of some other node downstream to compute the window within the imageproc generated rectified image to match the desires of the original roi request with dorectify set to true?

Should I be able to toggle do_rectify on and off and see the same parts of the image that are visible stay put, while the distortion border moves around?