AprilTag double-applying CameraInfo projection matrix

edit

I’m trying to use the AprilTag package to detect and localise tags in space using a stereo camera pair, and I’m running into an issue where tags are inaccurately localised due to how AprilTag applies the projection matrix to rectified images.

My workflow looks like this:

raw camera image ➝ stereo_image_proc ➝ rectified camera image ➝ apriltag_ros ➝ relative tag pose

Here’s the problem: stereo_image_proc uses the CameraInfo KDRP matrices to process and rectify the raw images. But the AprilTag module then applies the projection matrix onto the already rectified image to obtain the tag pose, and calculates inaccurate pose information as a result.

To test this, I’ve been publishing a duplicate CameraInfo topic solely for the AprilTag module and editing the matrix values manually. Despite what’s claimed in the wiki, AprilTag doesn’t seem to use the K (camera) matrix at all; there’s no change in tag detection or localisation if I modify these values or even set them all to 0. Conversely, modifying the projection (P) matrix does result in changes to where tags are localised with respect to the robot.

Additionally, if I provide AprilTag with unrectified images the tag localisation is correct for tags directly in front of the camera (but obviously incorrect towards the edges due to distortion).

My question is, what’s the correct way of dealing with this double application of the P matrix, or can I prevent it? I can publish a ‘hacked’ version of the CameraInfo topic to AprilTag that results in correct tag localisation, but this is obviously not a clean solution. Alternately, I could probably generate a projection matrix that results in correct tag localisation after double application, but then the output of stereo_image_proc would be incorrect for all other uses.

How have others resolved this issue?