why the left 3x3 portion of Projection matrix(P) is different from the Intrinsic camera matrix(K)?

edit

Have a look at this page, which is also linked in the sensor_msgs/CameraInfo comments: https://wiki.ros.org/image_pipeline/C...

Specifically this part:

4.1 Simple monocular rectification

In this case, there is a monocular device with distortion correction only. To transform a pixel in the input image into one in the output image, it is sent through the K - D - R - K' series of transformations. K - D gets to the normalized, undistorted image; the rotation R is the identity because we don't want to rotate the normalized undistorted image; and then K' converts back to pixel coordinates in the output image. In this case, since there is no need of in-image-plane translation, or scaling from the original image, so K = K' , and only the K and D elements of CameraInfo are needed. But since ROS Electric, the camera_calibration package does monocular calibration to get K' using OpenCV's getOptimalNewCameraMatrix() function with argument 'alpha'=0.0 that cause K' ≠ K.

Projection onto the output image is by the P matrix, which is formed from K' and the optional rotation and translation, in this case the identity and 0, respectively. So if K' = K, P is redundant information, and could be formed directly from K.

The important point here is that the left 3x3 portion of P is computed by getOptimalNewCameraMatrix() function with argument 'alpha'=0.0. If you undistort an image, it is no longer rectangular, and you'll get a pincushioning effect like this:

Since the image has to be rectangular to be stored in memory, the remaining pixels are filled with a black color.

The getOptimalNewCameraMatrix() scales the camera matrix K and computes the new image size so that the undistorted (rectified) image either contains no invalid pixels (alpha = 0.0), or so that all original pixels are contained in the image (alpha = 1.0) or something in between. Also, the new image size for the rectified image has to be specified. The camera_calibration package uses alpha = 0 (i.e., crop the image) and output image size = input image size.

And if you're working with stereo camera pairs, it gets even more complicated (see the page I linked above), so in that case P can never be replaced with just using K.

Short rule of thumb: If you're working with rectified images, use P. If you're working with unrectified (raw) images, use K.