i am abit confused about the camera matrix in calibration. Do IR camera and RGB camera have different camera matrix?

If i have a point in RGB image plane and i want to convert it to world coordinate frame (ignore depth for the moment), should i use RGB camera matrix or IR camera matrix ?