You've got all the information from the stereo calibration to construct an image_geometry::StereoCameraModel. You could use the projectDisparityTo3d function from the StereoCameraModel to get the 3D coordinates. Your input is a pixel location in rectified image (u, v) and the corresponding disparity (x2 - x1).