I am using a camera pair from Aptina, producing 1920x1092 px images. I created a ROS node for them and used the stereoimageproc/camera_calibration node to create a calibration. By now everything looks fine, the calibration looks good. Then I have the following tool chain for left and right camera respectively:

• aptina (delivers left/imageraw and left/camerainfo)
• bit_converter (node to convert to 8bit)
• image_proc/rectify
• imageproc/cropdecimate (cropping to 1920x880 for some project reason and reducing the resolution by a factor of 8/8 to 240x110px)

Having the same for both sides I use - stereoimageproc/disparity

And here the problems occur. I have the following left/camera_info:

header: 
  seq: 11793
  stamp: 
    secs: 1535645408
    nsecs: 924607110
  frame_id: "camera_102_left"
height: 1092
width: 1920
distortion_model: "plumb_bob"
D: [-0.009216, -0.092237, 0.009712, -0.010848, 0.0]
K: [1843.04496, 0.0, 920.519756, 0.0, 1842.589347, 578.078866, 0.0, 0.0, 1.0]
R: [0.999581, -0.027793, 0.008056, 0.027725, 0.999579, 0.008503, -0.008289, -0.008276, 0.999931]
P: [1958.152785, 0.0, 905.844528, 0.0, 0.0, 1958.152785, 565.23769, 0.0, 0.0, 0.0, 1.0, 0.0]
binning_x: 0
binning_y: 0
roi: 
  x_offset: 0
  y_offset: 0
  height: 0
  width: 0
  do_rectify: False

left/image_raw:

header: 
  seq: 1
  stamp: 
    secs: 1535645468
    nsecs: 953573126
  frame_id: "camera_102_left"
height: 1092
width: 1920
encoding: "bayer_grbg16"
is_bigendian: 0
step: 3840
data: [...]

And after the cropping left/cropped_info:

header: 
  seq: 16235
  stamp: 
    secs: 1535645708
    nsecs: 867018980
  frame_id: "camera_102_left"
height: 1092
width: 1920
distortion_model: "plumb_bob"
D: [-0.009216, -0.092237, 0.009712, -0.010848, 0.0]
K: [1843.04496, 0.0, 920.519756, 0.0, 1842.589347, 578.078866, 0.0, 0.0, 1.0]
R: [0.999581, -0.027793, 0.008056, 0.027725, 0.999579, 0.008503, -0.008289, -0.008276, 0.999931]
P: [1958.152785, 0.0, 905.844528, 0.0, 0.0, 1958.152785, 565.23769, 0.0, 0.0, 0.0, 1.0, 0.0]
binning_x: 8
binning_y: 8
roi: 
  x_offset: 0
  y_offset: 0
  height: 880
  width: 1920
  do_rectify: True

and left/image_cropped

header: 
  seq: 17126
  stamp: 
    secs: 1535645768
    nsecs: 963487656
  frame_id: "camera_102_left"
height: 110
width: 240
encoding: "bgr8"
is_bigendian: 0
step: 720
data: [...]

The resulting disparity_info looks:

header: seq: 0 stamp: secs: 1535646144 nsecs: 380223275 frameid: "camera102left" image: header: seq: 1677 stamp: secs: 1535646144 nsecs: 380223275 frameid: "camera102left" height: 110 width: 240 encoding: "32FC1" isbigendian: 0 step: 960 data: [...] f: 244.769104004 T: 0.175583317876 validwindow: xoffset: 70 yoffset: 7 height: 4294967281 width: 4294967218 dorectify: False mindisparity: 0.0 maxdisparity: 63.0 deltad: 0.0625

And here are my questions/problems: 1. The stereoimageproc/disparity output seems extremely slow, it produces at most one image per minute. 2. The stereomsgs/DisparityImage shows very interesting validwindow values. How are these produced? Are these the reason for the slow algorithm? They seem to be autogenerated somewhere, but I can't find out where.

1. As last step I try to convert the stereomsgs/DisparityImage into a pointcloud2 with stereoimageproc/point_cloud2 nodelet. This however produces messages but, and this is very strange, if I visualize in rviz are all located at one point in the room. It was hard to find them at all, they seem to have identical [x,y,z] position and are mapped to the same 3D point. Can someone explain that? The DisparityImage however does not look that bad, in the end it shows at least some regions very well.

The /disparity message looks:

header: 
  seq: 4
  stamp: 
    secs: 1535646474
    nsecs: 809055457
  frame_id: "camera_102_left"
height: 110
width: 240
fields: 
  - 
    name: "x"
    offset: 0
    datatype: 7
    count: 1
  - 
    name: "y"
    offset: 4
    datatype: 7
    count: 1
  - 
    name: "z"
    offset: 8
    datatype: 7
    count: 1
  - 
    name: "rgb"
    offset: 16
    datatype: 7
    count: 1
is_bigendian: False
point_step: 32
row_step: 7680
data: [...]
is_dense: False

To sum up everything, here is the right/camera_info message:

header: 
  seq: 9682
  stamp: 
    secs: 1535646684
    nsecs: 101092621
  frame_id: "camera_102_right"
height: 1092
width: 1920
distortion_model: "plumb_bob"
D: [-0.000306, -0.214565, 0.003271, -0.003276, 0.0]
K: [1831.564982, 0.0, 955.565706, 0.0, 1831.183872, 539.501467, 0.0, 0.0, 1.0]
R: [0.99996, 0.006618, 0.005947, -0.006568, 0.999943, -0.00841, -0.006002, 0.00837, 0.999947]
P: [1958.152785, 0.0, 905.844528, -343.818962, 0.0, 1958.152785, 565.23769, 0.0, 0.0, 0.0, 1.0, 0.0]
binning_x: 0
binning_y: 0
roi: 
  x_offset: 0
  y_offset: 0
  height: 0
  width: 0
  do_rectify: False

Can anyone see my mistakes? Please tell me if you need more information, launch files or pictures, I will provide them. However I assume there is a principal error in my tool chain? Thank you very much for your help.

P.S. A fourth question, I tried to do a calibration with matlab, which looks quite better than the ROS/OpenCV one. Unfortunately I have problems converting the matlab stereoParams object to ROS-specific camera_info messages. I think D and K are no problem, R, should be fine, too (the R matrix differs, but the rectified images look quite good). But the projection matrix P is a mystery to me.