How can I speed up image rectification?

edit

I hope to increase the rectified frame rate that I'm getting from image_proc. I'm feeding it a 64 fps 704x456 video stream, with that 704x456 rectangle being the region of interest sampled from a 1600x1200 sensor. The camera calibration is specified for the full-sensor size, and I'm using the Diamondback version of image_proc.

$ rostopic hz /camera/image_raw
average rate: 64.023
$ rostopic hz /camera/image_color
average rate: 64.113
$ rostopic hz /camera/image_rect_color
average rate: 27.558
$ top
  767 user      20   0  587m 109m  11m S  *115*  0.9 132:37.80 image_proc

Using earlier versions of image_proc and camera1394, I was able to get 38 fps rectified by choosing an MxN subrectangle [slightly larger than the current ROI] and calibrating with that. In that case, the Image and CameraInfo messages would both specify MxN pixels. Now, though, camera_info lists the full 1624x1224, and I suspected that something inside the rectify nodelet was copying my ROI onto a full-size image before undistorting anything.

$ rostopic echo /camera/image_raw
height: 456
width: 704
encoding: bayer_rggb8
is_bigendian: 0
step: 704

However, when I remove the ROI the undistorted framerate drops further:

$ rostopic hz /camera/image_color
average rate: 29.145
$ rostopic hz /camera/image_rect_color
average rate: 19.497

so I'm unsure of the cause of the slowdown. Can anyone see what's going on?