What's an algorithm or technique to solve this problem?

A 3D obstacle persists if there's lots of free space behind it when it moves away. I've observed this problem using costmap_2d's VoxelLayer plugin on ROS Jade.

Here's a side view of a ToF camera pointed at a green obstacle. The red triangle is the field of view, and the red dots are points returned by the sensor.

VoxelLayer will mark voxels as shown below. Green is free, yellow is occupied, and white is unknown. Here it correctly sees the green box as an obstacle.

Now imagine the green box moves out of the way vertically or sideways out of your screen. Behind the box is nothing but free space, so no points beyond it are returned by the ToF camera. Voxels need points behind them to be cleared, so the robot mistakenly believes an obstacle is still there.

The phantom obstacle is a problem, but I can't really fault VoxelLayer's logic. What if some sensor malfunction prevented the ToF camera from getting points on the box? How can I tell the difference between free space and an invalid sensor reading?