Hybrid Imaging 8 Years After

The hybrid camera project started at 2002 at Columbia University CAVE lab after I have noticed just how many of my images taken by my new Nikon CoolPix 995 came out blurred by motion blur.

This was definitely not the first attempt to deblur morion blur images, in fact INSPEC listed over 400 papers aiming at blind deblurring of motion blur images. Since the problem is very ill-posed, some assumptions were needed, usually some assumption about the type of the motion was taken – from linear and constrained speed motion to more complex model such as harmonic motion.

Unfortunately, these assumptions did not fit well the motion of a hand-held camera, which is cause by hand movement – in particularly during pressing the shutter release button.

This work was focused on measuring the camera motion during image capture, and it did it by utilizing a secondary video camera (see paper for detail). It is the first method that was able to compute the PSF and deblur images resulting from a relatively very long and complex motion path.

However, The deblurring algorithm itself (RL-deconvolution) was not very strong allowing for shift-invariant motion only (although the hybrid camera did compute rotation as well) and resulting with many ringing artifacts.

Later on, through the work of Fergus et at. and Levin at al and others, blind deblurring was greatly improved by replacing the assumptions about the motion path with statistical priors on natural images. Moreover, these methods results with much better deblurring nearly canceling all deblurring artifacts, and even enable deblurring of slightly mixed signals at occluded boundaries.  Flutter shutter photography by Raskar at al. can further improve the deblurring result by better conditioning the system by controlling the exposure pattern.

To date, hybrid imaging framework is still the best methods for measuring arbitrarily long and and complex motion path (and compute the resulting PSF). It can measures the PSF for each image during image capture, and when a single lens system is used (see paper) – it allows free change of lenses and zoom setting without needing to calibrate the system to the new settings.

A combination of hybrid camera, flutter shutter and prior based deblurring  should prove to be a very robust system for motion debluring including shift variant blur, and to some extent also deblurring of moving objects.

The work was cited over 200 time (average of 25+ citation a year) and appeared in the recent (2010/11) Eurographics State Of the Art (STAR) review on Computation Plenoptic Imaging. Hybrid cameras were also built and successfully used by other researcher.