On 6 June 1944, the famous war photographer Robert Capa landed on Omaha Beach in Normandy, France with the soldiers of the American 1st Infantry Division. He was commissioned by Life magazine to photograph the invasion. One of the photos he took amidst a rain of bullets on D-Day captured a blurry image of a GI crawling in water towards land. When Life published the photos, a caption explained that the ‘immense excitement of [the] moment made photographer Capa move his camera and blur [his] picture.’

Conveying the chaos and confusion of the day, this seemingly flawed image is one of the most famous photos of Capa and a classic of war photography. But under normal circumstances few people are happy to see their photos blurred. The good news is they can now use the new Robust Motion Deblurring technology developed by Prof. Jia Jiaya of the Department of Computer Science and Engineering and his post-doctoral fellow Dr. Xu Li to fix the problem.

Seeing the World through Machines’ Eyes

A photography buff, Professor Jia has been in love with taking pictures since childhood. When he was a PhD student, he used a month’s studentship to buy a newly-introduced digital single-lens reflex camera. He says that many artificial elements can be added to the process of converting three-dimensional objects into two-dimensional images in digital photography. And this hobby is related to his research interest, namely, computer vision.

The field of computer vision involves helping computers to acquire the abilities of human vision, enabling them to make sense of the photo or video they see. Professor Jia said, ‘For example, when a computer sees a robbery or a vehicle collision through a surveillance video camera, it’ll be able to know that an accident is happening and will automatically alert the police.’ Deblurring is one of the basic issues of computer vision. He says, ‘This problem was first faced by astronomy photographers. When you take photos of stars and galaxies through telescopes, small vibrations can be magnified to giant blurs because of the long distance.’

What if a concert photographer finds that his photos are blurry and he doesn’t have a chance to retake them? Professor Jia says, ‘He may use the sharpening function of image processing software currently available on the market to increase the contrast of the images, thus making them appear less blurry. But that’s not a real solution and does not in any way resemble what we’re offering.’

Pixels are the building blocks of a digital image. Each pixel represents a tiny point on the object’s surface. Motion blur is the result of the relative motion between the camera and the object during the integration time of the image. As a result, a point on the object will become a line in the image. That means instead of corresponding to only a point on the object, now each pixel in the image represents more than one tiny dot on it, thus making the image blurry. And those lines depict the motion paths.

Turning Back Time

Professor Jia and Dr. Xu backtrack the blurring process by means of mathematical derivation and computer algorithms. The deblurring process involves two steps: first, estimating the direction of motion; second, using proper mathematical models to fix the problem. Professor Jia says, ‘That involves the mathematical concepts of convolution and deconvolution.’ He says simple algorithms such as partial differential equations and simple mathematical models were used in the past to restore blurry images. But those older systems needed someone to tell them the types of the camera motion, namely, vertical, horizontal or diagonal, before they could handle the problem. ‘Now we don’t need that. You give our software a photo. It’ll automatically detect the types of camera motion.’

The software can recover about 50% of blurry photos. Professor Jia says that it works best with images with motion blur, especially those resulting from frontal plane motion. It can also help to improve the quality of images with slight defocus blurs. The blurs caused by the motion of objects are a hard nut to crack. He explains, ‘If the blurred image is a jumping kid, his head may turn sideways, while his body moves in a horizontal direction and his arms flail back and forth. In other words, there are blurs caused by different types of motion in one image. That complicates the problem because identification of each motion direction is very difficult. Restoration of such an image requires careful analysis and the mixed use of multiple mathematical models targeted at different blurs. That makes the calculation time very long.’

But as computer hardware and software develop, they will continue to improve their algorithms to deal with more difficult cases. ‘The limitations we’re talking about now may be nothing a year later.’ Besides fixing professional or amateur photographers’ photos, this new technology can be useful for recognizing license plates in surveillance images or restoring blurred medical images.

As for Robert Capa’s iconic photo, it was later revealed that the blur was not caused by Capa’s shaking hands as suggested by Life, but by the carelessness of a darkroom technician, who turned on too much heat while drying the negatives. The excessive heat melted the emulsion and ruined the film. With regard to a photo blurred in such a disastrous way, Professor Jia says, ‘Well, I’m afraid we’ll need a specially developed computer algorithm to handle it.’