Seeing Around Corners With Dual Photography

An anonymous reader writes "This project (which is part of this year's SIGGRAPH) has absolutely blown my mind. Basically they photograph an object with the photosensor at one point, and the light projector at another, and use the Helmholtz reciprocity algorithm to virtually switch the locations of the camera and projector, showing exactly what the light source "sees"! If that doesn't make sense to you, check out the research page and make sure to watch the 60MB video at the bottom. The playing card trick will leave you speechless!"

Why don't the editors link to mirrordot?

[cortana]

Mirrordot link to the video file [mirrordot.org].

Slashdotted already! Google cache here.

[aug24]

Clicky! [66.102.9.104]

Anyone please mirror the movie?

J.

Re:Does it work for...

[Spacejock]

A jail term. Or if you're really lucky, a fine and a photo in the local paper. ;-)

ARTICLE CONTENTS

[Anonymous Coward]

Dual Photography

Abstract

We present a novel photographic technique called dual photography, which exploits Helmholtz reciprocity to interchange the lights and cameras in a scene. With a video projector providing structured illumination, reciprocity permits us to generate pictures from the viewpoint of the projector, even though no camera was present at that location. The technique is completely image-based, requiring no knowledge of scene geometry or surface properties, and by its nature automatically includes all transport paths, including shadows, interreflections and caustics. In its simplest form, the technique can be used to take photographs without a camera; we demonstrate this by capturing a photograph using a projector and a photo-resistor. If the photo-resistor is replaced by a camera, we can produce a 4D dataset that allows for relighting with 2D incident illumination. Using an array of cameras we can produce a 6D slice of the 8D reflectance field that allows for relighting with arbitrary light fields. Since an array of cameras can operate in parallel without interference, whereas an array of light sources cannot, dual photography is fundamentally a more efficient way to capture such a 6D dataset than a system based on multiple projectors and one camera. As an example, we show how dual photography can be used to capture and relight scenes.

(a) Conventional photograph of a scene, illuminated by a projector with all its pixels turned on. (b) After measuring the light transport between the projector and the camera using structured illumination, our technique is able to synthesize a photorealistic image from the point of view of the projector. This image has the resolution of the projector and is illuminated by a light source at the position of the camera. The technique can capture subtle illumination effects such as caustics and self-shadowing. Note, for example, how the glass bottle in the primal image (a) appears as the caustic in the dual image (b) and vice-versa. Because we have determined the complete light transport between the projector and camera, it is easy to relight the dual image using a synthetic light source (c) or a light modified by a matte captured later by the same camera (d).

Re:around corners?

[Anonymous Coward]

Seeing around corners is really stretching it. You switch positions with the light source, so you can technically look at the scene from a point which is "around a corner". What they so casually mention as "structured lighting" is really the key to the whole algorithm and means that the light source shines a pattern on the scene which then allows the camera to retrace where every bit of light it sees is coming from. This means that the light source needs to be part of the scheme. You won't be able to switch yourself into the position of arbitrary lights on the street.

Re:Does it work for...

[Anonymous Coward]

Not what you think it does. The algorithm can only reconstruct points which participate in the light transport from the projector to the camera. Light which is absorbed in the scene won't be traced.

Structured light.

[Anonymous Coward]

They make the point that if you illuminate an object with a projector, you can get the image with a photocell. That's because the projector scans the image with a light beam. If you know when you see the reflection, you know where the light beam was when it reflected because you have prior knowledge of the scanning pattern. That technique has been used forever. It's like the flying spot scanners that predate camera tubes.

The 3D part is obtained when you offset the detector and the projector. If I look at a particular point on an object and scan the object with a beam of light, I can get the distance between me and the object as a function of the scanning angle.

It's all very impressive, but..

[tonywestonuk]

... a form of This technique has been done before. Take a bar code for example. A bar code could be read in 2 ways

• {usual method} laser scans over barcode, light sensor picks up changing intensity of light, as the light is either reflected, or absorbed by the pattern.... or
• Camera take photo of barcode in one go.

All these people are doing, are using the first barcode technique to, take a picture of the scene. Instead of using a laser, an animation of a moving white dot is sent to the projector. The Camera, is then treated like a light sensor, for each point in the animation, the camera is queried for the brightness of the perhaps, brightest dot in it's field of view. Gradually the picture is built up, pixel by pixel, untill, finally a picture is formed in memory. This picture would be from the perspective of the projector.

Re:Military applications?

[Technician]

It seems like this might have some military applications as a result. Imagine sticking a photo-resistor array under a door or through a window and then getting "viewpoints" from any of the lights in the room. Could aid in target aquisition and elimination.


If you can get to the article, it mentions the light source as a projector. The projector controls the resolution. How it works is a raster scanning video projector lights objects. A photoresistor (in my opinion way too slow. A fast photodiode would be better or photomultiplier tube) picks up the reflected light from the object scanned by the light projector.

A simple street light or the ceiling light in the room will not modulate the light to provide an image signal on a photo sensor slid under a door. On the other hand, if they were doing a video presentation, and the presenter walked between a projector and the screen and you had a photoresistor slid under the door, you would be able to see his arm movements.

You would get the best image when the projector was not showing a slide, but showing a blank screen. Use a CRT projector, not an LCD. LCD's don't raster scan.

University of Virginia Mirror to Video

[Pavan_Gupta]

Well, I've begun the download for this video, and seeing as how mirrordot is being slashdotted, I have only downloaded about 20 megs out of the 60 meg file, with an ETA of about 25 minutes. At any rate, I've put the mirror up linking to the file that's being created -- and in 25 minutes that file will be complete, until then it'll be some percentage of the total.

Enjoy. [virginia.edu]

Re:rays?

[Wyzard]

If you mean in the sense that POV-Ray does, then no, this is very different. It's an "image-based" rendering technique, which means that you create new images using photographs and other such real-world measurements as input. Conventional ray tracing gives you pictures of models built in the computer's memory, which might approximate a real-world object.

The important difference is that you don't have to build a computer model of the geometry you're trying to render. This is both a help because many real-world objects are hard to model accurately in a computer, and a hindrance because you can only render pictures of objects that you actually have in the real world.

IANAS, but it looks like reverse 3d rendering...

[capsteve]

I totally lack any scientific degrees, but this technique looks an awful lot like raytracing in reverse(or even real world application of algebra)... the projector is necessary to help map the way certain areas of the subject react to light based on the surface quality, and using pixel level illumination from the projector recreates the camera... FUCKING BRILLIANT.

this technique works because of the lcd/dlp array in a projector, but i wonder if it can be reproduced if the light source is already a pinpoint(chrismas light, or very small bulb). what happens when the light source is very broad, like that of a computer monitor/ TV? i wonder if this technique could also be used to extrapolate what someone is watching/reading/viewing on screen? taking another stab from a raytracing perspective, i wonder if an environment could be revealed thru image analysis, aka reverse-HDRI?

hats off to the dually photo boys of stanford and cornell... keep up the cool work.

Re:around corners?

[indy]

The parent is right. You will not be able to see things that were hidden to the camera.

All you are going to see is the scene as if camera and light source had switched places. Everything that was hidden to the camera in the original image will fall into black shadow regions in the generated image.

Re:Structured light.

[Technician]

It's like the flying spot scanners that predate camera tubes.


Wow, you remember those?

For those who don't know what they are, it's simply a CRT with a blank raster and a photo detector. Usualy a photomultiplier tube (fast and before photodiodes). The flying spot was simply the bright spot on the CRT. If you put movie film in front of the CRT, the brightness detected by the photodetector was modulated by the film in-between. This was the standard way of showing movies on television in the early days. The flying spot scanner was built into a movie projector with a CRT for the lamp and a photomultiplier tube where the projection lens would go.

In this example, it's a very big flying spot scanner. The lightsource is a projector. (raster scanning light source) The target is a 3D object instead of movie film, and the detector is offset so the 3D object casts shadows to the detector.

The scanned image looks like it would be viewed from the light source with shadows that look like the light source is from the photo detector.

Torrent

[spadadot]

Only the first part for now :

http://dload.digitalriviera.com/DualPhotography-pa rt1.mp4.torrent [digitalriviera.com]

Second part in 30 minutes !

First torrent I host, I hope it's ok.

Re:around corners?

[Anonymous Coward]

The virtual camera in the position of the projector will see things which the real camera can't see. The important point is that the camera becomes the virtual light source. If a light at the position of the real camera would not light the scene sufficiently for a camera at the position of the projector to capture a good picture, then the algorithm won't produce a good result either. In the pictures on the website you can see that the shadow on the computed picture is actually like you would expect when the scene is lighted from the position of the camera. The algorithm does not change the camera position without changing the light position.

Re:Torrent

[pairo]

http://bragi.wolfheart.ro/DualPhotography.mp4.torr ent [wolfheart.ro]

Re:Why don't the editors link to mirrordot?

[akadruid]

And not even mirrordot can cope with the load.

Try the nyud.net mirror [nyud.net] instead. Works for me.

Re:Why don't they just move the camera?

[Technician]

With this technique, 'any' light source can function as a point of view.

No it can't. The light source must scan the target, not just illuminate it.

The only place I know of with a scanning light source that might be exploited is the confrence room. A photodetector would able to get a raster image of the Power Point presentation in the room and the presenter when he walked in front of the screen and became a scanned object.

Optical TEMPEST

[Anonymous Coward]

Reminds me of this Slashdot article [slashdot.org] on an "optical TEMPEST" device to "eavesdrop" on the display put out by a CRT monitor. This has been around for a long time by picking up electromagnetic emissions, but this method used the light from the CRT itself, as reflected off of walls and such: a photomultiplier tube and a fast digitizer allows you to reconstruct the image from seeing how the light in the room very quickly goes from light to dark and back as individual pixes are painted on the CRT, given a knowledge of the CRT scan rate and resolution.

Another mirror...

[Malcolm Scott]

Another mirror here [retrosnub.co.uk]. No guarantees as to how long it will stay up; if it pushes me close to my monthly bandwidth limit I'll kill it...

Re:IANAS, but it looks like reverse 3d rendering..

[Solder Fumes]

It is already possible and demonstrated to view what is on a CRT by analyzing the brightness changes of the surrounding room through a telescope.

Torrent file

[Bisqwit]

BitTorrent file here. http://bisqwit.iki.fi/torrents/DualPhotography.mp4 .torrent [bisqwit.iki.fi]

Mirror

[Anonymous Coward]

http://www.nrsmith.me.uk/~neill/temp/DualPhotograp hy.mp4 [nrsmith.me.uk]

Re:around corners?

[MankyD]

Half truth:

If you watch the video, the very last demonstration is that of them generating the image of a King (of hearts?) that was not directly visible to the camera. Rather, its face was reflected onto the page of an open book - much more complicated that just, say, a mirror. The cards reflection is not visible in the still image of the book and is only made possible through pixel scanning with the projector.

In sum, they are seeing around a corner and are seeing something the camera could not see (directly).

Re:Why don't they just move the camera?

[Breakfast Pants]

Only if it is a CRT projector, which are increasingly rare.

Norwegian U. of Science and Technology Mirror!

[vidnet]

I will, in the mean time, here's another mirror [stud.ntnu.no]

Direct http mirror here

[stric]

http://ftp.acc.umu.se/mirror/media/DualPhotography /DualPhotography.mp4 [acc.umu.se]

Re:Why don't the editors link to mirrordot?

[j-beda]

Because it can't handle it either? Maybe this torrent [wolfheart.ro] will work?

Re:buh-bye server...

[j-beda]

Here is a working torrent [wolfheart.ro].

Re:Why don't they just move the camera?

[merlin_jim]

How far could you get with all the information escaping the window in your direction?

It's called optical tempest. With a high enough sampling rate you can reconstruct what is being shown on the monitor/TV. Each pixel as it illuminates causes a brief spike in the ambient brightness; by measuring this spike one can reconstruct the pixels being shown. After that, it's pretty simple to find the horizontal and vertical retraces.

more info [slashdot.org]

Re:Why don't they just move the camera?

[Anonymous Coward]

Except the OP was explicitly NOT interested in reconstructing CRT images, but rather images of the surrounding room.

Re:It's all very impressive, but..

[mrmojo]

Yes, and in addition, it's not a pixel by pixel scan across the scene. They use clever adaptive methods to take the minimum number of shots, which is the log of the resolution in the best case, and degenerates to a pixel by pixel scan only when the scene is horribly interreflective.

I work with these guys, it's a really cool project.

Re:Another mirror...

[Anonymous Coward]

Another mirror here. No guarantees as to how long it will stay up; if it pushes me close to my monthly bandwidth limit I'll kill it...

That's very generous of you. But since it's 2005, why not just participate in the existing torrent? It's faster and avoids a concentrated burden on one (generous) host...

Re:To find out if I understand this

[Rothron the Wise]

So, could you do something useful with a structured light source and a structured receiver? Or would you just get redundant information?

You get redundancy or rather parallelism which is used to speed up the process.

Re:Blame The Slashdot Editors

[cft_128]

OTOH, by having people go to mirrordot after the site goes down they can be assured that they got the maximum number of hits that they can handle (and therefor the maximum amount of ad revenue that you could at the time) before people starting viewing the mirrors and bypassing the ads. Flawed I know, but it should not be ignored.