Computer Scientists Scour Your Holiday Photos

Barence writes "Hundreds of thousands of images on Flickr are being used to teach a program to determine the geographic location of an image, simply by looking at it. The program attempts to mimic the way that humans can deduce the location of an image by searching for visual clues, such as similarities to pictures or locations they have seen previously. In its current state it can guess the location of a photo to within 200km, 16% of the time — extremely accurate given the complexity of the problem."

Where pictures are taken

[tomalpha (746163)]

The paper [cmu.edu] referenced in the article has an interesting density map of where their 20 million source photos were taken (ok, so they only ended up using 200 or so of these). It says it uses a logarithmic scale, and seems to imply that the vast majority of photos available to them on Flickr were taken in one of only a handful of locations:

• London
• Paris
• New York
• Washington
• Los Angeles
• Tokyo

Ok so there are a couple more than this, and my geography is appalling, but these seem to be the only areas that are are coloured red.

Re:Where pictures are taken

[elguillelmo (1242866)]

then... if there are 6 sources of pictures, by blindfold guessing you'll get it right 16.66..% of the time

Re:Where pictures are taken

[SteveAyre (209812)]

So it's actually less accurate than if it just guessed? :)

Re:Where pictures are taken

[Melfina (872932)]

It's like answering 'True' in a multiple choice test!

Re:

[Ethan Allison (904983)]

There are way more than 6 sources, check the map.

Re:Where pictures are taken

[Rich0 (548339)]

If you picked a random point on the globe, and I picked a random point on the globe, then they would be within 200 miles of each other a few percent of the time. If the only logic used by the software was to determine whether or not any land was visible it could probably increase that probability significantly - the earth doesn't have that much dirt poking out of the oceans. 200 miles is a VERY large area of land.

Re:Where pictures are taken

[Belial6 (794905)]

Now add to that, the fact that populations tend to bunch together, and you can massively increase your odds of those two point being within 200 km of each other. This is without any image recognition at all.

With the most basic of image recognition, you could narrow things even farther with things like, "Is there ocean in the picture?", "what is the height of buildings in the background?", or "how many people are in the background". One almost needs to ask how they got their accuracy so low...

Re:Where pictures are taken

[raehl (609729)]

Actually, the earth is pretty big - you'd have only a 0.0246% of being within 200km of someone, counting water. Get rid of water and you get to around 0.075%.

Re:Where pictures are taken

[1729 (581437)]

That's if you choose points at random. If you only choose points corresponding to cities with large populations that frequently use internet photo-sharing sites, then your chances of being within 200km of the location become much better.

Re:Where pictures are taken

[Falkkin (97268)]

If you RTFP, Figure 6 shows the difference between the performance of the algorithm and random guessing. It's pretty significant.

Re:

[Naughty Bob (1004174)]

then... if there are 6 sources of pictures, by blindfold guessing you'll get it right 16.66..% of the time

If they're smart, they'll only ever guess one of two points:

point a) Halfway between NY and Washington DC
point b) Halfway between London and Paris

This should give them a better than one-in-three chance of being correct to within 200km, as long as their program can take a decent stab at guessing which of the two sets is the more likely....

Re:

[elguillelmo (1242866)]

I'm not with you in the argument. Assuming there are just 6 cities, and that the proportion from each is the same: 1/6, if you guess randomly you are right 1/6 of the time. It's just like a die... Then, if there are zillions of sources but only six cities amount for most of the pictures, then randomly guessing among them will get you close to this 1/6...

Re:

[hagnat (752654)]

exactly, its like a d6 die... if i have to identify one (and only one) picture i'll have a 1d6 chance of getting it right

but when you have to roll this for 200 cities, also chosen by a 1d6 roll, you have two dies being rolled 200 times, and you want to know how many times both dies have the same value

Re:

[danaris (525051)]

No, I think you're misunderstanding how this works.

Given that there are 6 locations where all (or nearly all) of the pictures being used were taken,

Deriving from this that the probability of getting any 1 of these right by random guessing is 1 in 6,

Given that the accuracy averaged over 200 pictures is slightly less than 1 in 6,

The computer does slightly worse than chance.

However, my guess is that the first given is not entirely correct ;-)

Dan Aris

Actually, he kinda understands....

[raehl (609729)]

The dice analogy is right-on.

The problem is he just doesn't seem to realize that the chances of throwing doubles are 16.66%.

Dude where's my photo

[stainlesssteelpat (905359)]

Dude, that's as accurate as my girlfriends map navigation. *sigh*

Within 200km, 16% of the time?

[14erCleaner (745600)]

I'll guess...New York City, without even looking at the pictures that should get me in that ballpark.

Re:

[Bearpaw (13080)]

Wait, don't tell me, let me guess where you're from ...

What I need

[Intron (870560)]

... is a program that will remember the names of the people in the photos.

heh

[Kingrames (858416)]

Show it a picture of the andromeda galaxy and throw its statistics way off.

Re:heh

[somersault (912633)]

Nopes - it will guess "Starbucks", and there's always one within 200km, even in Andromeda.

Obligatory - I can get it within 6378km 100%

[SlashTon (871960)]

of the time...

(Not counting those rich bastards who can afford taking a holiday on the ISS).

Statistics is important

[UnknowingFool (672806)]

Just like all statistics getting a good sample population is very important. If this program were to sample the /. population, it would come to one of two conclusions.

1. We have no holidays as we don't socialize.
2. We all live within 1.0 km of a basement. :P

Re:

[raehl (609729)]

That was much funnier before I started working out of my basement.

Photosynth looks cooler

[Bombula (670389)]

The Photosynth multi-resolution and image-recognition tech demonstrated at TED looked cooler if you ask me:

metacafe link here [metacafe.com] and TED link here [ted.com].

Scientist make new discovery

[by Anonymous Coward]

Machine is shown hundreds of thousands of holiday pictures from Flickr.

Scientists surprised to discover it is possible for a machine to loose will to live.

Source code

[RandoX (828285)]

It's a for loop that spits out "Your mom's basement".

This is very hard

[mzs (595629)]

Look at this set of pictures:

http://htmlhelp.com/~liam/Hawaii/Kauai/WaimeaCanyon/ [htmlhelp.com]

Would you know simply by looking at the photos without the sign that this was not say the grand canyon? The whole correct to 200 km aspect is troublesome when the state of the art in computer vision cannot yet even answer that this is a picture of a canyon.

Re:This is very hard

[cheebie (459397)]

Would you know simply by looking at the photos without the sign that this was not say the grand canyon?

Yes, because there aren't 746 helicopters flying over it.

Actually, it's not as hard as you might think

[Khopesh (112447)]

There's an upcoming paper coming from MIT on this topic, Recognition of Natural Scenes from Global Properties: Seeing the Forest Without Representing the Trees [mit.edu] that proves this isn't as hard as you might think.

To sum up this massive paper in a very small (and likely highly imprecise) nutshell, building models up from basic objects (the traditional method) is only one way to approach this. Using this method, you are correct; it's impossible to understand what a canyon is. Using the new global properties

Re:

[Facegarden (967477)]

I would know it was Waimea Canyon because I've been there, and to the Grand Canyon, and as others have said the amount of vegitation and steepness of the walls differentiate the two very clearly, and i know that canyons that large are rare (on earth, above water), so it is unlikely to be some other place i just haven't been. :)
But i know a lot more than computers.

This all brings up an interesting point though... When you're in an unfamiliar place, but your friend knows the area, they can always tell you whe

Re:

[SpinyNorman (33776)]

Would you know simply by looking at the photos without the sign that this was not say the grand canyon?

Yes, because the GC doesn't have so much vegetation growing down the sides.

The whole correct to 200 km aspect is troublesome when the state of the art in computer vision cannot yet even answer that this is a picture of a canyon.

The two things arn't related. You don't need to know it's a canyon to be able to locate it - you just find the closest match in your database and give that as the location. You only

Just checked on flickr...

[stoofa (524247)]

OsamaBinLaden2001 has deleted his account

Re:Just checked on flickr...

[aadvancedGIR (959466)]

I don't think he has to worry, that program does not seem to be any better that GWB when it comes to determine the right country to attack.

Lies, Damned Lies, And...

[jesdynf (42915)]

200km, 16% of the time? I guess that sounds sorta neat... except that 84% of the time, it's off by more than 200km. Now, we know that the earth's circumference is 40000km, and it follows that nobody can ever be more than 20000km from any location on Earth.

So 16% of the time, it's accurate to within one percent of the TOTAL RANGE OF ERROR. The other 84% of the time, you're on your own. I wonder if I could manage that kind of accuracy just by sampling colors, classifying them by terrain, and then just picking a likely spot at random.

Re:Lies, Damned Lies, And...

[SQLGuru (980662)]

Surface area of a sphere = 4*pi*r^2
Radius of the Earth = 6 378.1 kilometers (from Google: http://www.google.com/search?hl=en&rlz=1T4ADBS_en__230US231&q=radius+of+earth [google.com] )

Surface area of Earth: 510,065,600 km2 (http://www.google.com/search?hl=en&rlz=1T4ADBS_en__230US231&q=surface+area+of+earth)

Percentage of surface area that is land: 29.2% (http://www.physicalgeography.net/fundamentals/8o.html)
Surface area of Earth that is land: 148,940,000 km2 (same source)

Area of a circle = pi*r^2
Radius of "target" = 200km
Area of target = 125663.7km2

Number of "target" areas that could fit on the surface of the Earth covered by land (assuming too few landmarks to identify pictures take over water, so they will be excluded): 1185.2

Chance of being right by pure dumb luck - 1 in 1185.2

Layne

Re:

[raddan (519638)]

Your post made me think of something-- the Earth is really BIG! But I'm a nerd, so I had to prove it to myself.

The surface area of the Earth, not counting water, is 510,072,000 km^2, according to Wikipedia. So, that's roughly 5.1 x 10^14 m^2. Again, according to Wikipedia, there are currently 6.67 x 10^9 people on Earth. That translates to about 7.65 x 10^4 m^2 for each person! In terms that Americans can understand, that's roughly 14 football fields (or to choose a landmark close to me, a little bi

Missing double blind

[RichMan (8097)]

From the looks of the test selecting London all the time would have a
1/6 chance = 16.67% chance.

They need better double blind testing and a more diverse set of geographical locations.

Re:Missing double blind

[zappepcs (820751)]

It's a bit worse than that I think. Trying to identify location of a picture by looking at it in the way that humans do requires that you know the location. As an example of why this implies intimate knowledge to be useful, everyone knows of the big statue of liberty. Not just anyone can guess that your holiday picture with the non-descript base in the background was taken at the base of the statue of liberty. The same goes for > 90% of other places in the world.

Another example: The forests on planets on the show Stargate One, are they in Missouri, Montanna, Canada? Just looking at them will not necessarily tell you anything unless you are intimately familiar with the actual location.

A photo in Syntagma Square in Athens may look like it was taken in Central Park in NYC if not enough of the background was included. It will take huge amounts of data and photos to get anywhere close to what a human can do at this job, and even then it is limited to only what it has seen before.

Other knowledge plays a part too. London bridge is now in Arizona (I think) as it was moved brick by brick and re-assembled. Seeing the bridge does not now mean you know where it is .... it's a trick question. The point is that you need additional information as well. A picture that is a beautiful park setting that has a kangaroo in it? is it in Australia, or a zoo? Additional information is required.

Hats off to them for working on it. It's a tough problem.

Re:

[darkstar949 (697933)]

Wikipedia to the rescue - http://en.wikipedia.org/wiki/London_Bridge [wikipedia.org] - London Bridge is actually pretty interesting as well as the original one was demolished, the second one was moved to Arizona, and the third one currently standing in London was built between 1967 and 1972.

Re:

[klenwell (960296)]

From the looks of the test selecting London all the time would have a
1/6 chance = 16.67% chance.

Indeed, not very impressive for London.

Look at this guy's [wikipedia.org] claim for basic audio analysis:

"Simply phonetics. The science of speech. That's my profession; also my hobby. Happy is the man who can make a living by his hobby! You can spot an Irishman or a Yorkshireman by his brogue. I can place any man within six miles. I can place him within two miles in London. Sometimes within two streets."

And that was almost a century ago!

Re:

[Woundweavr (37873)]

Accents used to be much more pronounced pre-radio/mass media. I know in Boston, individual streets had slight variations so that you could tell the neighborhood of a person by their accent. However, now that a large percentage of the words we hear are from movies/TV/radio our accents get washed out.

Blue screen your pictures

[BMonger (68213)]

I propose we all take pictures with blue screen in them (not the whole background, just "enough") and then write a script to randomly replace the blue screen with alternative locations every time the picture loads.

Moon Landing pictures!

[erroneus (253617)]

I'd like to present this with Moon landing pictures to see where the moon landing was staged! (hahaha... love it)

Automatic image recognition is no walk in the park

[hedu (1215514)]

Reminds me of the experiment done in a Dutch military lab a couple of years ago. They trained a neural network to recognize whether a photograph taken out on a country road had a military vehicle in it or not.

The system recognized the photos from the training set perfectly, but did no better than random on images fed to it that were taken at different times.

Turns out all the training shots with a military vehicle in it had been taken on a sunny day, and the control shots without one had been taken when it was overcast. The system had been trained to recognize a different thing from what they intended!

Google

[sckeener (137243)]

Google should get behind this. I think their Picasa would benefit from it.

Generate some autotags.

What would be nice also is if they had a feature where if you labeled someone in a picture, if you uploaded another picture with that person in the picture, the program would prompt to auto tag.

I've been going through old family photos and it would save so much time if the programs I am using autolabeled based off details in the picture.

Automatic Carmen San diego

[goombah99 (560566)]

Where's Goatse?

Re:Automatic Carmen San diego

[by Anonymous Coward]

Where's Goatse?

Uranus?

Re:Random pick is correct ~8% of the time

[by Anonymous Coward]

Re-check your math that is wrong.
Should be .08%

Re:It helps..

[gstoddart (321705)]

Do you think it helps or hurts that my photos on Flickr have titles like "Tokyo - Ueno park"?

For the researchers, it probably helps. They chose pics that had either GPS or location information -- so they could manually verify where the photos originated.

If they started out with a bunch of pics they didn't have any location information about ... they'd never be able to measure their results. ;-)

Cheers