Searching by Image Instead of Keywords

Content based image retrieval (CBIR), the technique to search for images not by keywords, but by comparing features of the images themselves has been the focus of much research ever since the web emerged. Consider for instance adding CBIR to Google Images, where you would be able to search for images similar to a query image instead of using keywords. A research project at Penn State University has recently been applied to the biggest aviation photo database in the world with close to 800,000 images. You can search for images similar to a photo already in their database (click "View similar photos") or submit your own query image. Some queries generate better results than others but CBIR is certainly here to stay and will be standard in many image applications of the future.

Some relevant research papers

[FleaPlus]

There's a bunch of interesting papers out there on content-based image analysis and retrieval. Below is a sampling from my bibtex file. Does anyone else have others they'd like to share?

* Finding Naked People [hmc.edu] (Fleck et al, 1996)

* Video google: A text retrieval approach to object matching in videos [ieee.org] (Sivic & Zisserman, 2003): web page demo here [ox.ac.uk]

* Names and Faces in the News [columbia.edu] (Berg et al, 2004)

* FACERET: An Interactive Face Retrieval System Based on Self-Organizing Maps [springerlink.com] (Ruiz-del-Solar et al, 2002)

* Costume: A New Feature for Automatic Video Content Indexing [www.irit.fr] (Jaffre 2005)

Re:Old photos

[znaps]

It is, I was on it in December.

It's an amazingly scary experience to sit there at night when a large plane is landing.

One more: automatic film character retrieval

[FleaPlus]

I forgot one more, where specific faces were automatically retrieved from feature-length movies and Fawlty Towers:

Automatic Face Recognition for Film Character Retrieval in Feature-Length Films [cam.ac.uk] (Arandjelovic & Zisserman, 2005)

The objective of this work is to recognize all the frontal faces of a character in the closed world of a movie or situation comedy, given a small number of query faces. This is challenging because faces in a feature-length film are relatively uncontrolled with a wide variability of scale, pose, illumination, and expressions, and also may be partially occluded. We develop a recognition method based on a cascade of processing steps that normalize for the effects of the changing imaging environment. In particular there are three areas of novelty: (i) we suppress the background surrounding the face, enabling the maximum area of the face to be retained for recognition rather than a subset; (ii) we include a pose refinement step to optimize the registration between the test image and face exemplar; and (iii) we use robust distance to a sub-space to allow for partial occlusion and expression change. The method is applied and evaluated on several feature length films. It is demonstrated that high recall rates (over 92%) can be achieved whilst maintaining good precision (over 93%).

Re:Wow

[theguyfromsaturn]

If you are only interested in searching for images on your own computer, have a look at imgSeek. http://imgseek.python-hosting.com/ [python-hosting.com]

It's been around for some time now. You can not only use an existing image to search, but also do a rough sketch. Check the screenshots: [sourceforge.net]

Nice complement to what has been presented in this article.

Re:Old photos

[stuffman64]

No, it still happens.

In fact, there is a bar [sunsetbeachbar.com] located right in the flight path of the runway. I just met a guy who came back from there, and said it's quite interesting to have planes landing so close to you.

Re:Old photos

[eander315]

They still fly that approach every day. Here's a picture [airliners.net] taken this year from the same beach on St Maarten (SXM). That airport is famous for it's low approach. That's a nude beach, by the way, and there are many photos to prove it if you dig around airliners.net. From Wikipedia [objectsspace.com]:

"The island is served by many major airlines that bring in large jets, including Boeing 747s, carrying tourists from across the world on a daily basis. This fuels the island's largest revenue source, tourism. The airport is famous for its short landing strip - only 2130 meters, which is barely enough for heavy jets. Because of this, the planes approach the island flying extremely low, right over the beach. Countless photos of large jets flying at 10-20 meters over relaxing tourists at the beach have been dismissed as photoshopped many times, but are nevertheless real."

Re:Pretty controllable test

[Anonymous Coward]

It seems that a favorite use of the image similarity search over there at airliners.net is for the spotters to run pix on airline and flightsim sites through the search, to see who on anet has been infringed upon copyright-wise.

Look up Bombardier in the forums on airliners.net, they have frequently asked a photog for permission to use their photos (for pay), then later say they elected not to use them (and therefore no payment to photog). But then they use the photos anyways without payment or acknowledgement to the photographer.

So these spotters trawl the web looking for aircraft photos to 'vet' to see if they are stolen from an a.net photographer and band together to stamp out the piracy (sound familiar??????)

Oh, you mean like imgseek?

[mr_zorg]

Oh, you mean like imgseek [python-hosting.com]?

Been going on for YEARS...

[FatSean]

IBM was pushing this in 1996/1997 with DB2 v...uh...4? Didn't work all that well then...and it was just basic shape outlines.

Pie. Sky. In.

Purdue University's 3D Shape Search

[karthik_r085]

Purdue also has a 3D shape search. More can be found at Here [slashdot.org].

altavista

[heatdeath]

I'm pretty sure altavista had this feature several years ago, but removed it. I remember that it worked fairly well. Does anyone know what happened to it?

Worked on something similar

[Sirch]

About a year or so ago, I and three other Masters students worked on a similar project at the University of Southampton.

I've not RTFA (not had the time), but our approach was to split the images into segments (based on colour and texture) which were assumed to be objects. The segments would then be analyzed for various feature vectors, such as shape, texture, colour etc. These vectors would then be added into a database of numbers, and finally the segments grouped, giving a collection of classified sections which (hopefully) represent similar objects.

From related metadata such as keywords, you could then hope to build up an idea of what keyword matches which section. You could also come up with a relevance between two images, and thus search for similar images.

We didn't have enough time to make it bulletproof by any means, but our limited results were very promising.

Sorry I can't find the paper, but we've got some screenshots of the application here [soton.ac.uk] and here [soton.ac.uk] (you can see false colouring applied to the original image to display the segments)