Searching by Image Instead of Keywords

Content based image retrieval (CBIR), the technique to search for images not by keywords, but by comparing features of the images themselves has been the focus of much research ever since the web emerged. Consider for instance adding CBIR to Google Images, where you would be able to search for images similar to a query image instead of using keywords. A research project at Penn State University has recently been applied to the biggest aviation photo database in the world with close to 800,000 images. You can search for images similar to a photo already in their database (click "View similar photos") or submit your own query image. Some queries generate better results than others but CBIR is certainly here to stay and will be standard in many image applications of the future.

Wow

[themoodykid]

I was just thinking about this the other day. I think content-based image search is one of the Next Big Things. Cameras are so ubquitous now (for better or worse), but having to rely on metadata to give meaning to images requires lots of effort up front.

It will be interesting if we ever get to a stage where we can just search for a random object (or person) in a database of photos. Then you could take pictures of everything with an always-on camera and if you need to find where you put your car keys, just do a search.

Re:wtf?

[Rei]

Because it still has problems - you'll note that the pictures seem to be compared simply based on color similarity. That's the same thing imgSeek [python-hosting.com] does (a great program for sorting and searching your photos) on photo searches. It works wonderfully if you're searching a very limited picture subset (say, airplanes), but if you search a wide variety of pictures, the results can be quite amusing.

Old photos

[green pizza]

Those must be old photos. There is no way that beach would be open to the public in the post 9/11 world.

Skin Cancer Detection

[Anonymous Coward]

Talking about greatness to society and a little bit of skin. At university one of my projects was a system that used CBIR to try and diagnose skin cancer. The doctor would take an image of the suspect area it then would be compared against a database of cancers. It would then return a suggested likelyhood of being cancer. It also allowed the doctor to build a history of images allowing easy comparision over time.

I always felt good about working on projects like this, gives a warm fuzzy feeling.

and then we have reverse "Googling" for images..

[dotpavan]

Here is a google game which is reverse of google's image search:

One has to guess the search word which generated a given set of 20 images in google's image search [robinson.name]

When things are moving forward, we have soomthing to talk about "those good ole days" but frankly the game is interesting initially but later gets boring due to the frequent repetitions..

Is it just colour?

[Bifurcati]

I just did a quick search based on this [designer.am] image of a Qantas logo (that's the main Australian airline, in case you're wondering...) It's red, with a white kangaroo in the middle. My theoretical aim was to find photos of Qantas planes.

What I got was an awful lot of red planes - some of which were actually Qantas planes, but I think more by coincidence (i.e., they're red) than design. Many images had nothing to do with Qantas, or even a red plane - they simply had a lot of red in the image.

This is impressive in some ways, but in others it seems like it's simply looking for similar patches of colour. I haven't done enough testing to see what happens if,say, I gave it a half red half green image.

Interesting, but not ready for public consumption just yet. A bit like A.L.I.C.E. the artifial intelligence system actually - neat, but not practical. Yet!

IP Enforcement Nightmare

[buckhead_buddy]

The Mona Lisa (famous and out of copyright) is often plagarized in whole or in part as part of commercial or satiric artistic works. These types of visual database engines have frequently been explained to me as being able to input the Mona Lisa and get a list of images that used the entirety of the image or just a part (such as the highly-praised subtle smile).

The big problem to me is specifying input. I know the "look" of the Mona Lisa's smile, but even with the best pen input methods I'd never be able to mimic DaVinci's subtle emotion of the smile; my hands just aren't capable of doing so. Using photos of the painting could simplify this, but this almost assumes that I'm only looking for the parody's and commercial exploiters of the image rather than the image itself (after all, I have the image to start with). And it raises the further issue that many photographic reproductions of the Mona Lisa that I can get my hands on are still under copyright and I'd be doing something legally questionable with an image long in the public domain.

Add to this the "infinite number of monkeys" issue where legally litigious companies will use technologies like this to scan the internet for litigation targets. Imagine Disney using a cell of Rafiki from the Lion King to find legally similar images that were created after the Lion King was released even if they were only superficially similar. Now do this for all movies back to Snow White or Steamboat WIlly and you could get to be a real visual mob boss with ownership (or at least threat of litigation) over huge libraries of works that weren't even created to intentionally violate Disney "Intellectual Property".

My need for this technology is small considering the input problems I'd have with my artistic abilities, while the litigation nightmare from large databases of "similar" visual data would seem to be more bothersome than helpful. I rather hope these visual search and categorizing methods don't catch on.

This reminds me of Gibson's Pattern Recognition

[WhiteDragon]

Pattern Rcognition [williamgibsonbooks.com] is a novel by William Gibson, basically set in the present day or very near future. Image based search plays a central role in the plot. It's a very good read.

There is a GNU project related to this GIFT

[capedgirardeau]

From gnu.org:

The GIFT (the GNU Image-Finding Tool) is a Content Based Image Retrieval System (CBIRS). It enables you to do Query By Example on images, giving you the opportunity to improve query results by relevance feedback. For processing your queries the program relies entirely on the content of the images, freeing you from the need to annotate all images before querying the collection.

GIFT [gnu.org] It worked pretty well for me in the demos they linked too. I have been waiting for this type of application to gain momentum.

Would it work for animated .gifs?

[Tzarius]

'Coz I'm looking for more information on this image. [pacific.net.au]

It says "multi lock on" and a date, but all Google reports is other forum posts looking for the creator of the image. Apparently, there's a high-res version of it too.

Re:wtf?

[chefmonkey]

Google actually did take this technology and try it. The first version of their image search had a "find similar" link next to every image. These tended to work okay at first (they weren't great, but you usually got enough photos back that you could visually scan them and find something of interest that was related to the original image). After a few months, for some reason, the "find similar" links started returning increasingly nonsensical results. After it degenerated to the point of near uselessness, they took the "find similar" link away from the image search results. I expected it to turn up again once they got the kinks worked out, but apparently they just decided to stop working on it.

Is this a joke?

[Daikiki]

I've tried two different images of airplanes; one of a bright red flying car on bright green grass and one of SpaceShip One against a deep blue sky. Both times, the results looked surprisingly like my query images in color composition only. Red planes on grass and white planes against a blue sky. Inauspicious start.

Next experiment: I took a picture of a highly distinctive plane, a harrier, climbing at a steep angle and viewed in profile. I got, in return, a list of passenger jets, and even a helicopter. Hardly surprisingly, all of the result pictures had the same bluish white sky as my original image. That was literally the only similarity.

According to the introduction on the search page the heuristics used compares colors, contrast and shapes in the images themselves. I saw no correlation whatsoever between shapes, and any correlation in contrast seems to be to be the result of the search engine simply looking for images that contain the same colors in a similar ratio to the original. In short, nothing to see here, move along.

On the other hand, one of the projects listed under the Penn State University link looks fairly fascinating. The Riemann a-LIP project [psu.edu] (automatic linguistic indexing of pictures) doesn't allow user input of images, unfortunately, but it does show some fairly fascinating attempts at verbally qualifying image data. For example, it describes a blue and orange mandelbrot as pattern agate shimer abstract scene, and a sunset over a lake as Berlin Devon Namibia landscape lake scene. Okay, it may still need some work, but it sure beats the hell out of the "find the same color airplane engine".