PhotoSketch Image Manipulation Tool Taking the World by Storm 193
PhotoSketch, a new image manipulation program that combines stick-figure sketches, internet image search and pattern matching, seems to be spreading like wildfire. Created by five Chinese students at Tsinghua University and the National University of Singapore, the tool takes a basic sketch and simple labels and turns it into a polished image. "Although online image search generates many inappropriate results, our system is able to automatically select suitable photographs to generate a high quality composition, using a filtering scheme to exclude undesirable images," say the PhotoSketch team in an abstract outlining the tool. "We also provide a novel image blending algorithm to allow seamless image composition. Each blending result is given a numeric score, allowing us to find an optimal combination of discovered images. Experimental results show the method is very successful."
correct links (Score:5, Informative)
Since the link to homepage in the article is some old-dated one, here's a correct one:
And the binaries [tsinghua.edu.cn] (it's a few command line programs, so no fancy UI)
Re:correct links (Score:5, Informative)
http://cg.cs.tsinghua.edu.cn/montage/main.htm [tsinghua.edu.cn]
Damn slashcode!
Re:How does it mask? (Score:3, Informative)
It says in the Vimeo link. Not gonna summarize it cause just look at the damn thing
Re:How does it mask? (Score:5, Informative)
It appears from the video that it's running a fairly sophisticated series of algorithms to compare backgrounds and determine how difficult it would be to do a convincing mask-out of the foreground object, of which it appears to have a sort of heuristic expectation of shape from the user's sketch.
For instance, if your background is a grassy field, the user has requested a dog running, and you have a photo of a dog running over grass and a dog running over pavement, the grass one will allow a greater margin of error in the masking and thus it gets selected.
Overall, this looks like a fantastic step forward for computer vision, bringing the computer ever closer to the non-Cartesian way our brains see.
Another awesome example of the power of photos (Score:4, Informative)
Re:Copyright implications, mashup (Score:3, Informative)
Re:correct links (Score:4, Informative)
Downloaded. NOTE: the as-compiled binaries require the OpenCV libraries [sourceforge.net] of the 110 variety (SourceForge holds the 200 version now). So, get older 110 binaries [sourceforge.net]. From the file list [sourceforge.net]
See the OpenCV Wiki [willowgarage.com] on setting up and checking you OpenCV installation.
I'm still setting up, but I'll post back when I get it working...
Re:This sums it up quite nicely (Score:3, Informative)
Come on, don't plagiarize! At least give credit to Gizmodo for your cut and paste.
http://gizmodo.com/5374890/this-is-a-photoshop-and-it-blew-my-mind [gizmodo.com]
Re:Rough around the edges (Score:4, Informative)
You make a Workspace folder and but a .jpg for each "item" with a simple line drawing. You make an ImageDownload folder, and in it a folder for each "item" with downloaded images in those folders.
It's missing exe's for these two lines from the ini file:
AttentioinCut = true
ShapMatching = true
They can be switched to false and it will run and create data in the Workspace folder. The images in the Segment folder are neat.... but looks like the executable to actually stitch the images together was not included.
Re:Copyright implications, mashup (Score:4, Informative)
Isn't this the visual equivalent of a mashup?
Aren't mashups already in a copyright gray area?
In the US we have this really fucked up way of dealing with derivative works - the more complicated the work, the less of it needs to be incorporated into another work before it is considered an infringement. Yes, that's right the more information in the original the smaller the percentage of that information is required to disqualify any fair use defense.
So you can quote a couple of lines of a short poem in a book or even have a character speak them in a movie and that's generally OK. But sample just 3 notes [wikipedia.org] of another song and you are in deep doodoo. Similarly, any background artwork in a movie - simply just pictures hanging on the wall in the background of a scene and thus mostly out of focus and of very low effective resolution require clearance and licensing fees, frequently absurdly high fees and of course just about any clip of video used in another movie or show - even on a television in the background of a scene - is going to require licensing too.
Most hollywood studies have an entire division devoted just handling these clearances (look in the credits for most movies and you'll see at least one person credited as head of the clearances group). This practice has the effect of keeping the "little guys" out of the motion picture business similar to the way patent pools are used to squash tech start-ups - all the studios have large "pools" of our culture under their copyright and the independent artist can't afford to license any of it for his work while the other studios can make each other sweetheart deals that guarantee cheap and easy access to each studio's "pool" of culture.
So no, mash-ups, since they generally are 100% composed of samples of other songs, aren't anywhere near being gray in the USA.
Re:correct links (Score:3, Informative)
No. I keep hitting a null pointer issue. I think they did some really bad documentation and/or pointer checking.
See a related thread [slashdot.org]
Re:Interesting point: This research is in China (Score:3, Informative)
An interesting point: This research is being done in China, not the United States. Whatever happened to basic research being done in the US?
This is cool research and all, but it's not like it happened in a vacuum. Below is a copy of the references from the PhotoSketch paper, showing the prior work the current paper was built upon, the vast majority of which are from labs in the US or Europe:
BELONGIE, S., MALIK, J., AND PUZICHA, J. 2002. Shape match-ing and object recognition using shape contexts. IEEE Trans.Pattern Anal. Mach. Intell. 24, 4, 509-522.
BEN-HAIM, N., BABENKO, B., AND BELONGIE, S. 2006.Improvingweb-based image search via content based clustering.In Proc. of CVPR Workshop.
DIAKOPOULOS, N., ESSA, I., AND JAIN, R. 2004. Content basedimage synthesis. In Proc. of International Conference on Imageand Video Retrieval (CIVR).EITZ, M., HILDEBRAND, K., BOUBEKEUR, T., AND ALEXA, M.2009. Photosketch: A sketch based image query and composit-ing system. In SIGGRAPH 2009 Talk Program.
FARBMAN, Z., HOFFER, G., LIPMAN, Y., COHEN-OR, D., ANDLISCHINSKI, D. 2009. Coordinates for instant image cloning.SIGGRAPH 2009.
FELZENSZWALB, P. F., AND HUTTENLOCHER, D. P. 2004. Effi-cient graph-based image segmentation. Int. J. of Comput. Vision59, 2, 167-181.
FERGUS, R., FEI-FEI, L., PERONA, P., AND ZISSERMAN, A.2005. Learning object categories from google's image search.In Proc. of ICCV.
GEORGESCU, B., SHIMSHONI, I., AND MEER, P. 2003. Meanshift based clustering in high dimensions: A texture classifica-tion example. In Proc. of ICCV.
HAYS, J. H., AND EFROS, A. A. 2007. Scene completion usingmillions of photographs. SIGGRAPH 2007.
HOU, X., AND ZHANG, L. 2007. Saliency detection: A spectralresidual approach. In Proc. of CVPR.
JACOBS, C., FINKELSTEIN, A., AND SALESIN, D. 1995. Fastmultiresolution image querying. In SIGGRAPH 1995.
JIA, J., SUN, J., TANG, C.-K., AND SHUM, H.-Y. 2006. Drag-and-drop pasting. SIGGRAPH 2004.
JOHNSON, M., BROSTOW, G. J., SHOTTON, J., ARANDJELOVI C,O., KWATRA, V., AND CIPOLLA, R. 2006. Semantic photosynthesis. Proc. of Eurographics.
LALONDE, J.-F., HOIEM, D., EFROS, A. A., ROTHER, C.,WINN, J., AND CRIMINISI, A. 2007. Photo clip art. SIG-GRAPH 2007.
LEVIN, A., LISCHINSKI, D., AND WEISS, Y. 2008. A closed-form solution to natural image matting. IEEE Trans. PatternAnal. Mach. Intell. 30, 2, 228-242.
LI, Y., SUN, J., TANG, C.-K., AND SHUM, H.-Y. 2004. Lazysnapping. SIGGRAPH 2004.LIU, T., SUN, J., ZHENG, N.-N., TANG, X., AND SHUM, H.-Y.2007. Learning to detect a salient object. In Proc. of CVPR.
MANJUNATH, B. S., AND MA, W. Y. 1996. Texture features forbrowsing and retrieval of image data. IEEE Trans. Pattern Anal.Mach. Intell. 18, 8, 837-842.
PEREZ, P., GANGNET, M., AND BLAKE, A. 2003. Poisson imageediting. SIGGRAPH 2003.RAJENDRAN, R., AND CHANG, S. 2000. Image retrieval withsketches and compositions. In Proc. of International Conferenceon Multimedia & Expo (ICME).
ROTHER, C., KOLMOGOROV, V., AND BLAKE, A. 2004. "grab-cut": interactive foreground extraction using iterated graph cuts.SIGGRAPH2004.
SAXENA, A., CHUNG, S. H., AND NG, A. Y. 2008. 3-d depthreconstruction from a single still image. Int. J. of Comput. Vision76, 1, 53-69.
SMEULDERS, A., WORRING, M., SANTINI, S., GUPTA, A., ANDJAIN, R. 2000. Content-based image retrieval at the end ofthe early years. IEEE Trans. Pattern Anal. Mach. Intell. 22, 12,1349-1380.
WANG, J., AND COHEN, M. 2007. Simultaneous matting andcompositing. In Proc. of CVPR, 1-8
Re:This sums it up quite nicely (Score:1, Informative)
The video mentions they came up with a novel algorithm for selecting pairs of images that are easy to composite, and blending algorithms that are aware of the messy parts of the boundaries and compensate for them.
And as you know, selecting your image is the hardest part of a shoop.
Re:correct links (Score:2, Informative)