Forgot your password?
typodupeerror
Graphics Input Devices Software

Erasing Objects From Video In Real Time 175

Posted by timothy
from the no-more-wires-visible dept.
Smoothly interpolating away objects in still pictures is impressive enough, but reader geoffbrecker writes with a stunning demonstration from Germany's Technical University of Ilmenau of on-the-fly erasure of selected objects in video. Quoting: "The effect is achieved by an image synthesizer that reduces the image quality, removes the object, and then increases the image quality back up. This all happens within 40 milliseconds, fast enough that the viewer doesn't notice any delay."
This discussion has been archived. No new comments can be posted.

Erasing Objects From Video In Real Time

Comments Filter:
  • by Jah-Wren Ryel (80510) on Wednesday October 13, 2010 @08:12AM (#33881166)

    We need this built into our televisions to automagically remove those network logo "bugs" and other crap they have started putting on the screen during the shows.

    • by leuk_he (194174) on Wednesday October 13, 2010 @08:16AM (#33881188) Homepage Journal

      In reality the networks wil use it to blur out any logo's from companys that do not sponsor the show. F1 cars will be red instead of filled with sponsors.

      • Re: (Score:2, Insightful)

        by Anonymous Coward

        Somehow I doubt that would work, unless the racing teams could get money some other way. Who would sponsor a car when your logo won't be visible on it? The race organizers would probably require networks that buy rights to air it to not scrub logos.

        • Re: (Score:3, Insightful)

          The race organizers would probably require networks that buy rights to air it to not scrub logos.

          I'm pretty sure they already do.

        • You're right (Score:3, Insightful)

          by ebuck (585470)

          Racing competitions, NFL, etc. own the copyright to the original footage. They're not going to license broadcast of that footage to any television station which threatens their revenue model, unless the station is going to pay so dearly that their previously existing revenue stream looks paltry in comparison.

          Even if the TV stations were to put more cash on the table, they still might not agree to such a practice as it gives a large degree of control to a single party (which means more finiancial risk if th

      • Networks won't be allowed to do that as per their sports contracts...
      • by John Hasler (414242) on Wednesday October 13, 2010 @09:07AM (#33881552) Homepage

        A simpler version of this has already been used to edit billboards visible in broadcasts of baseball games.

      • Or they'll just use it to remove rude gestures, nose rings and nipples, in a visual analogue to profanity bleeps. Welcome to the world of tomorrow.

        • Re: (Score:2, Insightful)

          by Agent0013 (828350)
          They could use it instead of the blur they use on people's t-shirts. I would much prefer this than having that stupid blur.
          • on the rare occasion that i watch trash tv, i have started to count the number of times you see black duct tape over logos on hats and sweatshirts in different 'reality' shows. it is interesting that they don't have to completely cover the disgusting ed hardy-esque tshirts that these idiots wear, since the whole thing is one big logo. maybe they have cleared the use of those designs, but not sports team logos or something.
      • by Sockatume (732728)

        I'm sure that for events like F1, preserving the sponsorship messages is part of the exclusive licencing agreement the broadcaster has to sign up to.

      • by Tom (822)

        F1 cars will be red instead of filled with sponsors.

        You mean like before everything was invaded by advertisement? Wow, what an improvement!

        (yes, slashdot, I can type more than one comment every 20 minutes, I have 10 fingers, not two... argh...)

      • I've noticed a steep drop-off of sponsorship in F1 in the past few years. The Ferrari, for example, has a solid red shell behind the cockpit, where it used to be littered with ads.

        Hard economic times will do the same thing as this German video tool.

    • Re: (Score:3, Interesting)

      by gstoddart (321705)

      We need this built into our televisions to automagically remove those network logo "bugs" and other crap they have started putting on the screen during the shows.

      First off, I don't think we'll get control over this on our TVs. The networks aren't gonna let us delete their "bugs".

      I'm actually more concerned over something like Running Man where you can't trust the news reports you see because someone selectively tweaked the image to hide/alter the bits they don't want you to see.

      Now, of course, the technolo

      • They can already do that - most news reports aren't in real time, they can tweak the video before they air it.

      • by sjames (1099) on Wednesday October 13, 2010 @12:37PM (#33883846) Homepage

        Or stripping out the thousands of peaceful middle aged protesters and the hundreds of uninvolved pedestrians being tear gassed to show only the one or two violent people who actually have nothing to do with the protest.

        Next up, witnesses will disappear from police video to discredit them in court.

    • by timeOday (582209)
      More importantly... pimple removal!
    • The Christian Coalition wants this for Super Bowl half time show boobs.

      I want it to remove the entire half time show.

  • by suso (153703) * on Wednesday October 13, 2010 @08:12AM (#33881168) Homepage Journal

    Pretty good, but take note that all the examples where objects sitting on pretty flat colored backgrounds. I'd like to see what happens when you try to remove an object in a complex environment. Like removing a single person standing in a crowd.

    • by smallfries (601545) on Wednesday October 13, 2010 @08:36AM (#33881300) Homepage

      Take a look at the explanation part of the video. The background texture is tiled. You can see some strange deformation in the regular pattern where the object used to be. Also in the drain example there is a strange crater effect as the camera angle changes.

      It seems like smooth colour graduations work well, but patterned backgrounds have more obvious deformations.

      • Re: (Score:3, Interesting)

        by stewbacca (1033764)

        All of their samples (except the brick one) use solid/high contrast surfaces that are somewhat evenly lit. Still kind of cool though, but they should have waited until it's more mature to impress us old timer motion graphics guys.

      • Re: (Score:3, Interesting)

        by RandCraw (1047302)

        Right. I'm pretty sure the underlying technology is based on Seam Carving , where a continuous background region is collapsed 'seamlessly' (or an object within such a region is removed). This doesn't work so well when the background is discontinuous, so it's not going to remove logos from clothing. It's also not going to work well on live video, since the object to be removed needs to be identified manually before the excision can occur. But it works nicely on prerecorded media. I'm impressed at how we

      • by HiThere (15173)

        In addition to dealing with reflections, which I consider just a part of polishing step one, step two will use the position of something in the video as an anchor and substitute the image of something else.

        How far off do you think *that* is? I give it two years to the the lab demo with problems.

    • Re: (Score:3, Interesting)

      by DrXym (126579)
      Pretty good, but take note that all the examples where objects sitting on pretty flat colored backgrounds. I'd like to see what happens when you try to remove an object in a complex environment. Like removing a single person standing in a crowd.

      There were some examples of that in the clip if you watched closely. The removal drain on a pebbly asphalt caused a weird swirly pattern to occur as the camera moved. I expect the same would be true for live attempts at the same. It probably works best on static th

  • by gknoy (899301) <<moc.smetsysizasana> <ta> <yonkg>> on Wednesday October 13, 2010 @08:13AM (#33881170)

    This has some frightening ramifications for how much we believe video. Videos similar to the ones Wikileaks leaked, or news videos "live" on scene, could be doctored in near enough to real time that we consumers might never know it. Scary.

    • Now we can frame someone for murder in realtime.
    • comes to life then?

      Its bad enough people believe lines said by comedians are the actual lines of some high profile people, how can we hope that people will care enough to know if the video they are seeing is not edited? Hollywood doesn't need the tech to make movies, maybe to "fix" reality shows, but I figure politics is where the mileage comes in.

      • how is that any different than now? how can i tell if a video i see now isn't either edited or staged? the only thing this changes is it makes it possible to do it live.

      • by cusco (717999)
        They don't care enough now to know that recorded video used to make important policy decisions, such as what country to invade, are routinely faked. I have no hope at all that they're going to pay any attention at all to this development, even when it gets beyond the alpha stage release. "Seeing is believing" is the sheeple's creed, even when the thing they're seeing is so blatantly faked that 10 year-olds recognize something is wrong with the recording (the case in at least one of the Binladdin videos).
    • by Muros (1167213)
      Not only video, but sound as well. If you reversed the purpose of the technology discussed here [slashdot.org] to delete particular sounds instead of focusing on them, you will soon be able to completely edit out anything you don't like, in real or near real time, from video feeds. Whatever about not trusting the spin or coverage of news these days, soon you won't even be able to trust what look like actual recordings of events.
    • Anything non-live can be doctored with since decades (including "Videos similar to the ones Wikileaks leaked"). The ability to alter live video in that way however is, AFAIK, new. But as long as you can't verify that any given video broadcasted on TV is actually life, that's more a moot point (see countries with a strict censoring that require "live" video to be delayed for a short time so censoring can happen).

      • by IBBoard (1128019)

        Yeah, just watch what they do with the top CGI films these days. Parts of Alice in Wonderland (the latest film I watched with CGI) are pretty much indistinguishable from reality, except for the bit where the thing is nigh-on impossible without huge expense and months of prep time for a single shot (like long shafts full of rough edges and trinkets that *could* be dug and prepared, but are realistically going to be graphics).

        LotR painted all sorts of stuff in, and Gollum was painted over the top of Andy Serk

      • by ebuck (585470)

        With live video, you don't need to doctor it. You just need to convince the audience that your sound stage is really "on-site".

    • Re: (Score:3, Interesting)

      by houghi (78078)

      The 'Live on scene' is pretty rare already. Almost everything is taped and edited. And even now many people believe what they see as they will have only one (if that) source of information. How many people will actually look at what others have to say? http://aljazeera.com/ [aljazeera.com] as an alternative? Nah, because that is propaganda from the enemy. Better just watch Fox News.

      People do not want to be informed. They want to be entertained.

    • Re: (Score:3, Informative)

      by Tom (822)

      This is only news for real-time feeds. For anything that is not live (and you can verify that it's live! A lot of what you see labeled "live" on TV actually isn't!), assume that the stream has been messed with, already today. Most of the times, it is "artistic" messing - improving picture quality, editing out distracting background content, cleaning up artifacts, etc.

    • So, we now need easy-to-use software for identifying video and image manipulation usable without much technical skill? I have no technical insights as to how these things are done, though.
    • Re: (Score:3, Informative)

      by ebuck (585470)

      Video can already be manipulated without restoring to high-tech wizardry. There's been plenty of examples of news reporters "on scene" when they're just in front of a blue screen. Cinema (which has much higher resolution, so it is harder to fake) constantly amazes us with simple tricks like flattening the depth of field, rotating the camera to make small inclines look like cliffs, adjusting zoom while moving the camera to distort depth perceptions, etc.

      And we aren't even getting close to the easier techni

      • by tlhIngan (30335)

        Cinema (which has much higher resolution, so it is harder to fake) constantly amazes us with simple tricks like flattening the depth of field, rotating the camera to make small inclines look like cliffs, adjusting zoom while moving the camera to distort depth perceptions, etc.

        Actually, it's not restricted to cinema. Anyone with a half-decent camera can do it. Still or video (especially since modern dSLRs support recording video). Heck, a surprising amount of video these days is captured on nothing more than

  • It is impressive, but not perfect...you can see shadowing and the outlines of the object when the camera moves in certain angles.

    Also, if there is video evidence presented in a courtroom, people should be aware that technology like this exists and it can and will be used.
    • Re: (Score:3, Funny)

      by Hognoxious (631665)

      Are you saying the pixels are wrong? Have you seen a lot of shops?

    • Also, if there is video evidence presented in a courtroom, people should be aware that technology like this exists and it can and will be used.

      And it seems likely an entropy analysis on the resultant images will show up bright problems with the re-created area.

  • by Defenestrar (1773808) on Wednesday October 13, 2010 @08:23AM (#33881224)

    Great - it'll start off by making eyesore real estate disappear from "live coverage," then be required as a precondition for live celebrity interviews (not just makeup to cover that acne), moving on to inconvenient points to the story that would take too much time and effort to explain, then images which might "disturb the children" (number of student bodies in Tienanmen Square?), and finally develop to ubiquitous studio-in-a-cameras such that we'll have little assurance of whether live coverage is fact or fiction.

    Of course that's just pessimism speaking. Really I'm looking forward to watching live reports without those obnoxious people waving at their mothers, or holding up witty slogans about taxation.

    • Re: (Score:3, Insightful)

      by HertzaHaeon (1164143)

      Here's an optimistic thought — it might make people skeptical of the images they see, which is a useful attitude reagrdless of this technology.

      • I sincerely hope you're right, but more likely cognitive dissonance will take over [innovations-report.com] and people will be more likely to believe what they see even if they know it likely to be false.

        Don't worry, I'll be as optimistic as you like on a Friday afternoon. The second morning back after a three day weekend still tracks closer to Monday before caffeine for me.

    • by nospam007 (722110) *

      Communal CCTVs could edit out the hookers, dealers, pimps, garbage and other 'untouristy' stuff on the fly.

      Now we just need it miniaturized and built-into our glasses, preferably rose-tinted, to see the world in a whole new light.

    • by jimwelch (309748)

      Anyone who uses Journalist and Integrity in the same sentence should be ... (oops I did it too)!

      • Re: (Score:3, Insightful)

        by Defenestrar (1773808)

        Ed Murrow was a journalist who practiced integrity [wikipedia.org]. There, I did it again. So what was supposed to happen? Perhaps I should be taken in front of a special committee hunting for non-American behavior. Except wait - they disbanded that one after some punk journalist risked his career and took out McCarthy.

        I actually think this is a pretty neat idea. I think the thing which concerns me is that, given the current state of affairs, the public will have even less reason to trust the press. If the public do

  • by Gordonjcp (186804) on Wednesday October 13, 2010 @08:24AM (#33881230) Homepage

    ... I'd pretend I was one of those deaf-mutes.

    • Seconded.

      The Ring of Gyges [slashdot.org]
    • by mister_playboy (1474163) on Wednesday October 13, 2010 @08:42AM (#33881332)

      That phrase was familiar to me, but I wasn't sure where I had seen it... now I remember:

      https://secure.wikimedia.org/wikipedia/en/wiki/Laughing_Man_(Ghost_in_the_Shell) [wikimedia.org]

      • Re: (Score:3, Informative)

        by Shikaku (1129753)

        Catcher in the Rye you meant right? That classic book? Bill Gate's favorite as well (just throwing that line out for comedy).

        • No, I recognized it from that anime. I've only seen a handful of episodes of GinS so I had no idea what was going on story-wise, but that distinctive logo stuck with me.

          Overlooking the Wiki article on the book makes me think I was not introduced to it because I went to a Catholic school. It wouldn't have been banned from the library, but it probably wouldn't have been allowed as part of a class curriculum, either.

          • by IBBoard (1128019)

            It wasn't part of my class curriculum either, and I've not read it. I only found out about it through GitS (actually, GitS:SAC - it is the "Stand Alone Complex" series rather than the film). They do mention that it is from Catcher in the Rye in one episode, but that comes after the phrase has been mentioned a few times. Someone has a rather aged copy and there is a bit of a "these funny people with their ageing paper books" tone, given the setting, but it is acknowledged in a fairly intellectual way.

        • by hoshino (790390)
          While Catcher in the Rye is the original source, Ghost in the Shell is the more relevant reference. In the series, everyone possesses cyberbrains that can be remotely hacked to modify the person's sensory data in real-life. The Laughing Man (think Anonymous of the future) used this to mask his face with a smiling logo containing the Salinger quotation above. Literally no one could ever see his face, except for homeless beggars who are too poor to afford cyberbrain implants.
    • by somaTh (1154199)
      It's beyond that, though. He just blurred his face. This is removing the entire person. In that world, he goes from being anonymous to being invisible.
      • by Securityemo (1407943) on Wednesday October 13, 2010 @08:58AM (#33881464) Journal
        He learnt to edit himself out completely towards the end, though. Leading to a very big-ham moment with Batou exclaiming "He stole my eyes!"
        • Re: (Score:3, Informative)

          by IBBoard (1128019)

          He did it before that, though. When he kidnapped Serano from his own home then he was visible to Serano, but not to his guards. As far as the guards were concerned then Serano was just walking out on his own and nothing was unusual. That would probably be more impressive because he did it to lots of people at once, not just a single person who was chasing him as he left a hotel and casually escaped (IIRC).

          • Yes. The original manga is much more detailed when it comes to technical subjects - Shirow Masamunes view of the cyberbrain-experienced cyberspace was that it was "incomprehensible" and not really visual as such. He allegedly has an engineer education, so I always interpreted that as meaning something like the experience of a code/data model in your brain, as you experience when you are programming, turned up to 12. This also puts the twisted inhuman minds of the kids in the cyberbrain autism care center in
    • Re: (Score:3, Interesting)

      by EdZ (755139)
      Not really. There are plenty of webcams that come with free software that can overlay an image (including the requisite spinning-text-around-face logo of the Laughing Man) over a tracked face in real time, but this software instead edits out a tracked area using surrounding data. I wish they gave more explanation, or any explanation at all, rather than the nebulous magical 'increase the image quality back up'.
  • I can beat that. I can erase everything in the frame in less than 40ms.

    Actually, this is really cool. They could generate hype for it by posting a demo on the web of Episode I with Jar Jar erased. They might be able to stave off Lucas's lawyers by calling it a parody, although in this case it would be more like the original was a parody.

    • Or re-edit Episode IV so that Han shoots first... again.
    • They could generate hype for it by posting a demo on the web of Episode I with Jar Jar erased.

      How would making Jar Jar Binks invisible improve The Phantom Menace? He'd still have all his lines. Better would be to re-edit the film to make Binks only a minor character, and that's called The Phantom Edit [wikipedia.org].

  • Jar Jar (Score:5, Funny)

    by veggiespam (5283) on Wednesday October 13, 2010 @08:25AM (#33881242)

    Finally, we can restore my childhood memories and eviscerate Jar-Jar from the last batch of Star Wars movies.

    • by Shrike82 (1471633)
      Either it's a setup, or somehow the neurons in your brain have become quantumly entangled with the neurons of the guy who posted just above you. Spooky.
  • increase image quality?
  • by bareman (60518)

    ChatRoulette, the PG edition.

    • That was actually my first thought - I always imagine that in live broadcast TV events, there are guys sitting at a console with blur-out circles at the ready (I imagine they hire top Quake 3 Arena players for this).

      This could automatically blur naughty bits, saving millions of uptight Americans from Janet Jackson boob, and also most of the people on Chatroulette from unwanted dongs (they have an adult section now if you want the dongs).

  • by Ecuador (740021)

    Perhaps the next BluRay spec should include this technology so that when you watch ET you can select for the walkie-talkies to be erased and your choice of guns/rpg's etc to appear!

  • by ledow (319597)

    Sorry, I started to call bullshit at the "increase the quality back up" sentence. No-one worth their salt in video processing would ever use such a phrase. This makes it equivalent to Bladerunner-esque infinite zooming and Hollywood-style deblurring to a perfect image.

    I'm not saying it's not possible, or even not possible in real-time, but that explanation sends all sorts of warning signals to me. Hell, we know you can do stuff like this because Hollywood does it all the time without having to use physic

    • The way I interpreted it was that they decrease the quality so they can get more basic shapes, from which they can infer the outline of the shape they want to remove and go back to the original quality stream to remove it.
  • Thwartable (Score:5, Funny)

    by srussia (884021) on Wednesday October 13, 2010 @08:38AM (#33881308)
    FTFA:"It does seem to be thwarted by reflections though; a cell phone removed from a bathroom counter is still visible in the mirror."

    "Zoom in on the reflection...ENHANCE!"
  • FTA:

    . It does seem to be thwarted by reflections though; a cell phone removed from a bathroom counter is still visible in the mirror.

    Ummmmm, no. The software reads an image with 2 objects, they only deleted one. Maybe the software can only delete one object at a time now. But that's not being "thwarted by reflections," the mirror has nothing to do with it. The software would behave the same if there actually were 2 objects on the screen.


  • In Stalin's Soviet Russia...
  • How does it know what to draw beneath the replaced object?

    • by _Shad0w_ (127912)

      The same way any filter of that kind does, interpolation based on the surrounding pixels. It's the video equivalent of Adobe's filter in Photoshop 5.

  • by Posting=!Working (197779) on Wednesday October 13, 2010 @08:51AM (#33881388)

    We need a name for this process. I suggest "to Jar-Jar." Examples:

    They Jar-Jared the cell phone and stapler off the desk.
    "Jar-jar the 3-D glasses off the chair."
    Al Pacino released the "Actor's cut" of Godfather 3 and Jar-jared himself out of the movie.
    I'd like to Jar-jar my ex-girlfriend from my brain.
    It was a guy! He Jar-jared his webcam!

       

  • then increases the image quality back up

    Enhance! [youtube.com]

  • So one step away from TV networks manipulating live video for their own / political masters requirements.

    You didn't see it, it never happened. The camera never lies - right!?

  • by mattaw (718560)
    Obvious limitations from the demo:

    1) Objects must be sitting on a consistent(ish) surface with a low rate of change compared to the object. Desk, Chair, Bathroom, Wall, Hubcap, etc.

    2) It doesn't handle strong shadows (or they are not showing us it doing so).

    3) It makes the greatest amount of mistakes with the shadows anyway.

    Please add anything I missed to future posts.

    I would like to see it erase a boat from a choppy sea where there are 5-7 waves for the length of the boat as I expect that to be a

  • Sounds like "the ugliest shirt in the world" from William Gibson's Zero History.
  • This is perfect for propaganda peddlers, er news networks, now they can more easily deceive the public.
  • Ten years ago, one of my friends, who works on movies' restoration and coloring, told me that they had software that was able to remove moving objects from a scene.

    The idea was to use the whole scene to recreate the missing parts.

    I also remember an article on compression.ru, with plugins able to remove logos or subtitles:
    http://compression.ru/video/logo_removal/index_en.html [compression.ru]
    http://compression.ru/video/subtitles_removal/index_en.html [compression.ru]
    and even TV ads removal:
    http://compression.ru/video/tv_commercial_detector/i [compression.ru]

  • If people didn't trust the media before... this is really going to give them pause for thought.
  • Fine..I will say the obvious.
    Never again will you need to see a penis in your porn..unless of course you want to in which case I am not judging you. ;)

  • I can't find the paper at the moment, but I've seen examples of something similar done with a CCTV camera and encryption. It has the advantage of having a fairly fixed view, so it can easily repaint the background in, but the idea was that it recognises and encrypts the imagery of a person in the shot. The CCTV then captures everyone who walks through the shot in case actions need to be reviewed in future (e.g. "did we see a guy matching the vague description of the perpetrator in the area before the crime

  • Did anyone else notice the matchbox appeared in the mirror in the video?

  • by Tom (822)

    I would pay for glasses that remove advertisement from the real world. In fact, I'd pay quite a lot.

    Ads are one of the "unseen evils". By now we know that the processing and even the filtering out that our brains do take up more of our awareness than we become aware of. Advertisement has been positively linked to road accidents, for example, as it is a distracting factor.

    In any big city today, you are literally bombarded by advertisement, and all of it has been designed by psychological warfare...sorry, "ad

  • What 6 million jews and 2 clowns? I don't see 6 million jews and two clowns.

  • Pope Benedict XVI has warned that people are in danger of being unable to discern reality from fiction because of new technologies
    http://idle.slashdot.org/article.pl?sid=10/10/12/1328215 [slashdot.org]
    • Re: (Score:3, Insightful)

      by Viol8 (599362)

      Given the nonsense in the Bible I'd suggest he manages it without technology.

    • Sorry, but I now got that new mythtv plugin that replaces any TV appearance of Pope Benedict with Pope Carla.
    • by RMH101 (636144)
      Much better we're all kept in the dark ages, so we can better hear his invisible friend talking.
  • If you wish to be seen, but are being spliced out. You could wear several different colored t-shirts over each other or something similar and then take them off to trick the camera at least temporarily. If you want people to see something that is being blocked out, you would have to probably spray them or it with some kind of colorant, or a bright flash of light might also do the trick, maybe some kind of a portable strobe light. This is just off the top of my head.

    Somehow l feel like like I shouldn't be g
  • by rlseaman (1420667) on Wednesday October 13, 2010 @01:16PM (#33884364)
    Invisibility is an ancient notion and tampering with video as old as the Lumière brothers. What is new here is the trend toward placing these capabilities closer and closer to the camera. Combine such effects with the face detection algorithm that is already in your phone's camera and the original picture can remove or replace individuals from the scene of the crime. "Ground truth" will be ever more difficult to establish.

The meat is rotten, but the booze is holding out. Computer translation of "The spirit is willing, but the flesh is weak."

Working...