Facebook's Photorealistic Simulator For AI Runs At 10,000 FPS

malachiorion writes: Facebook just open sourced a simulator for testing and training embodied AI systems -- like virtual robots. They worked with AR/VR researchers to release the simulator along with what they say are the most photorealistic 3D reconstructions of real world places available. [Facebook Reality Labs have named this "the Replica data set".]

The crazy part: Because more frames are always better for training computer vision in simulators, it can run at 10,000 FPS!
The simulator's ability to hit 10K frames per second prompted an interesting follow-up on the original submission. "It's a totally useless framerate for humans -- just absolute overkill for our brains/eyeballs -- but it's apparently a benefit for AI systems."

"As more researchers adopt the platform, we can collectively develop embodied AI techniques more quickly," explains Facebook's blog post, "as well as realize the larger benefits of replacing yesterday's training data sets with active environments that better reflect the world we're preparing machine assistants to operate in."

And if you're worried about privacy, Facebook assures readers that "The data used to generate Replica scans was anonymized to remove any personal details (such as family photos) that could identify an individual. The overall reconstruction process was meticulous, with researchers manually filling in the small holes that are inevitably missed during scanning and using a 3D paint tool to apply annotations directly onto meshes."

Re:

[LarryRiedel]

I think the simulator generates data for training the reinforcement learning model, and the faster the simulator generates the data, the faster the model can be trained.

Re:

[PPH]

Just guessing here: But it seems that what Facebook has developed is a bunch of virtual environments within a software-based training environment. There is no actual camera looking at a physical scene. So it's just a matter of feeding the system being trained the data as fast as it will take it. Great for training, because feeding a system many thousands of data sets takes time.

I hope the AI system gets tested with live data before going into production. Because latency problems could be hidden by this spe

Re: How Is Processing Possible

[Code Herder]

Its not meant to go to prod in that way. Its meant to pre train the neural net on real world sights/navigation. After its pre trained in a virtual environment they can use a real robot farm with cameras etc to do the final trainning.

There are already a few robot arm farms for deep learning to train the arms at picking up stuff but its SLOW vs trainning for anything else and its expensive to scale. Retrainning neural networks is all the rage right now, turns out you dont need to train on 60k balloon images

Re:

[DamnOregonian]

I'm not saying I can do 10kfps, or anything, but you're way off track with the claim that 60FPS is a challenge for any GPU these days with built in encoding hardware.
My 1080Ti can do 10-bit h.265 at 650fps, or h.264 at 1300fps.

Fast

[Kohath]

But still not fast enough to outrun the tide of Facebook corporate scandals.

Hilarious!

[sheramil]

... "Facebook assures readers that "The data used to generate Replica scans was anonymized to remove any personal details (such as family photos) that could identify an individual."

Ha, ha! Hearing stuff like that from them always cracks me up.

It'll be leaked before the end of the year.

Re:

[CanadianMacFan]

It's being trained on anonymous data. Once it goes live they don't care about your privacy.

Re:

[110010001000]

That made more sense than the summary.

Runs at 10k FPS

[TimothyHollins]

And is still easier to max out than Crysis.

Time is relative...

[madsh]

Pretty sure the second they talk about is different from the second I experience. I bet the simulator runs in about 1:100 to realtime. And the FPS really just tell you the update frequency the agent to be trained experience. If they really have managed to speed up learning experiences with a factor of 400-500 compare to human sensor input... the resolution in other dimension must be to the low side...