Meta Open-Sources Multisensory AI Model That Combines Six Types of Data

Meta has announced a new open-source AI model that links together multiple streams of data, including text, audio, visual data, temperature, and movement readings. From a report: The model is only a research project at this point, with no immediate consumer or practical applications, but it points to a future of generative AI systems that can create immersive, multisensory experiences and shows that Meta continues to share AI research at a time when rivals like OpenAI and Google have become increasingly secretive. The core concept of the research is linking together multiple types of data into a single multidimensional index (or "embedding space," to use AI parlance). This idea may seem a little abstract, but it's this same concept that underpins the recent boom in generative AI.

Meta must be desperate

[ls671]

Meta must be desperate to try to appeal the open source community. I mean, it's the first time I hear about Meta's interests in open source projects. Anyway, who knows? This might become handy to the open source community but I couldn't be less sure about it given it's Meta after all...

Re:

[gweihir]

Meta is clearly desperate. I mean they have nothing for the future at this time. But they are not the only one. Google is also showing significant signs of desperation and so is Intel. In some sense, the Internet revolution is eating its children.

Re:

[timeOday]

What are you talking about? They released LLAMA just a few months ago and it's the dominant open-source large language model of the moment. (Although this article repeats the notion that it was a leak, which has never been true).

Re:

[narcc]

I wonder if the "leak" rumor was just Meta trying to drum-up interest in their model. Pretend leaks are all the rage these days, after all.

Re:

[AleRunner]

Meta must be desperate to try to appeal the open source community. I mean, it's the first time I hear about Meta's interests in open source projects

They're the one of the big internet companies which doesn't have anything like their own cloud; that means that various things push them towards FOSS software. They've done a bunch of interesting open things, such as osquery and including open standards for data centres.

Re:

[drinkypoo]

Meta must be desperate to try to appeal the open source community. I mean, it's the first time I hear about Meta's interests in open source projects.

Meta/Facebook has in fact made voluble and significant contributions to OSS. They are offensive in many ways, but failing to give back is not one of them. Relevantly to this particular discussion, for example, they developed PyTorch.

You might reasonably but uncharitably say that corporate contributions to OSS always come from "desperation", in that if they could maintain the software themselves, they wouldn't need to give it away. On the other hand, it is often argued that opening your code is a pragmatic d

Re:Meta must be desperate

[test321]

* Adding "facebook", "Meta research" and "pytorch", Meta controls 1009 open source repositories on github. https://github.com/facebookres... [github.com] https://github.com/facebook [github.com] https://github.com/pytorch [github.com]
* Meta is behind React.js, a framework that is widely used to build web user interfaces.
* Meta has developed the zstd compression algorithm and its integration to btrfs (which makes sense since they have huge amounts of data to store).
* Meta is the 10-11th corporate contributor to the the kernel 6.3, consistently contributing 2.5-3% of the lines changed in the recent releases https://lwn.net/Articles/92958... [lwn.net] https://lwn.net/Articles/92341... [lwn.net] https://lwn.net/Articles/91543... [lwn.net]

You can browse all their OSS projects at https://opensource.fb.com/proj... [fb.com]

Robotics

[timeOday]

The summary makes it sound like the point of this is a multi-sensory Disney-type ride, or AR experience with a smell-o-scope [fandom.com]. Which in fairness to the article, is also the tone of the linked Facebook blog page. But the real benefit of this is more likely to be embodied robotics (such as self-driving cars, which could use all the modalities mentioned) that fuse many sensors and actuators to sense and act in the physical world. Maybe they are playing up the AR angle because the research was funded under that line of business.

The Metaverse... now with Smell-o-vision!

[Locke2005]

Am I the only person at Meta still working on the Metaverse?