Microsoft Details 'Planet-Scale' AI Infrastructure Packing 100,000+ GPUs (theregister.com) 45
Microsoft has revealed it operates a planet-scale distributed scheduling service for AI workloads that it has modestly dubbed "Singularity." The Register reports: Described in a pre-press paper [PDF] co-authored by 26 Microsoft employees, Singularity's aim is described as helping the software giant control costs by driving high utilization for deep learning workloads. Singularity achieves that goal with what the paper describes as a "novel workload-aware scheduler that can transparently preempt and elastically scale deep learning workloads to drive high utilization without impacting their correctness or performance, across a global fleet of AI accelerators (e.g., GPUs, FPGAs)."
The paper spends more time on the scheduler than on Singularity itself, but does offer some figures to depict the system's architecture. An analysis of Singularity's performance mentions a test run on Nvidia DGX-2 servers using a Xeon Platinum 8168 with two sockets of 20 cores each, eight V100 Model GPUs per server, 692GB of RAM, and networked over InfiniBand. With hundreds of thousands of GPUs in the Singularity fleet, plus FPGAs and possibly other accelerators, Microsoft has at least tens of thousands of such servers! The paper focuses on Singularity's scaling tech and schedulers, which it asserts are its secret sauce because they reduce cost and increase reliability.
The software automatically decouples jobs from accelerator resources, which means when jobs scale up or down "we simply change the number of devices the workers are mapped to: this is completely transparent to the user, as the world-size (i.e. total number of workers) of the job remains the same regardless of the number of physical devices running the job." That's possible thanks to "a novel technique called replica splicing that makes it possible to time-slice multiple workers on the same device with negligible overhead, while enabling each worker to use the entire device memory." [...] "Singularity achieves a significant breakthrough in scheduling deep learning workloads, converting niche features such as elasticity into mainstream, always-on features that the scheduler can rely on for implementing stringent SLAs," the paper concludes.
The paper spends more time on the scheduler than on Singularity itself, but does offer some figures to depict the system's architecture. An analysis of Singularity's performance mentions a test run on Nvidia DGX-2 servers using a Xeon Platinum 8168 with two sockets of 20 cores each, eight V100 Model GPUs per server, 692GB of RAM, and networked over InfiniBand. With hundreds of thousands of GPUs in the Singularity fleet, plus FPGAs and possibly other accelerators, Microsoft has at least tens of thousands of such servers! The paper focuses on Singularity's scaling tech and schedulers, which it asserts are its secret sauce because they reduce cost and increase reliability.
The software automatically decouples jobs from accelerator resources, which means when jobs scale up or down "we simply change the number of devices the workers are mapped to: this is completely transparent to the user, as the world-size (i.e. total number of workers) of the job remains the same regardless of the number of physical devices running the job." That's possible thanks to "a novel technique called replica splicing that makes it possible to time-slice multiple workers on the same device with negligible overhead, while enabling each worker to use the entire device memory." [...] "Singularity achieves a significant breakthrough in scheduling deep learning workloads, converting niche features such as elasticity into mainstream, always-on features that the scheduler can rely on for implementing stringent SLAs," the paper concludes.
Great! (Score:3)
Plus the billions of chips he injected in us, Skynet here we come.
Re: (Score:1)
Re: (Score:2)
You know slashdot commenter by name. I cannot remember anyone from the last 15+ years. Are you not a littly creepy? Besides, it's not about bitcoin this time.
why not AMD epic? get more pci-e lanes! (Score:2)
why not AMD epic? get more pci-e lanes!
Re: (Score:2)
Money talks. Intel has been convicted several times for illegal business practices...
Nobody sane would use Intel for any new DC application these days.
Fear not, humanity will prevail... (Score:2)
...we WILL run Crysis on max settings!
Re: (Score:2)
With a name like "Singularity" ... (Score:2)
... what could possibly go wrong.
Pro-tip: When taking about things that might scare people, don't use scary names.
Re: (Score:2)
Re: (Score:2)
The singularity is silly science fiction nonsense. No adult of normal intelligence should be frightened of it.
Pro tip: There are no monsters in your closet, nothing is hiding under your bed, the boggy man doesn't exist.
Re: (Score:1)
*shrugs* but refactoring, recasting the idea of OS scheduling? systems people, is this something to take note of or not? as opposed to popular machine learning techniques?
Re: (Score:1)
The singularity is silly science fiction nonsense. No adult of normal intelligence should be frightened of it.
Pro tip: There are no monsters in your closet, nothing is hiding under your bed, the boggy man doesn't exist.
You're right about the "silly science fiction" part. However, the technological singularity (to differentiate it from the star-eating monster of physics) has actually been described as the geek version of nirvana or heaven (depending on your quasi-religious orientation), something very desirable, a state of orgasmic bliss for the post-human human. So nothing to be afraid of?
Re: (Score:2)
Ah, yeah. I've called that Ray Kurzweil's promise salvation with a glorious video game afterlife. It's an odd little cult. I've always wondered who they thought was going to bother keeping the lights turned on...
Re: (Score:1)
Let's go with "Skynet" then (Score:2)
Though Microsoft did change SkyDrive to OneDrive, so ... Onenet?
Size doesn't matter (Score:2)
Speed and size affect how well you do what you can already do. They do not grant new capabilities.
We need a paradigm shift to get new capabilities. Perhaps quantum chips, perhaps new software.
Re: (Score:3)
Good points, however one can simulate more sophisticated neural networks with bigger size. The origin or the nature of consciousness is still being debated, some say it's an emergent property some say it's quantum in nature and requires a quantum device, if the former then considering that a human brain has about ~100e9 neurons with 100e12 connections, it's just a matter of the size of the simulation.
Re: (Score:2)
People believe a lot of dumb things against all reason. Computationalism is among them.
They picked the name "singularity" precisely to set the imaginations of that particular group of idiots on fire. I think I hurt my eyes rolling them when I read the name.
Re: (Score:2)
It's not that simple because in an organic brain signal propagation speed is determined by the laws of physics, but in computers it's determined by things like the clock speed of the interconnects and the need to synchronize units.
It's also impossible to completely simulate an analogue process digitally. The fidelity can be very good, but never perfect. As an example, look at SPICE packages that simulate electronic circuits. That may or may not matter much with AI, it remains to be seen.
When we finally do g
Here I am with a brain the size of a planet... (Score:1)
Re: (Score:2)
Will it also have a "terrible pain in all the diodes down my left side"?
Great choice of name - NOT (Score:1)
BFS (Score:2)
Microsoft re-imagining the brain fucked scheduler
Planet-scale? (Score:1)
That seems very arrogant. I guess they did deliver planet-scale blue screens of death, followed by planet-scale forced updates and reboots. So even though the number seems small, I guess 100,000 GPUs should be enough for anybody?
Deep Thought successor computer (Score:2)
For those who remember the HHGTTG.
I hope that Slartibartfast does the fjords again.
Spare GPUs? (Score:2)
Sounds like the XBox Cloud Gaming service is not very popular and they bought way too much hardware.
And here we are... (Score:2)
Businesses keep sending money (Score:1)
Won't work (Score:2)
MS can't even tell me why my Azure backup failed to restore when needed. They have not a hope in hell of making AI work.
MS's core problem is that no one with any pride in their work would go near them for employment, so it's staffed top to bottom with failures and egotists without any real skill. That's why their products are almost uniformly shit with one exception, but I doubt that they're going to put the XBox team onto this.
Re: (Score:2)
MS can't even tell me why my Azure backup failed to restore when needed.
More likely they won't tell you that it's their negligence.
Mid size mining farm (Score:2)
Only 100,000 GPUs? Not enough to take control of a blockchain from Chinese mining farms.
Obligatory Archer quote (Score:2)
Do you want Skynet? Because that's how you get Skynet.
Top use of AI (Score:2)
Companies need more powerful AI to slip wanker-pill spam past spam filters.
Just name it the Torment Nexus (Score:2)
It has a good ring to it.