FreeBSD Can Now Boot in 25 Milliseconds (theregister.com) 77
Replacing a sort algorithm in the FreeBSD kernel has improved its boot speed by a factor of 100 or more... and although it's aimed at a micro-VM, the gains should benefit everyone. From a report: MicroVMs are a hot area of technology R&D in the last half decade or so. The core idea is a re-invention of some of concepts and technology that IBM invented along with the hypervisor in the 1960s: designing OSes specifically to run as guests under another OS. This means building the OS specifically to run inside a VM, and to talk to resources provided by a specific hypervisor rather than to fake hardware.
This means that the guest OS needs next to no support for real hardware, just VirtIO drivers which talk directly to facilities provided by the host hypervisor. In turn, the hypervisor doesn't have to provide an emulated PCI bus, emulated power management, emulated graphics card, emulated network interface cards, and so on. The result is that the hypervisor itself can be much smaller and simpler. The result of ruthlessly chopping down both the hypervisor, and the OS that runs inside it, is that both ends can be much smaller and simpler. That means that VMs can use much fewer resources, and start up much quicker.
This means that the guest OS needs next to no support for real hardware, just VirtIO drivers which talk directly to facilities provided by the host hypervisor. In turn, the hypervisor doesn't have to provide an emulated PCI bus, emulated power management, emulated graphics card, emulated network interface cards, and so on. The result is that the hypervisor itself can be much smaller and simpler. The result of ruthlessly chopping down both the hypervisor, and the OS that runs inside it, is that both ends can be much smaller and simpler. That means that VMs can use much fewer resources, and start up much quicker.
Define "booted" (Score:5, Insightful)
While I'm sure they've made great strides in a particular baseline benchmark, I find that the definition of "booting" to be a bit vague.
Ask a firmware person how long their device takes to boot, and they'll stop counting when jumping to a boot loader.
Ask a kernel person and they'll count 'booted' as when 'init' starts executing, which I presume is the case here (specifically probably booting when the kernel is already cached in RAM). The kernel people can't control how long init takes, so to them it's not that useful to include most of the time.
Re:Define "booted" (Score:5, Interesting)
Right, and in this case "booting" is even more strained. Ordinarily, booting time would start at power-on and continue until some usable state. It's hard to boot in 25ms when PCIe devices takes 100's of milliseconds to become useable and systems don't even exit their firmware for far longer than that. "Boot" in this context is just how long a particular piece of software takes to execute.
And "replacing a sort algorithm in the FreeBSD kernel" did NOT "improve its boot speed by a factor of 100 or more", it improved 7% of CPU consumption by that amount. Over many years of work, boot speed has been improved that greatly, not by one relatively modest change.
If anyone is old enough to remember the transition from hard disk to SSD boot devices, they will remember making their slow boot times near-instantaneous. Now, boot times, with SSD, are agonizing once again. It's crappy software to blame, systems are not doing anything so sophisticated that it should take a minute to boot. The industry has been plagued by this since the 80's.
Re: (Score:2)
"There's something seriously wrong if your SSD based system takes a minute to boot."
Agreed.
"My crappy HDD based ten-year-old desktop PC boots to the CLI in about 8s with a stripped down distro..."
Hardly representative of modern computers OR operating systems.
Re: Define "booted" (Score:2)
Yea. >10yo hw devices here, and ever since SSD that all boots to desktop login in ~25s, without any tweaks, and network mounts and services are available by then.
'Minutes of boot time' is so 1999.
Re: (Score:2)
Do any server/storage farms use SSDs yet for their boot discs at least. I wouldn't expect them to be trusting their data to anything other than spinning rust fo
Re: (Score:2)
A lot of datacenters are using SSDs for most of their storage. Once upon a time it was only selectively used as small fast tiers, now a lot of scenarios just have all flash. HDD still wins on $ per GB, so some workloads still prefer them when performance doesn't matter so much, but the margin is much closer than it used to be.
Depending on your flash based device, your woes might not have much to do with expected desktop/laptop usage. Compared to USB keys, SSDs tend to have more carefully managed wear lev
Re: (Score:1)
That's not entirely accurate. What hasn't changed much is the volume of data required for the average business. If all you needed "back then" was 10TB to store your database over gigabit network, today, that database is likely still in the 10-20TB range, your access is likely still over gigabit.
Now we have 2 and 4TB of flash, but we don't have 2TB and 4TB spindles anymore (at least not new). Getting to the 10-20TB is not that hard anymore, you only need 6 spindles or 6 SSDs to get the required redundancies,
Re: (Score:2)
I was looking at back-up solutions at the same time. In the data range I'm looking at moving into, the cost of backing up to tape would be more, substantially, than the cost of just getting a second drive of $DATA_DRIVE size and doing an occasional write to that with the current contents of $DATA_DRIVE. But you do remind me that I need to put the network onto wire again, not having bothered after my
Re:Define "booted" (Score:5, Informative)
That's why the summary states it's for micro VMs - guest OSes that are aware of their environment and can skip a lot of the boot process. The firmware just has to set up the virtual processor just enough to jump into the kernel - there is no need to POST in a VM environment. The kernel doesn't need hardware like PCIe or other things because it knows its talking to virtual hardware. So instead of having real drivers, it uses VirtIO drivers that basically skip all the initialization stuff.
VirtIO is useful if you have it - instead of an OS writing to a virtual hardware device, it just sends the data straight to the VMM using the VMM interface. So networking - instead of having to package up the packet into a memory buffer and starting DMA, the VirtIO driver takes the buffer from the kernel, then passes it to the VMM directly eliminating the emulation and driver overhead and the serialization/deserialization of packets.
As the devices are already well configured, there is nothing to initialize so most of the VirtIO interface are stubs - you don't have to initialize hardware, you might have to set a flag telling the VMM your networking is up but that's it.
Effectively, the hardware needs no time to initialize as it's already initialized by the VMM, so your 25ms boot time is just what the kernel software needs to get ready - initializing page tables, data structures and other management things.
Re: (Score:2)
"That's why the summary states it's for micro VMs - guest OSes that are aware of their environment and can skip a lot of the boot process. The firmware just has to set up the virtual processor just enough to jump into the kernel - there is no need to POST in a VM environment. The kernel doesn't need hardware like PCIe or other things because it knows it's talking to virtual hardware. So instead of having real drivers, it uses VirtIO drivers that basically skip all the initialization stuff."
Right, which emph
Re: (Score:2)
That reminds me of booting Windows in the 90 and 2000s. You would technically get to the desktop after “only” about two or three minutes (yeah) .. but it was unusable for about 5 more minutes while the hard drive spun loading various BS stuff Windows needs. If you tried to launch anything before that 5 minutes were up not only would the application have seizures but you dramatically increase the chance of a BSOD.
Re: (Score:2)
I read TFA and quoted tweets. It sounds like sorting SYSINITs took 7% of boot time and was reduced by a factor of 100. Sooo, that makes it 0.07% of the boot time? Or that the resulting boot time is now 93% of what it used to be? I don't think that's what got it to 25 msec.
I'm also puzzled about why the diversion into the kernel/hypervisor responsibility split and AWS Lambdas. Seems like the author kinda buried the lead, explaining why this is good without first explaining what was done. The Reg needs some b
Re: (Score:2)
I read the actual commit message which says
"We can mergesort the sysinits instead of bubblesorting them, which shaves about 2 ms off the boot time in Firecracker."
So there you go. 2ms. I'm not surprised nobody outside of microvms gave a crap about bubble sort all this time.
FreeBSD rocks. Loving Bhyve and Jails (Score:5, Interesting)
Re:FreeBSD rocks. Loving Bhyve and Jails (Score:5, Informative)
Best of all, no systemd.
Re: (Score:2, Informative)
Don't forget good up to date documentation and native ZFS.
Re: (Score:2, Informative)
The FreeBSD documentation is only good and up to date for a very small surface of use cases. For the things which don't work, it's woefully inadequate.
Re: (Score:1)
Re: (Score:2)
It sounds like you're saying a lot without making any concrete quality differentiation.
Re: (Score:1)
Re: (Score:2)
Have you notified anyone the documentation was wrong or outdated? What issues did you run into?
Re: (Score:2)
I had. I'd found a hardware bug in the mptsas drivers years ago (freebsd 8/9 era) which was causing stability and data corruption problems and attempted to bring them up with the driver maintainer (via IRC). I was effectively told "not my problem, pound sand". One of the least hospitable communities I've ever had the displeasure of interacting with.
Re: (Score:2)
Excellent virtualization, solid fast containers, no systemd, good documentation, native ZFS ... Sounds more and more like Solaris by the second.
Just a damn shame where it landed.
https://www.youtube.com/watch?... [youtube.com]
Re: (Score:1)
Yeah, it's great having nothing to manage services on your system except some random init scripts.
I'm pretty sure everyone who whines about systemd runs a single system at home and never actually thinks about the big picture.
Re: (Score:2)
It's only a matter of time before Linux rips off the concept and people will come to believe they invented it.
Re: (Score:1)
What concept?
You realize that the virtualization in FreeBSD has been a cargo lift of the features from virtio/KVM/lxc, and this has existed on linux for years?
Re: (Score:1)
Re: (Score:2)
.... no, KVM is not a rename of qemu. Not "basically" or in any other way. lol They're entirely different, interoperable components.
Does KVM virtualization utilize qemu as an abstraction for ease/consistency of access? Yes. libvirt sits on the other side of qemu providing additional usability/abstraction.
Do you need qemu to use KVM? No, you do not. You absolutely can run kvm machines directly without the complexity of qemu via API ioctls.
Re: (Score:1)
Re: (Score:2)
I'd like to hear more about your hypothesis that a bunch of GPL code was stolen and put into an incompatibly licensed OS. VirtIO is a standard. Don't confuse supporting a standard with porting some code. KVM is Linux-specific and has nothing to do with FreeBSD except a couple failed attempts to port it before the emergence of bhyve, which is also not connected to Linux.
Anyway we were talking here about jails, which FBSD was blessed with over 20 years ago, well before the concept of Linux containers.
Re: (Score:2, Insightful)
I'm guessing you've not had to go through the upgrade process yet, then. Good luck. Jails are probably the least maintainable virtualization/container technology, and that's only because they're more niche than Docker.
Re: (Score:1)
Re: (Score:2)
So is it possible to update FreeBSD jails without a complete host/OS/kernel upgrade, first? Last I checked, there's still a fair amount of chicken/egg dependency crap going on there - eg. you've got to upgrade jail tools, but the jail tools require you to take the jails down before upgrade.
Because that's not really something necessary with docker, as much as it sucks.
Re: (Score:1)
Re: (Score:2)
Hah - it'd have been really nice having a master build box or freebsd-update possibility when I last had to do this. Everything I had was peacemeal one-off and way out of sync. buildworld ran for weeks with multiple failures... it was a huge mess.
Re: (Score:2)
That documentation didn't exist yet when I last used jails, largely because the tooling didn't exist. Had nothing to do with planning.
Re: (Score:1)
Re: (Score:2)
I inherited an environment which was built on STABLE initially, and this was some years since that. It was still long before any sane person would consider it STABLE, or usable for production. This was very early FreeBSD 7 RELEASE timeframe and I believe this was the first RELEASE with jail support. (It was still comparable to a barely usable beta and there was no way to really address the situation without migrating the jails elsewhere, or taking a significant outage.)
I have no problem with upstream implem
Re: (Score:1)
Re: (Score:2)
Are those Type-I and Type-II as measured on the Kardashev scale?
Type-1: Uses all the resources of a single CPU
Type-II: Uses all the resources of a PC
Type-III: Uses all the resources of a LAN
Type-IV: Uses all the resources available on the Internet
I still don't know... (Score:2)
Reminds me of IBM PowerVM... (Score:2)
On POWER machines, once you set up your two VIO servers (VIO servers are something odd to IBM, vms running a modified version of AIX called IOS, where they take the physical drives, virtualize everything, and hand it to the true VMs.) Because everything on the "virtual" size of the VIO servers is "generic", and requires no drives, the inner VMs that don't rely on any hardware can boot extremely quickly.
I was wondering when we would see this functionality on PCs, where a hypervisor can do all the heavy lift
Re: (Score:2)
POWER boxes are pretty interesting. I had VIOS running and then wanted to switch back to bare metal. The ram previously allocated to VIOS was unavailable for other OS's until doing a factory reset. Now I'm seeing V7R2 IPL in under 5 minutes under VIOS.
Re: (Score:2)
This functionality has existed for a very long time on linux. I'm not sure why it's even passingly mentionable on FreeBSD, except FreeBSD folks think they do everything better on the basis of a few small (but legitimate, highly useful) corner cases.
Re: (Score:2)
Lol, I was going to snark post that it only took them ~15 years to catch up (sort of) to IBM VIO/AIX/Power...
also (Score:2)
use much fewer resources, and start up much quicker.
And have a ton less code in ring 0 that could have (potentially exploitable) bugs.
Now run your VM host on a seriously hardened system with MAC and RBAC and we're getting somewhere.
Whoever wrote the summary needs to learn English. (Score:5, Informative)
The article being referenced says:
FreeBSD (HEAD) no longer spends time running a bubblesort on its SYSINITs. We're now running a mergesort which is ~100x faster
When the FreeBSD kernel boots in Firecracker (1 CPU, 128 MB RAM), it now spends 7% of its time running a bubblesort on its SYSINITs.
O(N^2) can bite hard when you're sorting over a thousand items. Time to replace the bubblesort with something faster.
Now, I don't see anything there that claims the boot itself is 100x faster, just that one small part of the boot is about 100x faster. And that part which is being improved only took 7% of the CPU's time.
Re: (Score:2)
Re: (Score:2)
WTF is a bubblesort doing anywhere at all and for all this time??? Linux I could see that being the case but in BSD?
There is no way it's this much of a performance hit; it's all the goofy virtualization that is saving 100x speed. goofy because VMs are proving how much OS have failed at their jobs and now we've got separations being removed like some 60s idea for a more abstracted HAL. If you did the HAL properly you'd not need part of what a VM is for.
I'm still hoping for a micro kernel system to get some
Netcraft now confirms (Score:2)
Still a VM? (Score:2)
At one point does it stop being a Virtual Machine and just become a protected memory space? I mean, if it's not really managing it's own disks and such, is it really an OS?
Re: (Score:2)
When it stops virtualizing the whole machine and instead only virtualizes memory.
"I mean, if it's not really managing it's own disks and such, is it really an OS?"
What do you think an OS is? What do you think "it" is here?
An OS *can* "manage its own disks and such" while those "disks and such" are actually virtualized because, you know, there's a VM. But they don't have to be because there's, you know, a VM. A VM uses virtualization hardware other than virtual memory.
Without tools... users can't do this. (Score:2)
Advertising for regulations to be created that require sharing details about hardware publicly! Thank you so very much!
Re: (Score:2)
Liberty for all! Enforced by the government! Freedom for me! The law for you!
Synthetic HDWE (Score:2)
Reads like the moment has arrived where compute become virtual and hardware is immaterial or synthetically abstracted away.
Can SNOW suddenly handle images, streaming and movies? MacOS flippantly run Windows applications by throwing it all over the FreeBSD fence? Maybe ChatGPT can LLM humungous datasets spread over clusters of supercomputers?
What hath freedom from OS’n wrought now that the data hath been set free, as in speech?
Re: (Score:2)
"Reads like the moment has arrived where compute become virtual and hardware is immaterial or synthetically abstracted away."
That moment arrived in the mid-60's. How old are you?
"Can SNOW suddenly handle images, streaming and movies? MacOS flippantly run Windows applications by throwing it all over the FreeBSD fence? Maybe ChatGPT can LLM humungous datasets spread over clusters of supercomputers?
What hath freedom from OS’n wrought now that the data hath been set free, as in speech?"
Nice try,. but it
Re: (Score:2)
Reads like the moment has arrived where compute become virtual and hardware is immaterial or synthetically abstracted away.
For me that moment arrived the first time I read the introduction to The Art of Programming... lol
Nothing new is described here. What they're talking about is simply shifting which part of the code runs where, and removing redundant parts where the OS and the hypervisor both knew about the details of the hardware. Saying that only the hypervisor needs to know that stuff is useful, but the hypervisor is still part of the "compute."
Re: (Score:2)
Scalar memory maps on the fly addresses economy of scale rather than compute homogeneity. Abstractions are huge wins but that’s not evidently what AWS is selling – just bigger silos and flex runtime. That boils down to hotswap compute with infinite RAM potentials bandwidth-limited. The heavy haul provider for large data over their data paths. Sighhoped it presaged more.
40 reboots per second (Score:2)
Reboots fix lots of issues, right? When something goes wrong, reboot, and see if that helps! So if you can reboot in 25 ms, that allows you to reboot 40 times a second. Your system should run flawlessly all the time!
This is an ongoing process (Score:2)
Colin Percival is the one who's been doing most of the work to speed up the FreeBSD boot process, for the last several years.
He's made a lot of small improvements, and a few pretty big ones.
His focus has mostly been Amazon VMs, but a lot of his changes have sped up the boot process on other hardware too.
When FreeBSD 14.0-RELEASE comes out in October, it'll boot much zippier on everybody's computer.
Good job (Score:2)
Is this really a VM, then? At this point, with no hardware emulation and just talking to the hypervisor, you're blurring the line between a virtual machine it becomes just a regular application... running on a platform locked into a particular vendor. So we rediscovered the wheel here, good job.
Approaching BeOS territory (Score:1)
Nice to see that modern OSs are finally catching up to where BeOS was about 25 years ago ;)