Slashdot Log In
Patch the Linux Kernel Without Reboots
Posted by
kdawson
on Thu Apr 24, 2008 10:00 AM
from the click-n-go dept.
from the click-n-go dept.
evanbro writes "ZDNet is reporting on ksplice, a system for applying patches to the Linux kernel without rebooting. ksplice requires no kernel modifications, just the source, the config files, and a patch. Author Jeff Arnold discusses the system in a technical overview paper (PDF). Ted Ts'o comments, 'Users in the carrier grade linux space have been clamoring for this for a while. If you are a carrier in telephony and don't want downtime, this stuff is pure gold.'"
Update: 04/24 10:04 GMT by KD : Tomasz Chmielewsk writes on LKML that the idea seems to be patented by Microsoft.
Related Stories
[+]
Linux: Ksplice Offers Rebootless Updates For Ubuntu Systems 211 comments
sdasher writes "Ksplice has started offering Ksplice Uptrack for Ubuntu Jaunty, a free service that delivers rebootless versions of all the latest Ubuntu kernel security updates. It's currently available for both the 32 and 64-bit generic kernel, and they plan to add support for the virtual and server kernels by the end of the month, according to their FAQ. This makes Ubuntu the first OS that doesn't need to be rebooted for security updates. (We covered Ksplice's underlying technology when it was first announced a year ago.)"
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
Needed that bad? (Score:5, Insightful)
Re: (Score:3, Funny)
Meaning, yes I agree that in most cases it is not needed, but I have internal processing servers that have up times of over 3 years, so if I had something like this probably all my servers would have up times of this long..
Re:Needed that bad? (Score:5, Insightful)
I have internal processing servers that have up times of over 3 years
I've never understood this boasting about uptime. Long uptimes are a bad thing! How do you know a configuration change hasn't rendered one of your startup scripts ineffective? If you have to reboot for some unexpected reason, you could be stuck debugging unrelated problems at very inopportune moments.
You need to schedule regular reboots so that you can test that your servers can start up fine at a moment's notice. Long uptimes are a sign a sysadmin hasn't been doing his job.
Parent
Re:Needed that bad? (Score:5, Informative)
A patch to the kernel almost never requires changes to startup scripts. They're not talking about adding new functionality with user-space-addressable interfaces with this tool. They're talking about being able to install about 84% of security hotfixes in a hurry outside your scheduled reboots then rebooting on your regular maintenance schedule.
Parent
Re:Needed that bad? (Score:5, Insightful)
Parent
Re:Needed that bad? (Score:5, Insightful)
dd your production box's system filesystems to another hard drive, put in an identically specced machine, boot that?
Parent
Re:Needed that bad? (Score:5, Insightful)
Isn't that what QA systems and effective approaches to change management are supposed to handle?
If I am planning a change, I should discover problems with the startup scripts in QA, not in production, especially if a production reboot is not required to implement the change.
Parent
Re: (Score:3, Interesting)
Two things:
The very fact that there is load balancing means that every server is likely to have active connections going through it. If you currently have connections going through a specific server, you don't want to drop those connections in order to reboot that particular machine. This allows updates to a live machine.
Second, this is telephony, meaning it is t
Re:Needed that bad? (Score:5, Informative)
If you have a load balanced environment then you have the ability to redirect new connections away from a given server. Then it's just a matter of waiting for the active connections to terminate before the machine ends up in an idle state where you can safely apply patches offline. I've worked in a number of telephony environments and this was always the way we would patch systems. Stop accepting new connections, wait for existing ones to end, then perform the patch, reboot, verify, and start accepting connections again.
Second, this is telephony, meaning it is the infrastructure on which the internet is based. There's no dns tricks or tcp/ip you can use to send people to a different "server" if that server is the switch connected to your fiber backbone. Basically, there are points in the infrastructure where there are by necessity a single chokepoint.
Any mission critical hardware, switches, routers, servers, etc. should be set up in redundant pairs (or triplets,
Redundancy is key, and any commercial datacenter will offer it all the way from their connections to the outside world to the connections they provide their customers. Every datacenter used by every company I ever worked for (about 10) offered redundant power and redundant network drops (using HSRP, VRRP, etc) for our equipment. If the datacenter needed to upgrade a router they'd move all traffic off one router so they could upgrade and test it, then move traffic off the other and repeat the process. Similarly if we needed to upgrade our firewalls, switches, etc. we'd fail over to the second redundant device first. In some cases we had bonded interfaces right on the end servers so as long as one path remained active we could power down an entire switch, router, firewall, etc. In other cases we relied on load balancing across servers that were alternately connected to one or another switch.
Parent
Re:Needed that bad? (Score:4, Informative)
This assumes that active connections will terminate in a timely fashion. I used to have internet service via an ISDN via a connection to my office. My ISDN calls would connected for a couple of months at a time. Yes, one connection lasting multiple months. There are other cases where a connection, context, or state between two systems would need to be maintained for extended periods of time. Many of these situations can not be solved by load balancing and would benefit greatly by the ability to make kernel changes without interrupting current work, or waiting for it to complete.
Parent
Re:Needed that bad? (Score:5, Informative)
For all you know, your apparent always-on connection was actually a virtual connection being frequently switched & reswitched over many different real physical connections. That would be a fairly standard architecture for having a network infrastructure which can have components being worked on while data is still flowing through the network.
When the telecom provider is "waiting for active connections to go away" on a particular device only means that all of the virtual connections that are momentarily being switched that device have been successfully switched to another device. It doesn't mean that any of those virtual connections have to be terminated.
Parent
Re:Needed that bad? (Score:4, Insightful)
A decent load balancer will obviously give you the choice of whether to take a server out of service immediately disrupting existing sessions or simply stop sending new sessions to it while allowing existing sessions to continue.
As for your comment about physical connections, that's what portchannels and multilink trunks are for. Or VRRP and HSRP depending on which level of "connected to" you mean.
Parent
Re:Needed that bad? (Score:4, Interesting)
So you have this security hotfix you really want to apply, but it's going to 25 hours on average to fix a box and 50 hours to fix them all.
You could, with ksplice and a good concurrent control system, make your average time to fix 5 minutes in over 80% of kernel upgrade scenarios rated "Critical". Your boxes could still be rebooted on a regular basis later.
Which do you prefer?
Parent
Re:Needed that bad? (Score:5, Insightful)
Methods like that usually suck in real-life, because right the day before you want to 'take it out of rotation', a circuit is opened through it that requires five nines (so you can't drop it), and it will remain open for months...
You will end up with 99 boxes waiting to 'get out of rotation' for every
single box that you don't need to update...
Murphy will make sure of that.
Parent
Re: (Score:3, Interesting)
start up a new virtual machine with the new kernel, then when your sure it's working, just switch everything across from the old to the new, and shut down the old virtual instance
No, No, No and No again. (Score:5, Interesting)
1. We connect to several load balanced systems and the complexity introduced by load balancers translates to inexplicable down time. No load balancers means a pretty steady diet of the latest and greatest server hardware, but no down time. The a few minutes of down time costs more than the server hardware.
2. High availability translates more roughly into nodes that can fail (ex. power off) and not take the cluster down. This boils down to active-passive application architecture more than just using heartbeat.
As an FYI, PostgreSQL clustering is a killer application for me. Erlang is also great in many ways, but requires application architecture with active-passive node awareness. Which isn't present in things like Yaws, or even my other favorite non-erlang app nginx. Heartbeat is the solution there, but I'd like to see yaws be cluster aware on its own. http://yaws.hyber.org/
Parent
Re:No, No, No and No again. (Score:5, Insightful)
I spent a decade in perimeter networking at a Fortune 50 US bank. My group didn't do the internal network, just the perimiter, and we still had dozens of network sites and thousands of pieces of equipment. The bank itself has hundreds of thousands of employees, millions of users. Online banking and brokerage are about as high availability as you can get save utilities (power, water, telephony, etc) or military. Seconds of online brokerage downtime equated to millions of dollars lost.
The idea that load balancing introduces inexplicable down time is completely unsupported by my experience.
"One-node clusters" seems like marketing speak for "single point of failure". A cluster by definition is two or more nodes.
Redundant routers, switches, firewalls, the works or you're not high-availability in my opinion. The fact that you're talking about Postgresql instead of Oracle or DB2 on mainframes makes me think that your idea of high availability is different than mine.
Parent
Re: (Score:3, Interesting)
Unless it fails. (Score:3, Insightful)
Re:Unless it fails. (Score:4, Funny)
Parent
Re: (Score:3, Informative)
Re:Unless it fails. (Score:4, Informative)
Parent
Amazing (Score:5, Interesting)
Re:Amazing (Score:5, Funny)
- Roey
Parent
Re:Amazing (Score:5, Insightful)
Parent
Re:Amazing (Score:4, Insightful)
"A remote attacker who successfully executes a privilege escalation exploit and gains root access will have an easier time taking control of your server and hiding their tracks".
Thanks for pointing that out
- Roey
Parent
Wrong way to solve the uptime problem (Score:5, Insightful)
Re:Wrong way to solve the uptime problem (Score:5, Funny)
Parent
Not only the CEO (Score:5, Interesting)
It always irks me to see people just not understand concepts like "peak" vs "average", or "failing over".
- A cluster of, say, 4 machines (small application, really) which are loaded to 90% of capacity, if one dies, the other 3 are now at 120% of capacity each. If you're lucky, it just crawls, if you're unlucky, Java clutches its chest and keels over with an "OutOfMemoryError" or such.
- if you're at 90% most of the time, then fear Monday 9:00 AM, when every single business partner on that B2B application comes to work and opens his browser. Or fear the massive year-end batch jobs, when that machine/cluster sized barely enough to be ready with the normal midnight jobs by 9 AM, so those users can see their new offers and orders in their browsers, now has to do 20 times as much in a burst.
Basically it amazes me how many people just don't seem to get that simple rule of thumb of clusters: you're either getting nearly 100% uptime and nearly guaranteed response times, _or_ you're getting that extra hardware fully used to support a bigger load. Not both. Or not until that cluster is so large that 1-2 servers failing add negligible load to the remaining machines.
Parent
Re:Wrong way to solve the uptime problem (Score:4, Funny)
Parent
Re:Wrong way to solve the uptime problem (Score:4, Informative)
Of course many of the reasons is a lot of commercial telecom apps are badly implemented and need better management controls.
Parent
Re:Wrong way to solve the uptime problem (Score:5, Insightful)
Parent
Re:Wrong way to solve the uptime problem (Score:4, Informative)
Why not 24/7/52 or 24/7/4.3/12 or just 24/365 (or 24/365.242 for the pedants).
Parent
Already been used (Score:5, Informative)
Re: (Score:3, Informative)
replace modules (Score:3, Interesting)
The real test... (Score:5, Funny)
Re: (Score:3, Informative)
So yes, ksplice can be installed/used without rebooting.
Impressive hack (Score:5, Informative)
He basically compiles a patched and unpatched kernel with the same compiler, compares the ELF output, and uses that to generate a binary file that corresponds to the change. That gets wrapped in a generic module for use, another module installs it along with JMPs to bypass the old code and use the new, and he performs the checks needed to make sure he can safely install the redirects.
He also has to differentiate real changes from incidental ones (the example given is changing the address of a function - all references to it will change, but they don't really need to be included in the binary diff).
The only human work required is to check whether a patch makes semantic changes to a data structure... whether eg. an unsigned integer variable that was being used as a number is now a packed set of flags - the data declaration is the same, but it's being used differently.
Interesting paper. Also a useful new set of capabilities for any Linux user who can't handle downtime for quarterly patching... worth its weight in gold in some businesses.
Erik
Re:Impressive hack (Score:4, Funny)
A silver dollar, from which bits were commonly cut, weighs about .77 troy ounces.
Today's gold price as of posting is about $889.95 US per troy ounce.
A silver dollar was typically cut into 8 bits, which gives us a weight per bit of 0.096 ounces. That translates to about $85.66 per bit weight in gold. Remember, this is per system being patched.
Since the patches being applied ranged from 1 line to 285 lines per the paper, and a reasonable estimate of compiled average bytes per line is something like 20, we get a value of $13,700 per line of patch in gold. Even for the smaller patches, this is significant. The largest patch would be worth nearly $4,000,000 USD in gold.
Of course, for 64 bit systems vs. 32 bit, the value would be twice as much :)
Erik
Parent
If it's that critical, shouldn't you have two? (Score:5, Insightful)
If you've got this system that's so critical you can't reboot it for a kernel upgrade, what do you do when the building catches fire or a tanker truck full of toxic waste hops the curb and plows through the wall of your datacenter?
I'd rather have a full second set of anything that critical. It should be in a different state (or country) and have a well designed and frequently used method of seamlessly transferring the load between the two (or more) sites without dropping anything.
If you can't transfer the workload to a location at least a couple hundred miles away without users noticing then you're not in the big league.
And as long as the workload is in another datacenter, what's the big deal about rebooting for a kernel upgrade.
Re: (Score:3)
Extreme example: Try to design a fail-over for the space shuttle's solid rocket boosters :)
Interestingly, I've found that the skill needed (and the pay gathered) to deal with systems that can't be made redundant is much higher than that needed to work on "grid" or cluster systems where multiple cheap pieces of hardware are used.
And they tend to be more reliable too.
Over-engineered solution to a non-existent problem (Score:4, Insightful)
Once again, we have an over-engineered solution to a non-existent problem.
Any enterprise-level customer is going to have a VERY lengthy Q&A process before deploying anything into production. This includes testing kernels, hardware, networks, interaction, application, data and so on. One pharmaceutical company I know of is federally mandated to do this twice a year, every year, for every single machine that reads, writes or generates data. Period.
So you hot-patch a running Linux kernel. How do you Q&A that? How do you roll back if the patch fails? Where is your 'control'?
The answer? A duplicate machine. But wait, if you have two identical machines... isn't that... a cluster?
Exactly. And THIS is how you perform upgrades. You split the cluster, upgrade one half, verify that the upgrade worked, then roll the cluster over to that node, and upgrade the second portion of the cluster. If you have more machines in the cluster, you do 'round-robin' upgrades. You NEVER EVER touch a running, production system like that.
Well, not if you want any sort of data integrity or control and want to pass any level of quality validation on that physical environment.
You are Wrong (Score:3, Insightful)
Hmmm. I happen to live by your words in an environment where this is theoretically possible, but practically impossible. Why? Because when the cluster rolls to a passive node, the application times out on the existing connections. The tim
And Microsoft claims to have invented it (Score:4, Informative)
Tomasz Chmielewski wrote on LKML: the idea seem to be patented by Microsoft, i.e. this patent from December 2002: http://www.google.com/patents?id=cVyWAAAAEBAJ&dq=hotpatching [google.com] In essence, they patented kexec ;)
Andi Kleen promptly provided prior art: The basic patching idea is old and has been used many times, long predating kexec. e.g. it's a common way to implement incremental linkers too.
It's Not For 100% Uptime (Score:3, Insightful)
But this isn't about 100% uptime. It's about not having to reboot for a kernel upgrade. You should still have hot failover if you want HA, this just removes one more thing that requires a reboot.
It's like people saying, "I don't mind rebooting after installing Office, I don't expect 100% uptime from my workstation." Of course you don't need to be able to do software installs without rebooting. But isn't it nice to have that option available?
Same with this. When (and if) it gets stabilized and standardized, you'll use it. Not for 100% uptime, just because it's nice to not be required to reboot to enable a particular software install.
What!? Unpossible! (Score:4, Funny)
This was the smallest part of the interview... (Score:4, Informative)
Re:Maybe... (Score:5, Funny)
Windows 7, it's not awful like Vista!
Parent
Re:In Soviet Russia, (Score:5, Funny)
Parent