Slashdot Log In
Linux Kernel Goes Real-Time
Posted by
kdawson
on Sat Oct 14, 2006 04:42 PM
from the just-this-instant dept.
from the just-this-instant dept.
Several readers wrote to alert us to the inclusion of real-time features in the mainline Linux kernel starting with version 2.6.18. (Linus Torvalds had announced 2.6.18 on September 19.) Basic real-time support is now mainline. This will ease the job of developers of embedded Linux applications, who for years have been maintaining real-time patch sets outside of the mainline kernel. The announcement was made by TimeSys Corp., a provider of developer services. Much of the work was done by Thomas Gleixner at TimeSys and Ingo Molnar at Red Hat.
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
What about media? (Score:4, Interesting)
What other benefits would the desktop see from this?
Re: (Score:3, Informative)
Re: (Score:3, Insightful)
Re:What about media? (Score:4, Informative)
In a "classic" unix kernel, only one process is in kernel context at a time. That's because if you have more than one process in kernel context at once, there's a good chance that they'll step on each other.
Say that two processes were in kernel context at the same time, and one wanted to delete data from a data structure and the other wanted to read from it. If the process that wanted to read from the data structure got a pointer to it, and then before it could do anything with it, the process that wanted to delete it came in and did delete it, when the process that wanted to read from that pointer it just grabbed came back, it would fail, and probably cause a kernel panic.
If all the data structures in the kernel are protected by semaphores (and spin locks in case of interrupt routines), and all the paths through the code prevented deadlocks (that's one of the tricky parts), then multiple process could be in kernel context at the same time.
You'll get the speed up because in a classic UNIX system if a process in kernel context took a long time with what it was doing, it effectively locks out all the other processes that want to go into kernel context until it's complete. When you have everything set up so that multiple processes can be in kernel context at the same time, you move the contention from the entire kernel down to individual data structure manipulation, and the wait times are much lower.
I believe (and someone that's actually still reading this correct me if I'm wrong), they do have semaphoring of code within the Linux kernel right now, but it's over big parts of the code, rather than individual data structures. There is some benefit of doing that, but not nearly the benefit of doing those individual data structures.
It's tough, but worth it in the end. I think the hardest part of this will be the drivers, and making sure that they behave well in a system like that.
Parent
Performance overhead of the -rt patch-set (Score:5, Informative)
I can definitely say that unlike some other approaches, the -rt Linux kernel does not introduce a "big speed penalty".
Under normal desktop loads the overhead is very low. You can try it out yourself, grab a Knoppix-based PREEMPT_RT-kernel live-CD from here: http://debian.tu-bs.de/project/tb10alj/osadl-knopp ix.iso [tu-bs.de].
From early on, one of our major goals with the -rt patchset (which includes the CONFIG_PREEMPT_RT kernel feature that makes the Linux kernel "completely preemptible") was to make the cost to non-RT tasks as small as possible.
One year ago, a competing real-time kernel project (RTAI/ipipe - which adds a real-time microkernel to 'above' Linux) has done a number of performance tests to compare PREEMPT_RT (which has a different, "integrated" real-time design that makes the Linux kernel itself hard-real-time capable) to the RTAI kernel and to the vanilla kernel - to figure out the kind of overhead various real-time kernel design approaches introduce.
(Please keep in mind that these tests were done by a "competing" project, with the goal to uncover the worst-case overhead of real-time kernels like PREEMPT_RT. So it included highly kernel-intensive workloads that run lmbench while the box is also flood-pinged, has heavy block-IO interrupt traffic, etc. It did not include "easy" workloads like mostly userspace processing, which would have shown no runtime overhead at all. Other than the choice of the "battle terrain" the tests were conducted in a completely fair manner, and the tests were conducted with review and feedback from me and other -rt developers.)
The results were:
(Smaller is better. The full test results can be found in this lkml posting [theaimsgroup.com].)
I.e. the overhead of PREEMPT_RT, for highly kernel-intensive lmbench workloads, was 4.0%. [this has been a year ago, we further reduced this overhead since then.] In fact, for some lmbench sub-benchmarks such as mmap() and fork(), PREEMPT_RT was faster.
(Note that the comparison of PREEMPT_RT vs. I-pipe/RTAI is apples to oranges in terms of design approach and feature-set: PREEMPT_RT is Linux extended with hard-realtime capabilities (i.e. normal Linux tasks get real-time capabilities and guarantees, so it's an "integrated" approach), while ipipe is a 'layered' design with a completely separate real-time-OS domain "ontop" of Linux - which special, isolated domain has to be programmed via special non-Linux APIs. The "integrated" real-time design approach that we took with -rt is alot more complex and it is alot harder to achieve.)
See more about the technology behind the -rt patchset [redhat.com] in Paul McKenney's article on LWN.net [lwn.net], and on Kernel.org's RT Wiki [kernel.org].
Parent
Re: (Score:3, Interesting)
Re:What about media? (Score:4, Informative)
Parent
Re:What about media? (Score:5, Interesting)
But desktop users, or at least me, don't care about doing things "fast". I care about things feeling fast. I care about latency. So, do these patches help the audio on my computer not skip when I move a window? I've tried the premptive kernel patches in the past with no noticeable difference. How are these patches similar, different, or complimentary to Ingo Molnar's (whose patches are mentioned in the article). Thanks.
Parent
Re: (Score:2)
Re: (Score:2)
But sooner or later, they'll have to. Multi-core/multi-processor single box systems will greatly benefit from it.
Re: (Score:2)
I have an M-Audio Delta 66 for occasional recording but mostly just music playback. I've read about others having similar problems to mine. Seems like there may be some specific issues with this card. I haven't had time to look into it too much, so I just thought I'd throw this question out there. I know there are a million variables as other replies have hinted at...
Re:What about media? (Score:5, Interesting)
Parent
Re: (Score:3, Interesting)
Actually, the human ear is capable of detecting two discrete notes if they are separated by at least 30 milliseconds. If you set your digital delay to anything shorter than that, it won't sound like an echo - it will sound like the delayed signal is part of the original signal. Your example of a synth producing a note half a second after you hit a key is spot-on though, you'll notice that and be pret
Re: (Score:2)
Re: (Score:2)
If you're really interested, I have some spectrograms of what OGG and MP3 do to a waveform. The data rate is undoubtedly important, but you also have to take into consideration t
Re: (Score:2, Interesting)
Re: (Score:2, Insightful)
I know what you mean, and IMO many times sensory limitations are generalized to the entire population while extraordinary indviduals who spend time training their senses will be able to exceed them. I had the benefit of learning from professionals in audio production in a collegiate setting. The ~30ms rule was one I heard from many of my instructors and
Re: (Score:3, Interesting)
I for one believe that it could be immensely useful for Linux to provide the option of switching to a real time scheduler when starting a video playing - and for it
Re: (Score:2)
Re:What about media? (Score:5, Informative)
And actually the grand parent is right to wonder about whether or not this will allow for a more responsive feeling system, because in general, it will. Much more so than the pre-emptive kernel routines already in = 2.6.17.*. If you've ever used a real time OS you'll know what I mean -- nothing feels so nice to use. The processor can be pegged, and have a ridiculous load average of say... 100 or something (100 tasks are trying to use 100% of the CPU), and you won't really notice any sluggishness... but, of course, tasks will just take a long time to complete.
It's not without it's downfalls though. Obviously being able to guarantee low latency interrupt responses is going to require some overhead. You'll definitely slow your computer down by using a real time OS, although unless you're a gamer you might not even notice.
All that said and done, unless you're using your computer for some very specific things like embedded devices, critical applications (medical, power station management, etc), and audio / video stuff, you'll probably never notice the difference.
I know a lot of audio people are happy about the real time patches because the delay between turning a dial, or moving the mouse, and the noises that come out of the speakers is quite noticeable even with really small delays.
Parent
Re: (Score:3, Interesting)
Real-time is all this and more.
Getting tasks to complete on time is actually not something that an operating system can guarantee: you can simply lie about how long your job will take. (But, like your compiler, if you lie to your operating system, it will get its revenge.) Or you can design your hardware so that it swamps the system with so many interrupts that it can't service any user tasks. This kind of problem is a property of the system as a whole, and the best that the kernel can do is guarantee t
For Audio Work! (Score:2, Informative)
Re:What about media? (Score:5, Informative)
Desktops and most servers do not get any benefit from a RTOS. RT makes it so that the system purposefully downgrades less-useful things like user input for maximum priority things like, say, polling a fetal heart monitor every n milliseconds or responding to an automobile collision to deploy an airbag.
RTOS in Linux is primarily useful for Linux-based routers. However, seeing that QNX has been in the industry for 25 years, has an extremely good reputation (it's the de facto standard in the auto industry), and is already POSIX compliant, Linux still has a long way to go. The price for QNX might be USD $10,000+, but if you actually have a need for a RTOS, licensing cost is not a major obstacle.
Parent
Re:What about media? (Score:4, Interesting)
Parent
Real-time OS (Score:3, Informative)
http://en.wikipedia.org/wiki/Real-time_operating_
"A real-time operating system (RTOS) is a class of operating system intended for real-time applications. Examples include embedded applications (programmable thermostats, household appliance controllers, mobile telephones), industrial robots, industrial control (see SCADA), and scientific research equipment.
A RTOS is valued more for how quickly and/or predictably it can respond to a particular event than for the given amount of work it can perform over time. Key factors in an RTOS are therefore minimal interrupt and thread switching latency."
Re: (Score:2)
Re: (Score:2)
Anyone with experience on unix-like RTOSes is probably glad to see RT being introduced into the main branch, as the responsiveness of a RTOS is, by far, much more satisfying. The O(1) scheduler was a nice step forward. The incoming
Re: (Score:3, Insightful)
Linux audio software will now be #1 (Score:2, Informative)
Re: (Score:2)
"The Ardour project is happy to be able to announce the involvement and support of Solid State Logic (SSL), one of the most respected and trusted names in the field of audio technology. SSL has chosen to support Ardour's development and to promote the idea o
Re: (Score:2)
What does this really mean? (Score:3, Insightful)
Anyone know if this is just the ADEOS micro-kernel patch being distributed as part of the vanilla kernel? If not, is it compatible with RTAI, Xenomai, Fusion, and RTLinux?
Re: (Score:2)
Re: (Score:2)
1. futexes (locking for app code) with priority inheritance
2. high-resolution timers
It's progress, but not everything you need yet (Score:4, Interesting)
Linux has made major progress in the real-time area. But it still doesn't have everything needed.
Many drivers are still doing too much work at interrupt level. There are drivers that have been made safe for real time at the millisecond level, but that's not universal. Load a driver with long interrupt lockouts and your system isn't "real time" any more. This is the biggest problem in practice. There are too many drivers still around with long interrupt lockouts.
The Linux CPU dispatcher is now more real time oriented. (Finally!)
Interprocess communication in Linux is still kind of lame by real-time OS standards. The tight interaction between CPU dispatching and messaging still isn't there, although it's getting better. Interestingly, it's there in Minix 3, and of course, that's been in QNX for years. For simple real-time apps, though, this may not be an issue. Generally, though, if you find the need to use shared memory between processes, it's because your OS's messaging lacks something.
While Linux has been getting down to millisecond level response, QNX has been moving towards microsecond level response. The goal moves. However, millisecond response is good enough for most applications. What really matters in real-time, though, is worst-case response, not average. Benchmarks for real time OSs put a load on the system and measure how fast it responds to a few million interrupt requests. The number people look at is the longest response time.
Despite this, Linux is probably now good enough for most real-time applications that can tolerate a big kernel.
Re:It's progress, but not everything you need yet (Score:5, Informative)
Linux has made major progress in the real-time area. But it still doesn't have everything needed.
Many drivers are still doing too much work at interrupt level. There are drivers that have been made safe for real time at the millisecond level, but that's not universal. Load a driver with long interrupt lockouts and your system isn't "real time" any more. This is the biggest problem in practice. There are too many drivers still around with long interrupt lockouts.
That's where my -rt patchset (discussed by Thomas in the article), and in particular the CONFIG_PREEMPT_RT kernel feature helps: it makes all such "interrupt lockout" driver code fully preemptible. Fully, totally, completely, 100% preemptible by a higher-priority task. No compromises.
For example: the IDE driver becomes preemptible in its totality. The -rt kernel can (and does) preempt an interrupt handler that is right in the middle of issuing a complex series of IO commands to the IDE chipset, and which under the vanilla kernel would result in an "interrupt lockout" for several hundreds of microseconds (or even for milliseconds).
Another example: the -rt kernel will preempt the keyboard driver right in the middle of sending a set of IO commands to the keyboard controller - at an arbitrary instruction boundary - instead of waiting for the critical section to finish. The kernel will also preempt any network driver (and the TCP/IP stack itself, including softirqs and system-calls), any SCSI or USB driver - no matter how long of an "interrupt lockout" section the vanilla kernel uses.
Is this hard technologically? Yes, it was very hard to pull this off on a general purpose OS like Linux (the -rt kernel still boots a large distro like Fedora without the user noticing anything) - it's the most complex kernel feature i ever worked on. I think the diffstat of patch-2.6.18-rt5 speaks for itself:
How did we achieve it?
The short answer: it's done via dozens of new kernel features which are integrated into the ~1.4MB -rt patchset [redhat.com] :-)
A longer technical answer can be found in Paul McKenney's excellent article on LWN.net [lwn.net].
An even longer answer can be found on Kernel.org's RT Wiki [kernel.org], which is a Wiki created around the -rt patchset.
Parent
complexity of the -rt patchset (Score:3, Informative)
It's painful reading how that works. It's an achievement. "613 files changed." "It's the most complex kernel feature i ever worked on." But it's one of those things that, for legacy reasons, is much more complex and ugly than it should have been.
I think you misunderstood my point. The reason why our patchset is so complex and so large is because we want to do it right. The quick-and-ugly shortcut is alot smaller (and it has been done before), and it brings problems similar to the ones you outlined -
Bad for windows embedded (Score:3, Interesting)
I was at the MEDC 2006 [medc2006.com] and one of the presentations was about how Linux was comparing to Windows Embedded.
We got bunch of benchmarks showing how much more consistent the kernel response time were from Windows CE (custom Kernel) versus the Linux kernel, and how -on average- the Windows kernel had shorter response time. Some people in the audience asked (more than once) if they had benchmark from the Linux kernel with the real-time patches applied ... and the answer was they were benchmarking a 'for desktop' Linux kernel versus a 'for real-time' Windows Kernel. I was expecting M$ to put biased numbers but I could not stand it and left the room after 10 minutes.
So the great thing about this announcement is that next year they won't be able to pull this off. Honestly I would not be surprised to see them use a 2.4 kernel, or show us how the stock Desktop kernel with a crap-load of unnecessary modules take so much more resources than the optimized Windows kernel.
(sorry to post Anonymously, I guess I'm a Coward)Re:Open source at it's finest (Score:4, Informative)
Do you even know what Real-Time computing means? Do you know what an interrupt is? Can you name even a single one RT OS ?
sigh... from wikipedia: http://en.wikipedia.org/wiki/Real-time Computing [wikipedia.org]
Parent
Well, the meaning has changed somewhat (Score:5, Informative)
Typically, it is indeed used in simulations, but very rarely on a 1:1 simulated-time-to-wall-clock-time basis. Simulating the formation of a solar system, or the results of runaway nuclear fission cannot be done on a 1:1 basis. Instead, real-time's guarantees permit you to firmly quantify a constant ratio between wall-clock time and simulated time - provided the programmer can firmly guarantee that a given simulated time interval will never require more than a certain amount of wall-clock time to simulate and a barrier can always be established to prevent more than the desired amount from being simulated. This rules out all event-driven simulations, herustics, variable N-body problems, fixed-precision variable-mesh CFDs, etc.
As soon as the number of variables or steps becomes indeterminate for a given simulated unit of time, you cannot pre-determine the wall-clock time needed to simulate it. If, however, the guarantees are enforced, it is STILL real-time in the accepted sense, even though you have no idea prior to running the program as to when a simulated unit of time will occur. An OS does not stop being real-time if it is fed a non-computable problem, it merely stops being able to tell you when it'll get done.
ADEOS - the microkernel used in RTAI - is "hard real-time", as is VxWorks. TimeSys' Linux patches are soft real-time. Soft real-time is good for audio/video, but is pretty naff when it comes to providing any guarantees. It's usually faster than hard real-time, as it can spend more time processing than synchronizing, so is more useful on a desktop system. With suitable patches (nanosecond timers, real-time clock chip drivers, etc), Linux can be a damn good hard real-time system, but there are performance penalties. Besides, broadcast-quality images are 30 fps in the US and 25 fps in the UK. Being able to align a frame on a nanosecond boundary is thus a complete waste of time. It's damn useful if you're running a particle accelerator ring or even a maglev train (350 mph requires absolute sub-microsecond guarantees on timings), but not many home users have these yet. A pity - it might be fun to have a synchotron source.
Does Linux need hard real-time, then? Oh, certainly. It's popular in the very groups of people who are interested in nuclear science, mass transit or quantum cosmology. I firmly believe that patches that provide such mechanisms are not merely useful but extremely desirable. Bleeding-edge needs drive technology precisely because that is where technology isn't, so keeping in reach of those folk is vital.
Does Linux need real-time in the kernel? Certainly, though the soft real-time that has been added is probably sufficient. (I hate saying things like that. I'm a geek and I'm proud of it, and therefore want Linux to have the maximum flexibility possible. However, poorly maintained code is worse than no code at all, and there just isn't the userbase to keep hard real-time in the kernel at an acceptable standard.) As I said, soft real-time is all you need for multimedia. It's also all you need for games and other stuff that Linux hasn't been nearly hot enough in in the past.
Hard real-time distributions are another matter. For that matter, a distribution that provided a good set of hard real-time tools, clustering tools and CAM drivers would probably be very popular nor only with the scientific community but also with industry. A distro that pushed Linux into markets that couldn't be reached without effort from any mainstream project would undoubtedly be a good thing. It would be very much a minority effort, and yet would require far more sophisticated QA and technical support, so no sane person would attempt such a thing. Of course, I've never been accused of being sane...
Parent
Is the -rt patchset hard-real-time? (Score:3, Informative)
ADEOS - the microkernel used in RTAI - is "hard real-time", as is VxWorks. TimeSys' Linux patches are soft real-time.
Small correction: if by "TimeSys' Linux patches" you mean the -rt patchset [redhat.com] that i'm maintaining (and to which Thomas is a major contributor), and in particular if you mean the CONFIG_PREEMPT_RT kernel feature, then the answer is a clear "no": it's not "soft real-time", it's intended to be "hard real-time" in the same sense as ADEOS/RTAI is.
The -rt patch-set implements a fully preemptib
Hard-real-time in Linux? (Score:4, Informative)
I hate saying things like that. I'm a geek and I'm proud of it, and therefore want Linux to have the maximum flexibility possible. However, poorly maintained code is worse than no code at all, and there just isn't the userbase to keep hard real-time in the kernel at an acceptable standard.
We have good news for the geek in you: the upstream Linux kernel is already more than 50% on the way to become hard-realtime :-)
The 2.6.19-rc2 kernel already includes the following features/subsystems, which are the precondition of hard-realtime (PREEMPT_RT):
Most of these features were written for and prototyped in the -rt tree and were split out and merged individually. (they all have other uses besides serving a hard-real-time kernel, so their merging was largely uncontroversial)
Granted, there's still 1.4+ MB of patches pending in our 2.6.18 based -rt tree [redhat.com] (such as the core bits of PREEMPT_RT, irq threading, high-res timers, dynticks and more), but roughly the same amount of code has been merged upstream already, and we can now see the end of the tunnel.
In fact, i'd say that the most controversial ones are already merged and the flamewars are largely over: such as the generic mutex code (which replaced semaphores half a year ago in 2.6.16/17) or the priority inheritance and rt-mutex code (which is now in 2.6.18).
Parent
glibc support? (Score:3, Interesting)
I have a question about the new mutex features: what glibc version is required to use this stuff? do I need other user space libraries?
The latest upstream glibc version (2.5) has it:
(See: http://sources.redhat.com/ml/libc-alpha/2006-09/ms g00065.html [redhat.com] )
No other userspace library is needed.
Re: (Score:3, Informative)
I'm not sure any normal person will really care or notice the real-time stuff added to Linux. but I'm sure many companies are pleased that it has made
Re: (Score:2)
Make them an official part of the kernel.
I'd ask if you RTFA, but of course it's pointless.
lets try this again... (Score:2)
Re: (Score:2)
We, those who surf the web, play music, write email, host web servers and run databases will see no improvments.
Those who control tiny systems with ARM and MIPS CPUs will find life more consistent, because now there is The Official Real-Time Kernel, instead of a variety of non-similar patch schemes.
Not entirely correct. (Score:4, Informative)
For servers, real-time guarantees each process (or each thread, depending on implementation) a deterministic amount of time, so guarantees that denial-of-service to those who are currently having queries processed is impossible. (You are guaranteed your time on the server, no matter what.) However, because execution time is bounded, it also guarantees that response time can never be below some pre-defined threshold, either. A lightly-loaded machine will provide precisely the same response times as a heavily-loaded one.
For most servers, this is a pretty useless thing to guarantee - well, unless you're about to be Slashdotted. There are some exceptions. A server providing financial information would not really need to be able to respond faster off-peak, but must not fall over under stress. A networked storage device can't afford massive fluctuations in access time if it is to be efficient - mechanical devices don't have a zero seek-time. The better the device can guarantee an operation will take place at a specific time, the better the device can order events to make use of what time it has. Network routers that are hard real-time should never fail or deteriorate as a result of network congestion, it should merely hit the limit of what it can handle and not allocate resources beyond that point. (A router or switch that is not real-time will actually deliver fewer packets, once it starts getting congested, and can fail completely at any point after that.)
Real-time is not going to be generally useful to the majority of individuals the majority of the time, although it will be useful to most individuals some of the time (eg: multimedia), and to some individuals a great deal of the time (eg: those setting up corporate-level carrier-grade VoIP servers, unmanned computer-controlled vehicles, DoS-proof information systems, maglev trains, etc). All classes of user will have some use for real-time. Even batch-processing needs some degree of real-time - jobs have a well-defined start-time and must never be given so much time that the next cycle of the same job is incapable of starting on time. It's not "hard" real-time, but it's still real-time in that it is still a very well-defined set of bounds on when a process is executed and how much time it gets.
"No improvements" is therefore entirely incorrect. No noticeable improvements - maybe, depends on the situation. Some situations will improve significantly, others may actually deteriorate. No overall, on-average improvement would be closer to the truth. As for tiny systems - I run Fedora Core in full desktop mode on a MIPS, RiscOS was designed for the ARM, and I've seen plenty of embedded systems that need real-time on Opterons. For that matter, VxWorks - the "classic" hard real-time OS - is designed around the Motorola 68040 architecture and is often used by military and science labs to handle bloody huge applications (think of something of comparable size and complexity to Windows Vista as it was going to be before all the interesting bits got chopped out).
Also, when you think of "tiny systems", you generally picture something the size of a matchbox that can be powered by a watch battery. In practice, data collection systems (a heavy use of real-time) will usually use VME crates that might easily have 16 CPUs, 128-bit instrumentation busses, a few tens of gigabytes of RAM, a fan that sounds like it was pulled out of a jet fighter, and power requirements that would seriously strain a domestic power grid. The last time I saw a truly small real-time system was at a micromouse tournament, although undoubtedly there are people who use them on tiny systems. The biggest money and the heaviest demands are to be found in areas needing much bigger iron.
Parent
Re: (Score:2)
Re: (Score:2)
BTW, it's a fallacy that "embedded" implies "tiny". Sure, a heart pacemaker is an embedded device, but so is a digital set-top box and so is an MRI scanner. Many modern embedded devices have demand for quite a bit of computing power.