Microsoft Developer Explains Why Windows Kernel Development Falls Behind 347
New submitter mha writes "In a response that truly seems to be from a core Microsoft developer, we are told about why Windows kernel development continues to fall further and further behind that of the Linux kernel. He says, 'The cause of the problem is social. There's almost none of the improvement for its own sake, for the sake of glory, that you see in the Linux world. ... There's no formal or informal program of systemic performance improvement. We started caring about security because pre-SP3 Windows XP was an existential threat to the business. Our low performance is not an existential threat to the business. See, component owners are generally openly hostile to outside patches: if you're a dev, accepting an outside patch makes your lead angry (due to the need to maintain this patch and to justify in in shiproom the unplanned design change), makes test angry (because test is on the hook for making sure the change doesn't break anything, and you just made work for them), and PM is angry (due to the schedule implications of code churn). There's just no incentive to accept changes from outside your own team. You can always find a reason to say "no," and you have very little incentive to say "yes."'"
I'm sure this is on the money, but (Score:5, Interesting)
These NIH type problems are hardly unique to Microsoft, or even proprietary software. It's human nature. Big success contains the seeds of its own destruction. Open source has the forking mechanism which provides an outlet against some of the worst abuses (only).
I regretted submitting this story immediately. (Score:5, Interesting)
I submitted this story. I am only human - what was I thinking? I guess I thought of the many strange comments I could elicit...
I am so sorry, guys. I must say that shortly after reading the story reason set in (but I was too quick on /.) - there is nothing unexpected in it. It is no big deal. It is a non-story. Everything described is not "Microsoft", it is human, including the complaints. I don't think the points are invalid, it's just that one can make a long list like this for ANY large (or even medium) project. Life is messy - but I got my first story submitted (which means nothing).
My apologies.
I just hope that the guys managers, should they find out, react maturely - by doing exactly nothing (at least no punishment). Stuff like this happens, and if it does so only once it should be overlooked.
PS: On the other hand, enough people voted this to the front page...
Keeping people. (Score:1, Interesting)
Another reason for the quality gap is that that we've been having trouble keeping talented people. Google and other large Seattle-area companies keep poaching our best, most experienced developers, ...
Well, what do you expect? Microsofties come to Slashdot, get picked on, insulted, called evil, etc ... then they have to go home and cry themselves to sleep. The days of the Microsofty crying his way to the bank to see his millions are gone. Now, he cries in his bed, over the insults on Slashdot and the insults spewed at him during his performance review.
Now, some of those poor poor basterds are at Google and Google is now Evil and they'll have to put up with the abuse AGAIN!
On another note:
We just can't be fucked to implement C11 support, and variadic templates were just too hard to implement in a year. (But ohmygosh we turned "^" into a reference-counted pointer operator. Oh, and what's a reference cycle?)
Dude, I feel for the compiler team. MS' programmer tools team is the BEST on the planet bar none. YOU guys should walk around with your heads held high and your pants down do I can kiss your asses! And as far as anything regarding C++ is concerned, well the creator of that language - He who shall not be named - should have stopped in 1998 with adding features to that language - maybe even in '90. He made 'C' OOP - awesome!! Then he went crazy with the "features".
As it is now, I won't touch C++ for new development. ASNI C for system/metal work or anything that needs high performance and Python for most of everything else. GUI work depends on the platform - Visual C# rocks for Windows dev!!
Don't get me going on a rant about Java's current state of stinky.
Re:I regretted submitting this story immediately. (Score:5, Interesting)
Re:And the retraction (Score:2, Interesting)
Sure, people vent about frustrations at work. But you do that privately with your friends, family or select coworkers. You don't post something like that about your company on the web, embarrassing the hand that feeds you in front of the whole world. I am sure he did not think it would end up on Slashdot, but who's fault is that except his own?
If I was his manager and knew who it was, I would fire him immediately. Otherwise I would be risking him "venting" again in the future and embarrassing me even further. He is probably in violation of his employment agreements, so legal action might also be warranted. If his criticism are valid, sure, I would take a look at how to improve them, but still fire him for making them public.
His retraction was too little too late, the cat was already out of the bag.
Re:NTFS (Score:5, Interesting)
"Oh god, the NTFS code is a purple opium-fueled Victorian horror novel [...]" -- lol!
Wouldn't that make it on par with XFS and ZFS? Modern filesystems have their advanced features by breaking the traditional layers, which makes them much harder to organize, and makes it seems like they have dirty tentacles branching out into everything else.
Re:And the retraction (Score:5, Interesting)
Why not? "The hand that feeds you"? What kind of corporate-slave joke world do you live in? The company owes him for his services just as much as he owes them for his salary.
If you fired somebody who is allegedly one of the only good engineers in the organization what value would you have brought to the table? You can fire people who make you look bad? Why is it about you in the first place? It seems his complaints about Microsoft target the kind of attitude that you yourself have - that politics, punishment, and "managing up" matter more than real engineering work.
If I was his manager I'd ask him to post the retraction a bit more publicly since its been buried under the initial criticism, but then I'd try and carve out areas where the barriers he described could be broken down and improvement could be made. I'd also reward incremental improvement and argue for my colleagues and managers to as well - whether that be a fool's errand or not.
Re:And the retraction (Score:5, Interesting)
No, I'd promote that guy (maybe put him in a position to have as many as four people working right underneath him). He cares enough to bitch.
Windows has no future (Score:0, Interesting)
Most of you have no idea where performance PC-like architecture is heading, but if you did you'd comprehend how important articles like this are in helping to understand why Windows is doomed.
Consider the PC from its origins as an IBM design to the (near) present. The focus has been on what is crudely known today as SINGLE-THREADED performance. This is a monumentally dumb programming approach that means at any one time, only one code 'branch' is doing heavy continuous work that uses the significant resources of your PC. Other threads are given periodic, massively inefficient, access to the CPU cores by the dreadful Microsoft scheduling system, to keep the background tasks functioning.
This lousy approach to coding and computer architecture has suited both Intel and Microsoft. It gives us computers that in 99.99% of cases cease to get any benefit beyond 2.5 cores (the 0.5 comes from the scaling efficiency- obviously you don't get half cores). As a result of this, we are still using an extraordinarily primitive 2x64 bit memory bus on all consumer grade PCs. The Microsoft OS and programming model simply doesn't show improvement with modern RAM systems.
Now the Sony PS4 is arriving with an incredible, futuristic, PC architecture- an architecture that ONLY makes sense because it neither runs Microsoft Windows, nor uses an Intel approach to interconnecting memory buses. The PS4 has a cluster of EIGHT CPUs, a massive number of GPU clusters that can be used for streamed GENERAL PURPOSE calculations, and a HSA memory architecture that means the SINGLE (yes, the PS4 has all of its computing elements on one chip) chip feeds an integrated memory space via a massive 256-bit bus (4x64) connected to incredibly fast GDDR5 RAM chips.
The PS4 is designed to run dozens scaling to hundreds of powerful worker-threads, each doing the same real work as that single thread on an ordinary PC computer. If Windows were used on the PS4, the mere overhead of handling so many worker-threads would reach > 100%, meaning that the time managing the threads would be greater than the time spent by the threads actually doing work.
Put simply, future computer designs MUST be coded "close-to-the-metal". An OS that tries to guess what resources a thread is going to need, and for how long, is doomed to complete and total failure. The application itself must be responsible for ALL SEMANTIC signalling. The OS must be prepared to allow the application to define the efficient use of the resources of the computer.
On Windows, the OS is always trying to guess the intent of a thread. Applications CAN crudely attempt to reserve system resources like 'memory', but this idea is confounded by the scheduler which crucifies memory efficiency by having no explicit thread-control of the cache-system of the CPU. The scheduler tries to GUESS which threads are going to keep appearing on the scheduler list in order to keep their cache use coherent. The best a "to the metal" approach can hope to achieve on the PC is one heavy thread per CPU core, and one-to-two threads to the GPU. Anything more and the "clashing" and "thrashing" will eat all theoretical improvements of more simultaneous heavy threads.
Radical changes to the PC architecture are coming- not from the hopeless Intel, but from AMD. AMD plans PS4 like parts for future general and server PC use. AMD's Kaveri part, later this year, is a 6-core CPU cluster and powerful set of GPU clusters on the same chip, with a shared memory architecture drawing from a GDDR5 memory bus. Unfortunately, Windows will take no advantage of the architectural improvements, so the part will show little general improvement to the ordinary PC user.
Ordinary PC users won't begin to fret UNTIL it becomes increasing clear that AAA games titles from the PS4 cannot be efficiently ported to Windows. The best PS4 coding practices of mid-2014 onwards will not be possible on any Windows PC. The Windows versions of such games will thus need to be simpler cruder implementations with many features missing
Re:Poor Management (Score:4, Interesting)
Actually, to do this would demonstrate very poor management skills - a good manager doesn't just tell people to do things while leaving a broken system in place.
A good manager would modify the work environment in order to incentivise the staff to act in a way which is more in line with the business goals (advancing the kernel) - then they would explain these changes and why they were making them to all involved.
Shouting "DO YOUR JOB" at people has a curiously poor track record for making people, y'know, actually do their job.
Re:NTFS (Score:4, Interesting)
The ZFS code is actually very readable and well organized. The choice of an alternate layering model is logical and useful [oracle.com]. That said, I would not be surprised if the NTFS source is truly a horror to behold.
Re:And the retraction (Score:3, Interesting)
What legal action would that be? Most likely he's an American. Criticism in the public interest is a first amendment right. Rights are not subject to contracts. In a legal action you would get mauled At this point Microsoft would be engaging in a policy of intimidation to cause employees to fail to disclose information in the public interest, their liability would be staggering.
Fire -- don't forget whistleblower laws apply here too.
Your reaction is why companies have HR. Because his little criticism is a bit of fun on slashdot and not grounds for a justice department inquest into officially sanctioned misconduct against the public interest.
Sounds like a classic IT department (Score:5, Interesting)
Generally IT people are operating under a a system where they are brutally punished if things go wrong, are vaguely rewarded if they do what someone wants, and not rewarded for doing things that people don't understand (like simplifying the usage of VPNs). So these IT departments see any change requests as increasing the possibility of disaster and thus bad. This results in a combination of refusing to adapt to the company's needs as both dictated through employe requests and through changing technology. This is evidenced through many larger older organizations still running a bunch of SUN servers or a Novell network.
But it is often far more vicious where you have IT people actively reaching out into the company and telling them what technology they may use and how they might use it. One advantage of the iPhone over the Blackberry was that generally iPhones were impossible to ruin through "Corporate Policy" and BlackBerries could be completely neutered through an easy to use interface. But out of control IT people need not fear for long as horrible companies came along to give them the tools to mangle even the iPhones.
IT people might blah blah about corporate security and various data management laws but the simple fact is that if companies don't exist for the sake of their IT departments. IT is a tool that most companies use to achieve their core goal. Yet you have IT departments treating say the head of marketing of a $20 billion dollar company like an infant "for his own good". Where I find it interesting is when IT meets the President or the CEO. Often the president will say something like "I don't want to change my password every 30 days" The IT people don't dare pull the "corporate policy" card but resort to whining about the rational with the CEO concluding, "I'm going to change my password at the exact same frequency that I change the head of IT. So set things up accordingly."
Again this is not all because IT is filled with evil trolls but because their rewards are structured incorrectly. The best run companies that I have ever seen structured IT really well so that when some guy comes in with his Vic-20 and wanted to use it for presentations they either showed him how bad an idea it was or made it happen but then billed his department for the effort. Saying NO just wasn't something they were insented to do. The result was the more stupid the requests from various departments the more budget that went to IT. This way you don't cut ITs budget you told the various department heads to be less stupid with their money.
Back to Microsoft. It sound like MS has created a similar case of fiefdoms that have perverse incentives that are not aligned with the basic goals of the company. I know in the old days of MS they would hand out stock options like candy. This resulted in many people becoming insanely rich. Maybe they need to go back to that same structure. If a small department does something extraordinary they get some big bucks. This would have to be carefully managed as I can see a few superstar programmers doing the heroic only to watch their manager pull up in a new Porsche on Monday and for them to quit on Tuesday.
Re:here's some incentive (Score:5, Interesting)
Instead of everyone getting upset because they have work to do while making adjustments to new changes, how about you just do your damn job and maybe things will get done faster, with better quality. It's not a war, it's software development. If you want to stay relevant, you will do everything in your power to understand this and become better at what you do. In the case you don't want a job, keep getting "pissed" every time changes come down the pike. Consumers don't care about your personal struggle with adapting to change. This isn't a daycare, it's business.
It has been said that the job of a good software manager is less about getting the developers motivated than it is about keeping them going while not de-motivating them. Geeks have a tendency to drive themselves, and often drive themselves harder than they can be driven by others. The challenge is less in getting them to work than in getting them to work on the right things.
I worked once in a company which practised Management By Intimidation, and swore afterwards that no amount of money would ever persuade me to work for another company like that ever again. Pushing phrases like "if you want to stay relevant", "do everything in your power", and "this isn't a daycare" will have me heading for the exit faster than you can scream "You're fired!".
Ultimately, I'm in the profession for 2 reasons. 1: I enjoy what I do. 2: I'm pretty good at it. I'm not really in it for the money. I could make a lot more doing other things in other places, but I like what I'm doing now and it suffices. In that, I think I'm a lot like most of the Linux developers. They have their own agendas, and while Linus may not be the ultimate diplomat, he's a leader, not a manager.
Microsoft, conversely, is a lot about driving the developers rather than persuading them, and if I was to be really cynical, I'd even suggest that their marketing-driven agendas passed on to low-cost developers has a lot to do with their current woes. The Linux developers are often unpaid, but there's never been a "Slaves of the Penguin" book to match "MicroSerfs" and the thought of Linus telling his minions that "if they want to stay relevant..." strikes me as outright comical.
Not sure about this (Score:4, Interesting)
Re:NTFS (Score:4, Interesting)
They are documented, they do change at a pace that's much less insane than Windows'.
Most of the driver APIs are the same as they were back in the NT days, just with some extra driver signing requirements because developers proved to be untrustworthy and incompetent. The only API that does change fairly quickly is Direct X, and that is mostly due to demand from GPU manufacturers who want support for new features and improved performance, as well as game developers.
If Linux fails to keep up with the pace of change in the graphics/GPU computing world it will never be able to rival Windows as a gaming platform.