Why OpenSolaris Failed To Build a Community

xtaski writes "Ted Ts'o, one of the earliest Linux developers, points out some serious flaws in OpenSolaris. There is a severe lack of developers, for one. Apparently, after 3 years, the OpenSolaris 'developer community' is still struggling to get the proper tools for developers to develop! Ted also points out some other flaws which make it clear just how disconnected the executives at Sun are from what's really going on in their 'open source communities.' He notes, 'It was never ... Sun's intention to try to promote a kernel engineering community, or at least, it was certainly not a high priority for them to do so.'"

mirror

http://tytso.livejournal.com/ [livejournal.com]

Re:mirror

Yeah, sorry about that. Thunk.org is a rather ancient machine (> 5 years old) living in a colo facility, and this is how I figured out I had been slashdotted. (The two uptime commands were about two minutes apart):

    14:21:06 up 121 days, 16:47, 2 users, load average: 40.47, 12.41, 4.55
    14:23:05 up 121 days, 16:49, 2 users, load average: 81.43, 36.97, 14.52

  Fortuantely I'm still mirroring my blog onto my old Livejournal account; please read it there for now! The two articles that you want are this one: What Sun was trying to do with Open Solaris and this one: [livejournal.com]Organic vs. Non-organic Open Source [livejournal.com], if you can't get through to thunk.org.

Re:mirror

Not so ancient Chinese saying: "It is not enough to install wp-cache2 and activate the plugin; you must go to options->wp-cache and press then "enable" button to REALLY enable wp-cache."

Doh!

(Once I actually really enabled wp-cache, my server seems to have been able to keep up, for now...)

For those too lazy too read the article:

The answer is: "They acted like a bunch of dicks."

OSS is a labor of love. You've got to want to work on the project, and you've got to be able to work on the project.

If you put a big chunk of your time into something and get rudely dismissed, then its hardly likely that you'll continue to contribute.

Re:For those too lazy too read the article:

Speaking as someone on the inside -- you're right. There are a lot of big egos here.

I didn't come to Sun because I like the Kool-Aid, I came by acquisition. I haven't decided yet whether or not this whole "we love Open Source" thing Jonathan keeps plugging is real or a charade. I'm optimistic, but we'll see.

On better days, I like to think that the people way up at the helm really "get it" and are just waiting for the rest of the ship to slowly (slowly!) turn. On not-so-good days, I start to wonder if maybe someone's trying to pull a fast one.

There are lots and lots of people here who really and truly believe that Linux is just an upgrade path to Solaris. In other words... Once people start running Linux on Sun hardware, they'll "want more", and "step into the big league" with Solaris. It's kind of sad, when it's not irritating.

Anyhow... I could bitch for a while, but I won't.

Re:For those too lazy too read the article:

The thing is, Andy's [wikipedia.org] back, and as far as I can tell, he doesn't give a rat's ass about Solaris. He just wants to make interesting hardware. That's where the money is, after all. Software is a pathetic fraction of corporate revenues here. All the more reason to be mystified about the internal hostility toward Linux.

So, for example, the Thumper [sun.com] is one of Andy's creations. It's pretty hard to beat the storage density you get for the price. Put a mess of those under a Lustre filesystem, and people start to take notice of Sun as a player in HPC. The recent TACC Ranger [utexas.edu] system is all Sun gear: storage, compute, and network (with sun-built Magnum [sun.com] switches). The OS? Linux.

There's more interesting stuff coming down the pike, and from my perspective, it seems that there's a shift toward making money on volume rather than margins. In other words, somewhat less awesome, but more of it.

I dunno. I don't profess to have much more special knowledge than anyone outside of the upper echelons. I'm hopeful, though. I read somewhere that many of the big Solaris egos were hired away by teh google. Hopefully they keep going. They can have our kool-aid-drunk sales and marketing people too. :P

Re:Not welcoming your Scott McNealy overlord?

Actually, no.

The lesson here is: If you're going to try to court people active in OSS development, then you're going to have to be nice to them, and you're going to have to let them take some ownership.

IBM is being smart; they're reaping rewards far in excess of their investment. Effectively they've outsourced their development, and while the terms of the "outsourcing" say that they have to share everything that comes out of the project, they're still in a position to steer, and support the product.

I'm not sure how you equate that with "control"; sounds just like more FUD to me.

Bureaucracy

I think Sun underestimated the importance of casual users. A lot of times the people choosing an OS for a project (be it enterprise deployment, inclusion with hardware, or just use within IT) go with what they are familiar with and also what their current interests are. When Sun open sourced Solaris, there was a lot of interest from the Linux and BSD communities. A lot of those people decided to download a copy and give it a try. The difficulty these casual users had in grabbing an installable copy and getting it running easily were significant. A lot of people just said, "meh" and moved on. The last time I grabbed a developer preview I still had to fill out a bunch of forms with my personal data then deal with Sun's "download manager" and then spend significant time getting it to install, even within a VM customized to run OpenSolaris in particular. That is still better than it used to be. I only have a success rate of about 50% in getting Solaris to install to date.

For most people I think it is just too much of a hassle and all the developer momentum is on Linux. I guess when Sun thinks about open sourcing Solaris, they see it as a way to try to stop their hardware customers from moving away from Sun, which is fine, but does little to leverage the real benefits of an OSS community such as Linux has been doing for a long time.

Download barriers

Downloading is a royal PITA. The registration is usually a deal-breaker. Almost nothing I've ever run across that's worth anything requires registration for download. However, as a (former) long-time Solaris / SunOS user and major FOSS user, I felt compelled several times to try to circumvent that. But then there's no real way do a network install and othewise week download choice.

That gripe aside, the article is a bit premature. Though time is running out and it could become true if Sun decides to keep downloads off of anonymous FTP, AFS and Bittorrent.

OpenSolaris fails to build community b/c it sucks

You have to have a good product before you can have a community. Linux built its early community based on tinkerers and hackers who found it easy to play with. Early Linux distributions, you may recall, were all inclined to integrate well with DOS. Some of them could even be installed _in_ DOS. You could install Slackware and be up and running with an editor and compiler in half an hour. OpenSolaris doesn't follow this example. Using it is a tremendous pain in the ass. Its installer runs for 2-4 hours on the midrange PCs I've tried to install it upon. Once it's "installed" you still have to grope around trying to find familiar tools, which are maybe under UCB or perhaps under GNU subdirectories. It's hard to download software from the 'net and ./configure it. Hardware support is very thin.

To get a hacker community, you have to offer fun. OpenSolaris is simply not fun. It reminds me of work.

Re:OpenSolaris fails to build community b/c it suc

How are the GNU distros built on the opensolaris kernel though? I'm thinking of Nexenta [nexenta.org] specifically. Seems like it would be the best of both worlds if done right. World class UNIX kernel + world class userland utils. But then if it's just thrown together, it could suck too.

Re:OpenSolaris fails to build community b/c it suc

Nexenta works fantastically - I love it. I would definitely use it for any storage servers, or high availability servers that do your normal Apache/SQL/P* stack.

However, for desktop and non-standard services, it still sucks. If it's not a web server, and it's not a storage server, don't use Nexenta, use Ubuntu Server. Or Debian if you know what you're doing :)

Re:OpenSolaris fails to build community b/c it suc

> So where does OpenSolaris fit in? It seems to be an OS lacking a niche.

The niche for OpenSolaris is the 64way mission-critical server. Unfortunately, even ultimate kernel hacking enthusiasts rarely have one of those at home.

Installation problems as well

Another issue with opensolaris for me was the installation. Being a fairly experienced *nix user, years of sunos, aix, linux, bsd, etc.. under my belt and a fairly competent programmer. I tried quite a few times to install OpenSolaris and there was always some major problem. I never did get a stable system working and finally gave up. That said, this all comes as no surprise to me whatsoever.

Hmmm - Linux Fan Boys speak out....

The OpenSolaris development community is alive and well, vibrant and resourcefull.
There have been a lot of great development work on OpenSolaris in both the x86/x64 and SPARC worlds.

OpenSolaris (much like it's big brother Solaris) does have a list of valid / tested hardware platforms that work out of the box without issue.

If your specific hardware isn't listed and it's fairly well mainstream, document what didn't work, submit it, and it will more than likely get fixed.

I've used OpenSolaris on IBM/Lenovo thinkpads, IBM xServer hardware, SuperMicro / Intel hardware, homebrew systems with rarely an issue.

I've enjoyed the support of the OpenSolaris community as a whole, and found them to be as resourceful as any *inux / bsd community.

It all depends on what you like / want.

For me, gaining the ability to work with Solaris during development cycles to help in some small way guide / assist with the efforts is worthwhile.

Re:Hmmm - Linux Fan Boys speak out....

As it turns out, Linux development community members are critical of competing operating systems. How is this news?

The point of picking and choosing your operating system is so that you can pick the best tool for the job. If that tool is Solaris, then use it. If not, so be it.

Since Sun actively develops Solaris (and thus parts of OpenSolaris), do they really need individual contributors?

What about all of the burned bridges?

It seems Sun is doing everything in its power to alienate a developer community.

-Wouldn't let the opensolaris board call the project opensolaris. Probably a legal quagmire of their own creation. The consequences of that lead to this resignation. http://mail.opensolaris.org/pipermail/ogb-discuss/2008-February/004488.html [opensolaris.org]

-There's this gem, most of which I don't pretend to understand. The punchline is on the bottom. http://cryptnet.net/mirrors/texts/kissedagirl.html [cryptnet.net]

-There's this gem, where even Ian Murdock links in suggesting the difficulty is happening above his level. http://ianskerrett.wordpress.com/2008/02/22/a-solution-for-suns-os-community-problems/#comment-17418 [wordpress.com]

Instant success

Yes, because Linux was such an instant success! Wait...no, it wasn't. Everyone forgets that any community, either real or virtual, takes time to build. I believe that counting OpenSolaris as a failed community is premature, at the least.

of course not

OpenSolaris was an attempt by Sun to throw some sand in the gears of Linux, not to build an open source project. They are doing the same thing with OpenJava.

I mean, who is going to contribute to such a project if (1) Sun engineers keep calling the shots, and (2) anything you contribute needs to be given to Sun so that they can sell it to paying customers?

If Sun were serious about making Solaris and Java open source projects, they'd release them under a single, open source license only. That would probably have to be BSD.

And why not? Solaris was BSD licensed to begin with; it was Sun that made it proprietary.

As a member of the community...

I guess I disagree. I'm on several of the opensolaris mailing lists, and they're ALWAYS busy. And not just with people from Sun, people from all walks of life. To claim that opensolaris has failed is preposterous to me. I guess I don't quite understand what this mans idea of *success* is, but apparently having users and contributers from both sun and the public abroad isn't *success*.

Is his complaint that the majority of code comes directly from Sun? If so... let me just say *DUH*. If you have thousands of PAID programmers writing code, nobody is going to waste their free time re-writing from scratch. On the flip side, there's TONS of public side-projects, I can think of several around zfs like the automatic snapshots. Or maybe that little side project called nexenta.

I think I understand what his issue is... he doesn't even know what the opensolaris community is. By his definition, one distribution of linux is a measure if its success or failure. Last I checked, when we talk about linux, we're encompassing ubuntu, redhat, suse, slackware, etc, etc, etc... Guess what, the same holds true for Opensolaris.

So... basically, it sounds like a linux zealot casting a stone because he's most likely upset that Sun wont' release solaris under the GPL so that linux devs can start ripping code.

Re:As a member of the community...

Please let me make this clear. I was not disparaging Open Solaris as an operating system. And I was quoting Jon Plocher, a Sun Engineer working on Open Solaris, when he admitted that Sun didn't get the community they were hoping for. So it you can call it failure in terms of Sun being to get the results that it had hoped for when it released (most of) Solaris under an Open Source license. Other people who were major Solaris fans, and who were excited with whatever scraps Sun might throw from the table, might be mightly pleased with what Sun did. But nevertheless, it is interesting that Sun hasn't achieved what they hoped to accomplish with Open Solaris after three years.

The reason why I found Jon Plocher's candidate statement for the Open Solaris Governing Board so interesting was that it was first that I had seen someone from inside Sun comment about the what Sun had been hoping to achieve by release Solaris under a Open Source license that didn't appear influenced by Sun's marketing/spin machine. I don't believe Sun's officially stated reasons (that show up on the CEO's blog, for example) because after three years their words have not been matched by their deeds.

So for me, it's more about correcting the marketing spin. If Sun salescritters want to pay analysts to create Total Cost of Ownership white papers which compare the cost of the most expensive get-someone-on-the-phone 24x7 Red Hat support with a support-by-email Solaris support subscription, I might mock their desperation.

Similarly, if Jonathan Schwartz wants to talk about how wonderful it will be that Open Solaris is Open Source, and how they will reap the benefits of having Open Source developers, but three years later still have processes that result in 0.6 patches/day being accepted into Open Solaris, then I think it's only fair that to point out the chasm between his words and his company's actions.

OpenSolaris is a very young opensource initiative

Give it some time - it's still growing, and while there are some adjustments to be made, the situation is far from catastrophic for its stage of development. After all, there's a number of people contributing to it, and hopefully as processes and community contacts improve, the contributors will increase in number. You have to take into consideration that it's a huge chunk of code and some people are still just lurking to find their place under the sun (no pun intended).

OpenSolaris is an interesting operating system, I don't doubt it'll grow in popularity among developers, however slowly. As I said, give it some time, we have only just begun.

Linux Partisan Disparages Non-Linux OS

This is news?

It's Difficult to Download OpenSolaris (rants)

I spent over a week trying to get Open Solaris build 85. Sun just doesn't get the free distribution thing. You have to register and log in to the Sun Download Centre, from where you can download the CD or DVD images. They try to persuade you to get the Sun Download Manager which is some Java app that gives you pause and resume buttons for the download.

I tried 5 or 6 times to download on different days with the download stalling at sometimes as much as 90%. On the 8th day, I got the whole image. So much for their download manager. You just have to overwrite the chunk you have and start again.

After all these years, they still haven't sorted out the auto-layout of the filesystems. There's not enough room partitioned to install their developer tools.

I went to build gcc-4.2.3. That took 5 days and about a day of CPU time. OK it's an ancient 500MHz USIIi that I got for nothing, but...

See, Solaris's /bin/sh is badly broken (archaic) and can't be used to build gcc. So you set CONFIG_SHELL to be ksh. Only the configure scripts in gcc are still broken from gcc-3.1.x days and two of the scripts it generates, bin/as and bin/collect-ld at each stage of the bootstrap are broken because they begin #!ksh instead of #!/usr/bin/ksh or whatever.

When I used to build gcc on Solaris, I just sed'd all the scripts to replace /bin/sh with /bin/bash or whatever.

So, for the casual SPARC/Solaris power-user/Linux developer myself, it's just too darn inconvenient.

And the stuff in /usr/sfw/bin, which is where the "Open Source migration" into Solaris proper was supposed to happen still looks like it did in 2005, 3 years ago.

Solaris has a brilliant kernel. Putting the DVD images on Bittorrent (officially) like OpenOffice.org, would be a great start. There are too many hurdles for the average user to go through who might have been interested in trying it out. I don't have to register to download Slackware, Ubuntu, KNOPPIX, NetBSD etc.

Sort out the default install so that the disk layout is sane and make it trivial to install the GNU toolchain.

But I've been through all this years ago, and it pains me to see that it still hasn't been fixed.

Re:I didnt bother.

Linux just works? Yeah, maybe for a small system running a simple app stack.

I had to setup an oracle cluster: Thanks to Oracle's support policies, we could not use Solaris x86. Nor could we use RHEL5 (no Oracle 9i support), so RHEL4.6 it was. Should be easy, right? Well tested "enterprise" class linux that can do everything the big boys can do.... We took the hardware we were going to use for solaris and switched it to linux. A pair of Sun x4600M2's, 128G of ram, 4 Dual core AMD's. Sun fully supports linux on this box and RedHat lists it on their HCL. Should be easy.

The basic OS install was more or less easy, once we battled through the serial port redirection setup (guess most linux users never used a serial port before. After all, why bother when the box sits under your desk). I stil like serial ports over video for one major reason: issue resolution (when bad things happen, having that panic string saved by a console server can really save the day)

Ok, so the system was kickstarted and now it is time to set it up for use as an oracle DB. This is a production system, and we need lots of space (4TB) and High Availability. This means redundant connections to everything, mirroring and clustering.

Issue #1: multipathing drivers for the SAN. With solaris, you just plug the thing into the san and all of the storage that the host has access to just showed up. Multipathing was instant and I didnt have to do jack. I could see what devices mapped to which physical array with a simple command. I didnt have to guess which array /dev/sde came from vs /dev/sdf. When you have 20 luns mapped to the same host from two different arrays, its kinda important to know which drives come from which array and what the corresponding lun numbers are. That said, most linux admins I've talked to didnt have a clue about what I was talking about since they never had a san.

Issue #2: dynamically add luns: With solaris, you just change the mapping on the array and the host picks it up and auto creates the dev links. That was easy. On Linux? you've got to be kidding me... You get to echo some crazy strings into several spots in /proc and watch the fun start.

issue #3: IP Multipathing. With solaris, dladmin is used to create a bond (if it is going to the same switch and the switch supports bonding) or use the built in ip multipathing to do an active/failover setup if you are going to multiple switches. Very well documented and very easy to do. With linux... yeah, bonding is a fun task. Need to go to multiple switches? no such luck, you are screwed. I eventually used VCS to take over the systems main IP and uses its IPMultipathing agent to do the job for me. VCS on solaris just hands the task off to mpathd since the OS already does it for you.

Issue #4: zones: dont get me started. I dont want to run another entire OS, I just want name space isolation and chroot is so primative it is not even funny. Zones gives me everything I want with minimal overhead. It would have been nice to have since there are a few oracle products that dont play nicely with clusters (*Warehouse Builder*) because they imbed the host name everywhere. We could put it under Xen, but this is an app that moves huge amounts of data around, not exactly a good candidate for virtualization. Zones let us get around Oracle's brain dead use of the hostname, no such luck with linux.

Issue #5: 3rd party drivers vs the new kernel patch. If I install a 3rd party device driver in solaris and upgrade the kernel, I dont have to rebuild/reinstall the driver. Linux (even redhat 4.x with their "back port") forces me to rebuild/reinstall every damn time. Its great if the driver is standard with the kernel, but if you need something outside of that (lsi multipathing drivers to get around #1 and 10G NIC drivers in my case) and you are screwed. No wonder up2date ignores all of the kernel* rpm's by default.

Issue #6: Whats the system doing? Solaris: `mdb -k` and dtrace. Linux: still trying to figure out what mdb -k does, let alone dtrace.

Lets not even get started on the fun with laptops. The Latest solaris builds just work better than the latest linux distros when it comes to things like sound and video. Yeah, solaris can cheat and bundle the nvidia driver, not their fault that linux has license restrictions.

Linux has a handful of apps that make it nicer (acrobat anybody?), but other than that, switching from solaris to linux makes me feel like I'm going back 5-10 years in OS technology.

Sorry, thats just how I feel about it (and in case you've wondered, I've been using linux since '94). Its still good if you just want a single box to run something like apache and mysql. Anything else is just annoying.