Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 


Forgot your password?
Close
typodupeerror
×
Debian Google Linux IT

The Story Behind Google's In-house Desktop Linux (computerworld.com) 60

Posted by EditorDavid from the I'm-feeling-Linux dept.
"For more than a decade, Google has been baking and eating its own homemade Linux desktop distribution," writes Computerworld.

Long-time Slashdot reader waspleg shared their report: The first version was Goobuntu. (As you'd guess from the name, it was based on Ubuntu.) In 2018, Google moved its in-house Linux desktop from the Goobuntu to a new Linux distro, the Debian-based gLinux. Why? Because, as Google explained, Ubuntu's Long Term Support (LTS) two-year release "meant that we had to upgrade every machine in our fleet of over 100,000 devices before the end-of-life date of the OS."

That was a pain. Add in the time-consuming need to fully customize engineers' PCs, and Google decided that it cost too much. Besides, the "effort to upgrade our Goobuntu fleet usually took the better part of a year. With a two-year support window, there was only one year left until we had to go through the same process all over again for the next LTS. This entire process was a huge stress factor for our team, as we got hundreds of bugs with requests for help for corner cases."

So, when Google had enough of that, it moved to Debian Linux (though not just vanilla Debian). The company created a rolling Debian distribution: GLinux Rolling Debian Testing (Rodete). The idea is that users and developers are best served by giving them the latest updates and patches as they're created and deemed ready for production.
Google's using what appears to be an automated build system (along with virtualized test suites, and eventually "incremental canarying"), the article points out. The end result?

"The entire gLinux development team consists of a single on-duty release engineer position that rotates among team members."
This discussion has been archived. No new comments can be posted.

The Story Behind Google's In-house Desktop Linux More

The Story Behind Google's In-house Desktop Linux

Comments Filter:

  • will have hit gold. With a little care and patience before deploying the payload, it doesn't even sound too complicated, what with the lone Google dude in charge or rolling out the updates.

    • Re: Whoever infiltrates this distro's supply chain (Score:4, Insightful)

      by klipclop ( 6724090 ) on Saturday July 30, 2022 @11:55AM (#62747194)
      Right-o, you'd make a great addition to the average no nothing "security team" at the typical company. i.e try to generate a bunch of work for other groups who're busy adding to the bottom line since you wouldn't need to implement the vague "security issues"

    • It sounds like they have a canary system and automated testing to detect much of that. It might not be perfect, but it doesn't make it any worse than any other distribution.

    • will have hit gold. With a little care and patience before deploying the payload, it doesn't even sound too complicated, what with the lone Google dude in charge or rolling out the updates.

      Hm, it seems your hypothesis (thought experiment) is based off this apparently paradoxical (but I suspect only poorly copy-edited) sentence that appears in both the summary and article:

      Today, thanks to Sieve, the entire gLinux development team consists of a single on-duty release engineer position that rotates among team members.

      If we remove the unnecessary modifiers, the sentence basically asserts: "The entire team consists of a single position that rotates among members of the team."

      So is the team plural

    • Doesn't the military have a system of rotating some thankless responsibility to the "Officer of the Day"?

      Something like Colonel Potter barking at Hawkeye,

      "Pierce, you are now in charge of keeping all the Linux boxes in the 4077 patched and up-to-date."

  • Why would Google have to upgrade their equipment every two years because of the distro? Wouldn't everything still work on the hardware they have?

    Someone fill me in because either I'm missing something completely obvious (wouldn't be the first time) or Google is making more out of this than is necessary.

    • Re:Am I missing something? (Score:4, Informative)

      by snookerdoodle ( 123851 ) on Saturday July 30, 2022 @12:09PM (#62747228)

      "Upgrade every machine" means upgrade the Linux on it, not upgrade the actual hardware. Yes, that perhaps could have been worded more clearly. So, they're upgrading to the supported version to stay in support. (Just keepin' it real here: I keep an unsupported Windows 7 box around because it can still see SMB 1 shares of logs on our way-out-of-support Red Hat servers. Sec Eng folks "see" and and ask about it every few weeks and let me keep it so I can see the logs. The Red Hat boxes are upgrading next month, though and I'll gladly shut it down. Yay!)

      • Ok, thanks, that sounds *somewhat* better. But why worry about the supported version? It still works, right? It's not going to stop working just because it's no longer supported (ala your Windows 7 box). Does it matter if they're behind a year or two on the distro?

        Again, to me, this sounds like Google is making more out of this than is necessary.

      • Re:Am I missing something? (Score:4, Insightful)

        by Entrope ( 68843 ) on Saturday July 30, 2022 @01:05PM (#62747408) Homepage

        I think I'm still missing something. Every Ubuntu LTS release since Ubuntu 12.04 has had five years of free support, with the option to pay for Extended Security Maintenance to extend that by two or five years. (Ubuntu 10.04 had a shorter support period for desktop use and no ESM.) Why did Google think they needed to update every two years rather than every five years?

        • Re:Am I missing something? (Score:4, Insightful)

          by OneOfMany07 ( 4921667 ) on Saturday July 30, 2022 @03:32PM (#62747668)

          I think it's the requirement to upgrade the distro version to get the latest package versions, not that support was removed from the distro after 2 years. Older releases wouldn't push to receive the bleeding edge versions of everything.

        • Re: (Score:2)

          by kenh ( 9056 )

          To upgrade from, say, 18.04 LTS to 22.04 LTS you need to install the interim release (20.04 LTS) first.

          • Re: (Score:2)

            by Entrope ( 68843 )

            That may be, but it seems like they could just do those updates back to back. Sometime after 22.04 LTS comes out, they finalize the update process for both, and jump from 18.04 LTS to 20.04 LTS to 22.04 LTS -- the disruption to a given workstation is minimized, they only have to update every 4 years, and the update process should probably be heavily automated for any enterprise of that scale.

        • Re: (Score:2)

          by ufgrat ( 6245202 )

          Because most LTS distros, while they're up-to-date on security fixes, are woefully behind on features. For instance, 18.04, which is good until April 2023 for support, has php 7.2 by default-- which hit EOL 20 months ago. Even with backports, there is no 7.4 option (which hits EOL in November).

          Yes, there are 3rd party repositories (and I support those for some of my customers), having it in the base OS is rather nice.

          I gave up on fixed version distros some time ago, because the LTS "stable ABI" paradigm w

          • Re: (Score:2)

            by Entrope ( 68843 )

            Isn't that the definition of a traditional OS release, though? You get definite versions of the packages in it. You don't get newer features, but you also don't have to deal with your applications or libraries having breaking changes.

            For my home system, I use Debian testing and an happy with the rolling release model. For systems I use at work, where we deploy an application and expect it to work for years, we use a traditional OS release.

            • Re: (Score:2)

              by ufgrat ( 6245202 )

              And now we have come full circle to Google's justification to moving to their own rolling release of Debian. :)

              • Re: (Score:2)

                by Entrope ( 68843 )

                No, we haven't. We've rehashed the difference between traditional releases and rolling releases. Did Google just not realize what they wanted from a desktop OS for the first however many years? Or did their requirements change sometime during those years?

      • Re: (Score:2)

        by t0y ( 700664 )
        There's a set of windows components I can install on my win11 box called "SMB 1.0/CIFS File Sharing support". You can just install the client, if you wish.

    • Re: (Score:2)

      by Pieroxy ( 222434 )

      Ubuntu LTS support is 5 years, not two. I don't know what this article is about but the whole premise is blatantly false.

      • Re: (Score:2)

        by ufgrat ( 6245202 )

        Because there's a huge difference between "security updates" and "application updates". LTS gets the former, but not the latter.

  • I struggled to understand what the heck the summary was saying, so I actually read the article. My takeaway:

    Sieve sounds like the cat's meow. One program that can automate a rolling distro production pipeline to the point where it takes only one engineer to maintain a desktop used by 100,000+ users? Sign me up!

    Better still, release Sieve's code so we can all start producing rolling Linux desktop releases.

    It is a shame that more of the work of making a distribution work reliably isn't as automated as it cou

    • You can click my name to see what was actually submitted vs what showed up.

      • Re: (Score:2)

        by kenh ( 9056 )

        That's an interesting example of providing a summary...

        • Re: (Score:2)

          by waspleg ( 316038 )

          I wouldn't do it at all if I could just submit stories. Google News has that shit automated - really most people only look at the title anyway.

    • Re:Best part of story (Score:5, Funny)

      by Patrick May ( 305709 ) on Saturday July 30, 2022 @02:06PM (#62747514)

      I struggled to understand what the heck the summary was saying, so I actually read the article.

      That's a good way to get banned here.

    • Re: (Score:2)

      by kenh ( 9056 )

      Microsoft built a testing lab with thousands of computers and after a certain point "ate their own dog food" (running beta OS in production in some limited fashion), just to try and produce a stable release - Linux employs something kinda similar, but it more closely resembles an "infinite number of monkeys with an infinite number of typewriters..." than a professional, automated testing facility.

  • While I totally believe Google has their own distro, the reasoning here doesnâ(TM)t make sese. They switched from Ubuntu LTS because itâ(TM)s support window is only 2 years (itâ(TM)s actually 5), and instead decided to roll their own rolling release, that isnâ(TM)t supported at allâ¦?

  • It exists, it works, they chose Debian because it worked - see also https://meetings-archive.debia... [debian.net] where much is explained.

  • Wasted effort. (Score:5, Interesting)

    by The Evil Atheist ( 2484676 ) on Saturday July 30, 2022 @01:53PM (#62747492)
    This is what you get when you view the principles behind the GPL to be hostile to business.

    Imagine, instead, that they decided to base their rolling distro on another publicly available rolling distro, like Arch (mentioned in the article). They could make both technical and financial contributions to make Arch even more smoother to upgrade (as well as bugfixes, usability improvements etc).

    Other users and even companies can see all the value that's been pumped into the project and will be more inclined to use it, which means it will be tested even more widely, resulting in bug fixes going back Google's way for free.

    On the face of it, companies can "freeload" off of Google's efforts. But Google would be in greater control of the upstream simply by the sheer mass of their own contributions in comparison. Those companies will find themselves having to maintain their own separate distro, with the upstream "breaking" them all the time. So the only sensible option for companies is not to freeload and to contribute back to the upstream.

    You get a virtuous circle of contributions that only improve on each other, creating value for each one involved - far more than what they could ever spend money on their own. Even Google.

    But instead, Google wastes money on maintaining a distro all on its own.

    • I'm guessing this is just they want complete control, and won't promise to match any other system. I doubt they wrote it all from scratch (sorry didn't read TFA yet).

      I get your point of putting that effort into a name that someone else already made, but then you inherit all their bad decisions. Past, present, and future.

    • Re: (Score:2)

      by kenh ( 9056 )

      But instead, Google wastes money on maintaining a distro all on its own.

      Wastes money? How much could it possibly cost them? $20/googler? So what? It's cheaper than a MS software assurance license, and in case you forgot, Google already had an OS group in-house, working on Android...

      • So that's $3mil a year for a rolling Linux desktop distro.

        "Cheap" does NOT negate whether it is WASTED or not. If something is unnecessary, then by DEFINITION it is a waste.

    • Re: (Score:3)

      by stikves ( 127823 )

      Sorry, but these assumptions are way off base.

      A fleet of hundreds of thousands of Linux desktops would probably need some customization. They might want to ensure all updates go though another security review. They might want to have pre-installed in-house applications, or maybe some security settings. Who knows, they might even have a custom wallpaper in the mix.

      And, maybe, just maybe, Google is actually one of the top upstream open source contributors out there: https://www.freecodecamp.org/n... [freecodecamp.org] (interest

      • "Some customization" does NOT need rolling out your own distro. I've worked in companies that customized company stuff on top of Fedora. Hell, in this article, it points out Google used to customize Ubuntu.

        The issue they have was NOT about customization but about, what ended up becoming yet another rolling distro, of which there already exist.

        I could go on, but "it works fine" does NOT negate that it's a "wasted effort".

  • I thought Ubuntu LTS... (Score:3)

    by kenh ( 9056 ) on Saturday July 30, 2022 @05:24PM (#62747836) Homepage Journal

    I thought Ubuntu LTS was supported for five (5) years after release, why the need to replace every install every two years?

    A new LTS version is released every two years. In previous releases, a Long Term Support (LTS) version had three years support on Ubuntu (Desktop) and five years on Ubuntu Server. Starting with Ubuntu 12.04 LTS, both versions received five years support. There is no extra fee for the LTS version; we make our very best work available to everyone on the same free terms. Upgrades to new versions of Ubuntu are and always will be free of charge.

    Source [ubuntu.com]

    • Common platform concerns? Maintaining currency while providing the update benefits? I don't quite understand how they would pull off a "major" upgrade, but I guess they thought of that...

  • Windows banned (Score:5, Informative)

    by Tough Love ( 215404 ) on Saturday July 30, 2022 @06:54PM (#62747974)

    The elephant in this room is that Windows is banned at Google because of the unacceptable attack surface, famously exploited by China.

  • If you know what you are doing, the number of servers you administer does not matter much.

    Congratulations, Google. :)

  • > With a two-year support window, there was only one year left until we had to go through the same process all over again for the next LTS

    They release an LTS every two years, yes, but they're supported for five. You're doing lots of extra work if you're updating every two years.
    • LTS has security fix support for 5 years, but it does lock you to specific package versions (ie, you will only get up to Python version X, git version X, php version X, etc.). Google is needing the latest stable versions of those applications (often Google themselves contributing to them in the first place). So they were doing a fleetwide upgrade, taking almost a year, then having a year before doing the whole thing over again just to have semi modern versions. As others have pointed out, the application ve
      • Saying "we want packages newer than the ones contained within the release" if very different from saying that the release has a "two-year support window." The former is understandable, the latter is simply untrue.

Slashdot Top Deals

Sentient plasmoids are a gas.

Close