Follow Slashdot blog updates by subscribing to our blog RSS feed

 



Forgot your password?
typodupeerror
×
Social Networks Linux

'Uncertainty' Drives LinkedIn To Migrate From CentOS To Azure Linux (theregister.com) 79

The Register's Liam Proven reports: Microsoft's in-house professional networking site is moving to Microsoft's in-house Linux. This could mean that big changes are coming for the former CBL-Mariner distro. Ievgen Priadka's post on the LinkedIn Engineering blog, titled Navigating the transition: adopting Azure Linux as LinkedIn's operating system, is the visible sign of what we suspect has been a massive internal engineering effort. It describes some of the changes needed to migrate what the post calls "most of our fleet" from the end-of-life CentOS 7 to Microsoft Azure Linux -- the distro that grew out of and replaced its previous internal distro, CBL-Mariner.

This is an important stage in a long process. Microsoft acquired LinkedIn way back in 2016. Even so, as recently as the end of last year, we reported that a move to Azure had been abandoned, which came a few months after it laid off almost 700 LinkedIn staff -- the majority in R&D. The blog post is over 3,500 words long, so there's quite a lot to chew on -- and we're certain that this has been passed through and approved by numerous marketing and management people and scoured of any potentially embarrassing admissions. Some interesting nuggets remain, though. We enjoyed the modest comment that: "However, with the shift to CentOS Stream, users felt uncertain about the project's direction and the timeline for updates. This uncertainty created some concerns about the reliability and support of CentOS as an operating system." [...]

There are some interesting technical details in the post too. It seems LinkedIn is running on XFS -- also the RHEL default file system, of course -- with the notable exception of Hadoop, and so the Azure Linux team had to add XFS support. Some CentOS and actual RHEL is still used in there somewhere. That fits perfectly with using any of the RHELatives. However, the post also mentions that the team developed a tool to aid with deploying via MaaS, which it explicitly defines as Metal as a Service. MaaS is a Canonical service, although it does support other distros -- so as well as CentOS, there may have been some Ubuntu in the LinkedIn stack as well. Some details hint at what we suspect were probably major deployment headaches. [...] Some of the other information covers things the teams did not do, which is equally informative. [...]

This discussion has been archived. No new comments can be posted.

'Uncertainty' Drives LinkedIn To Migrate From CentOS To Azure Linux

Comments Filter:
  • by fishfrys ( 720495 ) on Wednesday August 28, 2024 @09:19PM (#64745340)
    And yet, weirdly, the linux guys in your organization come up with a reason to really want to do it every 7 years or so. Honestly, getting acquired by a company that has it's own flavor and a huge fuck off cloud that runs on that flavor is a much better reason than most I've heard.
    • by niftydude ( 1745144 ) on Wednesday August 28, 2024 @10:53PM (#64745460)
      To be fair, CentOS's decision not to provide a migration path from 7 to 8 or even stream was absolutely insane.

      I have no idea why anyone would stick with CentOS now.
      • by Antique Geekmeister ( 740220 ) on Wednesday August 28, 2024 @11:40PM (#64745508)

        Red Hat made a lot of insane decisions since their purchase by IBM. And they've since lost a lot of the sensible people who kept it a stable and attractive operating system.

        • by saloomy ( 2817221 ) on Thursday August 29, 2024 @12:00AM (#64745530)
          We moved from CentOS to Oracle Enterprise Linux. It has been an absolutely seamless transition that was very easy and our tools and scripts worked out of the box. It is free, and you can get support. We did not need it. Some of our MySQL queries sped up on identically configured VMs and some of our PHP workloads slowed down, but all in all with single digit percentages. Nothing catastrophic. It is now the goto in our shop.
          • by DrXym ( 126579 ) on Thursday August 29, 2024 @01:45AM (#64745574)

            Honestly that sounds like taking your balls out of the fire and quenching them in some boiling oil.

            • You will always always always get burned trustng anything Oracle.

              • by dougmc ( 70836 )

                I understand where you're coming with that, but it doesn't really apply to Oracle Linux specifically.

                Oracle Linux is just RHEL repackaged, just like CentOS, RockyLinux and AlmaLinux. They do change it up a bit with their own "unbreakable" kernel, but I don't know that this is really any better than the stock kernel, though I'm pretty sure if I asked they'd tell me a thousand ways that it was.

                In any event, if Oracle Linux changes in some way that you don't like, you just jump ship to the equivalent version

                • by DrXym ( 126579 )

                  Oracle have form for changing the terms from underneath you. Either for the sources or the binaries. Look at how they fucked over Java users between 8 and 11 - runtime binaries that were previously free were now not and it opened up companies to suprise audits by Oracle's inquisitors. It's the reason that many orgs now use Corretto instead which is binaries built from the same sources but not by Oracle.

                  • by dougmc ( 70836 )

                    Yes, but you're missing my point, and that is Oracle Linux is a special case. They *can't* change the sources or the binaries (or the license), because Oracle Linux is just a repackaged RHEL, which is under the GPL.

                    With Java they could, because they wrote it and had all the rights, but not Oracle Linux.

                    All they could do with Oracle Linux is stop supporting it, or change future releases to be something other than a repackaged RHEL -- which anybody else repackaging RHEL could also do, and CentOS actually did

            • by e3m4n ( 947977 )

              And that kids, is how I met your mother.

          • We are moving from CentOS 7 to Alma. Has been tons of problems with incompatibilities, but I'm not part of the work so I'm not familiar with the details (= could be our own engineers).
          • by ls671 ( 1122017 )

            We did not need it. Some of our MySQL queries sped up on identically configured VMs and some of our PHP workloads slowed down

            I still maintain a central beefed up MySQL VM for all php vms using MySQL. It makes it easier to find out which vm does stupid queries :)

            Sometimes, I wonder if it's worth anymore. Any expert opinion around here in a virtualization context where the central database runs on the same bare metal hardware than the vms, that you can use ksm to deduplicate memory and that you can use zfs to deduplicate storage space?

            Any opinion is welcomed in advance.

          • by thegarbz ( 1787294 ) on Thursday August 29, 2024 @06:29AM (#64745820)

            Blink twice is if an Oracle rep is standing behind you.

        • But can anything top the insanity of moving from Redhat to Microsoft?

          • But can anything top the insanity of moving from Redhat to Microsoft?

            Changing from IBM to Microsoft has been a cost-effective move for most of history. It's difficult to say which is worse today, unless it's both of them. I see this as a lateral move, which does make it dumb, but not necessarily insane.

          • It's not insane if you happen to be Microsoft. It's not like your vendor is going to screw you. But it's Microsoft, so who knows?

      • I wouldn't call it insane but short minded greed. Surely they expected some migrations to paid RHEL.Maybe they even got some.

      • by e3m4n ( 947977 )

        To be fair, a company with the net worth of LinkedIn should be paying for their damn RedHat licenses. CentOS was unlicensed redhat thst benefited small companies where a redhat license was encumbering to the business model. Like say running your small 100 or less user phone system. Running your small website, etc. CentOS was 100% binary compatible with RedHat, meaning a paid for version was always an option.

      • I have no idea why anyone would stick with CentOS now.

        CentOS is dead now. The forums were taken down, with all their history.

      • To be fair, CentOS's decision not to provide a migration path from 7 to 8 or even stream was absolutely insane.

        I have no idea why anyone would stick with CentOS now.

        There's Rocky Linux now.

    • There's also a big difference between using a distro maintained by someone else, and being the maintainers of your own distro. Part of the reason is probably CentOS being EOL'd, and another might be the chance to add features they need to integrate with WSL more tightly.

      Plus, if they can offer support subscriptions to their version for half the cost of a Redhat subscription, perhaps they are exploring that as a new revenue stream?

      Linux365 anyone?

    • by sodul ( 833177 ) on Wednesday August 28, 2024 @11:15PM (#64745488) Homepage

      Since they are forced by their owners to use Azure I guess that Azure Linux makes some sense but I've never had a pleasant experience with rpm based distros (RedHat, Fedora, CentOS, Amazon Linux, ... to name a few).

      The Debian based distros have always been a breeze in comparison and even a few years ago we were forced to switch our AWS EKS nodes to Ubuntu because the Amazon Linux nodes had a known kernel bug, fixed upstream for years, that caused our containers to drop network connectivity. That kernel bug had been fixed for years but RH based distros have very old kernels with a bunch of back ports. It mostly works but having to deal with years old bugs or not be able to use features released years earlier is just frustrating.

      If you want RedHat because you need commercial support, pay Canonical, I don't have first hand experience but I heard their support teams actually help. My experience with RedHat support is similar to my experience with Azure support: waste of time.

      • Thank you very much. RedHat was a dog's breakfast from the get go, and IBM put the final touches on it by making sure you can't really use it without "support", i.e. consulting services to get that last 2% to actually work like it is supposed to. It's straight out of the IBM playbook. That ensures it barely works and always needs that sweet sweet recurring revenue stream from technical lock-in and consulting services.

        Problem is Microsoft has too much motivation to strangle Linux to death with its own propri
    • by 1s44c ( 552956 )

      Microsoft might have a cloud but they are not known for quality of that cloud or on any of their products.

      Ubuntu just works. So does Debian. And the new CentOS alternatives Alma and Rocky. There is no reason to go use a Linux distribution from a company that would have to make vacuum cleaners in order to make something that doesn't suck.

      • by Lproven ( 6030 )

        Article author here.

        > There is no reason to go use a Linux distribution from a company that would have to make vacuum cleaners in order to make something that doesn't suck.

        Sure there is. There are several.

        * You are already a customer of that company;

        * You want to minimise your number of suppliers;

        * You have already paid (for support or for hosting or whatever) and you don't want to pay more;

        * You have already got investment in that company and want to keep it

        * Guaranteed compatibility and/or interop

        Those

        • by unrtst ( 777550 )

          IMHO, those are all of the worst excuses to use a specific piece of software. Excuses, not reasons. Dunno how many times a company (usually one I was working for) has purchased some software (or licenses, or got free seats with some other purchase), and that was enough for them to foist it on everyone and have everyone migrate off of the existing, and usually either inhouse developed or open source/free/no license needed implementations, to said software. And we'd often be limited on admin seats then, where

          • In case you missed it, Microsoft owns LinkedIn this is why they have been migrating to Microsoft solutions for the past few years. They migrated part of their infrastructure to Azure but the migration was so bad that they have been allowed the pause that for a while.

            GitHub, also owned by Microsoft, runs part of their infrastructure in Azure now. For example GitHub Actions run on top of Azure Pipelines.
            You can see how well that works for them by looking at their status history.

            https://www.githubstatus.com/h. [githubstatus.com]

            • by unrtst ( 777550 )

              In case you missed it, Microsoft owns LinkedIn this is why they have been migrating to Microsoft solutions for the past few years.

              Oh, I didn't miss it :-)
              THAT sounds like the actual reason. All those other excuses are just that - BS excuses.

              I personally suspect that part of Microsoft cloud growth is boosted by acquiring companies with loads of servers, and force them to use Azure.

              Yup. Acquiring _OR_ using those stupid excuses to convince others to consolidate into their cloud.

              I do think there are plenty of great use cases for the various cloud offerings and services. The one that breaks my brain is moving established, large deployments from data centers one already owns and operates into the cloud. You're going to pay more long term; Might as well invest that into yourself

  • by 93 Escort Wagon ( 326346 ) on Wednesday August 28, 2024 @09:20PM (#64745346)

    But they'd have had a much easier path had they just shifted to AlmaLinux (or Rocky, for that matter).

    • If you're right, I'd say that was a pretty valid reason, because they can control its development. As long as they're using somebody else's Linux, the OS might change in undesirable ways, and MS might not be able to stop it.

    • But they'd have had a much easier path had they just shifted to AlmaLinux (or Rocky, for that matter).

      I came on here to say this. They should have just gone to AlmaLinux.

    • by Lproven ( 6030 )

      [Article author here]

      > a much easier path

      But it's not always about what is _easy_. It's about what is the long-term better course for your company.

      • The long term better course is to use the best product, because otherwise you will be bitten down the road. Fail to plan, plan to fail.

        • Yes but you have to be very careful how you define "best" product. Choosing a product that is somewhat technically superior but controlled by a competitor might not be the best way to avoid a bite down the road. Using your own product gives you more control of your own destiny and ensures that if you really do hit a business critical roadblock, you have a path to getting the support you need. I certainly would have made the same decision if I was migrating a Microsoft project from CentOS
          • Naturally it makes a certain amount of sense for a Microsoft owned company to use the Microsoft product. But that's not for technical reasons, so it will wind up leading to more pain in the future. In fact, Microsoft's special Linux for Azure has caused problems repeatedly, because... Microsoft.

    • by Dan667 ( 564390 )
      you can't embrace, extend, extinguish Alma Linux.
  • Wait! What? (Score:4, Funny)

    by PPH ( 736903 ) on Wednesday August 28, 2024 @09:53PM (#64745386)

    They jumped right over Fear and went straight to Uncertainty? Soon, it will be nothing but Doubt.

  • by wgoodman ( 1109297 ) on Wednesday August 28, 2024 @10:14PM (#64745414)

    I've been saying it for years, but they're slowly moving towards windows being an abomination linux distro.
    sadly that will be the year of the linux desktop. But itll still be called windows on the stat sheet.

    • The main hook of Windows is backwards compatibility. If the next version is just a reskinned Linux, it will have to rely on Wine (and have no advantage over any other distro), or MS will have to create a new proprietary solution that blows Wine away.

      • by Guspaz ( 556486 )

        They won't need Wine. Windows already runs in a VM for most people (Virtualization-Based Security is now the default in Windows), and Linux running in WSL is also doing that as well. Windows is still treated as the primary for all this, but if they decided to shift to Linux being the primary, it would probably just involve a reversal of roles.

        • by gweihir ( 88907 ) on Thursday August 29, 2024 @03:22AM (#64745612)

          Indeed. And there are rather strong indicators MS has lost control of the Windows codebase and can now only do cosmetics for fear of breaking too many things. Hence they need to scrap it and not in the distant future. The other factor is that MS makes most of its money with Azure these days.

          • Scrapping it is difficult. Eventually the technical debt swirling around in the middle of the Pacific needs to be cleared up, so it's time to fix it. But now you get the set of requirements which claim that thou shalt support 16-bit handles, thou shalt not fix this set of bugs because that will break everyone's workarounds, and thou shalt keep the CEOs favorite feature. It's honestly better to start over with a new product.

            I see this problem in my own job. We try to get rid of a feature that we're confid

            • by gweihir ( 88907 )

              Well, as an IT security person, I think scrapping Windows will become necessary and without alternative in the not too distant future. At some point MS will get liability for its constant screw-ups and that will be it for Windows. I do not think Windows can be cleaned up to meet modern standards and the "innovations" MS has done in the last 10 years or so show this pretty clearly.

              That said, you can still run it isolated if you really need to and there are some pretty good emulation layers around.

      • But it also causes slow progress to Windows because it always had to have some long term backwards compatibility. Of course, simultaneously, Windows will dump new stuff just because they were in that mood while keeping really ancient architectures in place just in case someone might want to run an ancient executable. So forget having a reliable long term backup solution because they will toss it out and reinvent it every couple of releases, but the 16-bit real mode support stuck around far long than made

    • This is some weird fantasy that a lot of people have, but it flat out won't happen. The whole point of Windows is perpetual compatibility. If MS made a Linux flavour it would have to work perfectly and performantly with Windows binaries, and windows DLLs, and all the things which makes Windows, Windows. That is just not something that makes sense to work on.

      And this from a company that can't even get partially decent ARM emulator running.

      • Most of Windows in our days is C# / .Net

        Good luck running a game like WarLords on modern Windows.

        The 'hype' of backward compatible is completely overrated.

        You hardly find a software older than ten years that runs on current Windows.

        Mac's or Linux: a completely different story.

        • Warlords was released in 1990, so in all likelihood it's a 16-bit app. If that's the case, more than anything else, would be the primary reason it won't run on modern versions of Windows. The latter are all 64-bit, but support running 32-bit applications through Windows on Windows (WOW) thunking. On a 32-bit OS WOW can thunk down to run 16-bit applications, but the last version of Windows with a 32-bit release was Windows 10 190x (out of support), 1809 (available as LTSC), or Server 2019. The last two a

          • On a quasi-related topic, and yes this is BLATANT shilling, if you have a burning desire to play Windows XP's Pinball on Windows 11, I have a PowerShell script here https://github.com/ElizabethGr... [github.com] that can install it for you. It needs the path to the \i386 folder on the CD or ISO and does the rest. To say this effort is not supported, endorsed, or even mentioned to my employer is a gross understatement. :D
      • by unrtst ( 777550 )

        They were able to switch to the NT kernel. They were able to get Windows on ARM running (even if that limited application compatibility). They long ago ported SQL Server to Linux. Was .Net not meant to replace all the old cruft? (https://xkcd.com/927/) They even have their own Linux distribution, and run Linux for much of their infrastructure.

        I'd call it fear more than a fantasy, but it's certainly within the realm of possibility, and especially so if we look very far out.

      • "The whole point of Windows is perpetual compatibility."

        Then it has no point. Microsoft severely compromised backwards compatibility with Vista and again with 64 bit, and the situation has become only worse since. Even apps which technically work frequently don't quite work correctly in many cases. I remember bringing up Excel 97 on multi monitor Windows 7 and having pull down menus appear on the primary display even when the windows were on the secondary display. Classy!

  • Why "uncertainty"? CentOS 8 is clearly not a successor to CentOS 7, Rocky 8 is. Alma is another one. CentOS = 7 was a bug compatible clone of RHEL. Rocky is the new CentOS (RHEL clone). Maybe the uncertainty is what will IBM and RH do next in their attempts to thwart the RHEL clones? Ultimately, they had to get off the EOL CentOS 7, and probably there was an expectation to consolidate on to Microsoft's own tooling.

    • by Malc ( 1751 )

      Fucking /. silently eating characters and lack of preview on the mobile version. Where I wrote "CentOS = 7", it should be "CentOS <=7".

  • by dhaen ( 892570 )
    It's nice to hear they've got "Uncertainty". Now wait for the "Fear" and "Doubt".
  • Did I wake up in the wrong universe?

  • Sounds more like there's an attempt here to put Azure in the headlines, first for the service they sell, and second to try and reinforce their adoption of linux (the thing that runs the service on their platform).

    I'd be more impressed if they were working closer with the distros to upstream hyper-v - can you move a hyper-v into KVM - genuinely I don't know - but that'd be more sane. Instead hyper-v (AFAIK) runs only with WinNT but the Linux kernel can be a guest.

    • by Lproven ( 6030 )

      [Article author here]

      > there's an attempt here to put Azure in the headlines,

      Nah. Read the whole article.

      They've tried this at least once before, at the end of last year, and very publicly failed.

      This is not about "hey look at us" because they did that before and it was not good. It is more like "hey, we made it work, listen, here's how".

  • Cool to see it on the front page of /. Thanks!

  • by Blymie ( 231220 ) on Thursday August 29, 2024 @06:10AM (#64745790)

    So LinkedIn (a Microsoft owned company), finds Azure (a Microsoft company), has a more stable Linux?

    This is just a weird ad. They're only switching because of corporate policy.

  • There are plenty of good reasons for a Microsoft division to adopt Microsoft linux. It would be really stupid if they didn't. XFS as an example might have kept other potential customers away.

    RHEL has plenty of problems (plus racial-discrimination lawsuits) and the known problems are not new. But to hear a Microsoft blog complain about IBM's competitor product
    while promoting their own is nothing to be taken seriously especially since they uaed the U in FUD in their lede.

    Oh, and the free version of the comp

    • by unrtst ( 777550 )

      ... XFS as an example might have kept other potential customers away.

      I don't get that part at all. I've run XFS for ages on multiple distros, and moved drives and volumes between distros, and have never had a single problem with that. They're already changing out the OS, so the OS disk can move to whatever FS (ext4, bcachefs, btrfs, xfs, etc..). OS taken care of, they can just modprobe in XFS support (assuming it's not built into the kernel already) and mount the XFS stuff. (FWIW, no - I didn't RTFA)

      Closest I had to a problem like that was with ReiserFS (way back in the day

  • One of the most eye-opening factoids, at least for me, is that Azure Linux uses signed device drivers and as such can't or won't use kernel modules.

  • Why is Microsoft dogfooding such big news? If anything I'm surprised they aren't doing another hotmail... It must be on windows server, even if that means we need to use 5x the compute resource to achieve a significant performance hit.
  • by rwrife ( 712064 ) on Thursday August 29, 2024 @12:59PM (#64746708) Homepage
    I work for Microsoft and last year we were forced to migrate everything to Az Linux, which I would functionally put somewhere between Alpine and RHEL...in most cases is a minor switch, but there were several instances where it's either overkill or doesn't have enough support to do the workloads needed.
  • While their and EPEL packages tend to be crustier than Debian, Ubuntu, Arch, etc., the solid kernel stability of RHEL/Cent is the winning part. You can build around old cruft by rolling newer RPMs or using nix or habitat. The same (stability) can't be said for Azure Linux; clearly, the Microsoft Linux group is using it as a job security program. RHEL/Cent has been beaten on at-scale for 10's of gigahours of uptime. Very few OSes come close, and change is risky and expensive.

The gent who wakes up and finds himself a success hasn't been asleep.

Working...