Forgot your password?
typodupeerror
Google Network IT Technology

The Cult of DevOps 114

Posted by Unknown Lamer
from the but-risky-deployments-are-exciting dept.
packetrat writes "I was at OmniTI's Surge conference today, which turned out to be, among other things, a meeting of the cult of DevOps. Ars Technica covered the keynote and some of the presentations, but some of the best stuff is in the comments. Google CIO Ben Fried told the tale of a really poorly engineered trading application at Morgan Stanley that he was associated with, and how the way IT was structured there contributed to that engineering and to its spectacular failure, costing the bank untold millions in stock trade processing fees from its institutional customers. He said what he learned from cleaning up the mess has informed how Google runs its IT operations, and a culture that promotes generalist skills. A lot of how he describes Google's approach sounds like the DevOps kool-aid a lot of the other speakers were serving, but it also sounds like common sense — are most IT organizations really that poorly run that developers are totally unaware their software is sending messages that are generating network storms, or network engineers are clueless enough about QoS to route leased lines into their data center through their public-facing Internet?"
This discussion has been archived. No new comments can be posted.

The Cult of DevOps

Comments Filter:
  • Cult of DevOps? (Score:4, Insightful)

    by merlinokos (892352) on Saturday October 01, 2011 @08:47AM (#37576182)
    I'm not sure I can take anybody who calls an attempt to make IT and Development more aware of each other a cult, seriously.
    The traditional way of doing things didn't work for 30 years. Why is it that when people are trying to make (and apparently making) a difference to how companies work, they're regularly denigrated by a large subset of the very people whose working lives they're trying to improve.
    Haters gonna hate, I guess.
    • by Lennie (16154)

      I read the article, I think the title is basically talking about:

      'he thinks Google gets this right. “We go to great lengths to hire people with engineering skills, put engineers in operational roles and give them power and accountability."'

      • By scripts.

        Add to that that devOps is much, much harder than system engineering. A DevOps engineer needs to know enough about multiple programming languages *and* algorithmics *and* data and telecom infrastructure to tweak the programs that are really controlling the network. There aren't many people like this at the moment (so if you want to move as a programmer to a $200k/year job, understanding the packet datapath and it's controlling plane is where it's at, provided you're good at optimizing algorithms.

        • by Lennie (16154)

          I don't want to be an asshole.

          But if you don't adapt, you might loose your job. Especially in IT.

          Is that something people are surprised about ?

          • Lots of people are going to lose their jobs, because only the better ones will be retained. Managing systems manually is a skill that will go the way of car building skills. That's a lot of people.

            • And IT is just about the only sector that is supposed to be adding jobs to the economy. It will not go down smoothly once this starts happening.

            • by Lennie (16154)

              It could also be that we end up being a lot more effective. That is after all the whole point of IT. I've never made much sence to me why people did the 'manual labor' in IT.

              But if I look at education in my home country: the Netherlands in Europe, I can tell you there aren't even enough people studing programming to fulfill the available positions (well, I don't know the real numbers, but that is what they keep telling us in the local IT-press and looking at the people that get hired I'm not all that surpri

    • I don't think DevOps is new or non-traditional. The *label* might have been invented recently but it's not like Dev and Ops never worked together before. Everyone should know something about everyone else's job, or maybe (from your post)IT and Development should maintain an awareness of each others' status. I've seen a bunch, they all pretty much mean the same thing, and they're all one sentence.
      The people who can take one sentence and turn it into this [wikipedia.org] might be a cult.
      • by cduffy (652)

        I don't think DevOps is new or non-traditional. The *label* might have been invented recently but it's not like Dev and Ops never worked together before. Everyone should know something about everyone else's job, or maybe (from your post)IT and Development should maintain an awareness of each others' status. I've seen a bunch, they all pretty much mean the same thing, and they're all one sentence.

        Huh?

        DevOps isn't about mere communication or awareness -- it's about operations being automated via tools maintai

        • it's about operations being automated via tools maintained by development, using traditional development methodologies.
          That's pretty much how it is where I work, it's been that way for as long as anyone remembers (at least 15 years). But no one ever called it DevOps.
          In any case, however you define it, it's a concept and as such it can be expressed in a sentence. Most of us are more than capable of extrapolating the implications. Well, at least those of us that understand all seven sigmas. ;)
          My view i
          • by cduffy (652)

            it's about operations being automated via tools maintained by development, using traditional development methodologies.
            That's pretty much how it is where I work, it's been that way for as long as anyone remembers (at least 15 years). But no one ever called it DevOps.

            15 years is quite some time -- by any chance were y'all an early adopter of CFEngine? That's the only toolkit I can think of under an OSS license for doing OS-level configuration programatically that was available at that time, unless y'all wer

            • by any chance were y'all an early adopter of CFEngine?

              All in-house tools back then (though I wasn't there). Well, still actually. There are hallway conversations about chef and puppet. Some very well thought out scripts and rsync still work so there isn't a very strong motivation to experiment. A few people want to be able to put Puppet on their resume so it ( or something else ) will probably show up somewhere sooner or later.

              I have to agree that labels are useful, but I also think this devops thing

              • by cduffy (652)

                A few people want to be able to put Puppet on their resume so it (or something else) will probably show up somewhere sooner or later.

                Personally, I can't recommend Puppet -- it makes simple things simple, but when you get to the point where you want to do hard things, you end up needing to make use of integration points that aren't so well-thought-through... whereas Chef has extra primitives ("data bags", search, &c) that are well-integrated and more power in the language (real Ruby, as opposed to a limi

      • by Jawnn (445279)
        I agree, to a point. Sometimes that point is forced by the size of an enterprise, as in - eventually, you just can't afford to waste developers' time on teaching them admin skills. By that time, operations leaders should have forged two things: 1) A set of policies and procedures that will define "the way things work", and 2) A good working relationship with development leadership so that each understands and respects the other's challenges. Development will know that they can't roll out something new witho
        • "eventually, you just can't afford to waste developers' time on teaching them admin skills"

          OK, maybe you can't afford it. But then you will be forced to afford the alternative -and it's not going to be any cheaper.

    • Re:Cult of DevOps? (Score:5, Insightful)

      by wisty (1335733) on Saturday October 01, 2011 @10:31AM (#37576704)

      I'm guessing the main haters are sysadmins, who see threats to their importance and way of working.

      It's no longer desirable to have a gradually grown system. Everything should be built on virtual machines from scripts. You no longer need a wise old guy with a beard to change anything - who cares if your modification might trash the system? You can spin up a test instance, and dump it if it doesn't work right. If you have two apps that don't play nice with each-other, just put them on difference machines. The philosophy of a DevOps guy is very different to that of a sysadmin - you don't do brain surgery on a horse while it's in the middle of a race. Just shoot the damn thing, and send in a healthy clone. (Or would you prefer a car analogy?)

      There was a world of difference between "works on the dev box", and "works in the production environment, without screwing up anything else". That gap is getting bridged by DevOps practices. OK, you still need people with sysadmin skills, but the way they work (and the things they have to learn) is becoming a lot more like development than sysadmin work, and sysadmins are not bred to like change.

      • by Anonymous Coward

        This guy gets it. How many manhours - manmonths even? - have I wasted at previous jobs trying to revive a system or piecemeal fix a misconfigured or broken system? How many other companies have done the same? It's just not worth it - we have the technology to spin up a new VM instance to replace it... and while that seems obvious, the implications on how your build process ought to be created and the benefits you can get from it are pretty significant.

        • Re:Cult of DevOps? (Score:5, Interesting)

          by gmack (197796) <.gmack. .at. .innerfire.net.> on Saturday October 01, 2011 @11:51AM (#37577298) Homepage Journal

          And how do you know the new VM won't have the same problem? If you never know what went wrong and fix the actual problem you will just end up restarting VMs constantly.

          • by tirerim (1108567)
            You don't, but that's okay -- spinning up a new VM is just one thing to try. If the problem doesn't reoccur, then you've saved yourself a lot of time and hassle trying to figure it out. If it does, then it can still help you eliminate some possible reasons. If you're an idiot, yes, you'll just keep creating new VMs and not solving the problem. But you're not an idiot, right?
            • by gmack (197796)

              Sorry but no. What you are suggesting is a more modern "reboot the server and see if that fixes it" and it solves nothing. Somewhere the bug is lurking and waiting to rear it's ugly head again. It could happen in a week or it could happen tomorrow but it's very likely it will happen.

              • by Lisias (447563)

                Sorry, but no. What the parent is saying is something like

                "reboot it once and let's see what happens. If the shit hits the fun again, clone the damn thing, reboot it again to keep the service alive and try to figure out what's happening on the clone."

                There is no point, for the patient, to have his disease figured out after his death.

                • by gmack (197796)

                  Better idea: bring in backup server/VM or whatever then figure out what went wrong on the original. If it happened once than it will happen again given the same config. You might have some bad hardware, or you could have a bug that only happens under select conditions.

                  Case in point: the sasl software most people use with Postfix has a memory leak when the wrong password is entered (yes they know, no they don't actually care) Some would say a memory leak isn't a problem because it would take thousands of

              • by slifox (605302) *
                You hit it right on.

                I think many admins are ok with the "reboot fix" technique, because they don't work in high-stakes environments. If Bob in Accounting's windows desktop goes down twice this week due to the same issue, who cares (just reboot it again). The same goes with most non-mission-critical services (and those without strict SLA). The same can sometimes also apply in vastly-redundant heterogeneous environments, where having one application instance go down is not a big deal since you have 1000s mo
            • Us old-timers call that "shotgun debugging"

              • by Xeleema (453073)
                "Shotgun Debugging" indeed. However, some multi-platform greybeards call it "Being a Windows Admin".
          • by cduffy (652)

            And how do you know the new VM won't have the same problem? If you never know what went wrong and fix the actual problem you will just end up restarting VMs constantly.

            Sure -- but that's a thing to be replicated and fixed as a development task, not a critical-production-maintenance one. Not like you can't keep a copy of the bad VM around for inspection.

            More to the point, if you're Doing It Right, the "spin up a new VM" process involves applying configuration-as-code, not replicating off some handcrafted mys

            • by gmack (197796)

              On a properly deployed system you should expect config problems for the first couple of days after you deploy something and someone does something you didn't expect even if the same config worked in testing and after that the bugs get into the "one off" variety. In either case you want to find out what went wrong.

              It's not the "load a VM to get the service working again" that I object to. I object to the idea that you can just reload a VM and never debug.

        • by Vanders (110092)
          If your server is misconfigured then you've already failed at DevOps. Your configuration management platform (I.e. Puppet, Chef etc.) should be keeping the server in a known state. If it's not then you need to fix your manifests/Cookbooks. Redeploying a new VM with the same incomplete configuration wont help.

          If you don't have configuration management then you're barely even doing Ops, let alone DevOps.
      • by mounthood (993037)

        I'm guessing the main haters are sysadmins, who see threats to their importance and way of working. ... Everything should be built on virtual machines from scripts.

        The sysadmins like virtual machines; they make life easier. What they don't like is developers telling them how to do their job. Internal software doesn't come with instructions, it comes with a coworker making demands about server resources, DNS/firewall/network configuration, licensing and backup requirements, etc... Not all those developer requests will be reasonable or correct; devs make mistakes too.

        • by cduffy (652)

          The sysadmins like virtual machines; they make life easier. What they don't like is developers telling them how to do their job. Internal software doesn't come with instructions, it comes with a coworker making demands about server resources, DNS/firewall/network configuration, licensing and backup requirements, etc... Not all those developer requests will be reasonable or correct; devs make mistakes too.

          Thing is, that's not how it works in a mature devops shop.

          Development writes the code that set up the DN

          • by segedunum (883035)

            Development writes the code that set up the DNS, the firewalls, even the kickstart files (or whatever your local equivalent is) that control OS installation. If development's code doesn't work, developers' dev VMs and QA's environments break first, because they're configured just the same way.

            Development does that? So developers know all about DNS, firewalls and OS installation? Errrr, no. They think they do and they think it's easy, but they simply don't. Those that do get burned. Badly.

            Backup? Development writes the code that automates it.

            Backup as well? How fabulous. In reality it's sys admins who should be encroaching on developer turf, not the other way around, and the reason why this Devops cargo cult crap has come into existence is because developers now need to know about it, they don't want to and as such they want to pretend it's somethi

            • by cduffy (652)

              Development does that? So developers know all about DNS, firewalls and OS installation? Errrr, no. They think they do and they think it's easy, but they simply don't. Those that do get burned. Badly.

              And that's why you have a crossdiciplinary team -- DevOps -- taking care of that stuff rather than foisting it on the application developers.

              Good DevOps folks tend to also be system-level developers -- ideally people who know everything from the kernel level up -- as well as having a good chunk of system adminis

          • by lennier (44736)

            Development writes the code that set up the DNS, the firewalls, even the kickstart files (or whatever your local equivalent is) that control OS installation.

            I'm unfamiliar with the term "DevOps", but in my kind of workplace, Operations do that kind of OS configuration stuff (including writing automated scripts to manage it), and then we have Application developers who don't know and don't care anything about the OS, but want their precious application to write to C:\Windows\WhyDontIhaveRightsHere and have an always-on two-way RDP connection on port 31337 to a data centre in Korea.

            OS and Applications Development will fight eternally, in my experience. It's nothi

            • by cduffy (652)

              OS and Applications Development will fight eternally, in my experience. It's nothing to do with "development" vs "operations" or automation vs manual configuration - the OS guys want to lock everything down and use standard systems, and the applications guys want to open everything up and do everything custom and bespoke. How do you resolve that conflict?

              I've always been lucky enough to work places where the app team's senior architects had actually heard of the concept of "security" and were able to beat t

      • by Vanders (110092)

        I'm guessing the main haters are sysadmins, who see threats to their importance and way of working.

        I started life as a traditional sysadmin. Now I'm very much a DevOps guy. I love it. DevOps is a massive improvement on the old throw-it-over-the-wall, compartmentalised way of doing things.

      • by gmack (197796)

        I think you have missed the entire point. You still need people who know what they are doing to make the scripts and setup in the first place.

        What DevOps fixes are places where any problem gets passed off on someone else. Software is too slow? Software devs blame the database people who blame the systems people who blame the networking people (if they can) and as a result pretty much any problem is fixed with hardware upgrades and when that becomes impossible you end up with a badly running system but ei

      • by mverwijs (815917)

        I'm guessing the main haters are sysadmins...

        And that is exactly the chasm of the "us" and "them" attitude that the DevOps folks (IMHO) are trying to bridge. Thank you for your fine example, sir.

      • by Colin Smith (2679) on Saturday October 01, 2011 @04:36PM (#37579250)

        Historically a sysadmin has been able to and does write, fix software, and administering large scale systems has always meant templating and then deploying from updated templates... which means having a build system which can build to a template usually automatically (it's boring) which apparently is now called continuous integration. You also need an infrastructure designed to easily deploy changes. VM or physical is almost totally irrelevant, VMs make life a little easier.

        Lets see... Infrastructures.org for example was definitely there mid nineties (and still is, cool.). That's almost a whole generation ago, how old are you?

        I have no idea what makes people think this devops stuff is new. Is it just the fact that there are now thousands of newbies trying to make names rediscovering good practices? Is it just that we now have more large scale systems, or is it that we now tend to have highly distributed vs centralised systems?

        And it's not development. It's engineering. The difference is maths.

        Go on, get off my lawn!

        • by segedunum (883035)
          I think it's more of a case of developers having to now do a lot of sys admin work, not wanting to do it, not having the expertise to do it, but trying to make out that they can 'develop' things away and it will all be easy. Small companies have this problem, and small IT companies like to believe they can just hire in sys admins and farm crap out to places like Heroku and Engine Yard.

          In my last job I pointed out to our developer when the development work dried up, as it does, that our clients paid for t
        • by wisty (1335733)

          I'm guessing in the 90s, it was pretty rare to actually use those practices. A new shop would have one server called "server". Then it would get a few, named after the Seven Dwarfs. Eventually, you would get to the point of needing to get a new sysadmin who could handle large-scale deployments (or have the old sysadmin change the way they worked), but that would be around the post-IPO stage (if ever), not the "three guys, one of whom is figuring out how to script EC2" stage.

      • "I'm guessing the main haters are sysadmins"

        And then you are wrong.

        There certainly is *some* uneasiness on *some* sysadmins, but not because what you seem to think but because of very valid concerns.

        Devops, as other have stated, while a new word is not a new concept -other highlighted infrastructures.org for instance, and it is basically having the sysadmin mindset to use the developers' tools. And here comes the uneasiness because using the developers' tool puts developers in a tactical advantage... for

      • by CAIMLAS (41445)

        Sorry, that's just a bit of bullshit. In today's IT organizations, a sysadmin can't be a curdmungeonly small-minded person. He's got to accept and embrace newer technologies for the specific purposes of reducing the amount of work he has to do.

        If the sysadmins are resisting, it's because it means it's more work, or they're simply not up to the task of newer things. The distinction may be hard to understand by someone who doesn't do sysadmin work. Often, changing to a new approach means they won't be able to

      • by mibus (26291)

        I'm guessing the main haters are sysadmins, who see threats to their importance and way of working.

        Spoken like a true developer ;-)

        Developers need to get used to the idea of Operations, as much as the Ops folk need to get in bed with the dev side of things.

        Devs and Ops have traditionally led very different lives, and it's that schism that's really made DevOps-style movements so hard (and important).

        As a random example - Ops people are typically on call, and get woken up (automatically by a monitoring system

    • by packetrat (245386)

      I'm not sure I can take anybody who calls an attempt to make IT and Development more aware of each other a cult, seriously. .

      Actually, the people who were espousing devops were the ones who called it the "cult of devops". I'm not hating, just wondering if it's just relabeling.

  • are most IT organizations really that poorly run that developers are totally unaware their software is sending messages that are generating network storms

    How is it the IT organizations fault the developers are unaware of the effect of their software? Is it the IT departments responsibility to debug software?

    • by bsane (148894)

      Is it the IT departments responsibility to debug software?

      Where I work? Yes.

    • by Junta (36770)

      Well, solely blaming 'IT organizations' for developer unawareness is unfair, but in answer to you second question, yes, a good IT department can and will debug software to the extent it does not take down other services in the process. Sometimes a behavior cannot be readily replicated and knowing enough to capture viable debugging data before having to revert it to a working state can be critical. You don't want to leave it messed up until a 'developer' can get to it because you need it back to working AS

    • by bobaferret (513897) on Saturday October 01, 2011 @09:42AM (#37576372)

      I think this is the whole point of the devops stuff. It's not IT's fault, it's not the developer's fault, nor is it QA's fault. The larger your organization, the larger communications gap between these 3 areas. The devops folks are there to coordinate and communicate between the three. They may review/write a little code, they may consult on infrastructure, they may make sure things get tested appropriately and meet the customers expectations. For small shops/teams this fairly straight forward. Big shops have to have someone who's job is to know everything about the project. There is a similar position in commercial building. It's the "site manager" they make sure that the contractor is building what the architect asked for. From the number of bathroom stalls, to the depth of the foundation. Which at the same time working with the architect to fix problems that arise from misplaced city sewers lines, an unavailability italian terrazzo tiles in rural Illinois. All the while making sure that the customer is getting what they paid for and understanding and approving changes. Sorry I didn't have a good car analogy.....

      • Wish I had mod points handy.

        The separate department structure for computer systems encourages everyone to take the approach of 'Not My Problem' even when they could fix it. From a broader perspective, the barriers among Dev/QA/Ops are artificial: it is all computer people doing computer-ish stuff. External constraints are vastly simpler than the construction analogy -- no opcode shipments stuck in Customs, no delays from rain or freezing temperatures. If the computer industry can not overcome these lesse

      • by guruevi (827432)

        The problem is not that organizations are large or any of that. Small shops have the same problem. The problem is management. It's a manager's job to make sure the lines of communications up/down and horizontal are open and that his team gets the resources (whether physical or not) it needs to do a better job. SOOOO many managers fail at that. Some think they are good programmers and get in the way of architecting the software, others think they are good talkers and start acting as a filter. That's why ther

        • by Rich0 (548339)

          Maybe my workplace is just unusual, but the problems I've seen include:

          1. A chargeback system that discourages anybody from helping anybody else with anything unless that help is formalized and expensive and planned in advance.

          2. Since only the most routine functions of internal service organizations actually get formalized and planned in advance, the ability to do anything else gets whittled out of those organizations in the name of efficiency. The windows guys know how to create/edit groups, the oracle

          • by dossen (306388)
            Just a quick counter point to your first problem. Lack of a (good, working, sane) chargeback system may cause needed work to not be done, if the people who need to do it are not on the same team/in the same department as the people who need it done. Think servers being built by developers - who can't continue work until they have been created - instead of having server specialists build them - who may in fact be able to build them better and faster. This happens easily when no chargeback system exist for t
      • by sgt scrub (869860)

        Communication between QA and dev is a given. If it isn't, you have a major problem. Communication between QA and tech support should also be a constant. The best leaders of dev teams, IMHO, work from within tech support.

    • by shish (588640)

      How is it the IT organizations fault the developers are unaware of the effect of their software?

      Because they didn't take the time to send a quick email saying "hey, your code breaks in production"?

      As far as I can see, DevOps is just a fancy name for "people who are working together on the same project should probably talk to each other", and I'm amazed at the number of people who are surprised by or even hostile towards this suggestion.

    • by mikael (484)

      The developers are using third party API's which can do all sorts of odd things, like send out multicasts or broadcasts for servers on the local network (eg. 127.0.0.1 -> xx.yy.zz.255, xx.yy.255.255, xx.255.255.255), while sys-admins are responsible for the network topology including firewalls, bridges, routers, and general IP routing.

      Sysadmins fail to set up the IP filter tables correctly, and local broadcasts intended for an small network can go circulating around the entire corporate network.
      Any in-ho

  • by prefec2 (875483) on Saturday October 01, 2011 @09:04AM (#37576242)

    A while ago I thought most OSS application and framework projects including such Gnome and KDE are in trouble, due to the large use of the fumble-around development approach. Also known as the first code then think approach. All the great model-based, model-driven and agile development methods seam to be far away from the way many OSS projects are developed.

    However, lately I came out of my ivory tower and stood eye in eye with experts from the industry who largely believe that they are professionals and really do great stuff. They also use the fumbling approach. The main difference is the call it agile. Even though it is far away from such an approach. I always thought that one of the problems of our new economy company back in the 2000 originated from being too deep in code and too light on design, planning and documentation. But it looks like, that tinkering is a more widespread way of software development. So I guess that leaves me with bad management (which was not my responsibility).

    Most of these tinkering approaches originate in the absence of developer discipline and the "Add this quick"-management method. but I am telling nothing new. We all know for decades now what is wrong in software development. A lot of people wrote books on patterns (design and otherwise), but in the end if no one follows these patterns the problems remain.

    The "Add this quick"-management originates at large by a misconception of IT and its importance for businesses and other organizations.

    • Re: (Score:2, Offtopic)

      by hitmark (640295)
    • by Lennie (16154)

      These problems will always exists.

      You can not predict everything and you can't generate all possible testloads.

      There is always friction between:
      - adding features
      - lowering operational expenses
      - trying to predict what part will fail first under load, increasing scalability
      - deploying quick fixes because you did not anticipate something
      - deploying real fixes for quick fixes

      I could be wrong of course :-)

      • by hitmark (640295)

        The last one appears to almost never happen. One just end up with a house of cards in the form of quick fixes until the whole system is scrapped and a new one erected (likely because some manager got a nice dinner out of a silver tongued sales rep).

      • It's like anything else. The better your initial plan or design the less likely you are to get surprised and the more likely that the surprises will be small. There's an old saying about plans in the military "The best plans only last until the first bullet flies" and it's true... but that doesn't mean I didn't spend hours and hour planning operations. Sure nothing ever goes exactly according to plan, but a good plan considers as many variants and contingencies as possible in order to anticipate reaction

        • by Lennie (16154)

          Now that I think about, getting back to what the article is about.

          I think the point of the article is: know the environment where your code gets deployed and have some idea of how it will probably get used and actually monitor that so you can better adjust.

          An other thing I got from the article is that this does not really fit with the idea of cloud computing.

          Most people would have no idea how the environment looks where their code will run if it 'runs in the cloud'.

          • by prefec2 (875483)

            You can do all the stuff in the cloud as well. However, I do not know why this cloud thing is such a hype. But that's a different story. Good cloud environment provide you with the right information on provided performance properties and the avail virtual machine/platform. The problem is that people think in code. And code is not good to think in. Especially when you have to handle multiple sets of aspects which even are not all part of the implementation, but of the requirements to the application.

            I know t

        • There's an old saying about plans in the military "The best plans only last until the first bullet flies" and it's true... but that doesn't mean I didn't spend hours and hour planning operations. Sure nothing ever goes exactly according to plan, but a good plan considers as many variants and contingencies as possible in order to anticipate reactions and minimize disruptions to operations.

          To quote Eisenhower, who possibly faced the biggest challenges of this sort one man ever had to: "Plans are worthless, bu

    • "However, lately I came out of my ivory tower and stood eye in eye with experts from the industry who largely believe that they are professionals and really do great stuff. They also use the fumbling approach. The main difference is the call it agile."

      You mean, cowboy coding is now called agile? I thought Agile still has documentation/process just somewhat lighter than RUP/waterfall.

    • Re:Recently (Score:4, Interesting)

      by Junta (36770) on Saturday October 01, 2011 @09:30AM (#37576334)

      Also known as the first code then think approach

      No, it's think *as* you code. You do think some before starting to code, a vague rough picture of the pieces, but you don't invest a lot of time in that 'pure' design phase because the more detailed you plan without proving it out, the more you *should* throw away as you start implementation and realize how ill-conceived your design was or that maybe your design was adequate, but a better way makes itself apparent when actually implementing. Generally when I see a development team incapable of operating at all in this manner and fail to achieve any measure of 'complete', they only 'complete' under other strategies of development on a technicality and produce low quality product. Some management people think that methodology can be used to make piss-poor (cheap) developers serviceable and avoid having to reward/retain good talent.

      A lot of people wrote books on patterns (design and otherwise), but in the end if no one follows these patterns the problems remain.

      I think there are two aspects to this. One is that most development an organization is predestined to operate in a specifc way depending on who comprises it, regardless of what name they pick to describe it. I know organizations that did 'waterfall' and 'changed' to 'Agile', but really just renamed things they did, acted the same, and called it 'Agile'. However, I don't know if this is a bad thing. I think getting too hung up on a specific 'pattern' others preach in conferences is bad, and organically feeling your way for a process that works for your team is better. Awareness is good, but getting locked in is bad.

      However, I have found in the sea of Agile and Waterfall and all sorts of buzzwords a methodology that really resonates with me:
      http://programming-motherfucker.com/ [programmin...fucker.com]

      • by dbIII (701233)
        It makes me think of a conversation from quite a few years ago.
        "Flowcharts? What is it with you engineers. It's software, it's not as if we're building a bridge or anything. We just keep on typing stuff in until it happens and that gets the job done. If it doesn't work we can clean it up later."
        • by lennier (44736)

          "Flowcharts? What is it with you engineers. It's software, it's not as if we're building a bridge or anything. We just keep on typing stuff in until it happens and that gets the job done. If it doesn't work we can clean it up later."

          And that's why the cats keep getting stuck in the tubes.

      • by Rich0 (548339)

        Yup - we use "agile" but we still start with a requirements spec and all the requirements are written before the first line of code. In fact, even if we have multiple "sprints" we still write all the requirements before we start the first one.

        The only thing agile about it is that we use the right buzzwords, and we do a chunk of requirements at a time (as if that wasn't what everybody always did), and we demo the software to customers and maybe refine the requirements from time to time (as if that wasn't wh

      • by russotto (537200)

        Some management people think that methodology can be used to make piss-poor (cheap) developers serviceable and avoid having to reward/retain good talent.

        That's exactly the point of methodology. It is to reduce the process of software development to a bunch of rote steps which low-skill people can perform consistently. Assembly-line software.

    • by CAIMLAS (41445)

      As a sysadmin dealing with these so-called 'agile' developers, I'd mod you through the fucking sky, if I could.

      You describe it as it is. They write what I would call "quick one-off shit scripts/programs" and call it production quality code. Sure, it gets a 'finished product' done quickly, but it sure as hell won't be fun for the people who get hired on months later to fix it.

      People need to realize that custom applications take a long damn time to develop properly. Maintaining custom software is expensive l

  • The answer to that question is "yes". In my experience what is it uncommon is to find a truly competent IT organization. And normaly the biggest it is the worse the problems are. But, what could you expect in an age where your IT group is composed with the cheapest guys you could hire with no experienced (and more costly) ones?. Or what about the clueless IT managers with little or no experience on IT?. They can't plan against what they don't know. Yeah, as a colleague told me not long ago, IT life is nowad
  • by Junta (36770) on Saturday October 01, 2011 @09:09AM (#37576262)

    I've been around enough to know that this 'new' 'DevOps' philosophy is just the way it always has been done at many successful companies making extensive use of technology.

    I have come to associate the phrase 'enterprise IT' with those who *don't* work that way, and make their lives needlessly complicated. Of course, every last party to the mess will generally recognize it and know what could hypothetically be done about it, but only bitch in private about it and rarely ever push for meaningful change. The reason is simple, so long as it is a complicated mess, it requires a great deal of human care and feeding, meaning job security. Management can't force things to change without huge risk as everyone has sufficiently entrenched themselves.

    • by dkleinsc (563838)

      Size of an organization definitely also makes a real practical difference.

      When you're part of (as I am) a technical staff small enough to all fit in the same room, it's very easy to consult across the admin-developer boundary, along the lines of a quick phone call or stop by somebody's desk. Admins and devs regularly interact, both professionally and socially. That also creates a culture where the admins and devs are very much on the same team, are working towards the same goals, and collectively creating l

  • by Anonymous Coward

    That last was a rhetorical question, right? You could have stopped at "Are most IT organizations really that poorly run". Yes. Upwards of 95% of CIOs are completely unqualified to hold their position. They are grossly overpaid talking heads with NO real understanding of the technical underpinnings.

    Seriously. The single largest problem that most IT organizations have is that the people running them are completely unqualified to execute the duties required. It is not that IT workers are overpaid. Good IT work

  • Yeah, that's exactly what we need. More rules, more books describing how to do *doing* something. Meta-meta-meta-everything...

    And more companies that take a methodology which has quite sensible premises and transform it to a paper-pushing-based freak-child.
  • Whenever you see someone dismissing what you think is a genuine concern, it's not because they don't get it. It's because they don't care. Everyone has their own priorities. Dev's priority is functionality. Security is just a necessary burden. But no one prioritizes burdens. Managing priorities is a management job. So if you see a system in which important priorities are not given weight, that's just poor management. Management's job is identifying priorities and then creating an incentive structure
  • by Anonymous Coward

    The project was large enough and management communications poor enough to prompt many members of the team to see themselves as contestants making brownie points, rather than as builders making programming products. Each suboptimized his piece to meet his targets; few stopped to think about the total effect on the customer. This breakdown in orientation and communication is a major hazard for large projects.

    This cancerous attitude happens in any project, across any set of roles, once things get big enough.

  • They see things as a triangle: developers, Q&A and IT, and in the middle DevOps.

    That's not at all how it works from my experience.
    While developers and Q&A are very close, developers are not particularly close to IT at all, while Q&A is.

    Testing is what requires large infrastructure set-ups. Development doesn't really: the Q&A guy directly gives you access to a working configuration that demonstrates the problem. As a software developer I never have to interact with IT.

    • by SuperQ (431) *

      And that's the whole point of DevOps. To bridge the gap and bring the teams closer together. This is one of those "if you don't know who the chump is, you're the chump" situations. Because you haven't experienced it means your company(s) are missing DevOps and the whole point behind it.

      Get rid of dev/qa/ops silos.

  • Devs need to drink more of the DevQA cool-aid too.

  • by Anonymous Coward

    In an attempt to answer the barely articulated question buried amidst the run-on editorializing, I will answer "yes". Many otherwise extremely talented developers carry little understanding of how their code performs once exposed to chaotic and nondeterministic production environments. Likewise many ops engineers hold little appreciation for the motivations of and the pressures on those developers.

    Much of this isolation stems from culture, but at least as much is from experience. Being paged at 3am because

  • The wiki page on "devops" sums up part of the idea to involve "deep cross-departmental integration"

    While there is a lot of agile practises around Google, including "stand-ups", iterations, scrum, planning poker, and whatever, there is hardly any "cross-departmental integration". Communication is as poor as at any other huge behemoth. Just like we've read about multiple internal teams fighting each other at Microsoft and Nokia, so do many teams at Google create similar products, both externally and interna
  • the software industry in silicon valley by and large does not have a role 'architect of product'. we have architects of technology. the product comes together as various teams: tech pubs, qa, tech support, professional services, it, ops, manufacturing, marketing -- all contribute their efforts to take the technology delivered by the typical development team and transform it into a product. DevOps is one attempt at mitigating or filling this gap to varying degrees depending on the folks involved. Now what is

  • by Opportunist (166417) on Sunday October 02, 2011 @06:46AM (#37582504)

    It can be summed up like that: If the problem will arise in an area he isn't responsible for, and if implementing it that way regardless is cheaper, faster or less problematic for his department, he will implement it that way.

    Running the traffic through the internet instead of the LAN? Why not? It's not going to be the development's problem (since that's operation's problem), and it's faster to do it that way, so it's done that way. A user interface that makes users beg to break the programmer's fingers and make him a consultant? Not the programmer's problem, it's the user's problem, and if he can implement it faster with a shabby interface, it will be done that way.

    The problem isn't that the people wouldn't know that these problems will surface. Often they know that quite well. The problem is that this is not a criterion, while the time they spend on solving the problem is. Hence, whatever lets him get it faster off his table will be the way it is done.

  • ...and I'll even try to remember to check back in case people respond.

    Assumption: the trading application being critiqued is the same one that was there when I was an IT consultant at Morgan Stanley. I left in August 2000.

    I know the application well. It was developed by a department headed up by Vinny (whose name is withheld because...I'm senile and don't remember it). I worked in the department that wrote the messaging infrastructure that was used by every application on the sell side of the firm.

    If the

  • cults are stupid.

    if you want to name yourself something stupid, i don't care. Stupid people do stupid shit because they are stupid.

You can measure a programmer's perspective by noting his attitude on the continuing viability of FORTRAN. -- Alan Perlis

Working...