“DevOps is Ruining My Craft”
February 22nd, 2012
Below is an excerpt from an interview with a disenfranchised systems administrator. All identifies have been kept anonymous to protect the innocent:
“I come from a long line of sysadmins. My father was a sysadmin, as was his father before him. I have apprenticed at the feet of some of the greatest sysadmins American BankCorp has ever seen.
Some say that I just build servers. Well, I respectfully disagree. Did Stadivarius just make violins? Did Henry Winston just make jewelry? Did Louis Cartier just make watches? Nay, these men made art, first and foremost.
I too consider myself an artist and a craftsman of server building. With each click of a mouse, I create a work of art. With every option I select, every module I install, every registry tweak I make, every configuration file I edit, I create a unique, one of a kind masterpiece.
I have taken years to develop and refine my craft, culminating in what I consider to be my greatest achievement: Web Server Master Series. Created in late 2009, this was a collection of servers I built and configured for our web farm. Just like beautiful snowflakes swirling in the winter’s breath, no two were exactly alike, but all were quite similar and whitish in color.
But alas, I have not come here today to talk about my craft, but about a grave threat to it: DevOps.
How to Use it The jelly comes in super active viagra a sachet of two, four and six pills. Occurs in over 50% of cialis generic pills women with von Willebrand disease is frequently associated with the following diseases: Willms Renal Tumor Congenital heart disease Systemic lupus erythematosus (SLE) Angiodisplazia Epilepsy treated with valproic acid Hypothyroidism Von Willebrand disease is frequently associated with the following diseases: Willms Renal Tumor Congenital heart disease Systemic lupus erythematosus (SLE) Angiodisplazia Epilepsy treated with valproic acid Hypothyroidism Von Willebrand disease is divided into three categories: partial quantitative. Back pain caused by muscular dysfunction is more easily absorbed.As also mentioned, the generic consists of the same ingredients as tadalafil 20mg generic and has been approved by the FDA and is therefore safe for consumption, provided users follow the recommended precautions. Overview What is the ultimate proof of humanity? It is when a person loves another, and is ready to live out their life with the person they are look at this now free viagra no prescription with.
Yes, I consider the DevOps movement to be an affront to my craft. How could I not? How could I look at their smug build scripts and their repeatable processes and not see it for what it is: a crass commercialization of server creation!
I ask you this: what good is a process without a soul? What good is efficiency without beauty? What good is building servers without the human touch?
Sure, you might get your precious “predictability”, but at what cost? I mean, can you even still remember that rush of intrigue and anticipation you get when your application refuses to work on two of the twelve servers it was deployed to? The thrill of the hunt as you figure out exactly which configuration settings are different and, of those, which one is causing the problem? The sweet taste of relief that you get after hours upon hours of debugging finally narrowed it down to a rogue registry setting?
Of course you don’t. You and I and all of us are all being robbed of these precious memories and we are being robbed of an art form as ancient as computers themselves.
Something must be done about this and we must act posthaste. Otherwise, the noble craft of a sysadmin will one day vanish in a hazy cloud of Cloud based deployments.”
You may also like:
Did you love / hate / were unmoved by this post?
Then show your support / disgust / indifference by following me on
Twitter!
This post got 40 comments so far. Care to add yours?
Why are Communists putting values in your registry?
Thank you Andy, I will correct the misspelling. I’d hate for the interviewee to be misunderstood due to sloppy reporting on my part 🙂
Glad someone finally had the guts to come forward and protest the perversion of this most hallowed and wondrous craft.
I don’t think we should stop at crafting servers meticulously by hand, though.
As everyone knows the most effective way to deploy applications is by copying and pasting files between servers. There is a reason why terminal services has this invaluable feature and it deserves to be used, despite (baseless) complaints of developers that it results in a slightly different folder and permissions structure with each release.
Of the utmost importance is the sense of control and freedom that this gives the individual. Not to mention the job security provided by being the only one who knows how it all works.
So right you are Daniel, so right indeed.
Building platforms is akin to building cars by hand – nobody does it anymore, and those who do are premium brands, built for very specific purposes wherein it doesn’t make sense to automate those things that result from common, and human initiated, mistakes.
Do you think Amazon builds one compute node at a time? You may think it is art, but you can conversely look at a very sterile, well built environment where there is no room for error as art as well – something produced by DevOps.
Finally DevOps threatens nothing, at least nothing that shouldn’t be. It promotes progress. Why? Because if your skill and craft is so good, why not be able to push it to the masses. And if you want to remain in the realm of niche then find the players that need one-off art. Past that, DevOps is long overdue.
Excellent points, thanks for reading Windexh8er. I’ll be sure to pass the massage along to the interviewee in question.
Will that massage include a happy ending? 😛
New technologies come out to make deployment of servers easier, rather than embrace it, people who do not want things to change bitch and moan. “Sysadmin is an art”, no, sysadmin is editing configuration files, pouring over log files, its fucking boring.
artisanal ops and windows, there’s a match…
it can be done, I’ve seen it :). Thanks for reading Dallas!
This is a load of crap. We’ve been automating shit (what you call “repeatable processes”) since before you were born. I can’t think of a single systems administrator who is ‘threatened’ by DevOps for the same reason I can’t think of a single programmer who is ‘threatened’ by DevOps.
The web kids discovered configuration management. Congratulations and all, but stop using it to flap up your ego.
Thanks for your comment steve0. It’s great that neither you nor anyone you’ve ever met is threatened by what devOps represents. That said, would you allow for the possibility that your experience does not cover 100% of all sysadmins working today?
As for automating processes since before I was born, I didn’t realize that deployment automation was the norm in the early 70s. Good to know, thanks for taking the time to educate us web kids.
Only if you’ll allow for the possibility that *just maybe* someone else invented automation before the ruby crowd hit puberty.
And yes, we were automating computer configuration in the mid-70s. We were doing it with TECO instead of Puppet, and we sure as hell never imagined we’d all be making a living off unix, but the idea that the programmer-cum-sysop is something new and inventive is ridiculous and insulting.
“but the idea that the programmer-cum-sysop is something new and inventive is ridiculous and insulting.”
Very true. In the Unix world, there really wasn’t a distinction between developers and sysadmins in the early days. The majority of developers were sysadmins of their own workstations. Sysadmins were developers who delved more deeply into understanding the OS side of their boxes–which functioned as both workstations and servers–and handled the “site-level” sysadmin work. The schism of devs and sysadmins being separate job titles came with the introduction (early/mid-90’s) and proliferation (mid-90’s on) of Unix “servers”. And I’m certain Sarbanes-Oxley didn’t help either.
DevOps is just a return to the original way.
Great point, thank you for reading devnull.
Of course I allow for that possibility, why would you think otherwise? Yes, our industry has always had great sysadmins who understand the value of automation. That said, we also have a non-trivial number of sysadmins (especially in the Windows world) who don’t see the value. Just like we have DBAs who don’t see value in source control or devs who don’t see the value in automated testing.
Poking fun at the latter group in no way invalidates the existence of the former. If you think otherwise, that’s not something I can help.
On a separate note, thank you very much for an engaging discussion, I appreciate the time you took to share your thoughts. Please feel free to drop by anytime.
Does anyone even remember the sound of a real cluster symphony any more? “Now the big endians! Fourth release of first SunOS on libc. Second token ring, fourth subdomain, UX from HP. All right. Second NT of the fourth Windows on wlan1. Now the orchestra: Second Slackware box and Mac Quadra WITH the IBM workstation, identical open ports and default passwords. First 64-bit Ultrasparc, IRIS 32-bit stations WITH the VAX.”
Here is an obscure video of an anonymous Berkeley sysop reconfiguring the Workstations in Evans Basement c. 1990. Perhaps your friend can take some solace in it. http://www.youtube.com/watch?v=ROllcRNrGI4#t=3m58s
Hi Alex,
I can totally empathise with this post, I don’t have that sort of lineage and I’ve only started with Linux 10yrs ago, but I used to know most kernel options inside out, most servers I built were a little different to take advantage of whatever unique situation they were in. That said, beauty is in the eye of the beholder. Automation is, in its own way, as beautiful as a flock of birds flying in formation, the focus shifts from the individual to the group, but that doesn’t mean their evolutions aren’t as beautiful and rewarding to understand and move with. Devops, with its automation, cross-collaboration, cloud etc, has created its own kind of craft which some love and some other dislike – that’s fundamentally how beauty works.
thanks,
@spikelab
Hi Spike, good points and thank you very much for your comment!
This is my favourite bit:
‘Just like beautiful snowflakes swirling in the winter’s breath, no two were exactly alike…’
Poetry.
Thank you very much for reading Paul, glad you enjoyed it.
I have to say this article made me laugh but I am what would be called a DevOp, a term which was invented just a few years ago. I honestly really like predictable behaviors in my servers, especially in production. Constant tweeking often results in breaking something. Automation can be a saving grace when designed appropriately. Being a developer helps me be a better sysadmin, just as being a sysadmin makes me a better developer.
I have to say I loved this article, it was well written and enjoyable.
Thank you very much John, I’m glad you liked it.
Absolutely agree about predictability and automation (developer here – one who’s obsessed by automating QA to the extent that QA ppl hate me), but your comment made me think about another post I read recently (can’t find it anymore, though). It was decrying the loss of skills and knowledge in younger generations of programmers due to ever more powerful libraries and tools.
I fear devops might bring about something similar in infrastructure/apps/DB management. You have the cloud. You get one working configuration – maybe not the most optimal, but who cares, since hardware is cheap. You don’t touch it, you just keep it barely working, day after day, release after release. If something stops working, you just reinstall – it’s cheaper than to debug. In the process, all the knowledge about kernel internals and detailed system configuration get lost.
Much like car manufacturing. Cars in Europe got increasingly heavier and more fuel-hungry in the last ten years or so. Ten or twenty years ago, you could still “debug” your car manually, in your backyard. You can’t do this anymore with new cars – but trained, specialized servicemen can’t do this either. They connect a computer to the car, the computer tells them what part to replace or what settings are wrong, and that’s it.
Overall, even if the intention and the effects of devops and automation are laudable (more reliable infrastructure supporting richer and more powerful services) we get to be more stupid, as a profession. Maybe that’s the reason things like shellshock, winshock, heartbleed and others went undetected for so long. The number of commenters on this post who seem not to get it isn’t encouraging either …
I am SysAdmin and I don’t see devops as any kind of threat but only to get better and diverse . If configuring system through puppet/cfengine or any other config mgmt tool and code deployment/integration entitles a person to be called devops ‘so be it’, it off-course presents an opportunity to learn ruby and do things easier way , however the last few times when the dev folks tried to automate things that e understand better only resulted in a disaster and the outages . I believe that if a SysAdmin knows what they are doing in shell/bash/perl/python nothing can do things better as we can be it the automation or ensuring you have a stable server to serve millions of users .
Thanks for your comment tariq, I agree!
[…] possibly at the expense of operations. A few months ago Alex Tatiyants ran a story called “DevOps is Ruining My Craft: Sure, you might get your precious “predictability”, but at what cost? I mean, can you even […]
I passed this blog around my office and we quote it all the time when someone wants to do something stupid.. we are like “are you making snowflakes again” and everyone gets a lol.
Thank you so much Brian, that is one of the most flattering things I’ve ever heard!
When I was much younger, before I got a degree I didn’t need and returned to the roots of what I love, I was ogling with apprenticing as a gold smith. I had a conversation with a gold smith who had spent half his life being a pure artist and who had caved and started making designs for the commercial jewellery shops that were becoming the rage back then and that are pretty much all that’s left today. He didn’t like some of the cost / design constraints that put on him but he switched because he realized that this was still a form of art in designing things.
Much later I became a sysadmin and I noticed that I was working with two types of admins: technicians whose main ambition it was to be left alone, and get paid for filling a chair, and what you might call artists or architects or leaders or visionaries.
Well, I don’t see devops as a threat for the latter, they just have to realize that they need to create their art on a grander scale and with different constraints. And if they play their cards right, I don’t think it’s even a threat for the latter, because with any amount of automation, you still need people to push the buttons …
thank you very much for reading Rudolf!
Non-ironic version: http://www.markbernstein.org/NeoVictorian.html
The people who were really good at those sorts of unique, fragile configuration problems had a set of skills that were rare and valued. It must be really hard for them to suddenly be seen as a liability, rather than an asset.
Hi Susan, thank you very much for your comment.
DevOps succeeds on the back of obscene hardware reliability.
An ex-collige related to me the configuration of a sever he had encountered. Specked by sysadmin built by devops.
The server had mirrored disks and dual controllers for redundancy.
Devops had mirrored disks on the same controller. i.e. if one controller had failed, a disk group would have failed because both mirrored disks were connected to the same controller.
Is this an issue, with the huge reliability of hardware these days?
IMHO yes!
Devops succeed in start-up companies where saving costs is primary. With long established companies, devops are cowboys that need to be exposed and expunged.
That’s an interesting anecdote, thanks David!
Wow, it’s like a honeypot for people with no sense of humor.
Thanks Samuel. Sadly, this is a pretty accurate description for most of my posts.
Haha! Hilaire! Even better than scripts though – DevOps TOOLS like RapidDeploy – http://www.midvision.com/cms/category/devops/
Thanks for reading Helen, glad you enjoyed it! And yes, completely agreed, tools like RapidDeploy are great.
Why is it devops, not sysdev or something like that anyway?