Software Developers Should Have Sysadmin Experience - <AUTHOR> http://blog.professorbeekums.com/2017/01/software-developers-should-have.html <comment><USER6> This might be controversial, but I don't think you get to be a half decent developer without being a reasonable sysadmin.

Maybe my experience is unusual, but I've never worked anywhere that the sysadmins knew more than the developers about how best to run their code in production. And when things go wrong with it how best to find the cause of the issue.

And I've never thrown code over a wall without having tested it in a representative environment.

The worst sysadmins get in the way of developers. Ones that scale down your CI server to the cheapest, throttled, one the hosting company has, leaving $800/day contract developers waiting for builds that run in 20 seconds on their laptops take nearly an hour. And then try and argue the toss about whether the CI server is cost effective and every few months keep switching it down despite the CTO saying it needs to be left alone.

When a sysadmin sees an issue in "their" environment that they understand there's a tendency for some of them to just see that issue as the only thing the developer has had to deal with that month. In all likelihood, in a productive company, it's the most trivial issue the developer has had to resolve that day.

Often this stuff goes more smoothly where the developers (I mean, it's not as though if you're going to drop one of the two groups of people it's going to be them going) manage production and there aren't people with separate job titles and the resulting friction between them.

Sorry. There must be great sysadmins out there struggling with terrible developers, I'm sure of it. I just haven't seen it.

<comment><USER7> I've done the dual sysadmin/developer thing for a small company, and the problem I experienced there was completely incompatible working modes.

Sysadmins must deal with interrupts (requests, crises, things driven by external schedules etc) and then in the rest of their time build systems to manage or reduce the interrupts. Developers are expected to produce work on a predictable schedule. This is disrupted by interrupts and obliterates the schedule for proactive work unless your management is very good at making it a priority.

The "prevention of information services" problem is certainly real though. Perhaps it could be addressed by embedding the sysadmins in the dev teams rather than having a department of their own, but then you have to fight org hierarchy.

<comment><USER8> _AMEN!!!_ In my current job, I am wearing both hats, and while I like that there is a certain variety in my work, users calling for help is highly disruptive when programming or doing some other stuff that requires deep focus.

The upside that in a three-person IT department there is very little bureaucracy to fight, just the odd "organically grown" legacy system.

<comment><USER9> As an aside, did you know that the word "Amen" actually is a acronym in the Jewish language that means "El melekh neʾeman" (or AMN) which translates to "God, trustworthy King". (source: [https://en.wikipedia.org/wiki/Amen](https://en.wikipedia.org/wiki/Amen))

I figured the etymology of that word was rather interesting. But yeah, I get the whole SysAd/Dev dual job. They're tough to balance and do effectively. SysAds are firefighters. When the nag(ios) alarm rings, we come a-callin.

<comment><USER10> From that Wikipedia same article, it says

 _The Talmud teaches homiletically that the word amen is an acronym_

The etymology section shows the word has much more prosaic roots. The Talmudic acronym seems to rather be an interesting backronym.

</comment></comment></comment></comment></comment><comment><USER11> Hi, I'm a Sysadmin, and I've been a grumpy one through a larger part of my 15 years experience. My main issue was that Developers were acting like Users: they don't care about what you have to deal with, they want things to 'just work'. In return, I've treated them like children, in some instances yelled at them when they did dumb stuff. I've tried to educate them when possible, and was angry when the education didn't stick. At the time i was the 'King of the Hill' type of sysadmin - natural leader of a very small and tight team, kind of irreplacable, and with enough years in the company behind me to consider myself as a demi-god.

When I switched companies, I came across better developers. Some had decent sysadmin skills, but the main difference was that they actually took interest in how things worked past the 'git push', and when I asked / required them to make some changes that would make my life easier, they listened, discussed and adopted when appropriate. With those same guys, I took interest in what they were doing, what their actual job was and came up with ideas that would make things easyier and run smoothly on both ends. After a while I figured out that they weren't actually better developers - they were better people. (Also, I figured out that being grumpy was not the best approach and that patience, kindness and gratitude could get people to do more than snark, humiliation and flame-throwers.)

I guess my point is: you don't really NEED to have sysadmin skills to be a decent developer; what you really need is to care about what sysadmins do - <USER2> curious, talk with them and trust them when they say that your brilliant idea won't work in production.

<comment><USER12> I think there are definitely developers out there that give little to no respect to systems administrators.

I've seen this ignorance even in college professors. In my first programming class in college I took a CS class that had both CS and IT students in it since it was required for both kinds of students. The (CS) professor kept trying to convince students how much better CS was and gave some good arguments (ie: salary) but the most arrogant thing he said is that IT is a subset of CS and that by doing a CS degree you would understand everything it takes to be in IT. He also mentioned how in IT you would be constantly fixing other people's computer problems but as a software engineer you wouldn't need IT's help since you can fix it yourself. The funny part is part-way through my degree I realized that college didn't even offer a real CS degree it was called "CIT with Computer Science Emphasis" which none of my advisers nor professors mentioned would cause issues getting jobs outside of Utah, the best thing I did was leave that school and finish my CS degree elsewhere which caused me to lose a lot of unnecessary credits and almost felt like I was starting over. I feel like I got scammed but that's beside the point I am yet to work for a company where a software engineer gets to manage his own computer without following IT guidelines like my CS prof had described.

</comment></comment><comment><USER13> As a grumpy evil sysadmin, I think the good Professor misses where the real disconnect is, at least nowadays: stack management.

Why do things like Docker exist? Because developers got tired of sysadmins saying "sorry, you can't upgrade Ruby in the middle of this project". Why does virtualenv exist? A similar reason.

Containerized ecosystems (which is to say basically all of them now) are really a sign of those of us on the sysadmin side of the aisle capitulating and saying that developers can't be stopped from having the newest version of things, and I think that's a bad idea.

15 years ago, when a project would kick-off, as a sysadmin I'd be invited in and the developers and I would hash out what versions of each language and library involved the project would use. This worked well with Perl; once the stacks started gravitating to Ruby and Python it was a dismal failure.

Why? Because those two ecosystems release like hummingbirds off of their ritalin. Take the release history for pip[1] (and I'm not calling pip out as particularly bad; I'm calling pip out as particularly average, which is the problem): in the year 2015, pip went from version 1.5.6 to 8.1.1 (!) through 24 version bumps, introducing thirteen (documented) backwards incompatibilities. Furthermore, there were more regression fixes from previous bumps than feature additions. You'll also notice that none of these releases are tagged "-rc1", etc., though the fact that regressions were fixed in a new bump the next day means they _were_ release candidates rather than releases. Ruby is just as bad; the famous (and I've experienced this) example is that an in-depth tutorial can be obsoleted in the two weeks it takes you to work through it.

Devs are chasing a moving target, and devs who haven't been sysadmins may have trouble seeing why that's a bad idea.

[1]: [https://pip.pypa.io/en/stable/news/](https://pip.pypa.io/en/stable/news/)

<comment><USER14> As a Sys Admin turned Automation/Tools Engineer, I think you're missing part of the point. You've got the beginning of it right in saying that Sys Admins used to be involved in pinning down versions, and even in why that was necessary, but I believe you're incorrect in saying that the containerization technologies are bad for removing that.

Those technologies don't exist so Developers can get around Sys Admins and ignore your helpful advice. They exist to solve that problem that makes the Sys Admins role there necessary. It removes the underlying need for a Sys Admin to worry about the versions. Admins should see this as a good thing, but in my experience many dislike it because it takes them out of their Gatekeeper role. We shouldn't WANT to stop the Developers from from having the newest version of things. They aren't kids playing with toys that we need to nanny over, they're doing work that creates values and the fewer things we do to get in the way of that, the better.

If something breaks due to version changes, their testing should catch it. If things are breaking in production, we ought to get involved because there's some other problem, but before that we, as a profession, need to learn to get out of the way and let people work by letting technology handle the problems. The "Gatekeeper" mentality needs to die as quickly as it possibly can.

<comment><USER15> > They aren't kids playing with toys that we need to nanny over

In my experience (university), yes they are, and they should do that at home.

Why do you need the latest bleeding versions in the first place?

In my sysadmin experience, people believe software gets bad and deprecated as soon as the glory next breaking version appears. I don't think I need to argue why this is an illogical stance.

With my developers hat on, bumping to the next version mid-process reliably introduces more friction than is worth it. People think the next version solves that one weird issue but ignore that it introduces two new ones and that the software must be changed to fix five new incompatibilities.

But the solution reliably is to just not use that weird feature that caused the bug in the first place, and think what a clean solution would have been. And guess what, the result is a cleaner and more compatible code base. It's a tip that works for me again and again: If there is friction, think - <USER3> spending the next hours with an update that will soon lead to new problems.

It's great that you can for example compile Linux without too much friction. It's great that arcane shell scripts can run on any system. Stability (in a compatibility sense) is not a nice-to-have, it's basic sanity.

<comment><USER16> My comment to you two is -- why not both?

Stability, sanity, all that is amazing, and a must have.

But also bug-fixes, security improvements, and performance improvements are wonderful too, which tends to come with using up-to-date dependencies.

The problem with the latter, as you mentioned, is when it introduces breaking API changes and is wholly not backwards compatible. This is not a "kids playing with toys wanting to experiment problem" this is a bad software problem, which is why I like Go, and why I liked Java when I was doing it full time. If the language you use has backwards compatibility as a first-class citizen, most likely the package authors will act that way too, and then the maintainers, and eventually the developers. Limit your software choices to those who care about not breaking everyone's shit every 2 weeks. Heck even when I write my own API's now that I know only my company is going to use internally I am thinking about this.

<comment><USER17> Backwards compatibility is seriously under appreciated. When I tell developers to ensure that their changes are backwards compatible, they tend to look at me like I'm green.

I do not understand the disconnect that developers have with understanding all of the benefits that it brings. Yes, you have some extra code in your code base so it's less clean. You also have a stable environment as a result. The first affects only your personal preference. The latter affects all of your developers and users.

Unless you have a situation where it's impossible to maintain, not insisting on it is pure self interest.

<comment><USER18> _I do not understand the disconnect that developers have with understanding all of the benefits that it brings._

Because they've never worked on an old codebase, because front-end technologies change so often and everything just gets re-written anyway. It's a waste of time worrying about this when the code won't make it to is first birthday.

If you were speaking to seasoned C and DB developers about stability in the tools and the platform, you'd be preaching to the choir.

<comment><USER13> This gets to the complaint that so much of the open source ecosystem gets to version 0.8.6 (whether it's named that or not) and then completely rewritten "this time the right way". That's not actually a good thing.

<comment><USER19> As jwz put it,

> It hardly seems worth even _having_ a bug system if the frequency of from- > scratch rewrites always outstrips the pace of bug fixing. Why not be honest > and resign yourself to the fact that version 0.8 is followed by version 0.8, > which is then followed by version 0.8?

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER20> I'd say there is too much effort in reasoning on the wrong problem. What worries me the most is the 'why': why do (too) many software developers don't know about sysadmin?

I have been involved as a consultant in large software projects in the last two years and a vast majority of money lost in delays and bugs was caused by devs not understanding: 1) the difference between virtual memory and physical memory 2) the difference between costs of data storage per storage medium 3) the concepts of network round-trips 4) and hardware bandwidths 5) how to install and configure a web server on a workstation 6) how DNS works 7) how AD authentication works 8) what ORM frameworks do 9) how to write a raw database query (not necessarily sql) 10) the difference between navigating through database records on a database server vs. an application server vs. a client, 11) HOW TO INSTALL THEIR OWN WORKSTATION AND TROUBLESHOOT IT!!! N) etc. and those are just the topics that I can immediately remember.

As I see it, it's not about "they should". For me it's about understanding how many devs deal with such a level of ignorance on the systems they interact with, on a daily basis. This situation hurt my feelings everytime it happened and I struggled to accept it. I am not a sysadmin nor a developer but my daily work is insanely improved by my (even basic) understanding of how my workstation works and how to manage it.

<comment><USER21> I've worked with all kinds - from windows devs who can't figure out how to install visual studio - to people who understand Windows, Linux, and macOS - as well as basic system administration for each platform. The people who are most successful at rapidly developing good high quality software are more in the later group.

Would you trust a RF engineer who couldn't troubleshoot his own radio designs? why would you troubleshoot a software engineer who can't troubleshoot his own software as deployed in a real world environment?

<comment><USER22> I recently questioned someone about this very subject. They wanted to hire a "CSS expert" because the "developer" didn't have a grasp of css after having developed the project in JS/HTML. I was so confused as to how that's possible.

<comment><USER23> There's a large gap between basic understanding of CSS and actually creating good CSS. Personally I avoid touching CSS as much as possible.

</comment></comment></comment></comment><comment><USER24> Should sysadmins have software development experience (e.g. DevOps)? For what values should 'X have Y experience?' Should we go as far as...

"A human being should be able to change a diaper, plan an invasion, butcher a hog, conn a ship, design a building, write a sonnet, balance accounts, build a wall, set a bone, comfort the dying, take orders, give orders, cooperate, act alone, solve equations, analyze a new problem, pitch manure, program a computer, cook a tasty meal, fight efficiently, die gallantly. Specialization is for insects." — Robert Heinlein, Time Enough for Love

<comment><USER25> I strenuously disagree with some of these.

> Plan an invasion

This is actually a massive undertaking. An undergraduate at MIT taking a semester-long course on this will barely scratch the surface of it. Furthermore, you're never going to suddenly and unexpectedly need to know this. Any situation where you plan an invasion is going to be preceded by spending a long time getting into the position where people trust you with their lives and the fates of their nation.

> die gallantly

You're only ever going to be in this situation once, and probably not even that. Why does it matter how gallant your heart attack is?

These skills make a bit more sense in a world where most of us need to march off to war. Happily, we don't live in that world.

You should prepare for the situations you are only mildly unlikely to be in and where your skill matters.

<comment><USER7> It's literature, and SF, so it would be a mistake to take the quotation too literally. But since we're here; the thing to remember about Heinlein is that he was a romantic and a futurist at the same time. Hence his militarism wasn't so much about the enemy as about providing an opportunity for romantic heroism. Similarly with the other items in the list. It's not grounded in practical necessity but in a "renaissance man" / hero of a novel approach to being able to not just handle situations but show off in them.

</comment></comment></comment><comment><USER26> Sysadmin knowledge definitely helps, but so does an MBA, knowledge of writing, public speaking, design, user experience, networking. Oh, and the domain of the problem itself.

The skills I require of my developers depend on the rest of the team and the project.

<comment><USER27> The difference with this is that server-side software has direct operational consequences.

A server-side developer who never deals with ops is like a chef who never tastes the food. It's in theory possible to get right, but in practice the results tend to be poor.

<comment><USER28> That's true for backend, but frontend developers don't necessarily need a lot of ops knowledge if the team has a good separation of concerns. If frontend devs have to worry much about sys admin issues, I'd say that likely points to a flaw in the way things are being done.

Of course, the more someone knows, the better. Knowledge and experience in any area of technology can improve understanding of all the others. But nobody has time to learn everything, and there's way too much to learn. Time given to learning more about sys admin issues is time lost to other potentially valuable knowledge.

It's a good area to learn about, and it is essential to being a strong backend developer, but through good architecture and management practices, there are still plenty of ways to make very high value contributions for a developer who hasn't spent much time focusing on sys admin.

<comment><USER29> I think your definition of "sysadmin issues" is probably a little more limited than mine or 'wpietri's. I've had frontend developers insist that they could bake configuration settings into their webpacked artifact...which needed to be deployed into multiple environments because of _course_ we weren't rebuilding something that had already been okayed in QA when we wanted to send it to prod.

You might say that's not a "sysadmin issue," but I have seen it happen three or four times now and in each case it was the "sysadmin" (read: devops engineer) who caught the problem and explained it to the offenders in question. (Maybe it's a "build engineer issue"...but at most companies I've seen, he or she is probably the "sysadmin", too.)

<comment><USER28> I would consider that sort of thing to be part of the required basic knowledge to be a competent frontend dev, so if you want to call that sys admin, then sure, there's a bit of it involved. If it's on your side of the fence then yes, it's your responsibility. I'm not sure how many actual sys admins would consider configuring webpack to be 'sys admin knowledge', but there is a spectrum and it's true that there are some ABCs that everyone needs to know, particularly when there are security implications. Still, it's a pretty far cry from unix, web servers, and databases.

<comment><USER29> You might consider it basic. I would have, insofar as I considered it pretty obvious even when I hadn't written a line of frontend code in five years, and webpack etc. weren't even on my radar then. ;) But my experience has led me to believe that it's not.

I think that's kind of the point of devops, though, is that something like a build system _is_ a cross-cutting concern, that architectural decisions for an application need to involve people across the stack. Classifying somebody as "a sysadmin" is the problem in the first place, which is why I caveated my post as heavily as I did. My experience is that your "devops" or "sysadmin" people functionally become the "junk drawer programmer" who are relied upon for all sorts of weird stuff. I've been at jobs (and at clients) where I had to teach senior backend devs how to use VisualVM or what the implications of using Kafka and CQRS are. I've been at gigs like the aforementioned where I had to explain the ramifications of webpack to people knowledgeable and capable enough to make React dance. And so my definition is probably necessarily more broad...but it's also stuff I've had to do in practice, so, enh.

And, 'cause fair is fair, I think a "sysadmin" who couldn't step in and write production-quality code (allowing for a little ramp-up) is probably an endangered species over the next ten years, too.

</comment></comment></comment></comment></comment></comment><comment><USER30> I used to work on a product where the three major teams were Client, Server and Ops. Tens of millions of customers used our stuff.

Client and Server folks were on separate floors of the same building. We didn't interact much at all, except by trading bugs back and forth. The management chains met at a VP. Three or four times a year we tried to coordinate a release, and it always took at least a month. Getting a feature out the door might take a year, with all the paperwork and pipelining of release schedules.

Ops was in another building. The only things that both Client and Server teams were sure of was (a) they did a bunch of customization of our stuff so that it would work, and (b) they hated us.

Support was in another state. We were not _allowed_ to talk to customers. Maybe once a year Support would fly in to talk to the teams about pain points. I think we did an okay job addressing these, but it took a _long_ time, and customers suffered a lot.

I won't talk about the disaster that ensued when Scrum was thrust upon us, or the splinter projects that spun off to try to fix things (but wound up being lots worse).

You _have_ to be close to the customer or you won't know if you're succeeding, or even on the same page. You _have_ to know what your software is doing in production or you're just sitting in an ivory tower pontificating about angels-on-the-head-of-a-pin nonsense. You _have_ to spend time in the trenches measuring and fixing stuff or you're hatching an unmanageable disaster. The good news is that most of this is actually kind of fun. The bad news is that, when managed badly, this can turn into a horrible and soulless grind of pager duty and making legacy code even more legacy to fix wee-hours downtime.

I still get a rush when I get feedback from a real, live customer, and I think that isolating your teams from customers is one of the worst things you can do to a team and to a product. Getting teams to work on ops and support aren't bad ways to improve this.

</comment><comment><USER31> We took it one step further on my team. The best way to ensure that the pager(duty) doesn't go off at 2am in the morning is to put developers in the first on-call group. Not only did it work, but the developers got a much more nuanced view of how systems operate. When we first started, I'd see developers using ping to determine whether a nameservice entry was correct. After a few months of handling almost all of our own ops, developers knew how each piece of the puzzle worked and how it all fit together rather than the hazy, largely abstracted view they had before.

But we learned that we needed it to go the opposite direction too. We had ops people who, when given a corporate-wide mandate to apply a security patch or some such task, would log into every machine and apply the patch, despite the fact that we'd been practicing immutable infrastructure with zero-downtime deployments. We hadn't given them the necessary exposure to the dev side to understand that you had to apply those fixes to a base image and trigger a redeploy. There was a lot of finger pointing a couple of weeks later after it was discovered that the fixes were overwritten by an application deploy.

<comment><USER32> I hope you increased their salaries together with the new 2am pager duty.

</comment></comment><comment><USER33> That's what I did at my tiny engineering team (3 folks) and results have been great.

If you have a small organization (say <10 engineers), it's crucial that every developer writing server side applications can also do at least some sysadmin work. As the article says, it leads to deeper understanding and it really helps when thinking about scalability, fixing certain kinds of issues, etc. It also often shortens the feedback cycle and requires less throwing over the wall. As a bonus, you increase the bus factor.

Automation is the key though. If everyone connects to the boxes and does random things manually over ssh, nothing good will come out of it.

Still, you need to have a person or two who are responsible for the vision of the architecture/systems and who make sure that things don't go off the rails.

</comment><comment><USER34> Somewhat related, I feel that mechanical engineers who design cars should have some experience servicing vehicles.

Sometimes engineers will not leave enough space, use weird fasteners, etc. that make a simple job much more complex.

<comment><USER35> It could be a deliberate form of obfuscation/discouraging of repair, or just the common trend of making things more complex than they really need to be.

<comment><USER36> That's possible, but lack of insight into what's easy or hard in manufacturing or construction is a pretty common problem. I saw it a number of times when working for a few different manufacturing companies, though none of them built cars.

A failure to understand construction concerns also played a role in the Hyatt Regency walkway collapse. The original design was poor, but redesign to address construction difficulties accidentally weakened the walkways further.

> Havens Steel Company, the contractor responsible for manufacturing the rods, > objected to the original plan, since it required the whole of the rod below > the fourth floor to be screw threaded in order to screw on the nuts to hold > the fourth floor walkway in place. These threads would probably have been > damaged and rendered unusable as the structure for the fourth floor was > hoisted into position with the rods in place. Havens therefore proposed an > alternate plan in which two separate sets of tie rods would be used: one > connecting the fourth floor walkway to the ceiling, and the other connecting > the second floor walkway to the fourth floor walkway.

> This design change proved fatal. In the original design, the beams of the > fourth floor walkway had to support only the weight of the fourth floor > walkway, with the weight of the second floor walkway supported completely by > the rods. In the revised design, however, the fourth floor beams were > required to support both the fourth floor walkway and the second floor > walkway hanging from it.

[https://en.wikipedia.org/wiki/Hyatt_Regency_walkway_collapse...](https://en.wikipedia.org/wiki/Hyatt_Regency_walkway_collapse#Investigation)

<comment><USER37> There was another problem -- the beams were spec'd as box section, but what was on the shop drawings was 2 channel sections with the flanges welded together. (like this: []). The attachment point that supported twice the design load was also compromised by the weld and less competent section. Whoever was checking the shop drawings didn't pick up on the importance of the change.

IIRC, either change on it's own would have been marginally ok, the two together weren't. (by marginally, I mean probably wouldn't have killed people but wouldn't be to code)

When I was going through civil engineering, there was a big push to use a statistical basis for loads and resistances, rather than using a blanket factor of safety. Loads vary, strengths vary, potentially normally, probably not. But they're described by statistics at any rate. Blunders of this sort aren't, at least on a per project basis.

</comment></comment></comment></comment><comment><USER38> I don't think "should have" is the right statement; more like having at least one person on the team with sysadmin experience is extremely helpful. I know it has been for me.

Of course in HS/college I ran a website that was a frequent target of hate in the late 90s/early 2000s and it taught me about XSS and CSRF before people invented fancy terms to describe them. It also taught me about HTML/JS escaping, DoS/DDoS, SQL injection, and how all your defenses are useless if someone social-engineers their way into root and nukes everything. I have the assumption that everything is compromised and user input is toxic waste burned into my subconscious.

</comment><comment><USER39> I worked for an ISP in the late 90s/early 2000s for about a year before my final year at Uni that when I started had 2,000 customers and by the time I left we'd got to 750k customers.

It was the best training I could have for writing software for the Internet. Just a couple of months ago we noticed one of our core apps was not scaling no matter how many docker containers we had spun up in Mesos. Spent a week breaking it apart using that experience and being able to make changes directly to the code base and being able to talk to devops in a language they understood, and in under 5 days we managed to identify and fix 7-8 different issues.

I would value a dev with sysadmin experience _far_ higher than one without in a tech business with a headcount under 500: it's going to lead to fewer problems and issues in the short and medium term.

</comment><comment><USER40> Anecdotal as it is, I started my career working as a sysadmin while studying CS. When I was younger I was quite interested in security (e.g. I followed defcon and CVE lists and read a lot of manuals and source code). I built server software, broke applications, and did a lot of reverse engineering. In that time I was forced effectively to learn about how the OS worked and how to utilize it, both Linux and Windows, from C APIs to scripting, package management, and everything else needed to effectively work in those environments, primarily around reverse engineering and vulnerability research.

That experience naturally lead to me becoming a sysadmin during my study at university. It was a fairly straightforward application of what I already had learned with a much larger scale of management. The primary thing I gained out of it was a drive to automate everything. When I started that job most of the sysadmin work was manual, but a few of us spent a huge amount of time focusing heavily on automation and when I left most of the work was automated and we were just doing meaningful firefighting and supporting development.

As an engineer, the main benefits have been understanding how my software is going to run in an actual software/hardware stack, easily jumping into a production environment and debugging complicated issues, being able to quickly have my OS do what I want, and that drive to automate everything. A lot of that informs how I build software and in general it feels like it makes me a lot more productive.

<comment><USER41> I have a very similar story as a background. To add to what you said: while maintaining and debugging software installs, I learned a lot about how and when things break. Especially, how important it is to keep things simple. This turned out to be invaluable when I began working as a software developer after graduating.

</comment></comment><comment><USER42> I would change it from "should have sysadmin experience" to "should have operating system knowledge, or ask the right questions".

The example in the article isn't what I tend to see in real life. What I do see are things like:

\- Not knowing about various limits (number of open sockets, or listen queue depth for example), how to know you've hit one, how to deal with it.

\- Not handling various error situations correctly (can't open file due to permissions for example)

\- Security issues. ( socket listening on 0.0.0.0 when localhost would work, for example)

\- Making assumptions about things like "current working directory" or "certain environment variables will be already set for me"

For many of these, including a sysadmin or system knowledgeable architect in the right discussions would suffice.

</comment><comment><USER43> This is a short little article that just barely touches on a much deeper, often hidden issue; The state of system administration in business is abysmal.

It's not the developers fault though, at least not as much as devops types would like you to believe. I think the author has a good point, in that its good to get Devs thinking about real world environments on deploy, but the real world is much more complex than concurrency of servers.

All that being said though, very rarely have I as a sysadmin of 10+ years seen problems so easily attributable to devs. Of course I haven't lived in hn/sv startup land either, so take that into account, but failures in systems I have seen have almost always been a failure of management, up to and beyond C level.

I could go into detail, but I'll save it for another time. Suffice it to say, what businesses need to be doing is getting better CTOs and CIOs who can bridge the gaping chasm between sysadmins and managment.

Devs, you keep being Devs, and let the sysadmins be sysadmins. Cross train and communicate when you can, but don't fool yourselves, it is management that bears the responsibility and burden of you both. Management just doesn't like to admit that to themselves or anyone else, so don't play into this Devs vs sysadmins dialectic too much, lest ye find yourself the scapegoat next go round.

<comment><USER29> I agree that the dialectic shouldn't be played...but not that it's "cross- training." Rather, it's the same skillsets being applied in slightly different ways. There was a day in which "sysadmin" very often meant "shit-hot Perl slinger." It was before my day, but I know some of the graybeards who can still lay claim to it.

The sysadmin who can't write good, maintainable code is going to rapidly see their positions reduced to sinecures in large and slow companies. That may be enough to finish out a career, but I wouldn't bet on it if I was under 50 (and I don't bet on it; I have always, as said elsewhere in this thread, framed myself as "a developer whose output is configured systems" rather than "a sysadmin" for this reason). Similarly, in an age where infrastructure-as-code is becoming the norm, you had _better_ be able to work with it or, as a developer, you are very limited in what you can do without being blessed by somebody else--and the set of environments where that's gonna fly is shrinking, too.

This is emphatically not to take any weight off of the shoulders of management, to be sure. But rather that I feel very strongly that what divide existed between these "disciplines" no longer exists, and both developers _and_ traditional sysadmins need to move to catch up.

<comment><USER43> > The sysadmin who can't write good, maintainable code is going to rapidly see > their positions reduced to sinecures in large and slow companies.

I completely disagree, but respectfully. What I think we are lacking in this discussion is a differentiation between what we mean by sysadmin as a product and as a job description. Systems Administration is a aggregate management of the technical infrastructure. (for $reasons).

In super small technical infrastructure, such as webdev startups, there is very little active sysadmining to do if done properly, but as complexities grow, you need multiple people to perform all the various duties needed to maintain a system. In the performance of these duties, there are various descriptions with varying levels of requirements.

What I postulate is that traditional businesses in seeing the agility of the dotcom and startup culture have been attempting to cut costs in the technical infrastructure, but for the most part because the managers of the technical infrastructure (the sysadmins) haven't had anyone to advocate on their behalf, this has resulted in poor business performance bottlenecks.

I would be curious to hear what other have to say, but in the "shit-hot Perl slinger." days, usually the senior sysadmin was the slinger, and he reported directly to the CFO and CEO, if not ownership.

What I argue is that systems have gained in complexity from a computing infrastructure standpoint such that this old structure no longer worked well, hence the creation of the CTO/CIO class(very similar). The problem is, they aren't doing a good job in my opinion. Therefore, in this current business climate, my argument is that the main problem is we already have good code slinging sysadmins on hand (or can train them), but what businesses need/the industry is lacking, is business/political game aware senior sysadmins to make up for that failing of the C's. Hence I disagree the days of non-code slinging sysadmins are going anywhere. Indeed, some of the best senior sysadmins I know live in meetings, but if they are performing well in fulfilling that role, and not slinging code, but instead making big picture, high-level overview decisions and then monitoring progress, I don't think there is anything wrong with that. Now, if we got CIO/CTOs back on the right track, the political/business game demand could fall and those sysadmins could get back into hard work, (which just happens to sometimes involve code-slinging to solve problems.) Devs are an entirely seperate entity, but still part of the process, in anything other than a pure web startup type businesses with little to no real infrastructure (eg, workfrom home contractors).

Of course I want to qualify this in that this is ancedotal and I admit I haven't seen every environment, but I have seen many environments (as a contractor, seeing more insides than the guy going for retirement), from fortune 500s to 2 man lawyer shops.

Any business that can see this issue coming and address it head on will be far ahead of the game. Those that don't, will one day have very rude wakeup calls as the complexity exponentially increases and they don't have the structures in place the handle the demand, mostly due to lack of foresight/vision.

<comment><USER29> To me, you're just describing another layer of management. Which is fine-- management is important!--but it's not the same thing as the actual implementor class that "sysadmin" usually refers to. And those people are inexorably going to be doing their administration via code. There isn't enough time in the day to waste with "pet" servers. They're cattle. Infrastructure as code is here, and it's just going to grow more extensive. If you can't write code effectively--both in terms of stuff like "hey, this code can be read by other people" and "this code can be automatically tested like any other code"\--you're in _trouble_. One of my clients is a top-10 commercial bank in the United States that has not only moved entirely into the cloud (save for some legacy mainframe stuff that they're working on) and every team is not only required to provision and operate strictly with automated tools (both instance and cloud-level provisioners), but their non-database servers are killed automatically every few months to ensure that they're rotating successfully and without human involvement.

I'm pretty confident that I've seen where we're going with this. The management piece of the puzzle is totally important...but the _practice_ is going to continue converging with every other bit of software development.

</comment></comment></comment></comment><comment><USER44> Software developers should also have customer-facing support experience (supporting other people's code) so that they realise, the hard way, the importance of having extensive logging and debugging capabilities accessible to those unfamiliar with the codebase.

</comment><comment><USER45> Having someone who is cross trained is always better than not, but they will be harder to find and more expensive.

A frontend UI person who can write their own backends is great. A backend developer who knows javascript and can build their own frontend is great.

A coder who manages their own cluster is great too, as is a sysadmin who who can write code.

And a developer who can do customer service, and therefore can fix a problem for every customer at once through modification of the application, is better than just someone who can answer the phone and make the customer happy.

All this is to say that someone cross trained will always add more value and will also cost a commensurate amount.

<comment><USER29> _> All this is to say that someone cross trained will always add more value and will also cost a commensurate amount._

Man...I _wish_ the latter was the case. I ended up going into consulting precisely because of the way that salaried employees are valued in tech right now. I literally _am_ all of those things, I am comfortable and have delivered at a high level mobile, web, backend, and infra projects--but the "market" loves valuing those roles individually, and not the synergistic capability of being able to do all of them.

Consulting is fun, but finding a place where those talents actually are valued would be rad. (Anybody out there: have an opening for somebody who can deliver value at literally every level of your engineering organization while _always_ being game to help bring your other developers up? Email's in my profile. ;) )

<comment><USER45> If you claim to be an expert at more than two, then chances are you're either 1) not really that good at any of them or 2) would make a great founder!

If you're good in multiple areas the only way to really get paid what you're "worth" is to start a company and use your skills to create an amazing product.

Because you're right: the market doesn't value a true polygot.

<comment><USER29> I don't claim to be an _expert_ at anything so wide as a full category of software. Hell, I don't even claim to be an expert at a _language_. (I learned yesterday about Ruby flip-flops, ferchrissake.) Merely (well, "merely") that I can deliver a solid product at any of those levels. =)

As it happens that's the other side benefit of consulting--being able to build a product with downtime. And I am! But it's still a lottery, and risk mitigation is a thing. So I like hailing for interesting offers to come my way.

</comment></comment></comment></comment><comment><USER46> While I agree with the general sentiment, the examples feel fairly contrived.

What I think is more practical is understanding when certain problems fall outside of your own scope or capabilities and when to engage someone else for their advice.

</comment><comment><USER47> Sysadmin or devops is responsible for uptime directly, being ultimately responsible for the services to remain performant. It's a mindset and temperament that may help developers see solutions or foresee pitfalls while designing and implementing code. Zero-downtime attitude. Making and testing deployments, upgrades and migrations to be least impactful as a core function is probably the biggest effect of putting a developer in a sysadmins shoes.

</comment><comment><USER48> What he's talking about is systems architecture not sysadmin experiece.

Of course developers have to write code for a logical model of a machine that isn't necessarily the same as the laptop they're working on. But I don't think anyone would ever dispute that.

Storing pictures on a local disk when the software is supposed to run on a cluster has absolutely nothing to do with sysadmin experience.

<comment><USER49> I came here to say this.

If a developer is building an app for 100K users the same way they'd build an app for 100 people, the systems architect dropped the ball- not the developer.

</comment></comment><comment><USER50> The post has a valid point even if the example is more about system architecture than system administration.

In my experience a developer doing one year in an Operations department gains some insight in what is going to be important in the longest phase of the software life cycle, that is when it's exposed to customers and the company has to react quickly to problems. Proper logging, anything that can help pinpointing the cause of problems and even hot patch them at 3 AM.

The usual lean startup might have little thoughts for that (is it even going to have customers?) but established businesses do.

If it's 6 months to develop and it runs for 10 years (with maintenance and new features), which phase is more important to a company, development or production? I'm a developer which did a couple of years of operations and I've little doubts about it. It's where the company makes the money from.

</comment><comment><USER51> With the rise of devops (docker, puppet, ansible, etc...) , and cloud VMs, how often are sysadmins different from software devs?

<comment><USER52> That question doesn't make any sense at all. That's exactly the same as saying: with the rise of do it yourself home repair, how often are plumbers different from carpenters?

<comment><USER51> Not the same at all. Let me rephrase the question.

Reading the article, the sysadmin is described as the person who constructs and maintains the environment the code runs in, including the physical machines, but also things like dependency management (I infer).

Given that just about everything except the physical machine can be handled through devops automation (from experience), and those things don't require "special" skills other than docker/ansible knowledge, why are sysadmins still considered separate from software devs?

On a more opinionated note, I think that using people, and not code to setup the operating environment of their application is pretty inefficient nowadays.

<comment><USER53> I started as a dev many years ago, then moved into sysadmin / devops stuff for over a decade, and then moved back to more hands on dev stuff again a few years ago. I think your statement that "those things don't require "special" skills other than docker/ansible knowledge" is over-simplifying things a bit.

Writing a production quality Dockerfile or Ansible Playbook requires more than just a bunch of "apt-get install" statements. You need a bunch of knowledge about how each bit of software should be configured and tuned for each individual use-case.

Once it's been written, then the devs can happily automate all the things as much as they like. Writing it in the first place is where a lot of what was traditionally sysadmin knowledge comes into play.

Anyone with a bit of patience and practice can follow a recipe in a cook-book. Not everyone can _write_ one.

<comment><USER51> I'll agree that I definitely oversimplified it. I think what I was getting at was that it was a similar set of skills to other types of software development.

</comment></comment></comment></comment></comment><comment><USER54> UC Berkeley's approach to teaching computer science I believe is the correct approach.

1\. In CS150 one has to build a hardware computer. 2\. In another class one has to build an OS. 3\. In another class one has to build a Database. 4\. In another class one has to build a Network.

Education is key. Understanding the TLB inside the CPU gives one an appreciation for context switching between the only 2 users the CPU has hardwired into it: kernel and user.

Being a sysadmin is insufficient experience as education. Every developer should know how to build a hardware computer from the ground up.

Computer science at Berkeley doesn't have a "Java" class. The language one programs in changes depending on the Professor and topic. That way one doesn't get too attached to a language.

Most developers do not understand the hardware implementation of a thread, or now-a-days a docker. As a result both get used wildly incorrectly.

Threads are useful blocking on IO to keep the CPU from starving. However, switching threads simply to swap CPU bound jobs is inefficient. Just adding a thread doesn't guarantee equal service to users just like adding processes does not.

Developers are using dockers today who have no clue why they are doing so...it's just that everyone is doing it and they do not want to appear stupid.

Experience as a sysadmin is no substitution for education.

</comment><comment><USER55> The unspoken elephant in this thread is that sysadmins and developers have broken relationships surrounded by organizational dysfunction.

Personally, I feel that system administration experience of some sort is essential to development skills, and development experience of some sort is essential to system administration.

Communication and leadership skills are essential to both. It _used_ to be OK if you were the jerk at the office if you were good enough at your job (sysadmin or developer). That's no longer the case because we've progressed well past the point where a single developer or sysadmin can be enough of an asset to look past their ability to work well with others.

Operations needs stability. Developers need to hit moving targets. Sysadmins need to keep things manageable, compatible, and secure. These aren't incompatible goals, and usually when you get the techies talking they are sympathetic towards each other's challenges and helpful to one another. With as many project managers, engineering managers, manager managers, directors, and executives in the mix I wonder if the problems in all these environments isn't more of a historic issue with leadership failings.

</comment><comment><USER56> You do not know pain until you've had to deal with government sysadmins as a contracted developer. And the govt has a strong preference for bringing sysadmins into the civil service rather than contracting that work out (like they used to) these days.

Which means the ones with actual, marketable skills aren't working for the government.

Spending _months_ waiting for a sysadmin to perform a task you _know_ takes about 3 minutes is great fun.

</comment><comment><USER57> I think the problem here is that decent admin jobs are getting harder to come by these days. Used to be any reasonable organization needed an admin just to keep systems straight, now they all rely on less-skilled help desk types and make it up with outsourcing companies, which in turn are shedding personnel and relying on big vendor contracts rather than individual skills.

I predict sysadmin culture and skills will be the next Fortran.

</comment><comment><USER58> The underpinning logic seems false.

To extend the analogy of the house on a slope, if you don't tell the developer you need a house designed to be built on a slope, that is a problem with the spec provided to the developer.

Although it can at times be more efficient for a dev to have sysadmin experience, making blanket statements _requiring_ all developers to come with such a background is just sensationalism.

<comment><USER59> Wholeheartedly agree.

I think the lesson the author is really trying to convey is the importance of learning about _distributed computing_ [1] concepts and not necessarily _system administration_.

[1] [https://en.wikipedia.org/wiki/Distributed_computing](https://en.wikipedia.org/wiki/Distributed_computing)

</comment></comment><comment><USER60> I'm a front-end developer with very little knowledge of sysadmin stuff, but I can't agree more, the best developers I've ever seen had massive sysadmin experience.

Can you point some good resources about sysadmin I could use to improve my knowledge on the subject? Note I'm a noob when it comes to all this server- side magic.

<comment><USER61> When I was getting into ops 10 years ago the book "The Practice of System and Network Administration" by Thomas Limoncelli was a great overview (very high level and not specific) that we all read.

The Phoenix Project by Gene Kim is kind of a more modern uptake on it, but both can be read easily by people outside of the field and I doubt the first one will ever not be relevant.

Both are more about establishing proper ops methodologies. If you're looking for something more low level, like a Programmers guide to sysadmin I can't think of anything off the top of my head.

<comment><USER62> Limoncelli's newer book _The Practice of Cloud System Administration_ is awesome. It's really more about how to build distributed systems to performant and operational.

<comment><USER29> I started writing a book about cloud system administration before I read this one, FWIW. It's worth your time.

</comment></comment></comment></comment><comment><USER63> I've had to work at start ups where I was one of the initial developers (and the most experienced in the team) so I've been forced to learn a lot about Sysadmin and design (and a lot of surrounding dev tools for instance) but all in all - knowing more about sysadmin has made my dev experience change - <USER4> think a lot more about how it's deployed, how it works in prod, how to get data to improve my code and so on.

I do feel all devs at one point should have some sysadmin experience - if not professional, then informally.

That said, sysadmin is strongly divided into 2 types of tools - <USER5> and code.

I love tools that can be coded (as an example - Gulp), over configuration (like Grunt) - as they generally have a lot less undocumented "gotchas" and getting started is generally a lot more simple. (I'm looking at you Webpack!!!)

</comment><comment><USER64> A great reason to have cross-experience at the management level is to understand the different motivations of the two roles: * devs are incented (paid) to introduce changes to the system...mostly in the form of new features. * sysadmins are incented to stabilize the system and make it scale.

In organizations with uncoordinated or siloed management, this leads to infighting... sysadmins try to stop "troublesome" devs from making changes at all and actively slow down changes in the name of stability and developers at the other extreme try to get changes in without caring about stability and blame sysadmins for that.

Great management with experience in both will find ways to incent the teams to introduce regular change while maintaining stability.

</comment><comment><USER65> Wherever I went, I have seen the war between developers and sysadmins being fought. In enterprise the sysadmins have the upper hand, in frontend, mobile, SMB the developers.

Developers don't have the burden of maintaining and upgrading multiple applications on the same server, with conflicting dependencies. 24/7\. Sysadmins don't feel the pressure by end-users to deliver new features yesterday.

At the moment developers are winning because Docker. But when they grab that responsibility they will be the ones called when their 50000 containers are being hacked because vulnerabilities. And they will be under fire for the system being down and that costing the company a lot of money, so everybody breathing down their necks.

</comment><comment><USER66> I think it's more accurate to say that good developers need to have some level of systems expertise which probably means OS & networking fundamentals + practical Linux experience. This is because you need to have some understanding of how your program will execute on (or across) machine(s), and some know-how to get your program up and running on a real machine available to users on a real network. Whether you need to be an expert in the latest best practices for managing a fleet of servers compliant with whatever regulations, that's more for the sys admin (though it never hurts to learn more, there's just so many things we developers could spend our time learning).

</comment><comment><USER67> It's hard for me to "get into" developers who don't know how the system works. I don't mean they have to be full-fledged sysadmins, but it just seems natural to me that they should have a decent understanding of the electronics, how to assemble the hardware, and what's involved in tuning it and keeping it all running.

It may have something to do with my education being EE, but that's not the heart of it. I never practiced EE. I taught myself (more or less in order) building circuits, programming, building PCs from pieces, and elementary system administration. I know there are lots who never "get" all three, but the best I knew did.

</comment><comment><USER68> I get the sentiment. Personally, I could say the same thing about control theory, systems theory, economics, operations research, and statistics. And I could also share dozens of anecdotes of bugs and suboptimal software that was produced as a result of this lack of experience.

But at some point, you have to acknowledge that developers can't know everything. Go deep or go broad, but accept that both paths have their limitations and benefits. Maybe in some cases you'll run into a bug you created due to your lack of sysadmin experience, but don't feel bad about it...your experience has led you to other types of expertise. Fix it, learn from it, and move on.

</comment><comment><USER69> In the case described by the author, it seems like most of the trouble would be resolved by the developer understanding the concept of "distributed file storage is hard" and coordinating with ops early - the dev doesn't necessarily need to know the details of setting that up, but they need enough understanding of the domain to realize there's work to be done on the issue.

For that matter, I suspect most sysadmin / ops teams would be happier if developers could only strip "just" out of their vocabulary (and vice versa) - nothing more annoying than "could you JUST do {thing asker doesn't understand and thinks is trivial}"

</comment><comment><USER70> Actually, I would rather language and tool designers have a go at a true System Admin job. There is a reason PHP gets installed by default and Ruby on Rails does not.

Frankly, at the end of the day, most businesses don't want something that was "working" breaking. All the conflict comes from that one desire. There is a reason System Admins have to do the "Patch Tuesday" dance and despise any software that just updates by itself. Getting yelled at because Chrome updated to a version not supported by your ISV or your local developers is an amazingly fun experience.

Yep, System Admins have to be the company nanny. It sucks, but the apps need to keep working.

<comment><USER71> True for the most part, however PHP is a language while Ruby on Rails is a framework.

<comment><USER70> PHP comes with all the stuff to make a website. Ruby needs a framework, and Ruby on Rails is the most popular. I think you proved my point for me.

</comment></comment></comment><comment><USER72> Just like how you typically distinguish front and back end developers, went wouldn't you distinguish dev from devops?

Sure, it helps if an individual is not entirely myopic, and would be great if that person can wear multiple hats with skillset spanning all those disciplines.

However those different labels exist primarily they align to unique mindsets and solutions to a problem in a mutually exclusive manner.

Do you want a jack of all, but master of non?

Seriously, how many individual out there do you think have the capacity to develop their career so they have deep capabilities for all of those disciplines?

</comment><comment><USER73> I think that there's good advice in this article, but a couple counter-points:

\- Plenty of software developers work for small- to mid-size companies where the Node or Rails server running on your machine is nearly identical to the one running in production.

\- Plenty of software developers work exclusively on the front-end, which means that the computer your code runs on in production is very very similar to the one you develop on (although then you have the added concern of testing on lower-end mobile devices, other browsers, etc)

</comment><comment><USER74> I think everyone would benefit if they learned just a small amount about what their team members have to deal with. For example, project managers should learn a little about what programming involves while managing programmers and graphics+UX designers should learn a little about coding to make their designs more practical (i.e. identifying cases where a slightly better design requires a massive amount more coding effort and probably isn't worth it).

</comment><comment><USER75> I think many professions should have sysadmin knowledge. In my work (biology) there is a lot of unnecessary clicking, not only in Excel (where Python/R would be much more efficient) but also in web interfaces to transfer data daily. The latter is often completely automatable using cron and rsync over ssh. Many repetitive actions are a shell script or command line pipe away. But people don't know.

<comment><USER76> > automatable using cron and rsync over ssh ... shell script or command line > pipe ...

Where I come from, that's called "knowing how to use a computer".

Or at least it _was_ , until Windows et al arrived.

(Interesting how the user-interface technology advance called the GUI, _decreases_ the efficiency of the human/computer interaction.)

Except ... for those who still regularly use a Unix-like OS, that knowledge still is quite basic and, thus, surprisingly difficult to trumpet on a resume.

</comment></comment><comment><USER77> I understand the point here; it can be useful to understand the abstractions on top of which you are working.

But given most engineering projects are crud apps without scaling problems, and PaaS companies like Heroku totally abstract away the system adminstration piece for those projects... I certainly wouldn't recommend a new engineer spend any time learning system administration.

</comment><comment><USER78> I'm moving from dev to devops for a while, because of similar reasons. Excited to learn all these new things and peer beneath all the tools and abstractions I rely on! I have no doubt in my mind that it will make me a better developer - most abstractions have a tendency to leak, eventually.

</comment><comment><USER79> This seems obvious to me. You'll never see your code again with the same attitude after being on call.

</comment><comment><USER80> As a developer with practically no sysadmin skills, how would I go about improving these skills?

<comment> FLUX-YOU Small/micro AWS/Azure instances. Set up your own CI/CD pipeline. Read sysadmin books. Get comfortable with the environment shell (Powershell/bash/etc.) and write some scripts.

Bonus points if you make an actual product that solves a problem and also practice all of these skills.

</comment></comment><comment><USER22> I have been at this since the early nineties and can't comprehend that someone can be a developer and not have any experience with How to set up and administer the environment. I guess it's a normal thing now?

</comment><comment><USER81> Just a small nitpick with the article: it probably meant "backend software developers". Because a graphics developer who mostly writes shader code would surely benefit more from artist experience.

<comment><USER82> It's incredible how apparently 99% of HN are web developers who don't even bother considering that other things exist. It's not at all representative of the larger industry.

<comment><USER43> Very much this. I don't pretend to be a dev, but I listen and work with them to accomplish business goals. Sysadmin work is very much more business management oriented in the real world, but to see so many Devs say "I know how to admin my software in prod therefore I am sysadmin too" just makes me realize just how little the Hn/sv web startup dev knows about real world business systems administration for established business.

Once again, like I said in my other comment, even in web startup land, this is a failure of management, in particular CIOs and CTOs, and the management that selects them.

The problem is that companies can usually manage to survive for quite a while ignoring these problems. There is little short-term feedback mechanism for duct taped systems... Until that day when shit starts dying and no one can fix it. Or a cryptovariant hits the file server. Etc.

</comment></comment></comment><comment><USER83> Careful with that should, or you might take someone's eye out.

</comment><comment><USER84> As a developer the way we resolve this is by including the sysops department as a stakeholder that produces requirements for us to negotiate and implement.

</comment><comment><USER85> The general trend in software development appears to be toward generalization over specialization. I'm not sure this is a good thing.

</comment><comment><USER86> Funny, I always see this from the other side. I always find myself thinking "I wish sysadmins had more programming background."

</comment><comment><USER87> Yes, sysadmin experience will give me vital insight into my work writing embedded code for dishwashers.

Snark aside, if the post was headed "Software developers need to understand the environment where their code will be running or they may not build it properly", which is the point being badly made by conflating "environment" with "some kind of commodity x86 with a standard OS", it would be a lot more applicable.

</comment><comment><USER88> Software developers should have experience in the things they need to get their job done. This is not a revelation.

</comment><comment><USER89> sysadmin really means "technical debt manager" most of the time these days.

I really spend the majority of my time trying to slot something into place in a way that no one will notice that doesn't disturb existing stuff...often that means duplicating bad behavior because its become a dependency.

</comment><comment><USER90> Maybe it's only because I've only ever worked with myself, but it's hard to imagine a software developer who isn't also a sysadmin by necessity. How do you debug things otherwise?

In what city or company is the barrier to entry for software developers so low?

I just can't imagine paying above median income for a software developer who can't install, for an example given in this thread, Visual Studio.

</comment><comment><USER91> One thing I like about DevOps is never having to hear "it works on my machine!"

<comment> yoz-y I always take the "it works on my machine" as a synonym for "you need to explain your problem better". In my experience most problems that "work for me" are caused by somebody installing an unstable build and patching it with random crap they have thought of.

</comment></comment><comment><USER92> I think the reverse is much more true, but maybe nobody argues about that anyway.

</comment><comment><USER93> This needs to be mandatory for developers working on printer software.

</comment><comment><USER94> Shameless plug perhaps, but this is free and will help here: [https://www.cncf.io/event/webinar- cloudnativenetworking](https://www.cncf.io/event/webinar- cloudnativenetworking)

</comment><comment><USER95> I worked in the home office of WalMart Stores, the retailer, from the mid 90s until 2009 as a hybrid network engineer/developer. That is, my team worked in Network Engineering, but we wrote tons of code, because when there are millions of different IP addressable nodes on a centrally managed network, there will be code to manage it. (:

In that era, I think WalMart handled this apparent developer/sysadmin confliact quite well. Believe it or not, even though we (the whole company) had exceptional uptime, we were also extremely agile. LOTS of code was written and rolled out across thousands of sites, multiple times a day.

There were a number of keys to this, and I'll enumerate a few.

1\. Most new hires (those that had minimal previous experience) to development positions had to spend their first six months working at one of the four or so major help desks. Note: they were paid their target salary, but in an hourly fashion. 2\. The various operations teams had complete and final say so over what went out, and how incidents were handled. A couple of the big operational areas were Unix Operations, Network Operations, Windows Operations and Mainframe Operations. I called them 'teams' but each had a number of teams, and their own help desk. 3\. Any time there was an impacting problem associated with a program developed by team X, a war room was called by the relevant operational area. A member of team X (a developer) had to stay in that war room (switching off between team members if it ran very long) until the production problem was not only fixed, but also deemed unlikely to recur, except in cases where that dept of a fix would take a long time. 4\. Every development team had a pager rotation, with rigorous expectations about responding to such pages. This was primarily to support the previous point. 5\. Because of the enormous operational scale, all of the major operational areas had dedicated teams focused on automation. My team was that team for the networking area. Furthermore, most of the rest of the operational folk read/wrote code to some extent.

In short, incentives were aligned. Teams that wrote externally facing code felt pain if the stuff they wrote and released caused problems. Operational folks wrote/managed/interacted with tons of their own code in order to manage the enormous infrastructure. Also, ops folk were far more willing to let things move with velocity knowing that the people who actually wrote the code would be required to support it, globally, any time of the day or night.

Another, perhaps even more important reason we were so successful during those years (and the years before) was a strong and vibrant esprit de corps. The entirety of Information Systems was, at the time, around 2000 people, and we were facilitating double digit year over year growth over a 150 billion dollar company. We had over 5000 remote sites in 15+ countries, with a diversity of software and infrastructure that was honestly pretty astounding. Each of those sites had quite a surprising amount of infrastructure.

We worked hard, and we produced huge velocity with fantastic uptime. For example, the network achieved six 9s of availability a couple of quarters.

In end, while things were sometimes contentious, we trusted each other, and only minority of teams were forced to work bad hours for any length of time.

</comment><comment><USER96> Agreed.

</comment><comment><USER97> Yes, yes. We get it, we should know all things, keep up with all the latest rends, work 80+ hours every week, and do our company's IT support on the side for free while we code their software. Or companies could compensate us for our time and hire more god damn people if they need it. You know, just saying.

</comment><comment><USER98> On top of the 20 years of experience we already require of junior developers?

<comment><USER99> Probably preferable to calling someone a developer after a 12 week bootcamp.

<comment><USER100> Just make sure they're 22, 23 tops. Can't have any of those 30+ year olds with their early onset dementia.

</comment></comment></comment><comment><USER101> As software developers, this should notice and kept in mind that all these experiences are very much important. Thanks for sharing an important article.

</comment><comment><USER102> Managers should have management experience and less nepotism experience. One step at a time, guys.

</comment><comment><USER103> Ironically I can't see the images at the bottom of that page.

</comment><comment><USER104> That's interesting.

</comment><comment><USER105> Sysadmin is getting automated. There is not reason to know it.

</comment><comment><USER106> I have ten years of experience in systems and have written tools software in Python and Javascript for Netflix, Stanford University and Google (current). I have never been a software engineer officially, but I want to be. I am a PC gamer, anime lover, and a proud SJW. Any takers?

<comment><USER107> What are you on about?

<comment><USER106> I am interested in finding positions. Sorry if my post was ambiguous, I know it is a bit out of place here.

</comment></comment></comment><comment><USER108> No. Developers do not necessarily need any experience as a sysadmin and I would advise against moving productive developers into this role for the purpose of misguided training.

The argument about understanding how code scales beyond one computer is fallacious. In fact it's quite easy to learn and practice all kinds of distributed high scale architectures without getting up from a desk.

I will agree it's important to understand the people and work connected to your job. It's probably a good idea to eat lunch with the sysadmin, or QA, or UX person.

Would also agree its important to have respect, empathy, and collaboration others, but it doesn't mean you need to switch jobs.

<comment><USER29> I _strongly_ disagree with this, from experience. There _is no difference_ between a "developer" and a "sysadmin" in a healthy shop in 2016. It's all the same job. Until you actually put something into practice in a hostile environment you have no idea what rakes are going to be in that yard. I would be a bad developer if I did not understand how the systems my code runs on worked. I would be a bad sysadmin if I couldn't marshal my systems through code and fully and completely understand the software that would be run on them. (As it happens, I'm not too shabby at either, and it's additive.)

The mindset that you are describing is why that "devops engineer" that gets hired so very often ends up being burnt at both ends being the savior of the team because, unlike the developers, he or she _does_ understand both sides of the equation and is able to solve the problems that the incurious developer cannot. I would strongly urge that you reconsider.

And, for your own sake, please excise the word "fallacious" from your vocabulary. It will help you, I promise.

<comment><USER108> >There is no difference between a "developer" and a "sysadmin"

Ridiculous. Many developers have specialties that take years to gain proficiency. If there were no difference we could just replace a math PhD doing computer graphics with a sysadmin. Sometimes it's possible, but the blanket statement does not hold up.

>I would be a bad developer if I did not understand how the systems my code runs on work

I agree, but so what? Becoming a sysadmin is not the only way to do that. You shouldn't confuse what you've seen work, with being the only way something can work.

>[your] mindset is why "devops engineers" get hired and very often end up being burnt at both ends

You're reading way too much into this. I never said a sysadmin rotation couldn't be helpful for some. I said devs don't _necessarily_ have to do it to be great at their jobs. I wouldn't pull a dev who was, in the zone so to speak, being highly productive into a sysadmin role misguidedly thinking it would have no impact on delivery.

>[sysadmins are] able to solve the problems that the incurious developer cannot

I never implied I wanted "incurious" devs who couldn't solve problems to the extent they needed a "savior" as you put it. Maybe you are projecting your past experiences onto others based on overzealous inference? I didn't say what you suggested, and I don't believe it.

> please excise the word "fallacious" from your vocabulary

What are you talking about? The definition is "based on a mistaken belief". I claim the argument that devs cannot understand scale out architecture without being a sysadmin is based on a mistaken belief. Why are you so against that word?

<comment><USER29> _> Why are you so against that word?_

Because, to be frank: it makes you sound like a tool. That's what 'greglindahl was referring to in his post, 'cause I said exactly that before I edited it to assume good faith and be nice about it. But leading off your reply with "ridiculous" confirmed that you're okay with that. I'm not gonna play this game with you; I'm having way more interesting and way better conversations in this thread with people who don't wanna dance that dance. Haveaniceday.

<comment><USER108> That's unfortunate it came across that way. I did not intend to do anything other than make a good faith (and maybe passionate) argument.

</comment></comment></comment></comment></comment><comment><USER109> Nope. SD should not be using DB. They should use API to make REST | Microservices, ex FireBase. SD should not be using servers. They should go serverless, deploy static HTML to CDN>

etc. If you are still using docker, puppet or worried about scale/security... you are living in the past. (one ex: [https://github.com/struts3/what- demos](https://github.com/struts3/what-demos) )

<comment><USER110> You're right.But like any big change it will take time.Say 5-10 years to most new stuff being serverless.</comment></comment>


The gray area of charcoal toothpaste - <AUTHOR> https://scienceline.org/2019/04/the-gray-area-of-charcoal-toothpaste/ <comment><USER2> What a coincidence! Curiosity got the best of me this past weekend and I picked up Tom's of Maine with Activated Charcoal. It's very interesting how this ingredient has been in and out of toothpaste for so long yet we don't know how it affects our teeth... I'm beginning to second guess my impulse decision.</comment>


Create simple product tooltips and onboarding tutorials - <AUTHOR> https://inlinemanual.com/blog/simple-product-tooltips-topic-launchers

<comment><AUTHOR> Aiming to get product documentation under control with a backend like github, so it is easily reusable. If you have any questions please do ask.</comment>


Deno 1.4 - <AUTHOR> https://deno.land/posts/v1.4 <comment><USER2> Note that Deno is missing locale support. So for example this:

        (1000).toLocaleString();

will get you "1,000" with Node, but "1000" with Deno:

[https://github.com/denoland/deno/issues/1968](https://github.com/denoland/deno/issues/1968)

<comment><USER3> Ecma-402 support (the 'Intl' object, locale dependent number and date formatting, etc.) requires linking with the ICU library.

We expect to support that in a future release.

<comment><USER4> In the meantime there's stuff like this:

[https://email@example.com/locale](https://email@example.com/locale)

</comment></comment></comment><comment><USER5> What's the killer feature that will lead to node developers switching over in droves?

<comment><USER6> It's just so different, lots of people could claim different killer features. A strong contender for one section of the community might be the first class typescript support. Other people might like the way you can run so much from a single executable.

Anyway, node is a pretty mature platform now, _if_ it dies, it'll die like java-the-language is dying. Very very slowly, and with parts of its platform holding out much longer than other parts. Or alternatively it'll evolve and slowly adopt things from other ecosystems.

I don't think we're at the stage where we can say that a bet on deno is definitely right, but it has a lot of interesting ideas and lots of people will be picking it up where they can.

If you're interested in the difference between node and deno, check out the talk '10 things I regret about Node' [https://www.youtube.com/watch?time_continue=1&v=M3BM9TB-8yA&...](https://www.youtube.com/watch?time_continue=1&v=M3BM9TB-8yA&feature=emb_logo)

<comment><USER7> Since when is java slowly dieing?

<comment><USER8> Java _mindshare_ has been dying for a long time. That said, it's still taught in schools and all sorts of companies still use it (including "cool" ones like Google and Apple).

<comment><USER9> Java mindshare might be slowly shrinking but it is very, very far from dying.

<comment><USER10> I'd ask, what is sustaining java? Where are the replacement techies picking up the torch, where are youngsters seeing java?

I have some answers. It's not a lost cause. Death is not likely, & I do appreciate a lot of java (cdi rocks, microprofile is doing great, performance is good, it has excellent big data tools & many serious pieces of infra are built with it). But how java can retain liveliness, over time, & how the experience & knowledge base continues, is a real challenge. Being on Android helps a lot but also it's a radically different immensely unlike the server side world, with its own elaborate platform specific architectures & libraries. There's not a lot of places java has a hold on UI/ux centered systems, outside Android, so it risks becoming too invisible.

<comment><USER11> When SAP rewrites their stack to something else other than Java, Amazon or Alibaba stop contributing to Java, then I start worrying.

Microsoft decided that Java death is so imminent that they are now an OpenJDK contributor, collaborate with Red-Hat on VSCode Java support, have bought jClarity and Java has first class support on Azure.

<comment><USER12> I guess a Roman, asked circa 300 AD how he feels about the decline of the empire, would have replied similarly.

Java's mindshare just isn't where the exciting things in this field are happening. It's more the domain of overengineered "big data" platforms and clunky enterprise software. Yes, SAP employs a ton of java developers I'm sure, but many devs would rather switch careers than work on anything resembling a crufty ERP.

Thus, the ecosystem stagnates, due to the dead see effect - everyone who could push it forward into new areas has no interest being anywhere near it. And stagnation is a long, drawn out death all the same. It still has a ton of momentum, and millions of outsourcing devs who only know Java, so it will be with us for a while - but make no mistake, the decline started quite a while ago.

<comment><USER11> On my domains only Java or .NET stand a chance to win RFPs, and if anything .NET is the one having troubles, because many are unsure what Microsoft actually wants to do with it, and slowly pissed with the multiple rewrites where there is always something left behind.

I only see RFPs for Java based technology going up.

<comment><USER13> Do you think .NET is currently being badly managed compared to Java? And do you think it will negatively affect .NET's long term future?

<comment><USER11> Kind of, Microsoft seems to trying to fix UWP, while trying to turn .NET into a cross platform runtime that has an eco-system that has been Windows only for the last 20 years.

So while they are doing lots of nice performance improvements, there are plenty of business not so happy that their 20 year investments don't run on .NET Core, and if a rewrite is needed (e.g. WCF vs gRPC) then why not just jump into something else.

Just see the lengthy roadmap, and the considerations that not everyone was happy with "AOT" (packing everything into an exe that unzips on execution).

[https://github.com/dotnet/designs/blob/main/accepted/2020/fo...](https://github.com/dotnet/designs/blob/main/accepted/2020/form- factors.md)

Also check the repositories from Project Reunion, it is pretty much WIP right now.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER14> A large thread about 1.0 from a few months ago: [https://news.ycombinator.com/item?id=23172483](https://news.ycombinator.com/item?id=23172483)

</comment><comment><USER15> Is anyone using Deno at scale or in production yet?

<comment><USER16> I don't think Deno supports SSL yet.

<comment><USER17> Deno not supporting TLS isn't a show stopper, in my opinion. Not many production environments expose Deno (or Node) instances directly, but rather have a reverse proxy sitting in front providing SSL and routing.

<comment><USER18> It's totally a breeze to get https set up with Caddy.

</comment></comment></comment></comment><comment><USER19> I wish the docs for the rust crates got some love. It seems super interesting but its hard to figure out where to start.

<comment><USER20> Out of curiosity, how would you improve them?

<comment><USER19> I just started digging into it a few hours back, so I can't give you a concrete answer. A hello world example for the deno_core to give an idea on how the Ops and ResourceIds should be used. The examples given are not really clear at a glance what they are doing and I had to read through a few of the test cases before I got some idea how JsRuntime should work.

</comment></comment></comment><comment><USER21> I gave Deno a try last night.

There seemed to be some few alternatives to work with databases but I would like to see some more maturity before using this on production.

</comment><comment><USER21> The installation steps do not include cargo, which is another way to get cargo installed if you happen to have rust installed.

<comment><USER22> Rustup is the way to install rust and cargo.

<comment><USER21> What I mean is that you can run:

        cargo install deno

...to install deno. This is not mentioned in the article.

<comment><USER22> Wouldn't recommend it, takes quite a while to build from source.

<comment><USER21> Like 2 min.

</comment></comment></comment></comment></comment><comment><USER23> Can someone explain like I'm five what Deno is and what it's for? Is it for the desktop or web applications? What practical things can you / should you do with it? I just can't get my head around what it does.

<comment><USER24> deno is 'node' shifted by two letters. What do you use nodejs for? That's what you use deno for. It exists in the same space.

So in short, deno is a javascript interpreter suitable for local scripting, server/services, etc. Just like node.

You can probably use it for desktop applications, but it more closely targets server code, again like node.

So how does deno differ from node? It has a different module system and stdlib (better, more modern). It has first-class typescript support. It has much better security controls so that it's possible to use it to interpret untrusted code (this is not security advice).

The 'untrusted code' bit means deno's also gunning a little for the niche lua and other embedded scripting languages occupy, but I don't think it has much of a foothold there yet.

<comment><USER25> Any experience embedding deno in a c++ application? I absolutely lovehow seamlessly lua integrates into my code, but I just find the language itself absolutely unpleasant to work with.

<comment><USER26> Deno uses V8. If you want to embed a JS runtime in C++ then just use V8 directly.

Deno is mostly written in rust.

<comment><USER25> Ah, I hadn't realized that. For some reason I fell under the impression that it was its own JS engine.</comment></comment></comment></comment></comment>


DataScript, a Datomic-Inspired Data Structure for the Browser - <AUTHOR> https://github.com/tonsky/datascript

<comment><AUTHOR> Related: Rich Hickey on Datomic

[https://www.youtube.com/watch?v=Cym4TZwTCNU](https://www.youtube.com/watch?v=Cym4TZwTCNU)</comment>


Risks B2B companies should avoid with in-house sales prospecting - <AUTHOR> http://blog.mobileworks.com/7-risks-b2b-companies-should-avoid-with-in-house-sales-prospecting

<comment><USER3> There are some worthwhile points in there - hiring too many people too early is always a risk and underprepared telesales people can harm your brand - <USER2> also quite a few suspiciously specific figures and claims (I particularly like the "more than _34%_ of the leads you purchase may be unusable" claim, but the assumption the enterprise salesperson isn't doing anything useful to generate business over the "6-18 months to ramp up" is probably the most dubious one).

The validity of most of the claims very much depends on the product, value proposition and industry: there are an awful lot of industries where finding apparently suitable contacts is the easy part, plenty of salespeople whose _strength_ is the network and knowledge that allows them to identify the right prospects at the right time - whether they spend more than 20% of their time doing that or not - and plenty of startups where _knowing who your potential customers are_ is the very last function that should be outsourced.

</comment><comment><USER4> If you're not a native English speaker, like myself, knowing that "SDR" means "Sales Development Representative" actually turns this into quite the interesting read.</comment>


Hundreds of Google accounts suspended for using emoji in live chat - <AUTHOR> https://twitter.com/markiplier/status/1193015864364126208 <comment><USER2> Whoops, looks like this already had a big thread at [https://news.ycombinator.com/item?id=21490674](https://news.ycombinator.com/item?id=21490674). We've merged the comments there.</comment>


Ask HN: Is app.net worth it? - <AUTHOR>

I'm thinking of subscribing to app.net but I'm in two minds. I'm sick of all the junk and advertising on twitter but I'm a little nervous it will be like a ghost town on there and I'm not sure what the tangible benefits would be.<p>Can anyone offer me any advice? <comment><USER2> Looks like it will be ridiculously dead. Would you REALLY do this just because of advertising on Twitter? Who even needs social networking..

<comment><AUTHOR> Can you say that with any authority or is that a guess?

> Who even needs social networking

I've used twitter in the past to find people and make sales. So I guess that would be me.

</comment></comment><comment><USER3> I am a subscriber, purely because I don't want anyone to get my username. This really pisses me off, but at the end of the day, I like being richo everywhere so I'm gunna suck it up, pay $5 a month for a service I don't use and write it off to being a vain fool.

</comment><comment><USER4> I guess you should ask the people who you'd most like to see updates from, and see if they are on App.net. I am on there. I only follow a few people. I hadn't checked it in at least a month until earlier today. I won't be renewing though.

</comment><comment><USER5> I've actually noticed that the low amount of noise (as well as the higher character limit) has caused me to have better conversations than I've ever had on Twitter.

</comment><comment><USER6> I'd be interested to hear from people that actually have accounts to find out if they think it is good.</comment>


Ask HN: Is SearchYC.com gone? - <AUTHOR>

I've noticed it offline the past day or two, I used it pretty extensively to find old threads on here. Any alternatives if it's gone for good? <comment><USER2> I tried it a few times but got better results using Google with site:news.ycombinator.com so I haven't kept up with what's been happening with SearchYC.

<comment><USER3> SearchYC used to take points into account, which is at least one advantage over Google. I don't think it had any special access in the current case, though.

<comment><USER4> It also let you restrict results to top-level submissions, while the common news.ycombinator.com/item url scheme for both submissions and comments makes it hard to keep comments out of Google results.</comment></comment></comment>


Superconductivity at Room Temperature Pressure - <AUTHOR> https://www.thehindu.com/sci-tech/science/finally-iisc-team-confirms-superconductivity-at-room-temperature-pressure/article27246496.ece <comment><AUTHOR> Well yeah. The amount of gold Indians has and they love it it will be making gold more expensive.

</comment><comment><USER2> This is a big deal right?

<comment><USER3> _If true_ , yes, it's a very big deal.

And a very expensive big deal - it's literally made out of gold and silver. People have been critical of the value of gold because "it's not an industrial metal"; well, that could change... if this is true.</comment></comment>