Last weekend I attended the ThoughtWorks Australia ‘Team Hug’ – a weekend away for all Australian TWers to get together and share ideas and have some fun. On the Saturday we run a conference program. This year we had so much interest (and quite a few international visitors) so sessions were kept short and punchy at 25 minutes.
My presentation was titled “DevOps – Ten tips for loveless developers”. It’s a collection of ideas for ensuring developers do their best to build a DevOps culture of collaboration. The ideas seem very obvious at a glance, but I find that while we all do some of these things some of the time, we don’t do all of them all of the time.
View more presentations from evanbottcher.
The slides don’t make any sense on their own, here’s a rough outline. I apologise for any agile jargon.
Understand the organisation context that your operations team is operating within, and the incentives that organisation is providing. Understand that acting to prevent change is a perfectly sane response to incentives that reward production stability. Understanding these incentives can remove some frustration and help you focus on overcoming the barriers.
Ensure that the operations is involved early in new development initiatives, making sure ops have representation early in project workshops. Ensure ops are given the full business vision for a programme of change – and that they have input into solution and direction equally with the development team members. Provide plenty of opportunity for ops feedback into the agile planning process. Involve ops in regular design sessions and technical showcases.
Be relentless in the message that ops and dev are one logical team. Ensure that team has a chance to work together and celebrate together – lunch, drinks etc. Ensure ops have invites to project standups, showcases, retrospectives. Go out of your way to help your team members – for example finding quick ways in development to reduce manual workload for sysadmins.
Obvious really. Never lose your dignity by resorting to email carbon-copy escalation wars. Many ops and support teams are monitored and measured using ticketing systems – most of the time we just have to accept this. Ask if you can speak face to face or on the phone, and offer to raise a ticket reflecting the conversation.
Imagine that the operator who gets up at 2am to fix your faulty system is a homocidal maniac who knows where you live. What can you do to ensure that the system is easy to manage, monitor and troubleshoot in production? I’ve written before about cross-functional stories, ensure they are in your project scope and champion those stories to ensure they are appropriately prioritised.
Developers should wear pagers and suffer the consequences of their own crap software (there I’ve said it). I’m not necessarily suggesting handing the keys over to production and letting the developers fix things, however they should be given the same information and feedback that the operator gets when the system falls over. I’ve seen this drive improvements to logging and visibility for ops.
There’s lots more to shared responsibility – many development teams I’ve worked with see nothing of the applications they build beyond QA. Good development teams (especially product teams not project teams) will watch production metrics regularly to inform their technical direction.
Share problems, issues, and proposed designs with your operations team, and ask for help in solving the problems – even if you *think* you already know the answers. Don’t just place an order for servers, networks, and software. Work together to use all of your combined talent and experience.
You are not the first developer to make promises about how much better things will be. Organisations are littered with broken promises, as operational concerns are the first items of scope cut. Don’t over-promise, instead build trust by working together continuously and delivering small incremental improvements.
When things are going well and you’ve built a good team relationship with ops, you might find yourself given extra privileges. You might be able to make a phone call and get a problem resolved, instead of entering a ticket queue. You might even be able to have a login for monitoring access to production systems. Be very careful about abusing this privilege, as you might end up losing your new friendship much more quickly than it took to build.
Take some time to learn some skills, for example a working knowledge of unix is essential for most developers. There’s no excuse at this stage not to have a deeper understanding of your target platform.
Finally some basic politeness still lost on some of my developer friends… say please, thankyou, and when you’ve screwed up say sorry!
Look down. Look up again. You’re on the agile team your team could be like.
It’s the end of the iteration, and there’s a showcase this afternoon (sprint demo if you prefer) demonstrating all the new functionality the team has built in the last two weeks. In the room are members of the project team, the product owner, and various stakeholders and interested parties from the marketing and customer service teams who use the product every day. Everyone’s very excited about the new features, and provide some great feedback on the spot.
This sounds great! But something is missing. Where are the ‘ops’ features?
Very few agile projects I’ve been on will demonstrate the ‘cross-functional’* or ops features they’ve completed in the same showcase, but they SHOULD. Features like monitoring, failover testing, deployment automation, performance improvements – these are all very important to our business. If you’ve truly got a DevOps culture, then they should be showcased and celebrated alongside the new whizzy UI features.
How do we make these achievements relevant to a wider audience? Start by describing the work in a different way – talk about the work that’s being done in terms of its benefit to our business.
Enable monitoring of JVM heap allocation.
By putting the business benefit up front (and always present) this should help make the story more interesting to showcase.
The regular showcase presentation is also an opportunity to report to the stakeholder group on the current state of the system in production. This can take the form of presenting some selected metrics plotted over time. For a website you might include metrics on site traffic, response times, performance and stability over time. The presentation should support the prioritisation of appropriate cross-functional work to improve those metrics over time.
Getting to the point where cross-functional work is celebrated by a wider stakeholder group requires some creativity and effort. When it works I’ve observed it makes the conversations around proper prioritisation and collaboration on DevOps work so much easier.
* I’ve taken to using the term ‘cross-functional requirements’ (thanks to Sarah) to describe requirements that are cross-cutting and not-directly-functional – for example performance, availablity, volume, maintainability. I think the term NFR has become a weasel-word, treated as ‘someone else’s problem’ rather than an important priority. It might just be a word game, but I think it’s useful.