The State of Agile Software in 2018
The State of Agile Software in 2018
On the surface, the world of agile software development is bright,
  since it is now mainstream. But the reality is troubling, because much of
  what is done is faux-agile, disregarding agile's values and principles. The three
  main challenges we should focus on are: fighting the Agile Industrial
  Complex and its habit of imposing process upon teams, raising the importance
  of technical excellence, and organizing our teams around products (rather than
  projects). Despite the problems, the community's great strength is its ability
  to learn and adapt, tackling problems that we original manifesto authors
  didn't imagine.
This is a transcript of my talk at Agile Australia, Melbourne 2018. I did the talk
    off-the-cuff, with just the outline to guide me. I have edited the
    transcript to make the text read less incoherently, while following the
    track of the talk. You can find a video of the talk at
    InfoQ.

How many people have seen me talk here before at Agile Australia?. And you
    came back? Wow, I'm impressed. If you've seen me talk before at Agile
    Australia, or indeed at any conference, you know that pretty much every time I give
    a talk, I call it “Software Design in the 21st Century”, or something like that:
    because it's a vague title and I can talk about whatever I like. And I was
    going to do that again here, but I decided that I'm going to do
    something specific: to talk about where we are: the agile community
    in 2018. I decided I'd do not something that's a particularly deeply
    planned talk with lots of slides and clever diagrams and beautiful transitions
    - but just me yakking. Which I've done before,
    but it's been a while, so we'll see how we go. 

As we look at the state of agile: at a surface level,
    things are looking very good.

As we look at the state of agile: in many ways, on a kind of surface level,
    things are looking very good. I mean, look at the size of this crowd for
    instance. We're huge, we're fitting in this big conference place - actually
    not fitting in very well because it was pretty crowded out there. You go to
    all sorts of places and you see agile scattered around. Somebody sent me a
    Harvard Business Review cover with “agile” on
    it. I mean, it's all over the place. That's a big shift from 10 years
    ago here, or even longer when we were in that ski place at Snowbird talking
    about what the hell we should call ourselves. And it sounds like success, but
    you talk to a lot of old-time agilists, people who were doing it before it was
    called “agile” back in the late 90s, and there's actually a lot of disquiet, a
    lot of disappointment, a lot of unhappiness in the air. 

photo: Agile Australia




That's actually not unusual because it's been like that pretty much the
    whole time as far as I can tell1. And that's
    actually a good thing, because that dissatisfaction is a sign of wanting to
    improve. But it does lead to that sense of: “why are we struggling”?

1: 
      I first remember hearing people complain about agile losing its way in 2005-ish.
    

What's the current
    challenge that we're having to face? Back 10 years ago, the challenge was
    people taking agile at all seriously. I remember going into one of
    the big Thoughtworks clients in Australia. They wanted me to
    give them the usual “Martin Fowler come and give us a talk” kind of routine.
    Somebody from the clients said, “Yeah, we want you to talk about whatever you like,
    but don't say anything about the agile stuff.” Which was a bit of an alarming
    thing in the mid-2000s when I was talking a lot about agile, but that was the
    sense then. A sense that this was something kind-of bad and you don't want to
    talk about it.

Now agile is everywhere, it's popular, but there's been an important shift. It
    was summed up quite nicely by a colleague of mine who said, “In the old days
    when we talked about doing agile, there was always this pushback right from
    the beginning from a client, and that would bring out some important
    conversations that we would have. Now, they say, 'Oh, yeah, we're doing agile
    already', but you go in there and you suddenly find there's some very big
    differences to what we expect to be doing. As Thoughtworks, we like to think
    we're very deeply steeped in agile notions, and yet we're going to
    a company that says, “Yeah, we're doing agile, it's no problem,” and we find a
    very different world to what we expect. 

Our challenge now is dealing with faux-agile

Our challenge at the moment isn't making agile a thing that people want to
    do, it's dealing with what I call faux-agile: agile that's just the name, but
    none of the practices and values in place. Ron Jeffries often refers to it as
    “Dark Agile”, or specifically “Dark Scrum“.
    This is actually even worse than just pretending to do agile, it's actively using
    the name “agile” against the basic principles of what we were trying to do, when we talked
    about doing this kind of work in the late 90s and at Snowbird. 

So
    that's our current battle. It's not about getting agile respectable enough to
    have a crowd like this come to a conference like this, it's realizing that a
    lot of what people are doing and calling agile, just isn't. We have to
    recognize that and fight against it because some people have said, “Oh, we're
    going to 'post-agile', we've got to come up with some new word,” - but that
    doesn't help the fundamental problem. It's the values and principles that
    count and we have to address and keep pushing those forwards and we might as
    well use the same label, but we've got to let people know what it really
    stands for. 2

2: 
      The term I use for terms losing their meaning is Semantic Diffusion - and in
      that post I argue why we should fight against that process rather than
      come up with new terms.
    

If that's the broad level of the problem, how do we focus on
    particular things?  I want to focus on three main problem areas that I think
    are the ones that I would like to highlight. The ones that I think are most
    worthy of our attention. 

Our first problem is dealing with the Agile Industrial Complex

The first one of these is what I would call the Agile Industrial
    Complex. To be fair, I'm part of it, right?  I'm
    standing on the stage here talking about agile, indeed we're
    all part of it to some degree, many of us are parts of some kind of agile
    consulting firms, probably with agile in the title. But a lot of what is being
    pushed is being pushed in a way that, as I said, goes really against a lot of
    our precepts. 

In particular, one of the really central things that
    impressed me very much about the early agile advocates was this realization
    that people are operating at the best when they choose how they want to work.
    

When we talked about the Agile Manifesto
    and laid out the four value statements, with most of those value statements,
    we didn't care very much about what order they came in. But we did have an
    opinion about the first one: which is “Individuals and Interactions over
    Processes and Tools”. To me that crystallized a very important part of what
    agile thinking is about. If you want to succeed in doing software
    development, you need to find good people. You need to find good people that
    work together at a human level, so they can collaborate effectively. The
    choice of what tools they use or what process they should follow is second
    order. I thought that's a very important statement to come from what was
    basically a gathering of process weenies. I mean we were all process guys to
    some extent or other, and yet we were acknowledging that what we were
    talking about was actually of secondary importance. What matters is that the
    team chooses its own path.

A team should not only choose its process, but continue to evolve
    it

It goes further than that for the team should not just choose
    the process that they follow, but they should be actively encouraged to
    continue to evolve it and change it as they go. One of the things about any
    kind of agile processes or agile method is that it is inherently very
    slippery. It changes week to week, month to month. One of the quotes that I
    used to flash around people was “if you're doing Extreme Programming the same
    way as you were doing it a year ago, you're no longer doing Extreme
    Programming”. Because if you don't take charge and you don't alter things to
    fit your circumstance, then you are missing the key part of it. There are
    various rituals and things that we can set up to make this work, and
    retrospectives are
    clearly a technique that lots of people find to be really very
    central. In fact, I think Ron Jeffries joked that Allistair Cockburn's approach
    to agile was “come together in peace and harmony, deliver software every
    week, and do a retro every time to figure out how you can improve”. The
    retrospective is really such a central part of the practice. 

Now, it actually doesn't matter whether you actually have a formal
    retrospective. It doesn't matter whether you have four or five labels of
    things on your retro board, or exactly how you do the retro. What does matter
    is the notion of thinking about what we're doing and how we can do better, and
    it is the team that's doing the work that does this, that is the central
    thing. 

This is a reaction against the whole Frederick Taylor, separate process
    people. How many people here know the story of Frederick Taylor and his
    approach? [a few hands go up] How many people have come across the name Frederick Taylor or even
    heard of it? A few more. A lot more of you should raise your hands. He's
    probably one of the most important figures in the history of the 20th century
    in terms of how he's actually affected people's day to day lives. He was
    from the late 19th century, in America, and he was very interested in
    trying to make people more efficient in the industrial workplaces that were
    developing at that time. His view of the average worker was that they were
    lazy, venal, and stupid. Therefore you didn't want them to decide how
    they should make a particular piece of machinery, but somebody else,
    somebody who was a more intelligent and educated, they should figure out
    exactly the best way to do it. Even going down to: do I do this and
    then that or do I do that and then this. This is a very scripted sense of motion
    and movement. The whole time and motion industry came out of that.
    At the heart of this notion was that the people who are doing the work should not
    decide how to do it. It should be a separate group of planners who does this,
    and that strongly affected manufacturing and factory work through much of the
    early 20th century. 

And it affected the software industry as well - people said,
    “we need software process experts to figure out how to do things, and then
    programmers just fit into the slots”. But
    interestingly, just as software people were talking about how we need to
    kind of follow this very Taylorist notion as the future of software
    development, (I heard people saying that back in the 80s and 90s), the
    manufacturing world was moving away from it. The whole notion of what was
    going on in a lot in manufacturing places was the people doing
    the work need to have much more of a say in this because they actually
    see what's happening. 

The agile movement was part of trying to push that, to try to say,
    “The teams involved in doing the work should decide how it gets done,” because
    let's face it, we're talking about software developers here. People who are
    well paid, well educated, hopefully, well-motivated people, and so they should
    figure out what in their particular case is necessary. 

photo: Agile Australia




“Particular case” becomes important because different kinds of software
    are different. I've lived most of my life in enterprise applications:
    database-backed, GUI/web frontend kind of world. That's what most of the
    people I know do but that's very different to designing telephone switches or
    designing embedded software. Even within the world that I'm
    relatively familiar with, different teams have different kinds of situations,
    we have different legacy environments to coordinate with, and we have
    different dynamics between individuals on a team. With so many differences,
    how can we say there's one way that's going to work for everybody? We can't.
    And yet what I'm hearing so much is the Agile Industrial Complex imposing
    methods upon people, and that to me is an absolute travesty. 

The Agile Industrial Complex imposing methods on people is an
    absolute travesty

I was gonna say “tragedy”, but I think “travesty” is the better word because in
    the end there is no one-size-fits-all in software development. Even the agile
    advocates wouldn't say that agile is necessarily the best thing to use
    everywhere. The point is, the team doing work decides how to do it. That is a
    fundamental agile principle. That even means if the team doesn't want to work in
    an agile way, then agile probably isn't appropriate in that context, and
    [not using agile]
    is the most agile way they can do things in some kind of strangely
    twisted world of logic. 

So that's the first problem: the Agile Industrial Complex
    and this imposition of one-best-way of doing things. That's something we must
    fight against. 

The second problem is the lack of recognition of the
    importance of technical excellence.

The second problem is the lack of recognition of the
    importance of technical excellence to what we do. A lot of agile conferences
    I go to don't tend to talk very much about the actual techniques of
    writing software. How many people here are software developers by the way?
    [a few people raise hands]
    Smattering, but very much a minority. The same problem was true at
    the Agile Alliance's main conference for quite a while, but they realized that
    they were getting a lot of people who were involved in the project management
    side and things of that kind, but not very many people who are the technical
    people who actually did the work. And that's actually quite tragic. It led to
    an even more tragic thing of some software developers saying, “Oh, we need to
    create a whole new world for ourselves. The software craftsmanship movement
    where we can go away, get away from all of these business experts and project
    managers and business analysts, and just talk about our technical stuff.” But
    that's a terrible thing because the whole point of agile is to combine
    across these different areas. The lowliest, juniorest programmer tapping away
    on JavaScript should be connected to people who are out there thinking about
    the business issues and business strategies of the group that they're
    working with. 



And I'll say a bit more about that in my third point, but that means we've
    got to pay attention to these technical skills, and we have to think about how
    do we nurture those, how do they grow, how do we make them important? I've
    spent the last couple of years as my primary writing exercise coming up with a
    new edition of the book about refactoring. How many people here have heard
    about refactoring? [a lot of hands] Good. How many people could accurately describe it to
    people? [less hands] Rather less. It's a very core technique to the whole
    agile way of thinking because it fits in with the whole way in which we can
    build software in a way that it can change easily. When I summarize agile
    to people, I usually say there's two main pieces to it. One, I've already
    talked about, the primacy of the
    team, and the team's choices of how they do things, but the other is our
    ability to change rapidly, to be able to deal with change easily. 

I always loved Mary Poppendieck's phrase, “A late change in requirements is
    a competitive advantage.” But to do that you need software that's designed in
    a way that's able to react to that kind of change. Refactoring is central to
    this because refactoring is a disciplined way of making changes. Now it's
    a horrible thing when somebody tweets something along the lines of:
    “I'm refactoring our software at the moment, so it's going to be broken for
    the next two weeks.” Bzzz - that's not refactoring. Refactoring is small changes,
    each of which keeps everything working. It doesn't change the observable
    behavior of the software, that's its definition. And I should know because I was
    the one who got to define it. 

Refactoring is lots of small changes, none of which change the
    observable part of the software

Refactoring is lots of small changes, none of which change the
    observable part of the software, but all of which change its internal
    structure.
    Usually (you refactor) because you want to make some new feature, and the current internal
    structure doesn't fit very well for that new feature. So you change the
    structure to make the new feature easy to fit in, all the time refactoring and
    not breaking anything, and then you can put it in. Kent Beck summarizes
    this. “When you want to make a change, first, make the change easy.
    (Warning, this may be hard.) Then make the easy change”. That's a very
    crucial way to use refactoring but it goes beyond that because
    refactoring is a constant thing. You're looking at some code and you're
    trying to say, “What does this do? I don't really understand what this does.
    Let me think about this. Let's burrow in. Ah, now I understand what this code
    does.” Then, before you move on, in the words of Ward Cunningham:
    “You take the understanding out of your head and you put it into the code,”: by
    restructuring it, by renaming it. Naming is a really important part of all of this.
    So that the next time you or somebody else comes through that same piece of
    code, you don't have to go through that puzzle exercise. 


for each desired change, make the change easy (warning: this may be
      hard), then make the easy change

-- Kent Beck


Why is this important? Because if you want to make changes, you want to add
    things quickly, you've got to quickly understand which parts of the
    program matter, what do they do, and how do I work so that I can quickly
    make that change. This also burrows up into modularity. If I've got a
    well-modularized piece of software, instead of having to understand the whole
    thing, I can just understand part of it. Technical excellence is about being
    able to build that kind of adaptive software. 

Then a self-reinforcing principle comes in. Once you realize I can change
    software quickly to add new things, I don't try to make the software do
    everything I want it to do right at the beginning. Because I don't need to.
    Because I'll be able to change things later on. This is the principle of
    Yagni - “You aren't gonna need it”. Don't add features to
    the software until you need them because if you do, it bloats the software and
    makes it harder to understand. But that strategy is totally hopeless without
    good refactoring techniques. And then refactoring relies on testing, and
    refactoring also relies on continuous integration, and together with
    continuous integration, you have the practice of continuous delivery and the
    notion that we're gonna be able to release the software very, very frequently.
    And in some organizations, we actually do release the software very, very
    frequently. 



Now, there's a notion out there that
    says that rapid release of software can only be done if you're prepared to put
    up with lots and lots of errors. And if you want software to be reliable, you
    do things in much more slow, deliberate way. Wrong. And we're seeing more and
    more evidence pile up, that this is very, very wrong. My favorite book of the
    year so far, I think it's going to be the best book of the year is
    “Accelerate” by Nicole Forsgren, Jez Humble and Gene Kim. (And I say that with
    great sadness because my book's going to come out this year, but I'm hoping
    for number two) In this book they, through looking at lots of
    surveys of organizations, show how different practices affect
    things. One of the things that they demonstrate is releasing many times a
    day and low defect rates go together. Because you can't release many times a
    day unless you figure out how to get your defect rates down. That requires
    the kinds of practices we talk about: automated testing, refactoring, yagni -
    all of this stuff flows together. The most important thing about Extreme
    Programming is that the individual practices of Extreme Programming reinforce
    each other. 

We should get rid of software projects and use a product-oriented
    view

The third thing that I want to stress is the importance of getting rid of
    software projects as a notion. Instead we want to switch to a product-oriented view of the world where instead
    of projects that you spin up, run for a while and then stop; you instead say,
    “Let's focus on things that are much more long-lasting and organize a product
    team around that.” Another way of thinking about it is: what are the business capabilities that your
    organization has, and then organize the teams around those. These business
    capabilities will be long-lasting and will necessarily mean combining together
    technical people and the people who are on the business side of things into
    the same team. 

 A popular notion at the moment is the Amazon Two Pizza Team - people talk
    about this all the time: “Organize yourself into two-pizza teams.” Maybe with
    a little side joke about how American pizzas are really, really big, so they
    can be bigger teams than you might think. But when they give that
    description, they often miss off something that's clear whenever I
    hear Amazon people talk about this; which is that each of those teams needs to
    connect - right the way through to the customer. Each team is focused on some
    aspect of the customer's experience, some aspect of what Kathy Sierra calls
    making the customer kick ass at what they do. This
    alters our notion of what that small team does because if that small team is
    focused on some piece of customer experience, some way of making the customer
    do what they do better, then that tells us how we should draw lines between
    our small teams. Now, it's not always easy to do this, but it should, I think
    be the driving notion. 

photo: Agile Australia




But again, we often see it violated. I went to one supposedly agile
    organization. I was chatting with a development team. There was about half a
    dozen programmers.  There were only four users of the software that they were writing,
    senior planners in retail planning. Six developers, four users. They had
    never spoken to each other. They weren't allowed to. Instead, there was a
    Scrum product owner who was managing the conversation between them. Well,
    actually, there had been four product owners over the course of a year. I
    mean, what nightmare world of agile is this that they can dare use the word
    “agile” to describe it? When [at Snowbird] we were talking about names for agile
    software development, Kent Beck suggested “conversational”, because there
    should be an ongoing conversation between software developers, business people,
    and anybody involved in making that customer experience better. 

If I was on that team as a developer, I'd want to be on first-name terms with
    all of the users. I would want to be talking. I would want to watch what they
    do, I'd want to understand how they think about making their customers happier
    and see that whole flow. So we need to think about that. That's my
    third challenge.

So my three things, that we should face as challenges: 


Get rid of
      the Agile Industrial Complex and the idea of imposing stuff on teams. Let
      teams work out the way they should work themselves. 

Raise the importance of
      technical excellence, and never forget that when writing software, the technology
      side is really vital, and 

organize around products. 


So that all sounds a little bit negative because I've talked about the
    problems of this current state of agile. I have two minutes and 45 seconds
    left to offer one reason why I'm not too depressed about the situation. It
    comes from probably my favorite thing about the whole Agile Manifesto. This
    didn't occur in Snowbird when we wrote it, but occurred about six months later
    in Tampa, Florida at OOPSLA where most of
    the people who wrote the manifesto were together with a bunch of other
    interested people. At this point, the Agile Manifesto had taken off in a way
    that we could never have imagined. Suddenly it was clear there was this huge
    opportunity for doing interesting things, and many people said, “We
    need to set up a real organization here.” 

The manifesto authors said “no” to a special role in the
    movement's future

One of the questions was: “Should the original 17 people that wrote the
    manifesto take a special place in this ongoing effort?” The thing I'm
    proud of, that we the 17 authors did, is that we said “no”. We wrote the manifesto. We did
    a nice piece of work, we'll be part of what goes on in the future, but we have
    no special role in that future. That role has to grow forwards. We said “new people will
    come in and do great things”, which is indeed what happened. 

One of the ways where that was important was about a big problem with the
    people who wrote the manifesto, particularly looking at it now with 2018
    glasses. 17 people: 17 white, middle-aged guys. 3 Not a thread of diversity
    there, but because we let go, the agile world could incorporate many people
    from all sorts of backgrounds who took part. Mary Poppendieck was one
    of the big leaders of the early agile alliance efforts. Rebecca Parsons, my
    boss at Thoughtworks, was the chair of agile alliance for a long time. I'd go
    to an agile conference, and I tend to see much more women than I do in other
    kinds of software conferences. That's a good thing. Certainly, it wasn't the
    work of us at the beginning. We weren't even thinking about that. But the
    point was: because we let go, we ended up with a community that could develop
    things themselves. That could tackle challenges that we hadn't even imagined
    and work on them. And it's that constant learning and growing and changing as
    a community, which is our greatest strength. 

3: 
      My statement there wasn't entirely accurate. The much missed Mike Beedle was Mexican. I only met him a couple times,
      and my memory did a typical bit of white-guy inattentiveness.
    

because we let go, we ended up with a community that could tackle
    problems we hadn't imagined

I spoke not that long ago to someone who is a latecomer to the agile world,
    who said that they found a degree of welcoming and openness that wasn't true
    of many other groups. That made me really, really happy, because as long
    as we can be that flexible, as long as we can
    always change, then whatever challenges we face, I think we have a future.
    Thank you. 


