abstract: When large industrial organizations change to (or start with) an agile approach to operations, managers and some employees are supposed to be "agile leaders" often without being given a clear definition of what that comprises when building agile teams. An inductive thematic analysis was used to investigate what 15 appointed leaders actually do and perceive as challenges regarding group dynamics working with an agile approach. *Team maturity*, *Team design*, and *Culture and mindset* were all categories of challenges related to group dynamics that the practitioners face and manage in their work-life that are not explicitly mentioned in the more process-focused agile transformation frameworks. The results suggest that leader mitigation of these three aspects of group dynamics is essential to the success of an agile transformation.
  .
  .
  .
  leadership, agile development processes, qualitative study
author: Lucas Gren; Magdalena Lindman
bibliography: ref.bib
title: What an Agile Leader Does:
  The Group Dynamics Perspective

# Introduction

The reported benefits of an agile approach includes increased customer collaboration, better estimation of tasks, and increased quality , but also higher job satisfaction as well as overall stakeholder satisfaction and, therefore, project success . All of which have contributed to the popularity of the agile approach to development work. Agile development, compared to the plan-driven/waterfall, implies more communication and stronger focus on people, which make the social-psychological aspects very important to understand, including leadership and management .

The research on leadership was focused solely on the leader as an individual for many decades, often referred to as *great person theory of leadership* . The challenge for leaders was then to be categorized as a leader, but once accepted, the leader could probably act as he saw fit to a larger extent (leadership was also categorized as a male trait). The research on leadership and management then shifted from trying to find the best leader to looking at what the best leaders actually do, since it turned out not all the behaviors in one accepted leader were towards effectiveness in general . After that, instead of a focus on finding the best leadership style, a more systematic and situational view of leadership has developed . What is the best leadership depends on the context, and, in some theories, on the maturity level of the co-workers, but it also contains a balance between task- and relation-focused behaviors connected to these phases . The leader role has thereby become more demanding and requires adaptability to context in a way that was not highlighted before. Trying to exert leadership from its context, the way researchers and practitioners have done, is probably a mistake. Schein writes that leadership and culture are two sides of the same coin, and Northhouse also emphasizes the importance of context, where he also models leadership as a collective ability of initiative. Dynamic team leadership is not new in psychological science, and also a property of teams that are in the more mature stages of small group/team development .

Recent studies in software engineering has shown that the definition of agile teams overlap with what is meant by a mature team in social psychology . We define team maturity in this study as the degree to which a team has navigated through the group development stages according to Tuckman . Teams transitioning towards agile ways of working are often in the middle of two paradigms where the more classical hierarchical management structure is in an organizational change process towards new forms of more dynamic and shared team leadership . Spiegler et al.  showed that the leadership function is gradually transferred from the Scrum Master to the team members over time. In more general leadership science, Millikin et al.  showed that self-managing teams have higher productivity even in multi-team settings and that the highly cohesive teams perform the best. In the agile space, e.g. servant leadership is advocated as the foundation for leading, but the definition of what that is remains vague . It is, therefore, still unclear what the behavior and challenges are in relation to group/team dynamics when implementing agile teams from the perspective of an appointed leader.

If the goal of leadership in an agile world is for the teams to eventually lead themselves within their mandate and technical expertise, we can also look at psychological theories on how to lead towards self-organization. We define self-organization in this paper as a state where the initiative, responsibility, and drive towards accomplishing team goals are dynamically shared between many team members in the same team. In past psychology research on self-organizing teams, team design has been found to be more important than team coaching when striving for self-organization . In this context, Wageman defines team design as including all the following eleven design features: (1) real team, (2) clear direction, (3) appropriate size, (4) skill diversity, (5) task interdependence, (6) challenging task objectives, (7) core strategy norms, (8) team excellence recognized/rewarded, (9) information for planning available, (10) training/technical consultation available, and (11) material resources available. We define team design in this paper as the eleven features above but want to stress that a role that is less integrated in the team compared to team members, needs to help design the team with regards to these eleven features. In our experience, agile teams are sometimes set up so that the teams themselves are expected do the design work, which is not what Wageman found to work well. She instead highlights the external leader role in enabling teams to self-organize over time.

Furthermore, Wageman describes the different roles of a leader as first having to be a designer, which includes (1) "setting a direction for the performing unit, design a team task and a team reward system," (2) "making sure the team has the basic material resources it needs to do the work," and (3) "establishing the team's authority over and its responsibility for its performance strategies." Only later can an appointed leader be what is refer to as a midwife that should act at natural breaking-points in the team's further development, which comprises "working with the team to establish appropriate performance goals." These goals should be measurable and specify "how a team will take on its work in ways that fulfill its overall direction." Only in the final step should the leader be a coach. Team coaching is only possible with the other two steps fulfilled and only then will the team make good use of the coaching. She continues to state that "because well-designed teams are robust enough to bounce back from inappropriate leader actions, the leader now has the latitude to unlearn old managerial habits and take the time that is needed to learn effective team coaching skills" .

We define leadership as a function of initiative or group action , and therefore, view all the described behavior as components of leadership. Our research goal is to understand what appointed agile leaders do when building and maintaining agile teams and where that fits into related work on leadership behavior. A qualitative research method allows for a deeper analysis of the complexity of a construct, and allows research participants to speak freely about their reality. Therefore, a qualitative approach is appropriate to study leadership in the context of agile teams. This study aims at investigating how agile leaders at different positions at different industrial development departments interpret the, vaguely defined, concept of agile leadership in relation to group dynamics in their real-world context. The research question is, therefore, **RQ: What are the behaviors and challenges in relation to group/team dynamics when implementing agile teams from the perspective of an appointed leader?**

# Method

This section presents the method we used to analyze the leadership situation in the agile development context.

## Procedure

The participants were obtained indirectly through our industry or research contacts. We asked these contacts to suggest fitting participants which we then contacted by email. A heterogeneous sample was achieved by recruiting people from many different companies, both with an overview of the development part of the organizations and people in new agile roles of newly formed teams. All of the people that we contacted participated. We conducted 45 to 90 minute open-ended interviews, and thirteen of them were conducted using teleconference. First, a personal introduction of the researcher(s) was done including research background, and what the researcher wanted to find out though the overall research project. The interviewer then asked for permission to record the interview and emphasized the anonymity of the data collection. Two interviews were conducted face-to-face and recorded on a mobile device, but were transcribed in the same way. Thirteen of the interviews were conducted by the first author in English, and two were conducted by the second author but in Swedish. The interviews were transcribed verbatim afterwards and Swedish quotes were translated into English.

#### Interview Protocol.

Most of our questions were descriptive in nature, however, some were also contrasting and reflective. The reason why we wanted to be concrete and not ask directly about emotions and interpretations was that we wanted to meet the engineering at their own discourse, i.e. use vocabulary that engineers are used to in their work situation. If the interviewee expressed frustration, emotion or problematized something we asked follow-up questions to prone the person's interpretation and experience around that topic.

The interviews were semi-structured and aimed to answer the research question on what main challenges agile leaders define in connection to group dynamics. We selected participant who saw themselves as leaders in the agile context an did not use any specific role or definition thereof. Examples of questions used to investigate such association were: "What do you think is working/not working with the agile implementation and why?" "Do you see a difference in how high performing teams adopt agile compared to new or less mature teams?" and "How do work processes evolve in agile teams?"

## Participants

The participants were practitioners working with an agile approach, according to themselves, on different levels of organizations, ranging from team Scrum Masters to founders or CEOs. The first thirteen participants were involved in software development and the second two were involved in hardware development complementing the sample since agile has spread to other areas than software development. However, the conclusions drawn from the hardware part should be considered with care since generalizations most likely cannot be drawn from only two participants, i.e. we have yet to see saturation in that data.

```latex
\begin{landscape}
\begin{table*}
\footnotesize
\renewcommand{\arraystretch}{1}
\caption{Company Information.}
\label{fig:companies2}
\begin{tabular}{p{20mm}p{53mm}p{41mm}p{41mm}}
\bfseries  & Interviewees \bfseries & Method used before agile & Reason for agile\\
\hline
Company $A$ & Agile coach  & Waterfall process &More engagement, job satisfaction, and quality.\\
Company $B$ & Project manager lead & Waterfall process & Improved business value.\\
Company $C$  & Project portfolio management responsible & Ad hoc process & Focus on project priorities instead of personal interest. \\
Company $D$ & Project Manager (initiated agile). & Waterfall process & Innovative ways to deliver value. \\
Company $E$  & Interviewee 1: Team leader sales and distribution. Interviewee 2: Lead of ~25 project managers & Waterfall process &Improve the company.\\
Company $F$ & Project manager in project execution & Waterfall process & Had read about agile methods, started a pilot project. \\
Company $G$  & Interviewee 1: Scrum Master/Project Manager. Interviewee 2: Scrum Master of two teams. Interviewee 3: Scrum Master  & Started as agile teams &Realized that the products needed to be developed faster. \\
Company $H$ & Scrum Master/manager in one of the first agile teams  & A culture of guessing what users liked & Better and faster feedback and solve organizational problem.s \\
Company $I$ & A multi-type supporting role & Something very similar to agile methods before they had heard about the concept.      &Started as an agile company.\\
Company $J$ & Founder of an agile company ($>$10 years experience) & A company built on agile principles and values in a very flat organizational structure. & Started as an agile company. \\
Company $K$ & Interviewee 1: Scrum Master driving the agile transformation of the team. Interviewee 2: Certified Scrum Master and developer & Waterfall process &Wanted to adapt to the fast development of new technologies \\
\hline
\end{tabular}
\end{table*}
\end{landscape}
```

Table  provides a brief guide to the variety of cases that we investigated with information about the interviewees, and their organizational situation. The interviewees worked in companies ranging in size from 35 to 56 000 employees, and represented work cultures in seven different countries. Three of the participants where in an environment that started with an agile approach to work from the beginning. However, since they still described the agile way of working in contrast to their own and their colleagues previous ways of working, we opted to analyze all the transcripts in the same way.

## Reflexivity

In accordance with Braun and Clarke , we believe that we "cannot free ourselves of our theoretical and epistemological commitments," thus, we acknowledge our previous knowledge of group dynamics as researchers as well as experience from working in various teams and from leadership. We, therefore, acknowledge that we cannot be completely objective when interpreting the challenges stated by the participants. However, we did not have a preexisting coding scheme and tried to let the participants speak freely about their experiences without us intervening. The guidelines by Dahlberg et al.  were applied in that a researcher must be prepared that data can present things differently than what was initially thought, i.e. we wanted to be surprised as researchers. The intention was to apply intellectual honesty, and thoroughness in reasoning and in view of condition and consequences. We then also want to avoid favoring one's own person, skewed sampling, omission of negative evidence, one-sided maneuvers and wishful thinking . The second author is an expert engineer without previous knowledge in agile methods but a lot of experience from product development, and could therefore provide an eyes-open-wide approach to the challenges studied. The first author is a researcher on the topic of building self-organizing agile teams and is therefore knowledgeable about the agile approach. However, he sees the agile methods as having a potential of positively transforming many parts of an enterprise, but also sees challenges with how it is sometimes implemented in practice. Therefore, the first author had no preconception about what results would be better or worse in conducting this study.

## Analysis

We analyzed the transcripts using the six phases of thematic analysis suggested by Braun and Clarke . The first step was to read the entire transcripts before coding took place. We then consistently focused our analysis on statements regarding challenges of agile leadership in relation to group dynamics in concordance with our research question. The analysis was inductive in nature since we wanted to keep an open mind to emerging themes in the data. The first author began by coding the first thirteen interviews and the second author coded the remaining two. We then cross-checked two transcripts and themes and we discussed and resolved discrepancies. Then, as the third step, different codes were put together into themes and (the fourth step) checked whether we agreed upon which themes fit with the codes. We applied descriptive coding since our research question is in relation to finding separate challenges of group dynamics that the practitioners working with an agile approach need to manage in their daily work. Therefore, connections or hierarchies in the found challenges were not sought for. The fifth step was then to assess the naming of the challenges and, as a final step, connect them to existing literature. Our epistemological approach leans towards a phenomenological view more than the social constructionist one, since we believe the challenges of group dynamics in relation to leadership roles can be described directly. We believe this partly due to the fact that our interviewees deal with group dynamics in practice every workday, that these concept are less emotional compared to other psychological constructs. They were therefore expected to be able to articulate their challenges as agile leaders accordingly .

# Results

The three main leadership challenges found in our data is summarized below. The challenges are shown below together with quotes and a discussion to support the claims.

## Team maturity

When the interviewees compared less and more mature teams, the latter were said to tailor their own agile process based on contextual knowledge. One interviewee saw a strong connection between more mature teams that have met for a longer period of time and how much initiative and responsibility they take for the process and collaboration, implying that their leadership role is easier and more consultative.

> "They reinforce the practices within the teams themselves." \[Project manager and initiator of the agile approach\]

Another indication that the level of team maturity is highly influencing the appointed leader's leadership style in agile teams, is the fact that more mature teams were said completely adopt the agile practices they find useful, while less mature teams need reinforcement of the practices, otherwise they are reluctant to use them. To enforce the practices is then something the leaders describe that they must do for the team.

> "Those are very visible high performing teams self-directive aspects vs. those needing reinforcement of the practices for them to be there." \[Project manager and initiator of the agile approach\]

A key seems to be to suggest best practices by the leaders for the team, have a minimum of what is allowed, and then let the teams tailor their process themselves. In such a way, less mature teams will resort to safety by adopting a predefined process but can then redefine their process as the team matures.

> "We really try to get the teams to focus on staying within the framework, but they have latitude and liberty within that framework to, based on their own team style or team makeup." \[Project Management Leader\]

Another interviewee clearly stated that even the most self-organized teams were different initially. The "agility" simply had to wait for the team to mature, which means that the needed leadership style is different across time.

> "Yeah, these days they don't need me in order to work. These days I am really a facilitator and the team is absolutely able to the normal Scrum process without me. They don't need any guidance any longer so I can easily go on vacations for 2-3 weeks, that's not a problem. /.../ Of course in the beginning I had to stop them in the dailies and say 'Stop discussing solutions,' just the tasks, please, and the three questions bla bla bla. Now it's more or less routine." \[Scrum Master\]

This indicates that the self-organization of teams emerges over time along with the team maturity from a psychological perspective, which also implies that the leaders must take on the function of leadership initially before that function can be shared. In addition, the built in flexibility of the agile processes is also something teams need maturity in in order to leverage in the intended way. One interviewee stated that more mature teams can easily change their process if asked, which is not something the less mature teams could do in the same way.

> "We are trying to adopt Kanban. But there are other teams here that have only worked with Kanban for a few months. They tried it because they saw some problems we had with Scrum. Some of the teams matured faster (mainly because they didn't break up the teams all the time as we do here) and they changed to Kanban." \[Scrum Master\]

For teams that are mature, and where the members are dedicated and have set clear roles, the interviewees saw that they could adopt self-organization and team agility without many issues, putting less focus and dependence on them as the designated leader or manager.

> "The team we have is an extraordinary team with a very open mindset and a very innovative team and always open to new things, so they were very open-minded so that was not much to say, and they trusted me." \[Project Manager Lead\]

To summarize, the first found challenge is that the agile leader needs to take a step back from mature teams and instead facilitate the team's work process in relation to the surrounding ecosystems of the teams. However, in order to implement agility in the less mature or newly formed teams, they also need to provide a lot more direction and guidance in order for such teams to become self-organizing agile teams.

## Team design

The team design process is very much connected to team maturity as described in the previous section. However, the following quotes are in this category because they are a symptom of a lack of team design from an external role, not a symptom of the team being new.

When teams were not given an initial structure, some interviewees were surprised and frustrated when newly formed teams expressed a need for clear and directive leadership.

> "It was the first retrospective that we had, they say that they lack some leadership there and then during the retrospective we were talking like 'no it's not leadership that you need, how about that you decide how you will do things and the new habits that we will create in the next sprint.' What kind of agreements that we have to have for solving that problem, and we're saying that the lack of leadership is the result. That's one thing that appears a lot because we are growing and new teams are being formed." \[Founder of an agile company\]

New team were also described as being open to any work practices and lack the insight into what is useful or needed in their context. This entails that the leader needs to step in and guide the team in making such decision, something that was described frustrating for the leaders since they did not expect that in the agile approach. However, the team must get help in its initial design since no team members can know the context simply because it is new to them.

> "Yeah, for teams that are younger, like formed more recently, they tend to be more open to all the practices, but they don't have the experience to decide which practices that would be the best." \[Founder of an agile company\]

Also in relation to new teams, one interviewee changed the agile practice of volunteering for task because the team was not ready to take on that responsibility on a team level. This implies that the leader felt the need to step in and be more directive, however, with the expressed frustration that such a leader behavior is not appropriate in the agile context. Teams, though, seem to need help in designing work processes initially.

> "In agile people should volunteer for tasks, but in most cases here we are obliged to… we do task assignment, by me or the person who already worked on this item takes this item. I know that this isn't a good practice in agile, but we do it for our team and for more productivity, but also for responsibility." \[Scrum Master\]

An interviewee from hardware development also highlighted that, since the company is expanding all the time and therefore consistently gets more team members, it is difficult to design and build self-organizing teams, which is a core part of the agile approach and frustrating for them as leaders.

> "You have to be very involved to be able to get a clear direction from it, and that's hard, especially because our group has grown so much and we get new people all the time who don't have that direction from the beginning. /.../ The further the team gets, and the more you have worked, the more autonomous the group becomes." \[Scrum Master and Software Developer\]

We also found support for the distinction between what is an organizational and enforced structure and what is up to the teams themselves, i.e. the right balance in team design. Teams with no provided structure were described as much less effective but it is about providing the right balance of flexibility and control by the people in leadership positions.

> "The teams can change as they want \[in the process\] all the time." \[Supporting role\]

Traditionally, it seems like most software development processes did not have any team reflection sessions by default. The agile process often adds the retrospective meeting, which is a structure for team reflection, which was described as helpful for the leaders in order to improve the teamwork.

> "The developers, I think, also feel that it provides them with a preset structure within which they can communicate with each other; they don't have to set up a meeting to do this. We have our Scrum identify that they need to meet to talk about something, and then they do. So it puts things in place for them and they don't have to think about it." \[Project Management Leader\]

To summarize, the leaders need to help transitioning teams to design their new agile structure and ways of working. If teams are not ready to tailor their own process, the leaders provide suggestions and best practices, but keep stating that the team should continuously improve their process based on what they learn about their ecosystem. This means that, when the teams are set up, they need an initial predefined team design and then be given the possibility to tweak their processes when the team is ready.

## Culture and mindset

The third and final challenge was that the leadership also needs to be adjusted to the existing more traditional structure and culture of the company. We found that even if members and teams can adopt self-organization, the context might hamper this way of working.

> "If the whole company was agile in the end, the teams could be more independent and talk to the business area more themselves, but now I spend a lot of time trying to make decisions on what the teams should do." \[Scrum Master\]

All the interviewees spoke about the different agile roles changing dynamically based on what is really needed at a point in time. A project manager acts as a Scrum Master, or trying to help people to not fall into old patterns of behavior in relation to the old structure instead of the new agile process.

> "After the first project I was able to spend time educating them at the beginning, but had to make almost daily conscious efforts at reminding them, or educating them with when I could see that their though process was tending toward waterfall. So I'll try to point out: 'oh actually, let's think about it this way.' Or, you know, helping them with definitions like what the basic function of the daily Scrum for example and remind them of that. That it's not a status meeting, for example." \[Scrum Master and Project Manager\]

One participant from the hardware-focused development also highlighted the fact that all individuals do not have the same possibilities, or motivation, to adapt to the new ways of working. The participants from hardware and software development differed in that the hardware-focused interviewees focused on the "old" ways of working as compared to the new agile way, while the software-focused participants reflected more on the team's place in the company as a whole.

> "Partially, I believe it's due to... from what I've heard, that \[a team member\] that is a bit more senior and has had previously bad experience from the agile ways of working in other areas and therefore doesn't think it worked well and actually only sees the negative aspects. \[The team member\] just does this because everybody else wants to do it, but does not think it will work. And therefore it does not fully work, because you... it's hard. I think it's really hard to have one leg in it, and one outside, you know..." \[Scrum Master driving the agile transformation\]

From the hardware-focused participants, they described their agile implementation as something that needed to be agile in itself. That means that they had to related their new process to also fit with the old, which was described as challenging.

> "We have tried to see the work process as something agile and adaptable too! Now there are more clear toolboxes with how to work in an agile way in the company. When we started that didn't exist, but, I mean that, when you talk about Scrum as an agile method it's not anything more than a toolbox with different practices, and you have to try to pick what suits your organization and your... we started with that mindset that we would have 'what is the smallest part of scrum we can pick?' or 'what is the smallest set of tools we can make use of?' and then, I guess, with time we have realized that, yes, 'we need this' or 'we need do do that one too'. \[Scrum Master driving the agile transformation\]

To summarize, the participants' leadership style is also adapted to where the company is in its agile journey, not just the internal process structure, as in the previous theme. The leaders act as both more traditional managers and as more agile (i.e. contextually adapted) leaders depending on what is needed at that point in time.

# Discussion

This study found that the main challenges in relation to group dynamics in the agile context is to adapt the leadership to the (1) teams' collaborative maturity, (2) design new teams well, and (3) balance the "old" ways of working with the intended new agile processes and their innate different culture. The first one, called (1) *Team maturity* included that the agile leaders saw a need to step back from mature teams and then be the facilitator with a strong focus on impediments external to the team. However, as agile leaders, the challenge from their perspective was that newly formed, or less mature teams, on the contrary, needed a lot of support in order to grow into agile teams. This was not seen a "agile" by the interviewees and expressed as frustrating.

The second category (2) *Team design* is connected to the first category but focuses on that teams need to be well designed in the agile context and get much more help setting up the agile team than the interviewees initially had thought. Teams do not seem to be able to design themselves when starting their agile journey, but instead need a suggested initial team design. New teams, that are less mature by definition, cannot tailor their own agile process, but need help to get started and can then improve their process based on what they learn about their ecosystem. The third category (3) *Culture and mindset* is about that the needed adaptability of agile teams are also in relation to where the whole company is in its overall agile journey. Agile leaders described that they acted as more traditional managers sometimes, if needed, but also adapted to where in the company teams were more allowed to drive themselves within their predefined mandate.

That team maturity is connected to team agility has been shown in previous studies (e.g. Gren et al. ). The third category also confirms previous results on the difficulty with integrating an agile approach to a traditional context (e.g. ). The second category, though, on the importance of team design, as conducted by a leader more external to the team, seems to be a novel finding. Even with regards to the other two categories, this study adds to knowledge since it focuses on the appointed agile leaders and what challenges they define in relation to building agile teams. More immature agile teams were said to be more open to all the practices, and that agile leaders need to provide a clear agile work process to the teams for them to get started in a constructive way. While this is not according to the description of an agile and self-organizing team, this is a typical trait of groups in the Forming stage according to group development . Team members are focused on dependency and inclusion when newly formed, and need to build some initial trust before sharing opinions and questioning each other. Giving new teams a structure and directives would therefore help them in their development as compared to trying to get them to self-organize too early. In the results of this study, we clearly see that the agile leaders describe a need for more guidance of newly formed agile teams as well. Not only do teams need more guidance in their internal collaboration, but just like Wageman concluded in her studies on self-organizing teams in general, someone needs to design the teams well also before they are deployed. This aspect seems to be largely ignored in the agile literature. After the team is set up, the role of the leader reassembles the leadership needed in different group development stages, which verifies their importance and clarified the connection between the two concepts. This study has shown that this temporal perspective of needed leadership when creating self-organizing teams is also essential when setting up agile teams, and Wageman's theory fits the result of this study exceptionally well. *If we aim for having only a facilitating and coaching leadership from day one, we hamper the teams' development instead of the opposite.*

The interviewees also described the more mature agile teams as aware of their context, e.g. surrounding teams and overall company strategies, and having the ability to tailor their own process in relation to responding to a change in requirements. This is at the very core of agility , and shows that self-organization is a property of collaboratively mature teams. In contrast, one interviewee thought that following the agile process was bad and thus that an agile process was inflexible. Adapting agile methods to a large organization was described as a balance between old and the new work methods, and therefore, there is a large risk of adapting the old organization to the new agile structure and by this not perform an actual agile transformation but a renaming of the existing structure. One example would be to keep a command-and-control approach by the line management instead of providing a structure for teams to grow and eventually self-organize, i.e. letting team members lead when they are ready to take on and share the leadership function more. This challenge was more stressed by our two participants from hardware development. A key to create agile teams, as described by our interviewees, seems to be to suggest good work structures for teams and then let them adjust their process as the development cycle moves along. This is a tricky balance for new agile leaders since new teams need more structure but the appointed leaders then need to take a step back and let the team self-organize when ready. Therefore, a mix of more directive leadership styles in combination with consultative and coaching leadership styles, seems to be what the successful practitioners do in practice. This is not then to be confused with traditional line management control, but instead the application of different types of support for teams guided by the end-goal of the team leading itself.

The guidelines from the agile community have been to be a facilitator of the process only instead of being a more directive leader . The problem with such simplified guidance to new leaders is that such behavior only works in very special cases. With a mature team, and with good organizational support, taking a step back, delegate and be a process facilitator is easy. The problem is that becoming a leader in such a context immediately is pretty rare. There is an awareness in practice that inexperienced teams need more guidance, but this awareness is only in relation to agile practices and not group dynamics over time. Therefore, that explanation model of agility fails to explain why some teams do not become agile over time even if they learn the agile practices. An understanding of group developmental psychology provides that explanation , and explains why different leadership styles and team support are needed at the different stages. And maybe even more novel in this current study, teams need to be designed well before they are deployed if they are to self-organize as fast as possible . This motivates the future inclusion of more group dynamics teaching in agile courses both at companies and at universities.

# Threats to validity

Since leadership is a complex construct, a qualitative method was suggested to provide deeper insights of the reality. However, with fewer participants is becomes more difficult to generalize to a larger population, which should be done with care for our study. The selection of a large variety of interviewees from various organizational settings in the sample, is one major strength of the study, but conclusions from our two participants from hardware development should of course be drawn with even more care and only seen a small additional comparison. The diverse sample with regards to companies provides a broad view of experiences from practitioners working with an agile approach in different stages of its implementation. A high risk in qualitative studies is that our interpretations of what the interviewees said could of course be erroneous and prone to confirmation bias. A way to counter such a threat was to be transparent in the description of our research method, and to be transparent with what quotes we believe supported our claims. In the analysis, we acknowledged our previous knowledge of group dynamics, which many of our participants did not have. Our participants could have a different perception of the constructs under investigation, but as researchers from different areas, one with and one without previous knowledge of agile but both with rigorous experience of leadership, we believe that this combination was advantageous for the analysis. For validation of interpretations and citations, feedback from the interviewees might have improved the quality of the analysis. We recommend that procedure for upcoming studies in the area. We also acknowledge that having a mix of two languages (English and Swedish), and conducting interviews in English with non-native speakers also threatens the validity of this study due to difficulty in obtaining exact translations between languages.

# Conclusion and Future Work

This paper set out to investigate what challenges appointed agile leaders see in relation to group dynamics. Through a qualitative method of interviews and a thematic analysis, we have found that leader adaptability to team maturity, the careful design of new teams, and a continuous balance between traditional and new work principles are all essential to the success of an agile transformation. These findings are an important contribution to both research and practice since it gives an in-depth view of leadership challenges of group dynamics in agile transformations at different scales. In terms of future research, we particularly suggest more use of qualitative research methods both when studying leadership in engineering, but also to apply narrative analysis to large agile transformations, since analyzing the complete transition from a waterfall process to an agile approach has not been done in its entirety. We believe the narrative analysis method is underused in software engineering research and fits well to the study of organizational changes over time.

# Acknowledgment

We would like to thank all the participants and everyone who helped us with making this study possible, and we would like to acknowledge Jennifer Strand and Petra Boström specifically for their excellent support.


abstract: Saturn's rings; astrophysical discs; kinetic theory; complex fluids Maxwell's investigations into the stability of Saturn's rings provide one of the earliest analyses of the dynamics of astrophysical discs. Current research in planetary rings extends Maxwell's kinetic theory to treat dense granular gases of particles undergoing moderately frequent inelastic collisions. Rather than disrupting the rings, local instabilities may be responsible for generating their irregular radial structure. Accretion discs around black holes or compact stars consist of a plasma permeated by a tangled magnetic field and may be compared with laboratory fluids through an analogy that connects Maxwell's researches in electromagnetism and viscoelasticity. A common theme in this work is the appearance of a complex fluid with a dynamical constitutive equation relating the stress in the medium to the history of its deformation.
author: Gordon I. Ogilvie
title: James Clerk Maxwell and the
  dynamics of astrophysical discs

# Maxwell 150 years ago

In 1856, the year that James Clerk Maxwell left Trinity College, Cambridge to become Professor of Natural Philosophy at Marischal College, Aberdeen, he was also writing his Adams Prize essay *On the stability of the motion of Saturn's rings*. The prize, named after the astronomer John Couch Adams who predicted the existence of Neptune, was awarded every two years by St John's College and the University of Cambridge for a mathematical essay on a specified topic. Among the examiners for the 1856 essay with its planetary theme were James Challis and William Thomson, later Lord Kelvin. Challis was the Plumian Professor of Astronomy whose lack of interest in Adams's prediction led him famously to fail to discover Neptune.

Such was the perceived difficulty of the chosen subject that Maxwell's was the only entry in the competition. The essay topic required Maxwell to investigate the stability of various configurations of solid, liquid and particulate rings orbiting around Saturn. Laplace had already shown that a uniform solid ring would be unstable, and suggested that the rings were solid but of non-uniform density. Maxwell's essay is interesting partly for its use of Fourier analysis and dispersion relations in the determination of stability criteria. For example, he considered the behaviour of a circular ring of equally spaced identical satellites under the destabilizing influence of their mutual gravitational attraction. He did this by introducing a perturbation in the form of a displacement that was a periodic function of the azimuthal angle $\phi$ and would depend on time through a complex exponential function, being proportional (in a modern notation) to $\exp(\ri m\phi+s t)$. An algebraic dispersion relation followed, which gave the growth rate $s$ in terms of the wavenumber $m$. If a solution existed with a positive real part for any suitable value of $m$ he deduced that the configuration was unstable and therefore untenable as a model of Saturn's rings. Maxwell's conclusion was that "the only system of rings which can exist is one composed of an indefinite number of unconnected particles, revolving around the planet with different velocities according to their respective distances". The following year he expressed himself somewhat more poetically in a letter to Thomson, describing the rings as "a great stratum of rubbish jostling and jumbling round Saturn without hope of rest or agreement in itself".

The published version of Maxwell's essay (Maxwell 1859) includes the results of work done in 1857 after the competition was closed. Brush, Everitt & Garber (1983) provide a comprehensive historical discussion together with reprints of all the relevant documents. A detailed scientific critique of Maxwell's essay has been given by Cook & Franklin (1964).

# Modern view of Saturn's rings

The modern view of Saturn's rings (*e.g.* Esposito 2006 and references therein) benefits, of course, from space exploration, notably the *Voyager* encounters of the early 1980s and the current *Cassini* mission. The principal rings of Saturn are the outer A ring and the inner B ring, separated by the 4500-km Cassini division. Much of the structure in the A ring can be attributed to its gravitational interaction with some of the moons of Saturn. The narrow Encke and Keeler gaps are cleared by moonlets orbiting within the A ring, which also generate trailing wakes and wavy edges. Also seen in the A ring are many discrete orbital resonances with more massive moons orbiting outside the ring system; at these locations spiral density or bending waves are launched, which propagate some distance towards or away from Saturn before being damped. The B ring, however, contains a rich irregular radial structure that cannot be attributed to satellite resonances, and which remains one of the major puzzles of ring dynamics (*e.g.* Tremaine 2003).

Although even *Cassini* cannot resolve individual ring particles, modern observations concur with Maxwell's conclusion regarding the constitution of the rings. The system consists of a dense 'granular gas' of iceballs up to several metres in diameter, the size distribution being inferred from radio occultation experiments. Ring particles are in circular orbital motion in Saturn's equatorial plane, at velocities on the order of $10~\mathrm{km}\,\mathrm{s}^{-1}$. In addition they have random velocities on the order of $1~\mathrm{mm}\,\mathrm{s}^{-1}$. The tiny ratio of $10^{-7}$ is therefore characteristic of the orbital eccentricities and inclinations of ring particles, and also of the angular semithickness of the rings.

Owing to their velocity dispersion, ring particles experience collisions that are very gentle but nevertheless significantly inelastic because of the nature of the material. Particles undergo a few collisions per orbit, placing the system in an interesting but challenging physical regime. It is different on the one hand from the situation in stellar dynamics, where the probability of a star undergoing a significant scattering event with another star within one orbit around a galaxy is typically very small. It differs on the other hand from gas dynamics, where molecules usually experience very many elastic collisions within the time-scales characteristic of the macroscopic flow and therefore establish a locally Maxwellian velocity distribution. The implication is that the rings do behave as a continuous medium similar to a fluid, but a complex or non-Newtonian one with a non-trivial rheology that needs to be explored. The velocity distribution can be significantly anisotropic.

Gravitational forces between ring particles are not negligible because the escape speed from the surface of a typical particle may be comparable to the velocity dispersion. As well as affecting the collisional dynamics to some extent, self-gravity as a collective effect makes the rings thinner and is especially important in the propagation of waves.

An important aspect of ring dynamics is the establishment of an energy equilibrium. Owing to Kepler's third law, the rings undergo systematic shear, with the inner parts rotating more rapidly than the outer. When particles collide their relative velocity results partly from their random velocities and partly from the systematic shear. Velocity dispersion can therefore be generated from shear in the same way that viscosity generates heat in a shearing fluid. On the other hand the inelasticity of the collisions diminishes the random velocities. The velocity dispersion is therefore set by the restitution law of the material. Experiments at UC Santa Cruz have determined how the coefficient of restitution of ice at low temperatures and pressures decreases with increasing collision speed (*e.g.* Hatzes, Bridges & Lin 1988). An energy equilibrium in the rings can be found when the typical collision speed is such as to imply a critical amount of inelasticity (Goldreich & Tremaine 1978). If the coefficient of restitution is always too small to allow this solution, another equilibrium may be found in which the velocity dispersion is comparable to the shear velocity across a particle diameter.

There are essentially two possible approaches to the theoretical study of Saturn's rings, $N$-body simulations and statistical theories, each of which has its merits and limitations. Direct $N$-body simulations (*e.g.* Salo, Schmidt & Spahn 2001) have the character of virtual laboratory experiments, and the accuracy of the model is constrained mainly by our uncertainty about the physical properties of the particles. An obvious limitation of this approach, however, lies in the limited scale of the simulations. For example, the number of metre-sized particles in Saturn's B ring is on the order of $10^{15}$, whereas a large-scale numerical simulation involves about $10^5$ particles. Simulations are therefore restricted to a tiny patch of a ring, corresponding to length-scales that are only just observable with *Cassini*; they are also limited to durations very much shorter than the global evolutionary time-scale of the ring. The extreme thinness of the rings is an obstacle to global simulations, because of the great ranges of length and time-scales involved. In order to understand the very direct and detailed information provided by simulations and to apply it to phenomena on larger scales or in different circumstances, the results need to be assimilated within a theoretical framework of some kind. Therefore it is important to develop analytical models in parallel with advances in computation.

A statistical approach to planetary ring dynamics is natural in view of the very large number of particles and the desire to formulate a continuum description of some kind. Nevertheless, the difficulties involved are severe. Such work builds on the foundations of kinetic theory laid by Maxwell. Indeed, the manuscripts held in Cambridge University Library (reprinted in Brush *et al.* 1983) show that Maxwell clearly conceived the idea of the energy equilibrium in Saturn's rings. He also started to develop a kinetic theory of the rings, but it proved much more difficult than the theory of gases because of the inelastic nature of the collisions. Their relative infrequency also means that the velocity distribution is significantly anisotropic and therefore non-Maxwellian.

# Kinetic theory of Saturn's rings

The aim of a statistical approach is to derive a continuum mechanical model of the rings, consisting of a set of partial differential equations related to, but more complicated than, the Navier–Stokes equations of fluid dynamics. To do this one may start from Enskog's equation governing the collisional evolution of the distribution function $f(\bm{x},\bm{v},t)$, which gives the number density of particles in the six-dimensional position–velocity phase space. This is similar to Boltzmann's equation in gas dynamics but allows for the dense nature of the medium. It is also necessary to modify Enskog's equation to allow for the inelasticity of the collisions (Jenkins & Richman 1985; Araki & Tremaine 1986).

In order to make analytical progress, it is supposed (in common with most $N$-body simulations) that the particles are identical, smooth, indestructible spheres. Collisions are assumed to involve only two particles at a time, and gravitational scattering or focusing is neglected.

The description can be reduced from six dimensions to three by taking velocity moments of Enskog's equation. The zeroth and first moments generate equations for the conservation of mass and momentum, while the second moment generates a dynamical equation for the kinetic stress or velocity dispersion tensor $\mathbf{W}$ (equation  below). This can be regarded, in part, as the constitutive equation for the complex fluid. Rather than relating the stress instantaneously and locally to the rate of strain, as in a Newtonian viscous fluid, the dynamical constitutive equation accounts for the fact that the medium has a finite 'memory' of its deformation history. To close the moment hierarchy it is necessary to neglect, or otherwise model, the third moments that appear. The collision term in the second-moment equation can be evaluated by modelling the distribution function as a triaxial Gaussian. Experience suggests that this relatively crude method of treating the velocity dimensions in Enskog's equation can yield quite accurate results because of the smoothing effect of the integral operator in the collision term.

In addition to the kinetic stress there is also a nonlocal, or collisional, stress requiring further integrals. This contribution originates from the direct transfer of momentum between particles during a collision, rather than the carriage of momentum by particles between collisions, and can be dominant in a dense gas.

The result of this procedure is a closed system of nonlinear partial differential equations of the form (Latter 2006) $$\frac{\partial n}{\partial t}+\bm{\nabla}\bm{\cdot}(n\bm{u})=0,$$ $$\frac{\partial}{\partial t}(mn\bm{u})+
  \bm{\nabla}\bm{\cdot}(mn\bm{u}\bm{u}+mn\mathbf{W}+\mathbf{P})=
  -mn\bm{\nabla}\Phi,$$ $$\frac{\partial\mathbf{W}}{\partial t}+
  \bm{u}\bm{\cdot}\bm{\nabla}\mathbf{W}+
  \mathbf{W}\bm{\cdot}\bm{\nabla}\bm{u}+
  (\bm{\nabla}\bm{u})^\mathrm{T}\bm{\cdot}\mathbf{W}=
  \mathbf{\dot W}_\mathrm{c},
\label{dwdt}$$ where $m$ is the particle mass, $n$ the number density, $\bm{u}$ the bulk velocity, $\mathbf{W}$ the velocity dispersion tensor and $\Phi$ the gravitational potential. The collisional stress $\mathbf{P}$ and the collisional rate of change of velocity dispersion, $\mathbf{\dot
W}_\mathrm{c}$, are complicated algebraic or numerical functions of $n$, $\bm{\nabla}\bm{u}$ and $\mathbf{W}$.

Having set up this system of equations, one may apply it to a local model of a differentially rotating disc known as the shearing sheet. This uses the extreme thinness of the rings to replace the global cylindrical geometry with a local Cartesian one in which the systematic motion is represented as a linear shear flow in a uniformly rotating frame of reference. An approximate procedure of vertical averaging is used to treat the dimension perpendicular to the ring plane.

First, solutions may be sought that are independent of position and time, representing the local uniform statistical equilibrium states of the rings. Such equilibria incorporate the energy balance mentioned above, and predict a certain anisotropic shape for the velocity distribution. These solutions depend on the restitution law of the material and on various dimensionless parameters such as the normal optical thickness, which is an observable quantity and a measure of the areal density of ring material.

Following Maxwell's approach, one may then study the stability of the equilibria with respect to small disturbances with a sinusoidal dependence on (radial) position and a complex exponential dependence on time, being proportional to $\exp(\ri kx+s t)$. It is possible to derive a complicated algebraic dispersion relation (with numerically determined coefficients) giving the growth rate $s$ as a function of the wavenumber $k$. If any of the solutions has a positive real part, an instability is present.

Two types of instability that have been discussed on the basis of viscous (*i.e.* fluid dynamical) models of the rings are the viscous instability and the viscous overstability. The former is favoured when the viscosity decreases with surface density, the latter when it increases sufficiently rapidly. In each case the instability would occur for all wavelengths exceeding a critical value, although the growth rate diminishes rapidly with increasing wavelength. A kinetic treatment of dilute rings shows that the viscous overstability is in fact completely suppressed because of the non-Newtonian nature of the kinetic stress (Latter & Ogilvie 2006). However, the overstability is found to occur in dense rings where the collisional stress is important, in agreement with $N$-body simulations, if the optical thickness exceeds a critical value (Latter 2006).

In Maxwell's problem instability meant that the proposed rotating configuration could be ruled out as a model of Saturn's rings. However, the instabilities that occur in a kinetic model need not be destructive to the rings. In fact, their nonlinear evolution may lead to a saturated state, most likely an equilibrium of a statistical nature, involving regular or irregular structure and dynamics. Since the linear instabilities exist on a broad range of wavelengths, only the nonlinear dynamics can determine the statistical distribution of energy between the available scales. Future work may determine whether this behaviour can explain the rich irregular structure of Saturn's B ring.

# The broader perspective

In astronomical terms, Saturn's rings are just one example of a universal phenomenon: the astrophysical disc. There are many situations in which a disc of fluid or solid matter is found in Keplerian orbital motion around a massive central body (*e.g.* Frank, King & Raine 2002). Protoplanetary discs, such as the 'solar nebula' that once surrounded the Sun, occur as part of the process of star formation and contain most of the angular momentum of the cloud that collapsed to produce the star. These discs of dusty molecular gas last a few million years and are the sites of planet formation. High-energy accretion discs consisting of a fully ionized plasma are found around black holes or compact stars in interacting binary stars within the Galaxy, and also around much more massive black holes at the centres of active galaxies and quasars.

Despite the very different physical constitution of these various systems, they share common dynamical features. In each case a shear stress (often called a viscous stress) that develops in the differentially rotating flow transports angular momentum outwards. In accretion discs this allows inward mass transport towards the central body. One of the challenges of accretion disc theory is to understand and quantify the effects that give rise to this stress — that is, to understand the rheology of the disc.

While in planetary rings the stress arises from the kinetics of colliding particles, in high-energy accretion discs it is believed to result from turbulence in the presence of a magnetic field. The magnetorotational instability is a robust dynamical instability of a differentially rotating flow in which the angular velocity decreases outwards (Balbus & Hawley 1998). Rather than completely disrupting the orbital motion, it generates small-scale anisotropic magnetohydrodynamic turbulence in the disc, which transports angular momentum outwards. While the identification of the role of this instability represents a great advance in the theory of accretion discs, understanding the behaviour of the turbulent plasma is still a major challenge that has been attempted almost exclusively with direct numerical simulations of the magnetohydrodynamic equations in three dimensions.

# A Maxwellian 'mechanical model'

In the world of nineteenth-century physics it was often seen as desirable to develop mechanical models of new phenomena. Maxwell constructed a device involving ivory balls to illustrate the patterns of displacements in his theory of Saturn's rings. In developing his great electromagnetic theory, he found it useful to formulate a mechanical model of 'molecular vortices' and 'idle wheels' (Maxwell 1861), even though he later adopted a more abstract Lagrangian approach (Maxwell 1865). Here I discuss a physical and mathematical analogy that links two quite different areas of Maxwell's work and connects astrophysical fluid dynamics with the laboratory.

The great rivals Newton and Hooke both derived constitutive equations for different types of medium. While Newton proposed that the viscous stress in a fluid is proportional to the shear rate, or rate of strain, Hooke's law stated that the elastic stress in a solid is proportional to the strain itself. In his paper *On the dynamical theory of gases* (1867) Maxwell wrote down an equation that interpolated between these descriptions and would later give rise to the theory of viscoelasticity: $$\frac{\rd\mathrm{(stress)}}{\rd t}=
  \mathrm{elasticity}\times\frac{\rd\mathrm{(strain)}}{\rd t}-
  \frac{\mathrm{stress}}{\tau}.$$ According to Maxwell, a deformable medium has a characteristic relaxation time $\tau$ on which molecular interactions attempt to establish an isotropic velocity distribution and the shear stress, which is associated with anisotropy, decays. If a deformation is applied that has a steady rate of strain or varies on a time-scale long compared to $\tau$, the first term above is negligible and a viscous response occurs with the stress being proportional to the rate of strain as in Newton's law. However, if the strain varies on a time-scale short compared to $\tau$, the third term above is negligible and an elastic response occurs with the stress being proportional to the strain as in Hooke's law. The reason for the elastic response is that the rapid strain prevents the molecules from relaxing towards an equilibrium distribution, and the stress is therefore 'frozen in' to the fluid during the deformation. In general Maxwell's equation allows a continuum of behaviour between viscous and elastic.

Modern constitutive equations for viscoelastic fluids (Bird, Armstrong & Hassager 1987$a$) are usually expressed in a covariant tensorial form based on the principles set out by Oldroyd (1950). His liquid B is one of the most widely used nonlinear models of a viscoelastic fluid, and provides an adequate representation of a dilute solution of a polymer of high molecular weight. It is based on Maxwell's viscoelastic equation above, but with a properly covariant interpretation of the time-derivatives. It can also be derived from the kinetic theory of idealized long-chain polymer molecules contained in a Newtonian solvent (Bird, Curtiss & Armstrong 1987$b$). If the viscosity of the solvent is negligible the model is known as the upper-convected Maxwell fluid. The dimensionless number characterizing the ratio of the relaxation time to the time-scale of the flow is the Deborah number; when this is large, the polymeric stress is effectively 'frozen in' to the fluid.

In an electrically conducting fluid such as the fully ionized plasma of an accretion disc around a black hole, the magnetic field $\bm{B}$ affects the dynamics through the bulk Lorentz force (e.g. Roberts 1967). This can be represented in terms of the Maxwell electromagnetic stress tensor, $$\mathbf{M}=\frac{\bm{B}\bm{B}}{\mu_0}-\frac{B^2}{2\mu_0}\mathbf{1},$$ the two parts of which correspond to a tension in the field lines and an isotropic magnetic pressure (for non-relativistic flows, the electric field makes a negligible contribution to the stress). It is well known that, in a perfectly conducting fluid, the magnetic field is 'frozen in' to the fluid, in the sense that magnetic field lines can be identified with material lines (Alfvén 1950). Even in a fluid of finite conductivity, the magnetic field is effectively 'frozen in' for motions of sufficiently short time-scale, or sufficiently large length-scale, corresponding to a large magnetic Reynolds number. It follows that the Maxwell stress is also 'frozen in' to the fluid in a certain sense.

Ogilvie & Proctor (2003) have shown that there is a physical and mathematical similarity between the dynamics of viscoelastic and magnetohydrodynamic fluids. Either the polymer molecules or the magnetic field lines are advected and distorted by the flow and exert a tension force on it. A formal mathematical analogy can be drawn between the Oldroyd-B fluid in the limit of large Deborah number and a magnetized plasma in the limit of large magnetic Reynolds number. Either the polymeric tension or the magnetic tension in this limit satisfies the equation $$\frac{\partial\mathbf{T}}{\partial t}+
  \bm{u}\bm{\cdot}\bm{\nabla}\mathbf{T}-
  \mathbf{T}\bm{\cdot}\bm{\nabla}\bm{u}-
  (\bm{\nabla}\bm{u})^\mathrm{T}\bm{\cdot}\mathbf{T}=0,$$ which states that the stress is 'frozen in' to the flow. This equation is also closely related to equation () for the kinetic stress in a planetary ring, although there is an important change of sign in two of the terms. The left-hand sides of these equations are both Lie derivatives, one for a contravariant tensor and the other for a covariant tensor.

Ogilvie & Proctor (2003) also showed that the magnetorotational instability, which, as mentioned above, is of primary importance in the theory of accretion discs, has a direct analogue in a viscoelastic fluid. Experiments have been carried out on the stability of viscoelastic Couette–Taylor flow between differentially rotating concentric cylinders since the 1960s. However, they appear to have neglected the regime in which the magnetorotational instability would appearmost clearly; this requires that both cylinders rotate, but in such a way that the specific angular momentum increases outwards while the angular velocity decreases, as in a Keplerian disc. It is likely that an experiment using a dilute polymer solution could quite easily demonstrate the existence of this astrophysically important instability and determine its nonlinear evolution. In contrast, experimental efforts to demonstrate the instability in its magnetohydrodynamic version still face considerable technical difficulties (Goodman & Ji 2002).

# Conclusion

Planetary rings provide the only examples of the universal phenomenon of astrophysical discs that can be observed with exquisite resolution. Investigations of the stability of Saturn's rings 150 years after Maxwell's still hold interest as they may explain the rich irregular structure in the B ring. Understanding the behaviour of planetary rings in detail has been difficult because of the complex physical nature of the medium: a non-Newtonian fluid consisting of a dense granular gas of particles undergoing inelastic and only moderately frequent collisions. Current research builds on the kinetic theories developed by Maxwell and others.

Other astrophysical discs also possess a complicated rheology owing to the presence of magnetohydrodynamic turbulence in a fully ionized plasma. A simple analogy with polymeric liquids may provide a way to understand aspects of their behaviour and even to study it experimentally. In all of this work the theory rests on the firm foundations established by Maxwell's pioneering researches in dynamics, kinetic theory, electromagnetism and viscoelasticity.


abstract: As deep neural networks become more complex and input data-sets grow larger, it can take days or even weeks to train a deep neural network to the desired accuracy. Therefore, distributed Deep Learning at a massive scale is a critical capability, since it offers the potential to reduce the training time from weeks to hours. In this paper, we present a software-hardware co-optimized distributed Deep Learning system that can achieve near-linear scaling up to hundreds of GPUs. The core algorithm is a multi-ring communication pattern that provides a good tradeoff between latency and bandwidth and adapts to a variety of system configurations. The communication algorithm is implemented as a library for easy use. This library has been integrated into Tensorflow, Caffe, and Torch. We train Resnet-101 on Imagenet 22K with 64 IBM Power8 S822LC servers (256 GPUs) in about 7 hours to an accuracy of 33.8% validation accuracy. Microsoft's ADAM and Google's DistBelief results did not reach 30% validation accuracy for Imagenet 22K. Compared to Facebook's recent paper on 256 GPU training, we use a different communication algorithm, and our combined software and hardware system offers better communication overhead for Resnet-50. A PowerAI DDL enabled version of Torch completed 90 epochs of training on Resnet 50 for 1K classes in 50 minutes using 64 IBM Power8 S822LC servers (256 GPUs).
author: Minsik Cho, Ulrich Finkler, Sameer Kumar, David Kung, Vaibhav Saxena, Dheeraj Sreedhar
title: PowerAI DDL

# Introduction

Deep Learning has become the de facto technique for an increasing number of cognitive applications, including vision, speech and language translation. The success is driven by the availability of an enormous volume of data and advances in deep neural networks, but it is these enabling factors that make Deep Learning one of the most computationally-demanding HPC applications. Hardware accelerators such as GPUs and their accompanying software stack have provided a significant amount of speed up. However, deep neural network training for speech and vision can still take days and even weeks. Therefore parallelization by distributing the Deep Learning training to many (upwards of hundreds) GPUs over a compute cluster is critical to cut the training time from weeks to hours.

Distributed Deep Learning (DDL) is challenging because as the number of learners increases, the computation time decreases whereas the amount of communication stays constant, resulting in unfavorable computation to communication ratios. Successful distributed Deep Learning requires an infrastructure in which the hardware and software are co-optimized to balance the computational requirements with the communication demand and interconnect bandwidth. In addition, the communication latency plays an important role in massive scaling of GPUs (over 100). If these factors are not kept under control, distributed Deep Learning can quickly reach the point of diminishing return.

In this paper, we present a software-hardware co-optimized distributed Deep Learning system that can achieve near-linear scaling up to 256 GPUs. The communication algorithm is encapsulated as a communication library, called PowerAI DDL, that provides a set of communication APIs for implementing Deep Learning communication patterns as an integral part of any Deep Learning framework. Currently, PowerAI DDL has been integrated with *Tensorflow*, *Caffe*, and *Torch*. PowerAI DDL provides a unified infrastructure for distributed Deep Learning over multiple GPUs for a single node, multiple nodes and a cloud environment. PowerAI DDL leverages an innovative multi-ring communication algorithm that balances communication latency with the communication overhead. It adapts to the hierarchy of communication bandwidths in any systems – bandwidths that are often highest within a node and then subsequently decreases between nodes and then between racks in a cluster. PowerAI DDL's adaptation enables the "optimal" distributed Deep Learning solution.

The paper is organized as follows. In section , we review prior art, focusing on the distributed Deep Learning result that Facebook AI Research recently announced. In section , we discuss what the methodology for benchmarking distributed Deep Learning should be. In section , we describe PowerAI DDL and our multi-ring algorithm. In section , we present and analyze our results on distributed Deep Learning using 64 IBM Power8 S822LC servers (256 GPUs).

# Prior Art

Massive scale distributed Deep Learning has been the holy grail of the deep learning research community for the past several years. There have been many attempts and reports on scaling to 64, 128 GPUs, but most of them only address the problem from a specific angle. In our opinion, the recent paper by Facebook is the first attack on 256 GPUs that addresses the entire massive scale Deep Learning problem, and provides truly impressive practical results. One thing we want to highlight is that the success of Facebook's paper hinges on being able to train Resnet-50 to the best accuracy while using a very large batch size, which runs counter to conventional wisdom. To train a neural network using 256 GPUs, one must solve the problem of converging to good accuracy with large batch sizes. The GPU computational efficiency at small batch size (4 or less per GPU) is very sub-par, and the compute to communication ratio is also too low for scaling to be efficient. Therefore the batch size per iteration must be in the thousands. The Facebook paper proffered a linear scaling conjecture for the learning rate for large batch size and selected a batch size of 8192 for the experiment, and demonstrated that it works, at least for Resnet-50. The linear scaling conjecture is plausible if the operating region in the parameter space is rather linear, which is not unreasonable when the training process has been warmed-up, as is being done in the paper. This paper is the first instance of a successful training of an "industrial" strength neural network to the best published accuracy with such a large batch size.

Other relevant details of are as follows. The hardware platform is 32 DGX-1 systems (256 P100-SXM2) and the communication algorithm used is the recursive halving and recursive doubling algorithm well known in the HPC community. Resnet-50 on data set Imagenet 1K is trained using Caffe2 with the usual data augmentation techniques. The learning rate schedule starts with a warm-up phase, followed by a ramp up and then the usual step-wise descent. The scaling efficiency in is defined as the ratio between the run time per iteration of the one node case and the run time per iteration for the 256 GPU case, which we believe artificially inflated the scalability metric slightly (as opposed to measuring scaling based on 1 GPU). The validation accuracy in reaches  76% in a hour, with scaling efficiency at  90%.

#  Distributed Deep Learning Benchmarking Methodology

To compare the efficacy of different distributed Deep Learning systems, we need to capture the various characteristics of the distributed Deep Learning problem. Several obvious candidates come to mind, namely, scaling efficiency, accuracy, end to end training time, the neural network itself, the GPU being used, and the Deep Learning framework used to perform the training. We will discuss these characteristics separately.

## Scaling efficiency

Given a specific batch size and distribution over n GPUs, scaling efficiency is defined as the ratio between the run time of one iteration on a single GPU and the run time of one iteration when distributed over n GPUs. This measurement is often extracted without any training to a given accuracy. Sometimes it is even calculated using synthetic data sets. In such cases, scaling efficiency on its own is completely meaningless, because one can satisfy any given scaling efficiency for any neural network by increasing the batch size. Using too big a batch size will result in converging to an unacceptable accuracy or no convergence at all, or such a long training time as to defeat the purpose of speeding up the training using distribution. Therefore, a high scaling efficiency without being backed up by convergence to a good accuracy and end to end training time is vacuous. In this paper, we only present results for neural network runs with demonstrated successful convergence – no artificial data or pure per-iteration training comparisons are used.

## Accuracy and end to end training time

In comparing two different distributed Deep Learning systems, it is imperative to have both system train to the same accuracy. A few percent improvement in accuracy can convert to adding many days of training time. Ultimately, users are interested in reducing the end to end training time at the same accuracy. Furthermore, accuracy has to be determined with a test data set of sufficient size that contains *all* classes used in network training and contains only images that have not been using during network training. Quoting training accuracy is unacceptable, since that just reflects over-fitting and over 90% training accuracy can be readily achieved for many neural networks. We have observed several unpublished (blog) claims of high neural network accuracy, which does not comply with this methodology. Results in this paper follow this methodology, as did .

## The neural network

The compute to communication ratio can vary widely for different neural networks. Using a neural network with high compute to communication ratio can hide the ills of an inferior distributed Deep Learning system. If there is not much to communicate, then having a sub-optimal communication algorithm or low bandwidth interconnect will not matter that much. Therefore it is important to select popular neural networks that are widely used for practical applications when performing the comparison.

## The Deep Learning framework

Currently, the computation time for one Deep Learning iteration can vary by up to 50% when different Deep Learning frameworks are being used. This increases the compute to communication ratio and gives the inferior distributed Deep Learning system an unfair uplift to the scaling efficiency. Therefore it is important to use a computationally-efficient Deep Learning framework to compare the merits of different Deep Learning systems. We have integrated PowerAI DDL into multiple frameworks, to demonstrate its robustness and generality.

## The GPU

Nvidia P100 GPUs are approximately 3X faster than Nvidia K40 GPUs. Using a slower GPU increases the compute to communication ratio and again gives the inferior distributed Deep Learning system an unfair uplift to the scaling efficiency. When evaluating the communication algorithm and the interconnect capability of a Deep Learning system, it is important to use a high performance GPU.

## The communication overhead

The communication overhead is the run time of one iteration when distributed over n GPUs minus the run time of one iteration on a single GPU. This includes the communication latency and the time it takes to send the message (gradients) among the GPUs. However it is not equal to the sum of the above quantities since the communication and latency could be hidden using clever algorithms. The communication overhead gives an indication of the quality of the communication algorithm and the interconnect bandwidth. To perform a meaningful comparison between distributed Deep Learning systems, it is important to start with a popular neural network that has been widely trained, use a Deep Learning framework that is computationally-efficient, and train to best accuracy on high performance GPUs. The measurements to be reported should include the accuracy achieved, the training time it takes to attain that accuracy, the scaling efficiency, and the communication overhead. The overall metric will depend on whether factors such as power efficiency, cost and so on. An example of a meaningful comparison is to use the same number of the same GPUs, same Deep Learning framework, same neural network and the exact same training script and strategy, and compare the run time it takes to train to the same best published accuracy. We follow this methodology in this work and show our contributions to be robust at these standards.

# The PowerAI DDL Library

The PowerAI DDL library provides functionality for high-performance distributed Deep Learning that can be employed in multiple frameworks, currently there are PowerAI DDL enabled versions of *Caffe*, *Tensorflow* and *Torch*.

## PowerAI DDL objects

PowerAI DDL is based around the concept of *PowerAI DDL Objects*(). An instance of a PowerAI DDL object combines a data structure, for example a tensor of gradients, with meta-data. The meta-data define the host and type of memory the data structure resides in as well as the type of device, e.g. a GPU, and device identifier on that host.

PowerAI DDL objects are analogous to variables and all PowerAI DDL functionality operates on these objects. The most basic operation is an assignment, where object $A$ is assigned the value contained in object $B$, creating a copy. The distributed Deep Learning applications employ the operations *ReduceScatter* and *AllGather* that operate on a set of variables that are distributed over a set of GPUs in a set of machines. For example, gradient reduction across 256 GPUs employs 256 PowerAI DDL objects distributed over 64 Power8 S822LC servers, one per process in the application.

## Current implementation

The current PowerAI DDL implementation is based on MPI, since MPI provides many needed facilities, from scheduling processes to basic communication primitives, in a portable, efficient and mature software ecosystem. We select Spectrum MPI, which offers functionality optimized for IBM's Power systems and Infiniband networks, leveraging the considerable investment in the Spectrum products. The core API can be implemented without MPI if desired.

## Topology aware communication

To synchronize the gradients $\vec{V_j}, j \in \{1\ldots N\}$ of $N$ learners which are distributed over $N$ GPUs, a reduction $$\vec{R} = \sum_{j=1}^N \vec{V_j}$$ has to be performed and the result $\vec{R}$ has to be distributed to all the learners. For a complex neural network such as *Resnet-101* for 22000 classes this vector has nearly 90 million floating point entries, resulting in about 350 Megabyte of data.

Deep Learning frameworks often use a tree communication pattern or a parameter server to perform the reduction among the GPUs. The communication overhead of these techniques has a logarithmic (or worse) dependence on the number of GPUs and leaves part of the network links idle during the reduction process. On the other hand, the ring (or bucket) reduction across $N$ ring nodes requires $N-1$ phases of operation, but it uses all links concurrently throughout the entire reduction process. Each phase consists of $N$ concurrent communications, each of which transmits $1/N$ of the data set. Hence, the elapsed time of the ring reduction may be estimated by $$T \approx \sum_{j=1}^{N-1} \left ( \frac{S}{N B_{min}}+L \right )$$ with $S$ denoting the size of the data set, $B_{min}$ the lowest bandwidth in the ring, and $L$ represents the latency incurred to initialize the communications and to synchronize between phases. Figure shows the basic communication pattern used by PowerAI DDL for a small number of GPUs, namely, a ring (shown with blue arrows) in which all nodes communicate concurrently with one other node. Even though the network topology forms a tree, it can be mapped into a ring topology by using every link in the tree simultaneously in both directions. The order of the nodes is important in this mapping. As an example, by switching the first node from the left with the third node from the left causes some network links to be used twice in the same direction, and thereby increasing the elapsed time. The ring reduction algorithm has a couple of drawbacks - the elapsed time for the ring reduction algorithm is proportional to the number of GPUs through the latency term, and it is also dominated by the slowest network link. While it is a good algorithm for a uniform network when $L$ is small and/or the number of GPUs is small, it is not the best algorithm when the latency $L$ is large, or the number of GPUs is large, or some network links are much slower than others.

Across 256 GPUs, nearly 90 Gigabyte of data have to be transmitted to perform the reduction and the same amount of data has to be transmitted to copy the result to all GPUs. Even with a fast network connection allowing a net transfer rate of 10 Gigabyte per second, bringing 90 Gigabyte of data into a single parameter server would take 9 seconds, not considering the time for the distribution of the result. PowerAI DDL achieves the entire reduction and distribution of 350 Megabyte in less than 100 ms by using a communication scheme that is optimized with regard to the topology of the network.

When the number of GPUs is large and the network links are not equally fast, PowerAI DDL uses a multi-dimensional ring algorithm. In the basic ring algorithm, we organize the $N$ GPUs in a one dimensional structure, a ring. In an $m$-dimensional ring algorithm, we organize the $N$ GPUs in a $m$-dimensional box, such that $N=d_1*d_2*...*d_m$, where $d_i$ is the size of the $i^{th}$ dimension of the box.

A key differentiation of PowerAI DDL is its ability to optimize the communication scheme based on the bandwidth of each network link, the network topology, and the latency for each phase. For hardware systems with different link speeds, PowerAI DDL adjusts its dimensions to take advantage of the fastest link, and to prevent the slowest link to dominate the elapsed time. For example in phase $1$, communication is performed via NVLink or memory bus, resulting in a bandwidth of 20 Gbyte/s or more. In the second dimension communication may be restricted to a single rack and achieve an effective bandwidth of roughly 10 Gbyte/s. The third dimension has to communicate between racks and typically achieves a slightly lower bandwidth, for example 9.5 Gbyte/s. In a complete ring, the slowest link limits bandwidth to no more than 9.5 Gbyte/s.

# Experimental Results

To evaluate the performance of PowerAI DDL, we used a cluster of 64 IBM Power8 S822LC servers, each equipped with 4 NVidia Tesla P100-SXM2 GPUs connected through NVLink. The systems were organized into four racks with 16 nodes each, connected via Infiniband. Each rack was equipped with a TOR switch and racks were connected via a director switch. The systems used IBM Spectrum MPI and PPC64LE Red Hat Enterprise Linux 7. As Deep Learning framework served IBM-Caffe (released as part of PowerAI), which is derived from the open source bvlc-Caffe with enhancements customized for IBM's Power8 S822LC servers. Additionally, a PowerAI DDL enabled version of Torch was employed.

We performed two types of experiments - one to extract scaling efficiency as a function of the number of GPUs, the other to report the run time to train a neural network to attain desired accuracy. For the former experiment, the batch size on each host was fixed, so that the total or effective batch size scaled with the number of hosts. We chose an effective batch size that has been demonstrated to converge to best accuracy, so the scaling efficiency reported is realistic.

We tested with two different networks, Resnet-50 for Imagenet with 1K image classes and Resnet-101 modified for Imagenet with 22K classes by increasing the width of the fully connected layer.

## Resnet-50 1K

We used Resnet-50 to extract scaling efficiency and communication overheads for 4 GPUs, 8 GPUs, up to 256 GPUs, while maintaining a fixed batch size of 32 per GPU. At 256 GPUs the effective batch size is 8192, the same as Facebook AI Research's 256 GPU result . Since Facebook's paper has already demonstrated successful convergence to best accuracy for 8192 batch size, the scaling efficiency number is meaningful. The first Resnet-50 1K we used is a port of the network described in to Caffe. Since Facebook did not publish the script for the Resnet-50 they used, we cannot guarantee that our Resnet-50 is identical to theirs. However, both have about 100 Mbyte of gradients, so a direct comparison on the communication overheads can be made. Since we used Caffe whereas Facebook used Caffe2, giving rise to different run times per mini-batch iteration, the scaling efficiency cannot be directly compared. The PowerAI DDL communication overhead of 64 Power8 S822LC servers (256 GPUs) over one Power8 S822LC server (4 GPUs) is about 23 ms. Extracting from figure 7 of the Facebook paper, the communication overhead of 32 DGX-1 nodes (256 GPUs) over 1 DGX-1 node (8 GPUs) is about 29 ms. Therefore our combined communication algorithm together with hardware and network is slightly better.

Figure shows the speed up and scaling efficiency for different numbers of GPUs for Resnet-50 with 1K image classes, with a batch size of 32 per GPU. All runs in figure were performed with IBM-Caffe, and thus included performance improvements to BVLC-Caffe. As a result, the intra-node communication between 4 MPI-ranks within a single node performed better than the intra-node communication in multi-threaded native BVLC-Caffe.

Furthermore, we trained Resnet-50 with 1K image classes with a PowerAI DDL enabled version of Torch based on the open-source single-node multi-GPU Torch application available at . A LUA interface to PowerAI DDL functionality for gradient reduction and distribution allows the adaption with minimal modifications to the LUA implementation of Resnet-50 for 1K classes . Learners are synchronized in every iteration.

Our implementation achieved an accuracy of 75.01% in a training time of 50 minutes, training over 90 epochs with a batch size of 32 per GPU. The learning rates and weight-decay were chosen as in . The original Resnet paper reported a peak accuracy of 75.3 % on Resnet-50 . However, the results in show that the accuracy could be pushed to as high as 76.2 % using hyper-parameter tuning for large effective batch-sizes. We have not incorporated those techniques in our work.

For these Torch Resnet-50 runs, figure shows scalability across increasing numbers of GPUs. A scaling efficiency of 84% is observed from 4 GPUs (single node) to 256 GPUs (64 Nodes). It should also be noted that in the current Torch implementation, the reduction of gradients is not overlapped with compute.

## Resnet-101 22k

Compared to Resnet-50 with 1K image classes, Resnet-101 modified for 22K image classes has a roughly three times larger gradient set of 350 Mbyte, but the corresponding increase in compute time for the forward/backward pass is not nearly as much. Hence, it is a harder problem to scale due to the lower compute-to-communication ratio relative to Resnet-50. Figure shows our results running IBM-Caffe using 1, 2, 16, 32 and 64 Power8 S822LC servers with a batch size of 20 per GPU. All runs in figure were performed with IBM-Caffe.

For batch size 5120 we achieved 33.8% validation accuracy within about 7 hours. This accuracy exceeds the previous published accuracies of 29.8% and $<$`<!-- -->`{=html}25% for Imagenet 22K. We used 7.5 million images from the full Imagenet dataset for training. Following standard practice, a separate dataset of 2 million images not used in training was used for validation.

# Conclusions

We have provided further confirmation that training complex deep neural networks with a large number of GPUs using a synchronous data parallel methodology can scale almost linearly to hundreds of GPUs, thus offering a viable strategy to reduce training time from weeks to hours. Such rapid turn around can accelerate the improvement of existing neural networks and design of new neural networks, and exploration of new application domains. To proliferate this technology to the masses, more research needs to be done. Massive GPU scaling relies on successful training to good accuracy for large batch size. FAIR showed an initial demonstration of distribution to 256 GPUs on a couple of Resnets; our work here demonstrates even better communication overhead, and uses PowerAI DDL to train a larger neural network to leadership Imagenet 22K accuracy. It is key that the community continue to extend demonstration of large-scale distributed Deep Learning to other popular neural network types, in particular, recurrent neural networks. The whole training has to be made resilient and elastic since it is very likely that some devices will malfunction when the number of learners increases. Automation and usability issues have to be addressed to enable more turnkey operation, especially in a cloud environment.