abstract: We present a fast, fully parameterizable GPU implementation of Convolutional Neural Network variants. Our feature extractors are neither carefully designed nor pre-wired, but rather learned in a supervised way. Our deep hierarchical architectures achieve the best published results on benchmarks for object classification (NORB, CIFAR10) and handwritten digit recognition (MNIST), with error rates of 2.53%, 19.51%, 0.35%, respectively. Deep nets trained by simple back-propagation perform better than more shallow ones. Learning is surprisingly rapid. NORB is completely trained within five epochs. Test error rates on MNIST drop to 2.42%, 0.97% and 0.48% after 1, 3 and 17 epochs, respectively.
author: Dan C. Cireşan, Ueli Meier, Jonathan Masci,
Luca M. Gambardella and Jürgen Schmidhuber
date: January 2011
title: High-Performance Neural Networks
  for Visual Object Classification

# Introduction

The human visual system efficiently recognizes and localizes objects within cluttered scenes. For artificial systems, however, this is still difficult, due to viewpoint-dependent object variability, and the high in-class variability of many object types. Deep hierarchical neural models roughly mimick the nature of mammalian visual cortex, and by community consensus are among the most promising architectures for such tasks. The most successful hierarchical object recognition systems all extract localized features from input images, convolving image patches with filters. Filter responses are then repeatedly sub-sampled and re-filtered, resulting in a deep feed-forward network architecture whose output feature vectors are eventually classified. One of the first hierarchical neural systems was the Neocognitron which inspired many of the more recent variants.

Unsupervised learning methods applied to patches of natural images tend to produce localized filters that resemble off-center-on-surround filters, orientation-sensitive bar detectors, Gabor filters . These findings in conjunction with experimental studies of the visual cortex justify the use of such filters in the so-called *standard model* for object recognition , whose filters are fixed, in contrast to those of Convolutional Neural Networks (CNNs) , whose weights (filters) are randomly initialized and changed in a supervised way using back-propagation (BP).

Despite the hardware progress of the past decades, computational speed is still a limiting factor for CNN architectures characterized by many building blocks typically set by trial and error. To systematically test the impact of various architectures on classification performance, we present a fast CNN implementation on Graphics Processing Units (GPUs). Previous GPU implementations of CNNs were hard-coded to satisfy GPU hardware constraints, whereas our implementation is flexible and fully online (i.e., weight updates after each image). It allows for training large CNNs within days instead of months, such that we can investigate the influence of various structural parameters by exploring large parameter spaces and performing error analysis on repeated experiments.

We evaluate various networks on the handwritten digit benchmark MNIST and two image classification benchmarks: NORB and CIFAR10 .

# Convolutional neural networks

CNNs are hierarchical neural networks whose convolutional layers alternate with subsampling layers, reminiscent of simple and complex cells in the primary visual cortex . CNNs vary in how convolutional and subsampling layers are realized and how the nets are trained. The CNN architecture considered in this study differs from the one of in the sense that after each CNN-layer an optional max-pooling layer can be used. Here we give a complete description of this independent implementation (Fig. ).

## Image processing layer

The image processing layer is an optional pre-processing layer of predefined filters that are kept fixed during training. Thus additional information besides the raw input image can be provided to the network, such as edges and gradients. In particular, we find that a contrast-extracting layer helps to improve the recognition rate for NORB.

## Convolutional layer

A convolutional layer is parametrized by the size and the number of the maps, kernel sizes, skipping factors, and the connection table. Each layer has $M$ maps of equal size ($M_x$, $M_y$). A kernel (blue rectangle in Fig ) of size ($K_x$, $K_y$) is shifted over the valid region of the input image (i.e. the kernel has to be completely inside the image). The skipping factors $S_x$ and $S_y$ define how many pixels the filter/kernel skips in x- and y-direction between subsequent convolutions. The size of the output map is then defined as: $$M^n_x = \frac{M^{n-1}_x-K_x^n}{S_x^n+1}+1; \quad M^n_y = \frac{M^{n-1}_y-K_y^n}{S_y^n+1}+1\\$$ where index $n$ indicates the layer. Each map in layer $L^n$ is connected to at most $M^{n-1}$ maps in layer $L^{n-1}$. Neurons of a given map share their weights but have different receptive fields.

## Max-pooling layer

The biggest architectural difference between our implementation and the CNN of is the use of a max-pooling layer instead of a sub-sampling layer. No such layer is used by who simply skips nearby pixels prior to convolution, instead of pooling or averaging. found that max-pooling can lead to faster convergence, select superior invariant features, and improve generalization. The output of the max-pooling layer is given by the maximum activation over non-overlapping rectangular regions of size ($K_x$, $K_y$). Max-pooling enables position invariance over larger local regions and downsamples the input image by a factor of $K_x$ and $K_y$ along each direction.

## Classification layer

Kernel sizes of convolutional filters and max-pooling rectangles as well as skipping factors are chosen such that either the output maps of the last convolutional layer are downsampled to 1 pixel per map, or a fully connected layer combines the outputs of the topmost convolutional layer into a 1D feature vector. The top layer is always fully connected, with one output unit per class label.

# GPU implementation

The latest generation of NVIDIA GPUs, the 400 and 500 series (we use GTX 480 & GTX 580), has many advantages over older GPUs, most notably the presence of a R/W L2 global cache for device memory. This permits faster programs and simplifies writing the code. In fact, the corresponding transfer of complexity into hardware alleviates many software and optimization problems. Our experiments show that the CNN program becomes 2-3 times faster just by switching from GTX 285 to GTX 480.

Manual optimization of CUDA code is very time-consuming and error prone. We optimize for the new architecture, relying on the L2 cache for many of the device memory accesses, instead of manually writing code that uses textures and shared memory. Code obtained by this pragmatic strategy is fast enough. We use the following types of optimization: pre-computed expressions, unrolled loops within template kernels, strided matrices to obtain coalesced memory accesses and registers wherever possible. Additional manual optimizations are possible in case future image classification problems will require even more computing power.

## Data structures

Both outputs $y$ and deltas $\delta$ of layer $L^n$ are 2D strided. Their original size is $M_x\times MM_y$, but they are horizontally strided with a pitch of 32 floats (we use this stride for all 2D data), resulting in coalesced memory accesses. The vertical stride avoids additional bounding tests in CUDA kernels.

All connections between maps of consecutive layers $L^{n-1}$ and $L^n$ are stored in matrix $C^n$. Each row of $C^n$ contains all connections that feed into a particular map in layer $L^n$. Because we aim for a flexible architecture with partially connected layers, in the first column we store the number of previous connections. This index is useful for Forward Propagation (FP) and Adjusting Weights (AW) CUDA kernels. The second column stores the number of connections, followed by corresponding indices of maps in $L^{n-1}$ connected to the current map.

For BP and FP, analogous information about connections is needed. We therefore store backward connections in $C_{BP}$. AW requires a list of all map connections (see Subsection ), stored as an array of map index pairs. Dealing with biases in BP kernel requires to know where the weights of particular connections start; this information is stored in a 2D array $WIDX_{BP}$ of size $M^n \times M^{n-1}$.

## Forward propagation

A straightforward way of parallelizing FP is to assign a thread block to each map that has to be computed. For maps bigger than 1024 neurons, the job is further split into smaller blocks by assigning a block to each line of the map, because the number of threads per block is limited (1024 for GTX 480). A one to one correspondence between threads and the map's neurons is assumed. Because of weight sharing, threads inside a block can access data in parallel, in particular the same weights and inputs from the previous layer. Each thread starts by initializing its sum with the bias, then loops over all map connections, convolving the appropriate patch of the input map with the corresponding kernel. The output is obtained by passing the sum through a scaled tanh activation function, and then written to device memory.

## Backward propagation

BP of deltas can be done in two ways: by pushing or by pulling. Pushing deltas means taking each delta from the current layer and computing the corresponding deltas for the previous layer. For an architecture with shared weights this has the disadvantage of being hard to code. Each delta from the current layer contributes to many deltas in the previous layer, which translates into a lot of programming. There are two ways of avoiding this: either writing partial deltas to a separated block of memory and then putting everything together by calling another kernel (slow because of a tremendous increase in the number of memory accesses, and the need of another kernel), or using atomic writes (to avoid data hazards) to update deltas (very slow because many writings are serialized). We implement pulling deltas, which has almost none of the above speed-limiting drawbacks, but is a bit more complicated.

The (uni- or bi-dimensional) thread grid assigns a (bi- or uni-dimensional) thread block to each map in the previous layer and a thread to each neuron in every map. Similar to FP, for maps with more than 1024 neurons, the 2D grid is further split into smaller 1D blocks by assigning a 2D block to each row of the map. Each thread computes the delta of its corresponding neuron by pulling deltas from the current layer. For every neuron in the previous layer we have to determine the list of neurons in the current layer which are connected to it. Let us consider neuron $(i,j)$ from a map in layer $L^{n-1}$, and then assume that $(x,y)$ are the coordinates of neurons in maps of $L^n$ that contribute to the delta of neuron $(i,j)$. The $(x,y)$ neuron is connected to kernel size number neurons ($K_x \times K_y$) from each connected map in the previous layer. The indices in $L^{n-1}$ of the neurons connected through a kernel to the $(x,y)$ neuron are: $$\begin{aligned}
  x(S_x+1)  &\le  i \le& x(S_x+1)+K_x-1 \nonumber,\\
  y(S_y+1)  &\le  j \le& y(S_y+1)+K_y-1 \nonumber.
\end{aligned}$$ We can now compute the inequalities for $(x,y)$: $$\begin{aligned}
   \frac{i-K_x+1}{S_x+1} &\le   x   \le&  \frac{i}{S_x+1} \nonumber, \\
   \frac{j-K_y+1}{S_y+1} &\le   y   \le&  \frac{j}{S_y+1} \nonumber.
\end{aligned}$$ Because $(x,y)$ has to be inside the map, the final inequalities are: $$\begin{aligned}
  \max{\left ( \left \lceil \frac{i-K_x+1}{S_x+1} \right \rceil,0 \right )} &\le x \le & \min{\left ( \left \lfloor \frac{i}{S_x+1} \right \rfloor,M_x-1 \right )}\nonumber, \\
 \max{\left ( \left \lceil \frac{j-K_y+1}{S_y+1} \right \rceil,0 \right )} &\le y \le &\min{\left ( \left \lfloor \frac{j}{S_y+1} \right \rfloor,M_y-1 \right )}.\nonumber
\end{aligned}$$ The above inequalities state that the delta of neuron $(i,j)$ from $L^{n-1}$ is computed from deltas of neurons in a rectangular area in maps of $L^n$ (Fig. ). After summing up the deltas, each thread multiplies the result by the derivative of the activation function.

## Adjusting weights

FP and BP have a grid on the list of maps, but the AW thread grid is on the list of kernels (filters) between maps of two consecutive layers. The 1D grid has a block for each connection between two maps. Thread blocks are 2D, with a corresponding thread for every kernel weight. The bias weight is included as an entire row of threads, thus requiring thread blocks to have $(K_x+1) \times K_y$ threads. Most of the time these additional $K_y$ threads will do nothing, thread (0,0) being activated only for blocks that have to process the bias.

# Experiments

We use a system with a Core i7-920 (2.66GHz), 12 GB DDR3 and four graphics cards: 2 x GTX 480 and 2 x GTX 580. The correctness of the CPU version is checked by comparing the analytical gradient with its finite difference approximation. On GPU this is not possible because all computations are performed with single precision floating point numbers. Hence the GPU implementation's correctness is checked by comparing its results to those of a randomly initialized net after training it for several epochs on the more accurate CPU version. Obtaining identical results after trillions of operations is a strong indication of correctness.

The implemented CNN's plain feed-forward architecture is trained using on-line gradient descent. All images from the training set are used for training and also for validation. If deformations are enabled, only the images from the training set will be deformed. Weights are initialized according to a uniform random distribution in the range $[-0.05,0.05]$. Each neuron's activation function is a scaled hyperbolic tangent: $y(a)=1.7159\tanh(0.6666a)$ .

We pick the trained CNN with the lowest validation error, and evaluate it on the test set (Test for best Validation - TfbV). The best test error (bT) is also listed for all experiments. The reported computation times per epoch include training, validation and testing as well as all data transfers.

## Experiments on MNIST

For the MNIST dataset the networks are trained on deformed images, continually generated in on-line fashion. Affine (translation, rotation, scaling, horizontal shearing) and elastic deformations are combined. We use a variable learning rate that shrinks by a multiplicative constant after each epoch, from $10^{-3}$ down to $3\cdot10^{-5}$ after 500 epochs.

|                                |          |          |
|:------------------------------:|:--------:|:--------:|
|            \#M, \#N            |    bT    |   TfbV   |
|        in Hidden Layers        |  \[%\]   |  \[%\]   |
|            20M-60M             |   0.95   |   1.02   |
|          20M-60M-150N          |   0.50   |   0.55   |
|       20M-60M-100M-150N        |   0.33   |   0.38   |
| 20M-40M-60M-80M-100M-120M-150N | **0.27** | **0.35** |

Error rates on MNIST test set for randomly connected CNNs with 2 to 6 convolutional layers with M Maps and an optional fully connected layer with N neurons. Various kernel sizes and skipping factors were used.

Fully connected convolutional layers lead to an exploding number of network connections and weights, making training of big and deep CNNs for hundreds of epochs impractical even on GPUs. Partial connectivity alleviates this problem and is also biologically more plausible. We reduce the number of connections between convolutional layers in a random way. Table  lists results of various networks with 2 to 7 hidden layers with random connections. Additional layers result in better networks, the best one achieving a test error of 0.35% for best validation and a best test error of 0.27%. The best previous CNN result on MNIST is 0.40% . A 0.35% error rate was recently also obtained by a big, deep MLP with many more free parameters. Deeper nets require more computation time to complete an epoch, but we observe that they also need fewer epochs to achieve good test errors. The deepest CNN from Table  reaches 2.42%, 0.97% and 0.48% after one, three and seventeen epochs, respectively. On the other hand, the network with 4 instead of 7 hidden layers reaches 4.71%, 1.58%, 0.68% after one, three and seventeen epochs, achieving a test error below 0.50% after only 34 epochs. This shows once more that deep networks, contrary to common belief, can be trained successfully by back-propagation. Despite the numerous free parameters, deep networks seem to learn faster (better recognition rates after fewer epochs) than shallow ones.

We consider MNIST an almost solved problem. The remaining errors stem from digits that are ambiguous or miss parts.

## Experiments on NORB

NORB contains stereo images of 3D objects. Hence there are two maps on the input layer. Rotation, scaling, shearing and elastic distortions seem to have a negative impact on generalization. These deformations improve recognition rates for digits that are intrinsically 2D , but seem inadequate for 3D objects.

Initial experiments on NORB show that unlike with MNIST where we use deformations, the CNN needs only 3 to 6 epochs to reach zero validation error. This allows us to quickly run numerous repetitive experiments with huge networks with hundreds of maps per layer. We decided to use a CNN with five hidden layers: layer1, a convolutional layer with 300 maps, kernel size $6\times6$ and skipping factors $1\times1$; layer2, a max-pooling layer over a $2\times2$ region; layer3, a convolutional layer with 500 maps, kernel size $4\times4$, skipping factors $0\times0$; layer4, a max-pooling layer over a $4\times4$ region; layer5, a fully connected layer with 500 neurons. The learning rate is initialized by 0.001 and multiplied by 0.95 after every epoch.

Table summarizes the results of four different experiments by switching on/off translation as well as the fixed image processing layer. We report the average error rate as well as the standard deviation of N independent runs with identical architectures but different weight initializations. For the first experiment without translation and no image processing (IP), an average test error rate of 7.86% is obtained. With additional translations of at most 5%, the average error rate drops to 4.71%, contradicting the common belief that CNNs are translation invariant. These results are on par or better than others in the literature: 5.90% error rate for a combination of CNNs and SVMs and 5.20% error rate for restricted Boltzman machines .

The best previously published result on NORB (2.87%) was obtained by a hierarchical neural network which to every convolutional layer provides a subsampled version plus edge information of the original image . This motivated us to implement a pre-processing layer with fixed filters. We tried simple edge masks (Sobel, Scharr) but obtained best results with a contrast-extraction layer realized by Mexican hat-shaped filters of size $21\times21$, one with a concentric on-center receptive field and one with a concentric off-center receptive field, similar to the filters automatically created by unsupervised *Predictability Minimization* applied to natural images . The first filter extracts positive contrast in brightness, whereas the latter extracts negative contrast. Each image from the original NORB is filtered, consequently the input of the CNN has six maps: the original image plus the positive and negative contrast for each of the two stereo channels. Using such a pre-processing layer results in lower average error rates, 3.94% without translation and 2.53% with translation. This result improves the previous state of the art on NORB .

| trans. \[%\] | IP  |     TfbV \[%\]      | runs | time/epoch \[s\] |
|:------------:|:---:|:-------------------:|:----:|:----------------:|
|      0       | no  |   7.86 $\pm$ 0.55   |  50  |       1141       |
|      5       | no  |   4.71 $\pm$ 0.57   |  50  |       1563       |
|      0       | yes |   3.94 $\pm$ 0.48   |  50  |       1658       |
|      5       | yes | **2.53** $\pm$ 0.40 | 100  |       2080       |

Average error rates and standard deviations of N runs for a five hidden layer CNN on the NORB test set (see text for details).

Experience with other image datasets tells us that NORB is unusual. The training set has only five instances per class. The resulting poor training set variability makes the nets learn quickly but generalize badly. NORB is the only dataset that profits from a fixed pre-processing layer in a substantial way. For MNIST and CIFAR10 such pre-processing has little or no effect. It is also worth noting that NORB's standard error rate deviation is bigger than CIFAR10's (see Tables and ). Identical nets with different initializations do not produce very consistent results. The best net had an error rate of 1.72%, the worst 3.69%.

## Experiments on CIFAR 10

CIFAR10 is a collection of natural color images of 32x32 pixels. It contains 10 classes, each of them with 5000 samples in the training set and 1000 in the test set. The images greatly vary inside each class. They are not necessarily centered, may contain only parts of the object, and have varying backgrounds. All of this makes CIFAR10 the hardest problem addressed in this paper. The CNN has three maps, one for each color channel (RGB). The CIFAR10 images are relatively small in comparison to NORB's, and force us to use small kernels. The tested CNNs differ only in the number of maps per convolutional and max-pooling layer. All have eight hidden layers: layer1, a convolutional layer with $3\times3$ kernels and skipping factor of $0$; layer2, a max-pooling layer over a $3\times3$ region; layer3, a convolutional layer with $3\times3$ kernels and skipping factors of $0\times0$; layer4, a max-pooling over a $2\times2$ region; layer5, a convolutional layer with $3\times3$ kernels and a skipping factors of $0\times0$; layer6, a max pooling layer over a $2\times2$ region; layer7, a fully connected layer with $300$ neurons; layer8, a fully connected layer with $100$ neurons.

Like for MNIST, the learning rate is initialized by 0.001 and multiplied by 0.993 after every epoch. Results in Table show that without translation the error rate does not drop below 28%; adding edge information does not help at all. Translations have a very positive effect, decreasing the error rate to almost 20%. Contrast extraction filters are better than the Sobel/Scharr filters but still worse than no pre-processing layer at all. Despite some CNN-inherent translation invariance, additional training image translations cause better generalization; additional image processing proved useless though.

| trans. \[%\] | maps |  IP  |      TfbV \[%\]      | runs | time/epoch \[s\] |
|:------------:|:----:|:----:|:--------------------:|:----:|:----------------:|
|      0       | 100  |  no  |   28.87 $\pm$ 0.37   |  11  |        93        |
|      0       | 100  | edge |   29.11 $\pm$ 0.36   |  15  |       104        |
|      5       | 100  |  no  |   20.26 $\pm$ 0.21   |  11  |       111        |
|      5       | 100  | edge |   21.87 $\pm$ 0.57   |  5   |       120        |
|      5       | 100  | hat  |   21.44 $\pm$ 0.44   |  4   |       136        |
|      5       | 200  |  no  |   19.90 $\pm$ 0.16   |  5   |       248        |
|      5       | 300  |  no  | **19.51** $\pm$ 0.18 |  5   |       532        |
|      5       | 400  |  no  |   19.54 $\pm$ 0.16   |  5   |       875        |

Average error rates and standard deviations for N runs of an eight hidden layer CNN on the CIFAR10 test set (see text for details). The first five nets have 100 maps per convolutional and max-pooling layer, whereas the sixth, seventh and eighth have 200, 300 and 400 maps per hidden layer, respectively. IP - image processing layer: edge - $3\times3$ Sobel and Scharr filters; hat - $13\times13$ positive and negative contrast extraction filters.

To see if bigger nets are better, we increase the number of maps per layer from 100 to 200, 300 and 400, respectively (last three rows in Tab. ). Training time increases exponentially, but the test error decreases, reaching a minimum for nets with 300 maps per layer. Our 19.51% error rate is better than the previous state of the art for this dataset, 20.40% and 25.50% . Unlike , however, we use the original images without any particular input normalization. Note that the error rate standard deviations are smaller than those obtained on NORB, that is, different initializations yield consistent results.

## Speedup factor of GPU code

The GPU code scales well with network size. For small nets the speedup is small (but still over 10) since they fit better inside the CPU cache, and GPU resources are underutilized. For huge nets (ex: Table ) the GPU implementation is more than 60 times faster than a compiler-optimized CPU version. Given the flexibility of our GPU version, this is a significant speedup. One epoch takes 35 GPU minutes but more than 35 CPU hours.

# Conclusion

We presented high-performance GPU-based CNN variants trained by on-line gradient descent, with sparse random connectivity, computationally more efficient and biologically more plausible than fully connected CNNs. Principal advantages include state-of-the-art generalization capabilities, great flexibility and speed. All structural CNN parameters such as input image size, number of hidden layers, number of maps per layer, kernel sizes, skipping factors and connection tables are adaptable to any particular application. We applied our networks to benchmark datasets for digit recognition (MNIST), 3D object recognition (NORB), and natural images (CIFAR10). On MNIST the best network achieved a recognition test error rate of 0.35%, on NORB 2.53% and on CIFAR10 19.51%. Our results are raising the bars for all three benchmarks. Currently the particular CNN types discussed in this paper seem to be the best adaptive image recognizers, provided there is a labeled dataset of sufficient size. No unsupervised pretraining is required. Good results require big and deep but sparsely connected CNNs, computationally prohibitive on CPUs, but feasible on current GPUs, where our implementation is 10 to 60 times faster than a compiler-optimized CPU version.

# Acknowledgment

This work was partially funded by the Swiss Commission for Technology and Innovation (CTI), Project n. 9688.1 IFF: Intelligent Fill in Form.


abstract: While Microservices promise several beneficial characteristics for sustainable long-term software evolution, little empirical research covers what concrete activities industry applies for the evolvability assurance of Microservices and how technical debt is handled in such systems. Since insights into the current state of practice are very important for researchers, we performed a qualitative interview study to explore applied evolvability assurance processes, the usage of tools, metrics, and patterns, as well as participants' reflections on the topic. In 17 semi-structured interviews, we discussed 14 different Microservice-based systems with software professionals from 10 companies and how the sustainable evolution of these systems was ensured. Interview transcripts were analyzed with a detailed coding system and the constant comparison method.
  .
  We found that especially systems for external customers relied on central governance for the assurance. Participants saw guidelines like architectural principles as important to ensure a base consistency for evolvability. Interviewees also valued manual activities like code review or boy scouting, even though automation and tool support was described as very important. Source code quality was the primary target for the usage of tools and metrics. Despite most reported issues being related to Architectural Technical Debt (ATD), our participants did not apply any architectural or service-oriented tools and metrics. While participants generally saw their Microservices as evolvable, service cutting and finding an appropriate service granularity with low coupling and high cohesion were reported as challenging. Future Microservices research in the areas of evolution and technical debt should take these findings and industry sentiments into account.
author:
bibliography: references.bib
title: Assuring the Evolvability of Microservices: Insights into Industry Practices and Challenges

# Introduction

Fast moving markets and the age of digitization require that software can be quickly adapted or extended with new features. If change implementations frequently happen under time pressure, the sustainable evolution of a long-living software system can be significantly hindered by the intentional or unintentional accrual of technical debt . The quality attribute associated with software evolution is referred to as *evolvability* : the degree of effectiveness and efficiency with which a system can be adapted or extended. Evolvability is especially important for software with frequently changing requirements, e.g. internet-based systems.

To provide sufficient confidence that such a system can be sustainably evolved, software professionals apply a set of numerous activities that we refer to as *evolvability assurance*. Activities are usually either of an analytical nature to identify issues or of a constructive nature to remediate issues . This includes for example techniques like code review or refactoring, standardization, guidelines, conscious technical debt management, and the usage of tools, metrics, or patterns. For larger systems, these activities often form a communicated assurance process and are an important part of the development workflow.

Microservices constitute an important architectural style that prioritizes evolvability . A key idea here is that fine-grained and loosely coupled services that are independently deployable should be easy to change and to replace. Moreover, one of the postulated Microservices characteristics is *evolutionary design* . While these properties provide a nice theoretical basis for evolvable systems, they offer no concrete and universally applicable solutions. As with each architectural style, the implementation of a concrete Microservice-based system can be of arbitrary quality. The "service cutting" activity has been described as especially challenging and several approaches have been proposed by academia to support it . Apart from this, very little scientific research has covered the areas maintenance, evolution, or technical debt for Microservices. Examples include the tracking and management of Microservices dependencies , a service evolution modeling technique for Microservices , and the applicability of service-based maintainability metrics for Microservices .

In addition to this sparse scientific state of the art, there are also very few empirical studies on the industry state of practice. Very little is known about what evolvability assurance processes and techniques companies use for Microservices or if these are different compared to other architectural styles. In the general area of service-based systems, Schermann et al. even describe a mismatch between what academia assumes and what industry actually does. An analysis of industry practices in this regard could identify common challenges, showcase successful processes, and highlight gaps and deficiencies. This would also provide insights into how industry perceives academic approaches specifically designed for service orientation, e.g. service-oriented maintainability metrics. Results of such a study could help to design new and more suited evolvability assurance processes or techniques.

We therefore conducted detailed interviews with 17 Germany-based software professionals from 10 different companies. They described 14 different systems with various Microservices characteristics and their concrete evolvability assurance process including tool, metric, and pattern usage. We also talked with them about the evolution qualities of Microservices, how Microservices influence the assurance process, and their perceived challenges for evolvability in the context of Microservices.

# Related Work

Several empirical studies about challenges for the adoption of Microservices exist. Baškarada et al. conducted 19 in-depth interviews with experienced architects. They discussed opportunities and challenges associated with the adoption and implementation of Microservices. Four types of corporate systems with different levels of suitability for Microservices were identified. Especially large corporate systems of record like Enterprise Resource Planning (ERP) were not seen as appropriate targets. In this context, organizational challenges such as DevOps methodologies would be less serious for Information and Communication Technology enterprises than for traditional organizations, where IT was perceived as a "necessary evil" .

Ghofrani and Lübke conducted a similar empirical survey among 25 practitioners that were mainly developers and architects. Their objective was to find perceived challenges in designing, developing, and maintaining Microservice-based systems. The results reveal a lack of notations, methods, and frameworks for architecting Microservices. Several participants named the distributed architecture as responsible for the challenging development and debugging of the system. Participants generally prioritized optimizations in security, response time, and performance over aspects like resilience, reliability, and fault tolerance.

In their interviews with 10 Microservices experts from industry, Haselböck et al. focused on design areas of Microservices and associated challenges. The study identified 20 design areas and their importance as rated by the participants. Design principles and common challenges from earlier mapping studies could be confirmed by the authors. Similar to our qualitative study, interviewees' rationales are discussed as well. Microservices design is a fundamental aspect of their evolvability later on. As such, the study can be seen as a valuable contribution to the topic at hand.

Some studies also focus on evolvability-related areas like technical debt and architecture smells. Carrasco et al. conducted a literature review to gather migration and architecture smells. The authors derived best practices, success stories, and pitfalls. By digesting 58 different sources from academia and grey literature, they presented nine common bad smells with proposed solutions. This study extends the work of Taibi and Lenarduzzi who defined 11 bad smells by analyzing 265 bad practices experienced by 72 developers from 61 companies. The results of their two-year interview phase were mapped onto a list of main pitfalls extracted from grey literature and practitioner talks. Out of 16 extracted pitfalls, only six were confirmed by the interviewees. Wrong service cuts during the split of a monolith turned out to be the most critical challenge, resulting in potential maintenance issues later on.

The same authors investigate technical debt interest by means of a long term case study where they monitored the migration of a monolithic legacy system to Microservices. The study aims to characterize technical debt and its growth comparatively in both architectural styles. As a preliminary result, they found that the total amount of technical debt grew much faster in the Microservice-based system.

Bogner et al. surveyed 60 software professionals via an online questionnaire to assess maintainability assurance practices in industry as well as notable differences with service-oriented systems. The authors inquired information about the used processes, tools, and metrics to learn about treatments specific to such systems. Explicit and systematic techniques turned out to be beneficial for maintainability. Very few participants reported the usage of techniques to address existing issues related to architecture-level evolvability. Similarly, barely any participants had established a systematic process for maintainability control. Since 67% of participants neglected service-oriented particularities in the assurance, the study revealed a weak spot in industry practice that may impair the lifespan of service-based systems.

In summary, the majority of mentioned studies does not focus on applied assurance techniques in industry. Bogner et al. are the only ones to do so in their survey. However, they also include service-based systems and say very little about participants' rationales. We therefore conducted our own qualitative Microservices study focused on industry evolvability assurance processes, techniques, and challenges.

```latex
\begin{table*}
    \centering
    % table caption is above the table
    \caption{Company and Participant Demographics}
    \label{table:demographics}
    \begin{tabular}{llrrlrr}
        Company ID & Company Domain & \# of Employees & Participant ID & Participant Role & Years of Experience & System ID\\
        \hline
        \hline
        C1 & Financial Services & 1 -- 25 & P1 & Developer & 6 & S1\\
        \hline
        \multirow{3}{*}{C2} & \multirow{3}{*}{Software \& IT Services} & \multirow{3}{*}{$>$100,000} & P2 & Lead Architect & 30 & S2\\
         & & & P3 & Architect & 24 & S3\\
         & & & P4 & Architect & 30 & S4\\
        \hline
        \multirow{2}{*}{C3} & \multirow{2}{*}{Software \& IT Services} & \multirow{2}{*}{26 -- 100} & P5 & Architect & 20 & \multirow{2}{*}{S5}\\
         & & & P6 & Lead Developer & 8\\
        \hline
        \multirow{2}{*}{C4} & \multirow{2}{*}{Software \& IT Services} & \multirow{2}{*}{101 -- 1,000} & P7 & Architect & 9 & S6\\
         & & & P8 & Architect & 17 & S7\\
        \hline
        C5 & Software \& IT Services & $>$100,000 & P9 & Lead Developer & 7 & S8\\
        \hline
        \multirow{3}{*}{C6} & \multirow{3}{*}{Tourism \& Travel} & \multirow{3}{*}{1,001 -- 5,000} & P10 & Developer & 9 & \multirow{2}{*}{S9}\\
         & & & P11 & Data Engineer & 7\\
         & & & P12 & Architect & 12 & S10\\
        \hline
        \multirow{2}{*}{C7} & \multirow{2}{*}{Logistics \& Public Transport} & \multirow{2}{*}{101 -- 1,000} & P13 & DevOps Engineer & 5 & S11\\
         & & & P14 & Architect & 17 & S11a\\
        \hline
        C8 & Retail & 5,001 -- 10,000 & P15 & Lead Architect & 9 & S12\\
        \hline
        C9 & Software \& IT Services & 101 -- 1,000 & P16 & Architect & 18 & S13\\
        \hline
        C10 & Retail & 1,001 -- 5,000 & P17 & Architect & 22 & S14\\
        \hline
    \end{tabular}
\end{table*}
```

# Research Design

We generally followed the five-step case study process as described by Runeson and Höst to structure our research. The different steps are described in the following subsections.

## Study Design

As a first step, we defined our research objective and related research questions. The primary goal of this study can be formulated in the following way:

*Analyze the applied **evolvability assurance**
For the purpose of knowledge generation
With respect to common practices and challenges
From the viewpoint of software professionals
In the context of **Microservices** in industry
*

We formulated three research questions to set a direction and scope for our study:

**RQ1:** How do software professionals structure the general evolvability assurance activities for Microservices and for what reasons?

**RQ2:** What tools, metrics, and patterns do software professionals use for assuring the evolvability of Microservices and with what rationales?

**RQ3:** How do software professionals perceive the quality of their Microservices and assurance processes and what parts are seen as challenging?

Since quantitative survey research with questionnaires would not be in-depth enough to cover participants' rationales, we chose a qualitative research approach. Qualitative methods analyze relationships between concepts and directly deal with identified complexity . Results from such methods are therefore very rich and informative and can provide insights into the thought process of participants. We selected semi-structured interviews as a concrete method. This left us with a basic agenda, but also allowed us to dynamically adapt our questions based on responses. For our interview participants, we defined the following requirements:

-   Significant professional experience (minimum of five years) and solid knowledge of service orientation

-   Technical role (e.g. developer or architect) that at least sometimes writes code

-   Recent participation in the development of a system with Microservices characteristics

We recruited participants via personal industry contacts of the research group and by attending industry meet-up groups on Microservices, where we approached companies from different domains and of different size. To ensure a base degree of heterogeneity within our population, we only allowed a maximum of three participants per company and if two participants worked on the same system, they needed to have different roles.

## Preparation for Data Collection

Before conducting the interviews, we created several documents. We prepared an *interview preamble* that explained the interview process and relevant topics. To make participants familiar with the study, they received this document beforehand. The preamble also outlined ethical considerations like confidentiality, requested consent for audio recordings, and guaranteed that recordings and transcripts would not be published. As a second document, we created an *interview guide* that contained the most important questions grouped in thematic blocks. This guide helped us to scope and organize the semi-structured interviews and was used as a loose structure during the sessions. We did not share it with interviewees beforehand. Additional interview artifacts used as a reference for certain topics or questions, like e.g. an exemplary list of assurance tools, were collected in a slide set which was was also not shared with participants before the interviews. For the analysis, we created a preliminary set of *coding labels* and a *case characterization matrix* containing the most important case attributes.

## Evidence Collection

In total, we conducted 17 individual interviews (no group interviews). Six of these were performed face to face and 11 via remote communication software with screen sharing. All interviews except for an English one were conducted in German. Each participant agreed to a recording of the interviews, which took between $\sim$`<!-- -->`{=html}45 and $\sim$`<!-- -->`{=html}75 minutes. We loosely followed the structure of the interview guide and adapted based on how a participant reacted. As an initial ice breaker, participants were asked to describe their role and the system they worked on. Later on, the topic shifted to the evolvability of the system and potential symptoms of technical debt. The next thematic block was the concrete assurance process for the system. We presented a maturity model with four levels that ranged from implicit and basic to explicit and systematic (see Fig. ). Participants were then asked to place themselves on the level that corresponded the most to their current assurance activities. From there, we discussed the details of the process such as techniques, tools, and metrics. Lastly, we asked questions about challenges and participants' satisfaction with the current process. The satisfaction and reflection questions relied on a five point Likert scale from -2 (very negative) to +2 (very positive) with 0 being the neutral center. After the interviews, we manually transcribed each audio recording to create a textual document. We then sent these documents to participants for review and final approval. During this review, interviewees were able to delete sensitive paragraphs or change statements of unclear or unintended meaning. The approved transcripts were then used for detailed qualitative content analysis.

## Data Analysis

As a first step, we performed the coding of each transcript. Using the created preliminary set of codes, we assigned labels to relevant paragraphs. During this process, several new labels were created and already finished transcripts were revisited. Labels were also renamed, split, or merged as we acquired a more holistic understanding of the cases. These coding activities followed the *constant comparison method* that is based on grounded theory . After the coding of all transcripts, we analyzed the details and code relationships of each individual transcript. This activity resulted in a textual description for every case.

In the second step, we applied *cross-case analysis* to identify important generalizations and summaries between the cases. We used the coding system and the created case characterization matrix as well as *tabulation* . For each research question, important findings were extracted from the transcript and documented. During this process, we also refined the case characterization matrix. General trends and deviations were documented and later aggregated into results and take-aways. All interview documents and artifacts as well as the results of the analysis are available in our anonymous online repository[^1].

```latex
\begin{table*}
    \centering
    % table caption is above the table
    \caption{General Case Characteristics: Assurance Maturity Level, Tools, Metrics, and Patterns}
    \label{table:cases}
    \begin{tabular}{
        p{0.01\textwidth}
        p{0.12\textwidth}
        p{0.06\textwidth}
        p{0.08\textwidth}
        p{0.2\textwidth}
        p{0.2\textwidth}
        p{0.13\textwidth}
    }
        ID & System Purpose & \# of Services & Assurance Maturity Level (0-3) & Tools & Metrics & Patterns\\
        \hline
        \hline
        S1 & Derivatives management system (banking) & 9 & 2 & IDE linting & -- & Event-Driven Messaging, Service Registry\\
        \hline
        S2 & Freeway toll management system & 10 & 3 & SonarQube (FindBugs, Checkstyle, PMD) & Test coverage, cyclomatic complexity, clone coverage, \# of defects per service, \# of failed tests, \# of code smells, \# of endangered requirements & Event-Driven Messaging\\
        \hline
        S3 & Automotive problem management system & 10 & 1.5 & SonarQube (FindBugs, Checkstyle, PMD) & Test coverage, clone coverage, defect resolution time & Event-Driven Messaging, Strangler\\
        \hline
        S4 & Public transport sales system & $\sim$100 & 1.5 & SonarQube (FindBugs), VersionEye & \# of code smells, test coverage, \# of outdated dependencies & Event-Driven Messaging, Backends for Frontends, Consumer-Driven Contracts, Tolerant Reader\\
        \hline
        S5 & Business analytics \& data integration system & 6 & \makecell[tl]{P5: 1\\P6: 1.5} & SonarQube (FindBugs, Checkstyle, PMD) & Test coverage & Event-Driven Messaging\\
        \hline
        S6 & Automotive configuration management system & 60 & 3 & SonarQube (FindBugs, Checkstyle) & Test coverage, \# of failed tests, \# of code smells, cyclomatic complexity, clone coverage, LOC & Event-Driven Messaging\\
        \hline
        S7 & Retail online shop & $\sim$250 & 2.5 & SonarQube (FindBugs, PMD, Checkstyle), Cobertura, IDE linting, custom static analyzer for coding conventions & Test coverage, cyclomatic complexity, clone coverage, \# of rule violations, velocity & API Gateway\\
        \hline
        S8 & IT service monitoring platform & 9 & 2 & SonarQube (FindBugs), IDE linting & Cognitive complexity, \# of code smells, test coverage & Event-Driven Messaging\\
        \hline
        S9 & Hotel search engine & $\sim$10 & \makecell[tl]{P10: 1.5\\P11: 3} & \makecell[tl]{P10: Checkstyle, IDE linting\\P11: SonarQube (FindBugs,\\Checkstyle)} & \makecell[tl]{P10: Defect resolution time,\\burndown\\P11: \# of code smells,\\\# of rule violations} & Event-Driven Messaging, Request-Reaction\\
        \hline
        S10 & Hotel management suite & 20 & 2 & SonarQube, IDE linting & Test coverage, \# of endangered usage scenarios & Self-Contained Systems, Backends for Frontends\\
        \hline
        S11 & Public transport management suite (S11a: human resource management part) & \makecell[tl]{10\\products} & \makecell[tl]{P13: 1.5\\P14: 2.5} & \makecell[tl]{P13: SonarQube (FindBugs,\\Checkstyle)\\P14: FindBugs, PMD, Cobertura,\\custom tool for architectural\\conformance checking} & \makecell[tl]{P13: \# of code smells\\P14: \# of architectural violations,\\\# of rule violations} & --\\
        \hline
        S12 & Retail online shop & $\sim$45 & 2.5 & SonarQube (FindBugs, Checkstyle, PMD), IDE linting, Codecov, Cobertura & Test coverage, cyclomatic complexity, \# of code smells, \# of rule violations & Event-Driven Messaging, Consumer-Driven Contracts\\
        \hline
        S13 & Automotive end-user services mgmt. system & 7 & 2 & SonarQube (FindBugs, Checkstyle, PMD) & Test coverage, \# of code smells, \# of rule violations & Event-Driven Messaging\\
        \hline
        S14 & Retail online shop & $\sim$175 & 2 & SonarQube (FindBugs, Checkstyle, PMD), Structure101, Codecov, Cobertura, Codacy & Test coverage, CI/CD pipeline duration, LOC & Event-Driven Messaging, Self-Contained Systems, Event Sourcing\\
        \hline
    \end{tabular}
\end{table*}
```

# Interview Case Characteristics

Our interviewees were from 10 different companies (C1–C10) of different sizes and domains (see Table ). Half of these were software & IT services companies that mostly developed systems for external customers. The companies from other domains always had an internal system owner. Every participant was located in Germany, even though some companies had sites in several European countries or even globally. From our 17 participants (P1–P17), 11 stated architect as their role while four were developers. The remaining two roles were data engineer and DevOps engineer. All participants possessed a minimum of five years of professional experience, with a median of 12 and a mean of 14.7 years. Altogether, we discussed 14 systems (S1–S14) and their evolvability assurance processes, where in three cases, two participants talked about the same system (S5, S9, S11).

For the sake of brevity, this publication only contains the aggregated results. We provide a detailed description of every case in our online repository$^1$. The descriptions include general information about the system, the details of the evolvability assurance process, and lastly reflections and challenges. Table summarizes the self-assessed assurance maturity levels as well as the usage of tools, metrics, and patterns to analyze and improve evolvability. Fig.  presents the summary of participants' reflections of their assurance processes. Lastly, the most important challenges reported by participants are listed in Fig. .

# Results and Discussion

By analyzing and comparing the individual cases, we identified several trends or common relationships. These generalizations, summaries, or notable deviations from common assumptions are presented in the following subsections that correspond to our three research questions.

## Assurance Processes (RQ1)

The intention of RQ1 was to find out what general techniques participants used to assure the evolvability of Microservices, how they structured these activities, and what participants' rationales for these decisions were. Since one Microservices characteristic is the decentralization of control and management, we also wanted to analyze how much central governance was applied.

In general, every analyzed assurance process had some degree of explicit and conscious addressing of evolvability, even though the sophistication and extensiveness of the applied techniques varied greatly. When looking at the larger systems, there were **two different approaches** for assuring evolvability: very decentralized with very autonomous teams (e.g. S9, S10, S12, S14) vs. centralized governance for macroarchitecture, technologies, and assurance combined with a varying degree of team autonomy for microarchitecture (e.g. S2, S3, S4, S7, S13). The latter kind was usually applied for systems that were built for external customers and that exhibited some project characteristics.

In the decentralized variant, the internal system was managed in a continuous product development mode, which created quality awareness by making people responsible and simultaneously empowering them. This variant is more in line with the Microservices and DevOps principle "you build it, you run it". Techniques in this variant also were by no means basic or implicit. Even though teams were allowed to choose their own assurance activities, they usually created a more or less structured processes that did not depend on external governance. This was hard to replicate for IT service providers that often did not operate the systems themselves (S2, S3, S4, S7, S13) and had to coordinate with external customers or even other contractors (S4, S7). Therefore, they relied more on central governance. Architect P4 described it as follows: "*In our case, the main challenge is to convince 300 people to move in the same direction. For that, we created a very large number of guidelines and rules for service creation.*"

Such **guidelines, principles, or standardizations** were nonetheless seen as important parts of the process in both variants. These coding labels were among the most frequent ones. Nearly all participants reported their usage in various areas such as architectural principles, rules for service communication, skeleton projects, style guides, cross-cutting concerns like logging or authentication, candidate technologies, or Docker images. The degree of enforcement varied between companies and was usually higher within the centralized variant. In the decentralized variant, pragmatism and simplicity were often more important than the strict adherence to rules and were reported as key principles ("KISS") by several participants (P5, P8, P10, P12, P15, P17).

To make assurance activities more efficient and objective, all participants saw **automation and tool support** as useful, albeit with varying enthusiasm. Several participants reported the integration of quality analysis tools into the CI/CD pipeline (P2, P3, P8, P9, P11, P14, P15, P17). This was often combined with quality gates that could prevent merging or deployment. For P17, the pipeline's execution time was very important: only tools that were absolutely necessary should therefore be integrated. Additionally, several participants advocated for a sensible usage of quality gates. P11 was frustrated with how difficult it would be to get a passing merge request due to the strict rules. Similarly, P10 mentioned that strict quality gates could hinder the deployment of important production bug fixes. P7's team did not use any quality gates because of continuous experimentation and prototyping. Lastly, P2's team circumvented some of these issues by applying quality gates only for releases and not for merge or pull requests.

Nearly all participants agreed that test automation was an important part for the assurance process of Microservices. While unit tests were very common, several participants also reported automated end-to-end tests for the integration of Microservices and stressed their importance (P2, P3, P7, P12, P16). Some teams also had more elaborate strategies that linked tests to requirements (S2) or usage scenarios (S10). Participants with only unit tests (P5, P6, P9) or barely any tests (P10) also saw the importance to bring their test automation to a higher level.

Despite the reported importance of automation and tool support, several participants also highlighted the usefulness of **manual assurance activities**. Code reviews were seen as an important practice to increase code quality and to share knowledge within the team (P1, P3, P4, P5, P7, P8, P10, P11). Pair programming was used for the same reasons by two participants and the downside of additional man-hours was willingly accepted (P8, P15). Lastly, refactoring was highly valued and some participants also explicitly mentioned the use of boy scouting during feature implementations (P11, P15) to efficiently increase code quality over time.

Even though some participants were proponents of concise documentation or architectural decision records within the source code repository (P10, P11), several systems relied on more elaborate **architecture and service documentation** in a system like Confluence or SharePoint (S1, S4, S5, S6, S8). Common types of documentation were system architecture, service dependencies and contracts between teams, service functionality and API descriptions, reference architectures and service blueprints, design rationales, or architectural principles and guidelines. For IT service providers, parts of this documentation was also used to communicate with the customer. Lastly, only P7 and P14 reported the conscious tracking of identified technical debt items for later debt management. P7's teams held an explicit meeting every two weeks, where the most important technical debt items were discussed and their prioritization was decided.

## Tools, Metrics, and Patterns (RQ2)

Our second research question targeted the application of and rationale for tools, metrics, and design patterns. Automation and tool support is an often cited Microservices characteristic and seen as necessary to manage a large number of small components. We were also interested if participants used tools and metrics specifically designed for service orientation. Lastly, we wanted to explore the usage of design patterns for evolvability construction, since there is a large body of patterns for Service-Oriented Architecture (SOA) and more recently also for Microservices.

While the usage of over a dozen different **tools for evolvability assurance** was reported, 14 of 17 participants named SonarQube as a central tool that was usually integrated into the CI/CD pipeline. Since P1 and P10 planned to introduce it soon, P14 remained the only participant that would not use SonarQube in the foreseeable future. Reported reasons for its popularity were the OpenSource license, the easy installation, plugin availability, and configurability. In Java-focused systems, SonarQube was often extended with tools like FindBugs, Checkstyle, and PMD. Additionally, specialized tools for test coverage like Cobertura (P8, P14, P15, P17), Codecov (P15, P17), or Codacy (P17) were used. For a basic degree of local and immediate quality assurance, IDE linting via e.g. TSLint, ESLint, and PHPLint was reported by some participants (P1, P8, P10, P12, P15).

With respect to **metrics**, 10 of the 17 participants reported the usage of test coverage, even though some perceived this metrics as less important than others and were very aware of possible quality differences with automated tests. P12 termed it as follows: "*Even I could fake the coverage for two classes you give me in like five minutes. You can write a test that brings coverage to about 60%, but actually it covers like 2%.*" Some participants also focused on additional metrics for testing and functional correctness like the number of failed tests over time (P2, P7), the number of defects per service (P2), or the number of endangered requirements (P2) or usage scenarios (P12). Most SonarQube users also payed attention to standard findings like code smells, code duplication, and cognitive or cyclomatic complexity. Participants with rule-based tools like FindBugs, Checkstyle, or other linters used the number of rule violations as a simple metric that had to be zero.

Overall, most applied metrics were focused on source code quality, even though their effectiveness for the whole system was seen as controversial by a few participants (P8, P17). P17 described it as follows: "*Most of these metrics relate to a single project, which is very useful when I have a monolith with a million LOC. However, if I have a service with 1000 LOC which code base is separated from all other 150 Microservices, most of these metrics lose their importance.*" With respect to productivity metrics, some interviewees reported the usage of defect resolution time (P3, P10), velocity (P8), sprint burndown (P10), or deployment duration (P17). These were important for them to control and manage software evolution.

While architecture-related topics like Microservices dependencies were very prevalent during our interviews, participants generally did not apply **architecture-level tools and metrics**. P14's team used a custom tool for architectural conformance checking in the monolithic code base for a sub product of S11 and P17 reported the intermittent usage of Structure101 for a larger subsystem that also consisted of one code base. Apart from that, tools or metrics were exclusively focused on code quality with a local view for a single service. No (semi-)automatic efforts were mentioned to evaluate the architecture of a Microservice-based System.

Likewise, no participant reported the usage of a tool or metric specifically designed for service orientation. When we explicitly asked about service-oriented metrics like the coupling of a service, the cohesion of a service interface, or the number of operations in a service interface, several participants indicated that these sounded interesting and useful (P1, P5, P6, P7, P8, P15). Some interviewees also noted that the underlying principles of these metrics were important guidelines in the architecture and design phase (P7, P8, P10, P15, P17). They tried to manually respect these principles during e.g. service cutting, even though they currently had no concrete measurements in place to validate them.

Another common theme in this area was the **healthy and non-patronizing usage of tools and metrics**, which should be respected when developing Microservices in decentralized and fairly autonomous teams. As already mentioned, several participants voiced reservations against test coverage (P10, P12, P15). P14 also warned that a strict metric focus would pose the danger that people optimized for measurements instead of fixing the underlying problems. Moreover, P15 perceived it as difficult to interpret measurements of a single service without a point of reference or a system-wide average (P15). P17 advocated for a sparse usage of tools, because too many metrics could not be analyzed by developers and their collection could slow down the deployment pipeline. Only tools that would support the analysis of current problems should be kept. Lastly, P8 and P14 highlighted the agile principle of "individuals and interactions over processes and tools": the usage of tools and metrics should support developers in their daily work and not be a frustrating and alienating experience for them.

We also analyzed the usage of **service-oriented design patterns** as conscious means to increase evolvability. In general, we did not find a widespread usage of them. Most common was *Event-Driven Messaging* that was partially applied in 11 of the 14 cases. While several participants stated that the pattern was used to decouple services, another intention was to implement reliable asynchronous and long-running communication. The pattern was sometimes paired with *Request-Reaction* (P10, P16). Apart from messaging, most participants applied activity patterns like *Service Refactoring*, *Service Decomposition*, and *Service Normalization*. In line with the philosophy of evolutionary design, Microservices were frequently split and merged.

Other patterns were used sporadically. P12 and P17 applied the *Self-Contained System* paradigm to achieve vertical isolation between subsystems. In a migration context, P3 and P16 reported the usage of the *Strangler* pattern to extend an existing monolith with new Microservices until its final replacement. To place an intermediary between service consumers and producers, P4 and P12 implemented the *Backends for Frontends* pattern that would also prevent too many concurrent long-running HTTP requests. Similarly, P8 chose the *API Gateway* pattern which also brought benefits for security. The patterns *Consumer-Driven Contracts* (P4, P15) and *Tolerant Reader* (P4) were applied to make service interface evolution more robust and to prepare consumers for future changes. Lastly, P1 was the only participant to explicitly report the usage of the *Service Registry* pattern for dynamic service discovery, even though some participants may have used similar functionality via Kubernetes.

## Evolvability Reflections and Challenges (RQ3)

With RQ3, we wanted to analyze participants' perception of the general evolution qualities of their Microservice-based systems as well as their satisfaction with their current assurance processes (see Fig. ). We also tried to summarize what participants experienced as the most important challenges for the evolvability of their Microservices (see Fig. ).

Generally, our interviewees perceived the **evolvability of their Microservices** as positive (mean: +0.88, median: +1), especially in cases with a migration context where a monolith had been rewritten. Only two participants chose a negative rating (-1). Architect P4 saw the high degree of technological heterogeneity and the very different service granularity as threatening for the large project, especially once S4 would be handed over to the smaller maintenance team. Data engineer P11 described the chosen service cuts as inefficient and politically motivated and worried about significant issues with the consistency of the data model as well as the inaccessibility of code due to distributed repositories.

As for more specific quality attributes, the analyzability of individual services would be much improved (P1, P8, P10, P16, P17), even though grasping and understanding the whole system would be difficult (P7, P8, P11, P17). When compared to most monoliths, the modularity of Microservices would make it very convenient to change or add functionality (P1, P3, P6, P9, P12, P17) and would also allow to efficiently scale-out the development with multiple teams (P7, P16). Even though reuse is usually a theme more common in SOA, several participants reported a positive reusability of their Microservices (P1, P3, P7, P8, P10, P17). To reduce coupling, some participants avoided the sharing of non-OpenSource code between services via duplication (P7, P10, P15, P17). Others tried to consciously increase reuse via shared libraries (P3, P9) or by slightly generalizing service interfaces (P1). Lastly, participants reported that individual services would be easy to test (P3, P7, P10, P13, P15) and to replace (P2, P15, P17).

Since most systems were fairly young or even still in the process of being migrated, individual services were usually of a good quality. Basic symptoms of technical debt or bad code quality were rarely seen as an issue, especially since a single service would be easy to replace. However, **problems related to architecture and the data model** were reported as serious threats for long-term evolvability (P3, P7, P11, P13, P15). This was sometimes exacerbated, because coordination between autonomous teams would be difficult (P4, P10, P11, P15). Moreover, finding the appropriate service granularity was a prevalent theme and service cutting was by far named as the most challenging activity that was also associated with frequent refactoring (P2, P3, P4, P6, P7, P9, P11, P12, P15). Harmful inter-service dependencies sometimes led to ripple effects on changes (P3, P5, P9, P11, P15) which made adding or changing functionality slower and more error-prone. Breaking API changes caused similar effects for service consumers (P1) or automated tests (P2). Participants did not use any tools to support service decomposition or metrics to evaluate the quality of the chosen cuts, e.g. via coupling or cohesion. P2 described it as follows: "*In my opinion, there are no useful tools to split up a monolith. It's always a very difficult manual activity. You can use something like Domain-Driven Design, but that's just a methodology which doesn't give you a concrete solution.*"

Participants were divided when it came to **technological heterogeneity**. In very decentralized environments, it was generally perceived as overall beneficial (P10, P15, P17), as it would allow choosing the best solution for problems at hand, broaden developers' experience and skills, and make a company a more attractive employer. Other participants perceived it as potentially dangerous and wished for a more sensible handling of technology hypes (P3, P4, P12, P16). Similarly, the mix of legacy and modern service technology would sometimes pose additional problems (P2, P11, P13, P14), like in the case of S9 where additional tooling was necessary to integrate legacy PHP components. Most participants also noted that significant efforts had to be spent on mastering new Microservices and DevOps technologies and it would be problematic to find skilled developers (P1, P3, P5, P6, P9, P10, P12, P13, P16). Overall, participants were very aware of the human factors of evolvability and sometimes even saw them as more challenging as technical ones. Knowledge exchange between teams was therefore a high priority for some interviewees (P10, P13, P15).

Concerning participants' **reflection of their assurance processes** (see also Fig. ), most saw the effectiveness of their assurance activities (mean: +0.76, median: +1) as well as overall impact on productivity (mean: +0.59, median: +1) as positive. Only three interviewees (P9, P11, P12) reported that activities would hinder development efficiency (-1) and would sometimes slow down feature development. Moreover, participants generally wanted to invest more effort for the assurance (mean: +0.76, median: +1) and try out new techniques or metrics. No one reported that he would like to reduce efforts.

However, the influence of Microservices on the assurance process was seen as controversial. While testing a single service would be easy, integration testing would be more complex because of an additional layer (P2, P3, P13). This would be especially critical, if Microservices were developed in independence for a long time and integrated at a later stage. Furthermore, root cause analysis of issues would be more complex in such a highly distributed system (P3, P11). A very commonly named concern was that keeping a system-centric quality view and assessing the macroarchitecture would be much more difficult (P4, P6, P7, P8, P10, P11, P15, P17), which P8 described as follows: "*I'd say we are pretty good when it comes to assuring the evolvability of single services. However, we have a lot of catching up to do for everything that crosses product or service boundaries.*" Distributed code repositories and autonomous teams would make the access to code as well as static analysis more complicated. It would also be hard to compare metrics between services and relate them to system-wide averages (P8, P11, P17).

Nonetheless, participants also named positive factors. Small services would not only be easy to replace, people would also be much more motivated to fix a small number of issues for a project (P15, P17), which P15 described as follows: "*In a monolith with 100.000 FindBugs warnings, you are completely demotivated to even fix a single one of those. In a Microservice with 100 warnings, you just get to work and remove them.*" If adopted correctly, Microservices would also bring a cultural change with respect to quality awareness and responsibility (P10, P15, P17). P17 highlighted the importance of continuous product development in this regard: "*If you work in a project mode, evolvability assurance usually annoys you, because you have short-term goals and want to finish the project. In a product mode, the team knows that they sabotage their system's evolvability in the long run, if they take too many short-cuts.*" Lastly, P15 noted that while they were relatively satisfied with their current evolvability assurance activities, they did not really invest much efforts into researching and designing a fitting evolvability assurance strategy for the future. Finding out which approaches, tools, and metrics worked best for their Microservices could be a vital advantage in the long-term.

# Threats to Validity

A number of possible limitations have to be mentioned for this research. Concerning **internal validity**, it could be possible that interviewees were not completely honest during their answers, which is a common problem in survey- or interview-based studies. Nonetheless, we think that the risk for this study is rather low. Most participants were quite comfortable to describe negative aspects of their systems and the discussed topics were only of low to medium sensitivity. Furthermore, confidentiality and anonymity was provided. As a second threat, participants could have provided incorrect answers, because they misunderstood questions or concepts. To limit this risk, important concepts were defined before the corresponding topics. Moreover, we posed additional clarifying questions if participants used terms of unclear meaning. Some interviewees also asked questions themselves, if questions or terms were not fully clear to them. In some cases, our analysis relied on the subjective opinion of participants, e.g. how they rated the evolvability of their system. Since participants could interpret such ratings very differently, detailed comparisons between systems may be difficult. Moreover, we only relied on one single opinion for most systems and people may have different opinions of systems, as is visible in the case of S9.

To increase interpretation validity and reduce researcher bias, every transcript was proofread by both moderators. Additionally, we strongly encouraged our participants to adjust wrong or unclear statements in the transcripts. Nevertheless, there still remains the small possibility that some of the more figurative paragraphs were misinterpreted. Overall, the largest threat in qualitative studies is that summarizing and aggregating results depends heavily on the subjective interpretation of researchers. We tried to limit this bias by relying on a precise coding system and by critically discussing all results between the moderators.

Concerning **external validity**, no distributions can be generalized from our 17 interviews, e.g. the industry usage of SonarQube for Microservices. Because this is a qualitative study, the center of this research are the intentions and opinions of participants as well as the relationships between concepts. An additional threat for generalizability could be that our participants were exclusively based in Germany, even though we included European and international companies. We also tried to achieve diversity with respect to company domains and sizes, but had nine participants from software and IT services companies ($\sim$`<!-- -->`{=html}52%). Lastly, possible selection bias could only be reduced in the case of C2 where we applied a random sampling to pick three out of seven candidates.

# Conclusion

We conducted 17 interviews with software professionals from 10 companies and talked with them about the evolvability assurance for 14 different Microservice-based systems. We found that systems developed for an external customer in a project-like manner generally relied more on central governance for the assurance. Continuous product development of internal systems exhibited more decentralization and team autonomy for the assurance process. Guidelines like architectural principles or rules for service communication were seen as important to provide a consistent basis for evolvability and to frame an otherwise diverse technology usage. While participants saw automation and tool support as important for evolvability assurance, they still relied a lot on manual activities like code review or boy scouting. Tool and metric usage was very focused on source code quality. No architectural tools and metrics were applied, even though most reported challenges and issues like service cutting were related to software architecture. This may indicate the importance of Architectural Technical Debt (ATD) management for Microservices. Likewise, no tools or metrics specifically designed for service orientation were used, even though most participants stated the significance of their underlying design principles. Participants generally perceived their fairly new Microservices as decently evolvable, even though things like distributed code repositories and difficult macroarchitectural assessment would make the assurance process more complex.

Future work that covers the areas of maintenance, evolution, and technical debt in the context of Microservices should take these findings and industry sentiments into account. In particular, academia can support industry with methods, metrics, or tools that aid macroarchitectural assessment of Microservices or provide a more system-centric view. We perceived support for service cutting activities and metrics to evaluate service granularity, coupling, or cohesion as concrete gaps that could save industry a lot of refactoring efforts. Finally, issues in the area of human evolvability factors with Microservices like the handling of hyped technologies as well as coordinating and exchanging knowledge between decentralized teams were described as important by participants.

[^1]: https://doi.org/10.5281/zenodo.2586916