# Abstract

In the world, in which acceptance and the identification with social communities are highly desired, the ability to predict the evolution of groups over time appears to be a vital but very complex research problem. Therefore, we propose a new, adaptable, generic, and multistage method for Group Evolution Prediction (GEP) in complex networks, that facilitates reasoning about the future states of the recently discovered groups. The precise GEP modularity enabled us to carry out extensive and versatile empirical studies on many real-world complex / social networks to analyze the impact of numerous setups and parameters like time window type and size, group detection method, evolution chain length, prediction models, etc. Additionally, many new predictive features reflecting the group state at a given time have been identified and tested. Some other research problems like enriching learning evolution chains with external data have been analyzed as well.

# Introduction

Network science is a very interdisciplinary domain focusing on understanding the relational nature of various real-world phenomena using for that purpose diverse network models. Commonly, networks consist of smaller, more integrated structures called groups, communities, or clusters. In practice, both the groups and whole networks evolve and change their profiles over time. Hence, their analysis demands advanced computational methods to understand and predict their future behavior. For that reason, group evolution prediction is an essential component of computational network science.

One of the domains explored by network science are biological networks. Viruses are as old as life on earth. At the same time, they are very young, as they constantly mutate to change their lethal attributes. Influenza, unlike other viruses which are rather stable, evolves much more rapidly and kills up to one million people worldwide every year. We can try to protect ourselves using vaccines. However, the rate of mutation is too rapid to provide an effective cure. What is more, the development of a new drug requires a huge amount of money and lasts from a few to a dozen or so years. Despite these difficulties, new drugs are introduced to the market every year. For example, antagonist drugs (also called blockers) are designed to bind to specific receptors to block the disease's ability to attach to these particular receptors, thereby immunizing the body to the disease. Unfortunately, diseases react to drugs and eventually mutate, creating a variety that will bind to other receptors. Therefore, we need methods that will be able to track the evolution of the disease, and based on the history of its mutations, will be able to predict the most likely future mutations. To track diseases mutations, we can focus on the group of receptors that it binds to, and observe how such group evolves. Based on the history of changes in the lifetime of this group, we can try to predict what will be the next change. Predicting the direction of the mutation could significantly reduce the amount of time and money needed to study the disease. With such knowledge, we would be able to start preparing the drug in advance and bring it to the market much faster and cheaper.

Another area that widely applies network science, especially its branch called social network analysis (SNA), is marketing, in particular advertising. Let us imagine that a start-up company invented a new generation of diapers – *Smart Diapers*, which are extra soft, super absorbing, and additionally, can communicate with parents' smartphones to notify when their change time comes. The company invested very much in their development, therefore, it has a limited budget to advertise the product. The owners decided to introduce the product to discussion groups on the Facebook platform where parents from different countries/cities create and join independent groups to talk about and comment on new products for babies, share general advice about raising children, sell used clothes, etc. Convincing members (parents) of such relevant, targeted groups to use and buy the new diaper product would be much more effective and cheaper than advertising the broader community using expensive TV commercials. Additionally, the word-of-mouth recommendation is commonly believed to be the most powerful marketing tool. However, the vital question rises here: which Facebook groups the company should invest in its limited resources, i.e., time and money? In the newly created relatively small groups that might be very active and are expanding fast, or in the larger groups that might be not very active in the nearest future? Which of these groups will be still running or growing in a few weeks/months/years and which one will disappear? That is why the knowledge about the history, current state, and future evolution of groups is crucial at decision making on where to allocate the resources.

In 2007, Palla et al. have defined the problem of group evolution identification. In the following years, dozens of solutions to this problem have been proposed. One of them was the highly cited GED method . Existing surveys describe as many as 12 or even over 60 methods . All of them are focused on defining possible events in the community life, hence, tracking the historical changes. This, in turn, has led to emerging a new problem – predicting future changes that will occur in the community lifetime. Some of the first methods concerning prediction of some aspects (e.g., determining lifespan) of the group evolution were: (1) Goldberg et al. – they focused on predicting the lifespan of evolution for a group; (2) Qin et al. – analyzed dynamic patterns to predict the future behavior of dynamic networks; and (3) Kairam et al. – they investigated the possibility of prediction whether a community will grow and survive in the long term.

Note that the methods for tracking group evolution can be also utilized to other similar prediction problems, like link prediction, churn prediction, as well as to understand evolution of software (Unix operating system networks) or dynamics of social groups forming at coffee breaks.

In 2012, we proposed a new concept, in which the historical group changes were utilized to classify the next event in the group's lifetime. In this first trial, we have used only event type and size of the group to describe its state at a given time. Over the next year, we have investigated the concept and adopted it to two methods for tracking group evolution – the GED method and the SGCI method. This resulted in the first method for group evolution prediction. It was the predecessor of the GEP (Group Evolution Prediction) method described in this paper. Since then, a few more methods have been proposed. At the end of 2013, İlhan et al. presented their research with several new measures describing the state of the community and a new method for tracking group evolution. In 2014, Takafolli et al. applied the binary approach to classifying the next change that group will undergo. They used 33 measures to describe the state of the community. We have presented new results in 2015, where, apart from new measures, the influence of the length of the history used in the classification was examined. Later the same year, Diakidis et al. adapted the GED method to conduct their research with 10 measures as predictive features. In 2016, İlhan et al. presented new results and proposed a method to select measures, which should be the most useful as predictive features for a given data set. More recently, Pavlopoulou et al. used 19 measures already validated in other works and studied whether employing the temporal features on top of the structural ones improves prediction, as well as what is the impact of using a different number of historical community states on the prediction quality.

Unfortunately, all of the methods proposed to this day have some drawbacks (see the Comparison with other methods section) and have been designed to solve a particular problem, hence, their application area is rather narrow. Therefore, in this paper, a new generic and comprehensive method to predict the future behavior of the groups, based on their historical structural changes as well as experienced events, is proposed, evaluated and discussed.

Some of the contributions of this work are: decomposing the group evolution prediction problem, proposing and extensively evaluating the modular method that can be applied to any dynamic network data, proposing new predictive features, performing the features' ranking, proposing a new concept of data set enriching, initial evaluation of the transfer learning technique, an example and discussion on the concept drift problem in group evolution prediction, reviewing all proposed methods in the field.

# Methods

## Decomposition of the group evolution prediction problem

The crucial matter in developing the modular method predicting group evolution, called *GEP*, was the identification and separation of the components of the entire group evolution prediction problem. The appropriate problem decomposition and information flow between particular components (dependencies) are depicted in Eq  and Fig. .

$$\label{eq:decomposition}
IS\xrightarrow[S_1]{TWT}TW\xrightarrow[S_2]{NT}TSN\xrightarrow[S_3]{CDM}G\xrightarrow[S_4]{CETM}EC\xrightarrow[S_5]{FE}PF\xrightarrow[S_6]{classification(CH)}Q$$

The data from the input stream $IS$ is divided into time windows $TW$ using the time window type definition $TWT$. For each time window $TW$, a complex/social network is created using the network type definition $NT$, resulting in the temporal complex/social network $TSN$. Within each time window $TW$ in $TSN$, some groups $G$ are identified using a community detection method $CDM$. Next, similar and consecutive groups are matched using a community evolution tracking method $CETM$, as well as the transition is labeled with an event type out of the set of possible changes $CH$. The matched groups are combined into evolution chains $EC$ that may consist of many successive changes. For each community state in $EC$, the feature extraction process $FE$ is applied in order to obtain a set of predictive features $PF$ describing the community state at a given time. Using features $PF$ in the form of a vector representing each evolution chain $EC$, classification of possible changes $CH$ is performed. The classification task (stage $S_6$) is to learn and finally label the next change(s) in community lifetime. The output of the classification process is a set of classification quality (performance) measures $Q$, for example, F-measure, accuracy, precision, or recall. The identified components were converted into six stages $S_1$-$S_6$ of the GEP method, Fig. .

## GEP method

The GEP framework consists of six main stages (Fig. ): (1) time window definition, (2) complex network extraction for the defined periods, (3) community detection in periods, (4) group evolution tracking, (5) evolution chain identification for communities together with feature extraction and computation for each chain and (6) classification, containing classification model learning and testing. Each of them can be implemented by means of different methods and approaches depending on research need and prerequisites, e.g., complexity level. The formal definition of the GEP method is as follows:

**Definition 1**. *The GEP method is defined as an octuple $<IS, S_1, S_2, S_3, S_4, S_5, S_6, Q>$, where: $IS$ is an input stream of activities, e.g., phone calls, linking two actors (network nodes) $x, y$ at time $t_i$; $S_1$ is a set of considered time windows of the given type $TWT$; $S_2$ is a set of considered approaches to temporal complex / social network $TSN$ creation from $IS$ using time window definitions from $S_1$; $S_3$ is a set of considered approaches to community detection methods $CDM$ for each time window in $TSN$ from $S_2$; $S_4$ is a set of considered approaches to tracking community evolution methods $CETM$ for communities from $S_3$; $S_5$ is a set of considered approaches to feature extraction for evolution chains from $S_4$; $S_6$ is a set of considered approaches to classification, including learning, training, validating, undersampling, oversampling, and feature selection techniques; $Q$ is a set of considered classification quality measures, for example, F-measure, accuracy, precision, recall, estimated based on the classification results from $S_6$.*

The methods enumerated especially in $S_1$, $S_3$, $S_4$, $S_6$ also include the space / set of their parameters.

The output of one stage $S_i$ is the input for the next stage $S_{i+1}$, e.g., communities detected in $S_3$ are used to discover their evolution in $S_4$. All these stages, together with parameters of the methods used, are more in-depth described in . They also require an appropriate definition of data structures to facilitate hassle-free implementation.

## CPM method

The Clique Percolation Method (CPM) proposed by Palla et al. is the most widely used algorithm for extracting overlapping communities. The CPM method works locally, and its primary idea assumes that the internal edges of a group have a tendency to form cliques as a result of high density between them. Oppositely, the edges connecting different communities are unlikely to form cliques. A complete graph with $k$ members is called k-clique. Two k-cliques are treated as adjoining if a number of shared members is $k$–1. Lastly, a k-clique community is the graph achieved by the union of all adjoining k-cliques. Such an assumption is made to represent the fact that it is a crucial feature of a group that its nodes can be attained through densely joint subsets of nodes.

## Infomap method

The Infomap method proposed by Rosvall and Bergstrom uses the information-theoretic approach to cluster nodes within a network. It focuses on information diffusion across the graph and compression of the information flow description obtained from a random walker, which is chosen as a mean of information diffusion. Infomap changes the problem of finding the best cluster structure into finding the partition with the minimum description length of an infinite random walk. It follows the intuitive idea that if the community structure is present, the random walker will spend more time inside the community because of its higher edges density. It means that the transition to another cluster will be less likely.

## GED method

The Group Evolution Discovery (GED) method is one of the best methods for tracking community evolution. It uses inclusion measure to match similar communities from neighboring time windows. This measure takes into account both the quantity and quality of the group members. The quantity is reflected by the first part of the inclusion measure, i.e., what portion of the members from group $G_1$ also belongs to group $G_2$. The quality is expressed by the second part of the inclusion measure, namely, what contribution of important members from group $G_1$ is in $G_2$. It provides a balance between the groups that contain many of the less important members and groups with only few but key members. The inclusion measure and the group size determine the type of community change. The authors defined seven possible event types: forming, dissolving, continuing, growing, shrinking, merging, and splitting. The method can work with any community detection method and with any group similarity measure, thus, providing great flexibility.

## İlhan et al. method

The İlhan et al. method works with the disjoint type of communities and utilizes the function by Hopcroft et al. to calculate the similarity between two communities. The event types that can occur in the community lifetime and also the classes being classified are: survive, growth, shrink, merge, split, and dissolve. The measures used as predictive features are divided into two categories: structural and temporal community measures. In total, nine features per timeframe are used, i.e., number of nodes and edges, intra and inter measure of community edges, betweenness, degree, conductance, aging, and activeness. If one calculates four network measures beforehand (average path length, betweenness, clustering coefficient, embeddedness), the method can also identify features that should be the most prominent for a given network profile.

# Results

Suitable decomposing the problem of group evolution prediction (see the Methods section and Fig. ) was crucial in solving the problem. It allowed to analyze distinct phases of the process and to propose multiple solutions for each phase. The GEP method was extensively analyzed on fifteen real-world data sets (see for their profiles), for which more than 1,000 different temporal networks were created, and in total, more than 5,000,000 individual classification tasks were performed. However, to keep the article clear and concise, only selected results are presented for each stage.

## Stage 1: Time windows creation

At first, the data is divided into time windows. Three main approaches can be considered in this context: (1) equal length periods – the events and relations are segmented based on their timestamp; (2) the same number of relations in each time window; (3) the arbitrary division, based on the data context. Additionally, the type and size of time windows have to be decided, which may be a challenging task. There are three most common types of time windows: disjoint, overlapping, and increasing.

A proper choice of the time window type and size has a direct impact on the following GEP stages, especially on the number of evolution chains discovered by the tracking method (Stage 4). If relations between individuals in a data set have a tendency to change rapidly, then disjoint time windows would be a poor choice since there may not be too many relations lasting between two consecutive time windows. As a result, the tracking method will not provide any events (Stage 4), so there will be no input to a classifier resulting in no event to predict (Stage 6). The too large size of the time window, in turn, might lose some information about community changes that occurred in the meantime.

So far, there is no formula which determines the right type and size of the time window, but a few guidelines can be provided based on our extensive experiments:

-   If the network is sparse or changes rapidly, the overlapping time window should be used. Usually, the offset equal to 30% of the time window size is enough to obtain a reasonable number of events between the consecutive time windows;

-   The time window type and size should be adjusted to the context of the given data set, e.g., the co-authorship network, referring to researchers who often publish only once a year, should evolve smoothly with the 1-year disjoint time windows;

-   If the persistent groups are the goal of analyses, the increasing time window should be utilized, as it provides mostly the continuing and growing events;

-   If relations between individual nodes are recurrent and the network is rather dense, one may try using disjoint time windows to lower the computational cost;

-   It is acceptable and even preferable to repeat the selection of the time window type and size several times to see which approach yields the best results.

The most common choice in our studies was the overlapping time windows with the offset between 30% - 50% of their size.

## Stage 2: Formation of networks

The parameters that can be adjusted at the creation of networks for each time window is the set of edge attributes, in particular, their weights and direction. The weighted/unweighted, as well as directed/undirected profile of the network, did not yield a significant impact on computational complexity nor classification accuracy. Some community detection methods, however, may be incompatible with the networks of particular characteristics or may ignore some attributes, e.g., weights. The CPM and Infomap methods, used in the experimental studies, are capable of handling the most important network attributes.

## Stage 3: Community detection

Some community detection methods can produce both disjoint and overlapping communities, but there are only a few methods for tracking the evolution (Stage 4) that can deal with the overlapping groups. Overall, the methods extracting disjoint communities perform faster than the ones providing overlapping groups. In some extreme cases, when the network is very large, the CPM method is unable to extract groups due to its enormous memory requirements. It is hard to compare two types of the grouping methods in terms of their impact on the classification accuracy, as each type of clustering delivers a different set of communities resulting in a different distribution of evolution events. Besides, the profile of the groups may be diverse, e.g., networks grouped with the CPM method tend to have a single giant component with many small overlapping groups alongside. This method also inclines to leave out nodes that do not belong to any clique, thus, excluding them from further consideration. If the network is sparse, a major fraction of the network may be omitted. In the most extreme case, the CPM method neglected even as many as 97% of network nodes, what resulted in a deficient number of communities and evolutions (Fig. A), and eventually in very low classification accuracy, Fig. B. At the same time, the Infomap method performed very well, identifying a large number of communities. Furthermore, the overlapping groups are likely to generate more merging and splitting events in Stage 4, since there are plenty of similar and overlapping communities in the consecutive time windows. On the other hand, the Infomap method tends to produce many communities having only 2 or 3 nodes. In general, while considering which type of grouping method to use the data context should be a crucial factor.

## Stage 4: Stepwise evolution tracking and chain identification

Regardless of the method, tracking the evolution of community is a computationally demanding task. The method has to iterate over all time windows and compare all the communities in order to detect similar ones. Although the methods for tracking group evolution can be very distinct, especially while defining the possible event types, our earlier study showed that the selection of the method has no significant impact on classification accuracy. In this evaluation, we use the GED method since, in the last evaluation of existing community evolution tracking method, it was selected as the one giving the most satisfying results.

The parameters of the selected method might influence the classification results, e.g., the alpha and beta parameters of the GED method have a direct impact on the number of evolution events discovered – the lower the threshold, the more events obtained (see for details). In the experimental studies, the most common value for the alpha and beta parameters was 50%. If the network is dense and relations are recurrent, the alpha and beta might be even increased to 70%. On the other hand, when the method provides a small number of the evolution events, the alpha and beta should be reduced to, e.g., 30%. Apart from the selection of the evolution tracking method, the length of the evolution chain has to be decided. The longer the evolution chain, the more predictive features for the classifier in Stage 6, hence, the higher computational complexity. Nevertheless, the results presented in Fig. C revealed that it is worth dedicating some more time and resources to extract longer chains since it can boost classification accuracy. The overall score achieved with the evolution chains containing six community states was 32% higher than the results achieved with shorter 2-state chains. In case of limited time or resources, the chains with the length of 2-3 states should be reasonably good.

## Stage 5: Feature extraction

In order to predict the future evolution of the group, we need to describe its recent and historical states by means of predictive features. Based on these features and previous evolutionary changes used to learn the model, we are able to forecast the next changes. The crucial features that are at our disposal are structural network measures computed for the previous group states. Calculation of all measures may be a very demanding task since they need to be evaluated for every community state in the evolution chain. Additionally, some measures, e.g., betweenness centrality, require finding all shortest paths for each pair of nodes in the community or network. The experiments revealed, Fig. D, Fig.  that the set of predictive features has a significant impact on classification accuracy, as they are used to build the classification model, see also , Feature Selection section. Therefore, it is highly recommended to compute as many predictive features as possible to deliver to the classifier a wide variety of descriptions to choose from.

To significantly enhance the already existing approaches, many new predictive features are proposed in this paper (see , Predictive Features section). We have clustered structural features into three general types: (1) *microscopic* – calculated for individual nodes, e.g., node degree, (2) *mesoscopic* – quantifying single groups, e.g., group size - no. of nodes, and (3) *macroscopic* – describing the whole network, e.g., network density. Mesoscopic features also include normalized group measures like the group size divided by the network size. Besides, node-based (microscopic) measures can be aggregated (usually averaged) at either *local* (group) or *global* (network) level resulting in *microscopic local* or *microscopic global* features, respectively.

All computed features were thoroughly evaluated in terms of usefulness for the classifier and rankings of the most prominent features were built, see , Feature Selection section, especially Tab. 5-9. For the evolution chains of a variable length, different rankings were obtained. For the shortest 1-state evolution chains, only macroscopic (network) features were helpful, which may result from the fact that communities with a short history are considered unstable and vulnerable to the environment they are a part of. For the evolution chains with the increasing time windows, the features describing the local structure, especially the centrality- and distance-based measures, were more informative for the classifier, as the changes between the consecutive increasing time windows were delicate and occurred at the microscopic rather than macroscopic level. The neighborhood-based features were among the most valuable features for the longest 8- and 9-state chains, which lead to believe that for the long-lasting communities, the relations with their surroundings are a better predictor of the forthcoming change than, e.g., the macroscopic features. In general, the variations of the eigenvector-, eccentricity-, and closeness-based features were present in most of the selective rankings, which suggests that centrality- and distance-based measures obtained on the node level are the most prominent ones. Hence, in case of limited computational capacity, these features should be respected before any other. However, out of all features considered by the classifiers, the Backward Feature Elimination selected only up to 34% of them as prominent, i.e., used by the classifier to make a decision, Fig. A.

Additionally, it turned out that usually over 90% of the selected prominent features were obtained from the last three community states, Fig. A1. For example, when the evolution chain length was 8, and the next change was classified, all the prominent features were from the 8th, 7th, and 6th group profiles. It means that the most recent history of the community has the most significant impact on its next change. This is an extremely useful conclusion if one has limited computational capabilities and cannot calculate community profiles for all states or does not possess data about older history. The number of features has a direct impact on the duration of the entire learning process, Fig. C.

## Stage 6: Prediction

In the last stage, the machine learning techniques, such as oversampling, undersampling, feature selection, and first of all, model training and adjustment are applied to achieve the highest possible prediction quality. The common problem with the training data is an imbalanced distribution of output classes, Fig. A. In extreme cases, when one class greatly dominates over the other ones, a trained model tends to assign the dominant class to most observations. Then, the solution is to apply additional preprocessing techniques like oversampling and undersampling to generate additional observations or to filter out predominant ones, thus providing a distribution closer to flat. Another common problem is overfitting the classifier by providing too many features or observations. In order to prevent from such case, feature elimination technique may be applied, which unfortunately is very expensive in terms of computational complexity.

Additionally, the proper classifier should be selected, and its parameters need to be accordingly adjusted. In the experimental study, fifteen different classifiers were compared in terms of the classification accuracy, Fig. . The tree-based classifiers and meta-classifiers (equipped with decision trees) performed best. Many classifiers could not efficiently handle imbalanced data, so the undersampling and oversampling techniques were applied, resulting in notably better prediction quality, Fig. B. On the balanced data set, a classifier focuses on the predictive features computed for the community states instead of focusing on the event distribution.

The Friedman statistical test  with the Shaffer post-hoc multiple comparisons  was performed to obtain rankings of classifiers on the imbalanced and balanced data sets (cf. , Tab. 10). In both cases, the Bagging classifier (with the REPTree classifier) was the winner, and the Random Forest classifier was ranked second. What is essential, the p-values confirmed that the results were statistically significant.

Furthermore, classifiers often have their parameters to tune them accordingly, which can substantially affect the classification accuracy, cf. for detailed discussion. For example, the logarithmic correlations were observed between the number of bagging iterations for the Bagging classifier and the average F-measure value, as well as between F-measure and the number of generated trees by the Random Forest classifier. The results prove that the process of adjusting the classifier parameters should always be performed, as long as the computational time and resources are available.

## Comparison with other methods

The GEP method was compared to other approaches. The existing methods for group evolution prediction were additionally analyzed, and many of their drawbacks have been identified. The most severe were: a narrow application area, methodological issues (e.g., inappropriate computation of the conditional probability), insufficient validation of the methods (e.g., a single sampling into two folds instead of the 10-fold cross-validation), superficial descriptions of the methods and conducted experiments (often insufficient to repeat and validate the experiments), and lack or unreliable comparisons with other methods.

Despite GEP is so flexible and has so many options, it is competitive with other approaches, designed to deal with a specific problem or data set. For example, a special version of the GEP method, in which only features from the last three states (out of all 8 or 9 states) were used as an input for the classifier, performed noticeably better than the method by İlhan et al., Fig. D.

After all, it needs to be emphasized that none of the existing methods is as adjustable and versatile as the GEP method.

# Discussion

Across its six stages, the GEP method utilizes various approaches, methods, and techniques, which can be adjusted with respect to a given data set and a particular study purpose. These approaches, methods, and techniques are considered as the GEP method parameters. To provide a concise summary of their impact on overall computational complexity, and first of all on the final classification accuracy, the crucial parameters were listed in Tab.  and discussed throughout the article.

```latex
\begin{adjustwidth}{-2.25in}{0in}
\centering
\caption{{\bf The GEP method parameters and their impact on computational complexity and classification accuracy.}}
\begin{tabular}{p{2.3cm}|p{4cm}p{5.2cm}p{2.3cm}p{2cm}}
\hline
Parameter group & Parameter & Parameter value & Impact on computational complexity & Impact on classification accuracy \\ \hline
\multirow{3}{*}{time window} & window division & timestamp / relations count / arbitrary & none & low \\ \cline{2-5}
 & window size & time unit or number of relations & medium & low \\ \cline{2-5}
 & window type & disjoint / overlapping / increasing & medium & medium \\ \hline
network type & edge attributes & directed / undirected, weighted / unweighted & low & low \\ \hline
\multirow{5}{*}{methods} & group type & disjoint / overlapping & medium & low \\  \cline{2-5}
 & grouping method & a method & high & medium \\  \cline{2-5}
 & tracking method & a method & medium & low \\  \cline{2-5}
 & GED alpha and beta & (10\%, 100\%] & none & low \\  \cline{2-5}
 & GED social position measure & a measure & medium & low \\ \hline
 \multirow{2}{*}{classification} & classifier used & a classifier & medium & medium \\  \cline{2-5}
 & machine learning techniques & undersampling, oversampling, feature selection & high & high \\ \hline
 \multirow{2}{*}{other} & evolution chain length & number of community states & medium & medium \\  \cline{2-5}
 & predictive features & a set of features & high & high \\ \hline
\end{tabular}
\label{tab:parametersInfluence}
\end{adjustwidth}
```

Many different classifiers were evaluated on various data sets. The tree-based classifiers and meta-classifiers (equipped with decision trees) performed best. Many classifiers could not handle imbalanced data sets, so the undersampling and oversampling techniques were applied. Balancing data sets notably improved the results confirming the usefulness of the undersampling and oversampling methods. The experimental studies showed that adjusting the classifier parameters can significantly improve classification accuracy. The logarithmic correlations were observed between the number of bagging iterations in Bagging classifier and the average F-measure value, as well as between the number of generated trees by the RandomForest classifier and the average F-measure value. The confidence factor parameter of the J48 classifier was found also correlated with the average F-measure value. The maximum improvement in average F-measure value achieved by adjusting the classifier parameter was 17%, and it was obtained by increasing the number of generated trees by the RandomForest classifier. The results prove that the process of adjusting the classifier parameters should always be performed, as long as the computational time and resources are not limited.

The GEP method enables us to consider different new scenarios, which are hardly available without this generative framework like transfer learning, class balancing by adding external data, or decreasing the concept drift effect.

The transfer learning technique was adapted to the problem of group evolution prediction for the first time in this field. Its main idea is to learn the classification model on one data set and test it on another one. Such an attempt was quite successful, and the preliminary results were satisfactory. The key to success is finding a data set with a likewise profile. Moreover, in some cases, learning the transferred model on the balanced data set can boost the classification quality for the data set to which the model is adapted. The initial experiments also suggest that the underlying similarity of two data sets (e.g., the same habits of actors or ideally the same set of actors) can help to create a model that if transferred can outperform the primary model built for a given data set.

Very promising results, although at an early stage, were achieved at enriching the learning phase of the classification model with additional evolution chains from a different data set. By partially balancing the original training set with extra evolution chains from another external data set, it was possible to improve the model and thus produce better results for minority classes, without affecting the outcome for the dominating classes, Fig. A. This phenomenon is especially important because the existing techniques of balancing a data set always affect the classification results of the dominating classes.

Another way to enhance the classification model, initially considered, is an appropriate selection of the observation time span to reduce the effect of non-stationarity of data – a.k.a concept drift. Our preliminary research shows that for a network spanning over a long period or changing rapidly, updating the classification model every once in a while might improve the results, as the model reflects the current characteristics of the network in the better and more up to date way, Fig. B. Nevertheless, in order to rebuild the model every now and then, the number of observations (evolution chains) extracted from such shorter time span must be high enough.

The GEP framework can be applied to any dynamic network data, i.e., to any complex network changing over time. In this paper, we have explored popular social network data, see Table 2 in the Supporting information section. However, the entire GEP method, its stages and component solutions may be used for diverse complex networks like evolving clusters of web pages , co-citation and bibliographic coupling networks extracted from citations between scientific papers , biological and medical networks , linguistic networks linking word meanings - WordNets , multimedia networks and many more.

# Conclusion

The main subject studied in this paper is group evolution prediction in social/complex networks. Its primary goal is to foresee a change like shrinking, growing, splitting, merging, or dissolving that the recently existing community will experience in the nearest future. To be able to perform any prediction, the most common approach is to process a temporal complex network $TSN$ extracted from the stream of user activity traces. Communities and their changes are identified and predicted within such $TSN$. However, the existing methods are often limited to operate on a particular data set or to solve a specific problem, which makes them useful only in a particular and narrow domain.

Therefore, a new generic method called Group Evolution Prediction (GEP) has been proposed in this paper. The GEP method has a modular structure, which makes it very flexible and allows us to successfully apply it to any data set and under any specific requirements. The method consists of several stages; each of them involves a suitable selection of methods, algorithms, and attributes – the GEP method parameters. The evaluation process of the GEP method included: (1) analysis of numerous parameters (time window type and size, community detection method, evolution chain length, classifier used, set of features, and more), (2) comparative analysis against other existing methods, (3) adaptation of the transfer learning concept to group evolution prediction, (4) enriching the classification model with evolution chains from a different data set, and (5) enhancing the classification model with a more appropriate training set.

Regarding the time window types and sizes, the main finding is that for rapidly changing or sparse social networks a shorter overlapping time windows (in relation to the context of the data) are a better choice than longer or disjoint periods. On the contrary, if relations between individuals are recurrent and the network is rather dense, one may try disjoint time windows to obtain more concise results and to lower the computational cost. If long-lasting, persistent communities are the goal, then the increasing type of time window is the best choice as it generates a high number of the continuing, growing, and shrinking events.

Two most commonly used community detection approaches were analyzed: the CPM method detecting the overlapping communities, and the Infomap method identifying the disjoint communities. It turned out that the CPM method was not a proper choice for sparse networks, as it left out nodes that did not belong to any clique. However, if a network is not so sparse, then generating overlapping communities may be a better choice, especially if the context of the data suggests overlapping communities. For example, when the nodes tend to belong to more than one community at a given time. The Infomap method, however, performs better if computational complexity is an essential factor, and computational time is limited.

The results yield that evolution chains with more community states (longer chains) provide better classification results. However, there seems to be a threshold of the number of states, which make the evolution chains too short, resulting in a lack of possibility of improving the accuracy level.

Even over 70% of the most prominent features were obtained from the last three community states. It means that the most recent history of the community has the highest impact on its next change. This is an extremely useful conclusion if one has limited computational capabilities and cannot calculate community profiles for all states. Additionally, many new predictive features are proposed in this paper. In particular, some aggregations of node measures were used to compute the local and global microscopic features. Network structural measures were adopted as macroscopic features, and ratios of community measures to network measures were utilized as mesoscopic features. In general, the variations of the eigenvector-, eccentricity-, and closeness-based features were present in most of the selective rankings, which suggests that centrality- and distance-based measures obtained on the node level are the most valuable features.

The GEP method flexibility enabled us to investigate some other interesting scenarios, i.e., (1) adapting the transfer learning technique to the group evolution prediction problem, (2) enriching the classification model with evolution chains from a different data set, (3) appropriate selection of the observation time span to reduce the concept drift effect. All of them appeared to be quite successful.

Even though the GEP method is a flexible, generic framework, it is competitive with other approaches often dedicated to a specific problem or data set.

# Supporting information

#### S1 File.

**Supporting information file.** Contains additional results and discussion.

# Authors Contributions

**Conceptualization:** SS, PB, PK. SS. SS, MK. PK. SS, MK. SS, PB, MK, PK. SS. SS, MK. PB, PK. SS, PB, MK, PK. SS, PB, PK.

# Acknowledgements

This work was partially supported by The Polish National Science Centre, the projects no. 2016/21/D/ST6/02408 and 2016/21/B/ST6/01463; by the European Union's Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No. 691152 (RENOIR); and by the Polish Ministry of Science and Higher Education fund for supporting internationally co-financed projects in 2016-2019, agreement no. 3628/H2020/2016/2. This research was supported in part by PLGrid Infrastructure.


abstract: Optical systems with wide field-of-views (FOV) are crucial for many applications such as high performance imaging, optical projection, augmented/virtual reality, and miniaturized medical imaging tools. Typically, aberration-free imaging with a wide FOV is achieved by stacking multiple refractive lenses (as in a "fisheye" lens), adding to the size and weight of the optical system. Single metalenses designed to have a wide FOV have the potential to replace these bulky imaging systems and, moreover, they may be dispersion engineered for spectrally broadband operation. In this paper, we derive a fundamental bound on the spectral bandwidth of dispersion-engineered wide-FOV achromatic metalenses. We show that for metalenses with a relatively large numerical aperture (NA), there is a tradeoff between the maximum achievable bandwidth and the FOV; interestingly, however, the bandwidth reduction saturates beyond a certain FOV that depends on the NA of the metalens. These findings may provide important information and insights for the design of future wide-FOV achromatic flat lenses.
author: Kunal Shastri; Francesco Monticone
bibliography: bib1.bib
date: June 15, 2021
title: Bandwidth bounds for wide-field-of-view dispersion-engineered achromatic metalenses

# Introduction

For more than the last three hundred years, optical systems have mostly relied on refractive elements, in the form of lenses for focusing/imaging applications. From a wave optics perspective, these lenses are based on the principle that a continuous phase profile can be added to the incident wavefront through propagation in a dielectric material of inhomogenous thickness. Although dielectric diffractive lenses have a flatter and thinner profile than conventional lenses, as they introduce phase modulo $2\pi$, this phase is still obtained in the same way as in a refractive lens, through wave propagation in a dielectric material of varying thickness .

In recent decades, new intriguing opportunities in lens design have emerged with the introduction of metasurfaces, flat and planar optical components that can abruptly change the phase of the incident wavefront over the distance of a wavelength or less . This is achieved not (or not only) through wave propagation inside a dielectric material, but through wave interaction with arrays of sub-wavelength nano-structures, called meta-atoms, which scatter light with the desired phase shift, allowing for a local control of the wavefront with great flexibility . Lenses designed using metasurfaces, called metalenses, offer flat, compact and lightweight alternatives to conventional lenses, features that are especially important in consumer electronics such as high-performance smartphone cameras and augmented/virtual reality glasses . By engineering the response of complex nanostructures used as meta-atoms, metalenses have also been shown to enable additional functionalities, such as polarization control, that cannot be easily achieved using conventional refractive and diffractive lenses .

Unfortunately, the abrupt phase shift introduced by metasurfaces is typically strongly dependent on the wavelength of the incident light making it challenging to design broadband metalenses. An elegant strategy to overcome this limitation is to develop a library of meta-units that provide, not only the desired phase pattern, but also the appropriate group delay and group delay-dispersion, as recently proposed in Refs. . This can be illustrated by considering a metasurface that changes the phase of an incident plane wave to a generic phase profile given by $\phi(r,\omega)=-\zeta(r)\omega/c$, where $\omega$ and $c$ are the angular frequency and speed of light in the surrounding medium, respectively, and the function $\zeta(r)$ has units of length. For example, for a linear phase gradient that redirects an incident beam at an angle $\alpha$, we have $\zeta(r)=r\sin(\alpha)$, and, for a metalens with focal length $F$ that focuses normally incident light, $\zeta(r)=(F^2+r^2)^{1/2}-F$, where $r$ is the in-plane linear/radial coordinate, respectively. For an ideal broadband achromatic metasurface, the meta-atoms at each location on the metasurface must be chosen to perfectly satisfy the above phase requirement at every frequency. An alternative, equivalent way to interpret this requirement is that the meta-atoms must be chosen to satisfy the required phase profile at the center frequency $\omega_c$, along with a constant group-delay profile given by $t_g(r)=\partial\phi(r,\omega)/\partial\omega|_{\omega_c}=-\zeta(r)/c$, and a group-delay dispersion equal to zero (achromatic wavefront manipulation, for example achromatic focusing or deflection, would still be obtained even if the group delay dispersion is non-zero, as long as it remains spatially constant, but pulses would be distorted). Metalenses designed using this method, implementing the required phase and group-delay profile, are called dispersion-engineered achromatic metalenses, and they can correct for chromatic aberrations over a continuous frequency range centered around the chosen center frequency, as originally demonstrated in .

Recently, by drawing parallels between a dispersion-engineered achromatic metalens and a series of delay lines and phase shifters, it was shown in Ref. that the range of group delays that can be implemented by a meta-atom cannot be arbitrary but is bound by delay-bandwidth limitations (the product between group delay and signal bandwidth is fundamentally limited in any linear time-invariant system ). Consequently, this places a theoretical upper bound on the bandwidth of the dispersion-engineered metalens, regardless of the chosen meta-atom library. In this paper, we generalize the bandwidth bounds to wide field-of-view (FOV) achromatic metalenses. In particular, we show that the maximum achievable bandwidth for dispersion-engineered metalenses decreases not only with an increase in numerical aperture (NA) , but also with an increase in the FOV.

# Results

*Location of the focal spot –* One challenge in the design of wide-FOV metalenses is that plane waves incident on the lens at various angles need to be phase shifted by different values in order to avoid aberrations. In particular, a metalens with focal length $F$ that changes the phase of an incoming plane wave to a hyperbolic profile given by, $$\label{eq:phase1}
\phi(r,\omega)=-\frac{\omega}{c}\left(\sqrt{F^2+r^2}-F\right),$$ achieves aberration-free focusing only for normally incident light . While some workarounds for this problem exist, such as using a curved (aplanatic) metalens or a curved focal surface , these devices are not "flat", negating some of the potential benefits of using metalenses. Thus, in this work we focus on flat-optic systems consisting of a planar metalens and a planar focal surface, as sketched in Fig . In this case, while normally incident rays are focused at the center of the focal plane, to create an aberration-free image over a wide FOV, obliquely incident rays need to be focused on the same plane at a different location $r_0(\alpha)$ relative to the center. Moreover, to create an undistorted image of the considered scene, it is necessary that $r_0(\alpha)=F\tan(\alpha)$, where $\alpha$ is the angle of incidence and $F$ is the focal length of the metalens . Note that since the problem is cylindrically symmetric, it is sufficient to use a single polar angle $\alpha$, with $\alpha=0$ corresponding to normal incidence, to describe the phase shift. Since this mapping from angle of incidence to focal spot position requires an infinitely large sensor for grazing angles of incidence, another frequently used choice is $r_0(\alpha)=F\alpha$, which however introduces significant distortions for large incident angles.

*Ideal metalens phase profile –* To focus a planar wavefront incident at a given angle $\alpha$ on a flat focal plane, the lens must impart a combination of a linear phase gradient given by, $$\label{eq:phase2}
\phi_l(r,\omega,\alpha)=-\frac{\omega}{c}\left(r\sin(\alpha)\right),$$ to redirect the wavefront to the surface normal, and the hyperbolic radial profile given by Eq. () but focusing light at $r_0$ instead of the center of the plane , $$\label{eq:phase3}
\begin{split}
\phi_s(r,\omega,\alpha)&=-\frac{\omega}{c}\left(\sqrt{F^2+(r-r_0(\alpha))^2}\right.\\
&\left.-\sqrt{F^2+r_0(\alpha)^2} \right),
\end{split}$$ Thus, the phase shift that a wide-FOV metalens must introduce for focusing without aberrations is, $$\label{eq:phasef}
 \phi(r,\omega,\alpha)=\phi_l(r,\omega,\alpha)+\phi_s(r,\omega,\alpha).$$

The spatial dependence of the two individual phase profiles and the resultant net phase shift for $r_0(\alpha)=F\tan(\alpha)$ are plotted in Figs. (a)-(c). Furthermore, the calculated net phase profiles for two different choices of $r_0(\alpha)$ are shown as a function of the angle of incidence in Figs. (d),(e). Importantly, it can be seen that the ideal phase profile has a strong dependence on the direction of the incident light and, for a metalens of finite lateral extent, the required range of phase shifts does not increase monotonically with the angle of incidence. In fact, while the required total phase range would be the same regardless of incident angle for an ideal, infinitely extended metalens, for a finite metalens the phase range to be imparted is minimal for grazing angles of incidence, as seen in Figs. (c),(d), and is maximum at a certain angle that depends on the maximum radius of the metalens.

An important point to note is that a "local" metasurface can only introduce a position-dependent phase shift and not an angle-dependent shift. Hence, while a local metasurface can be used to focus light incident at a specified angle, using the angle-appropriate phase profile above, it will form a distorted and aberrated image for light incident at other angles. On the other hand, a lens with a FOV equal to $2\alpha_0$ should be able to focus light from all angles between $-\alpha_0$ and $\alpha_0$ without any aberrations.

The easiest way to achieve an angle-dependent phase shift with local metasurfaces is to consider a system with at least two surfaces separated by a gap. The gap here is crucial since it utilizes the angle-dependent phase accumulated by plane waves propagating through a free-space (or dielectric) slab to create an overall angle-dependent transmission phase. In practice, this has been implemented using two metasurfaces separated by a certain distance or a single metasurface preceded by a circular aperture . The aperture followed by the free-space gap maps incident beams from different angles onto separate (or minimally overlapping) spatial regions on the local metasurface, which can then implement the necessary spatial phase shift (a nice example of this design strategy is demonstrated in Ref. ). In general, however, the bandwidth limitations for dispersion-engineered achromatic metalenses that will be derived in the next section are not affected to the presence of the aperture, since the required range of group delays remains unchanged.

To achieve an angle-dependent phase shift with a single metasurface, it would instead be necessary to design angle-dependent meta-atoms providing different phase shifts for different angles or, more broadly, "nonlocal metasurfaces" with an overall angle-dependent response . As further discussed in the Discussion section, nonlocal metasurfaces may provide some intriguing opportunities to overcome the limitations discussed in the following, which are based on the assumption of local interactions between incident wave and meta-atoms.

*Maximum achievable bandwidth –* Next, we proceed to calculate the fundamental bandwidth bounds for wide-FOV metalenses that have been designed, through dispersion engineering, for broadband achromatic operation (i.e., for diffraction-limited focusing at the same distance over a broad frequency range). As discussed in the Introduction, the meta-atoms composing these structures are designed to implement not only the phase shift given by Eq. () at the center frequency, but also the required group delay for broadband focusing. The group delay required for a pulse incident on the wide-FOV metalens at a location $r$ relative to the center of the lens, and at an angle $\alpha$, can be calculate by taking the angular frequency derivative of Eq. (),

$$\label{eq:delt}
\begin{split}
t_g(r,\alpha)&=-\frac{F}{c}\left[\frac{r}{F}\sin(\alpha)+\sqrt{1+\left(\frac{r}{F}-\frac{r_0(\alpha)}{F}\right)^2} \right.\\
&\left.-\sqrt{1+\frac{r_0(\alpha)}{F}^2}\right].
\end{split}$$

In this equation the relative group delay is negative at any position $r\neq0$, which can be interpreted as a time advance for a pulse at the edges of the metalens with respect to an undelayed pulse at the center, or as a (more physical) time delay at the center with respect to an undelayed pulse at the edges. Importantly, adding a global (i.e., position-independent) time delay is irrelevant for our purposes. Note that the plots in Figs. (d) and (e) can also be used as representations of the group delay profile. This is because the spatial phase profile is proportional to the group delay profile (as can be seen by comparing Eqs. () and ()) and they can be made numerically equal using an appropriate choice of normalization. The plot in Fig. (d) corresponds to the group delay calculated for a choice of focal spot location $r_0=F\tan(\alpha)$, and (e) for $r_0=F\alpha$.

Since the thickness of typical metalenses is of the order of the wavelength, this group delay is primarily acquired during longitudinal propagation through the metalens (namely, delay acquired through lateral propagation can be neglected, as in ). Thus, each point on the metalens can be modeled as a 1D delay line designed to implement the local delay given by Eq. (). This group delay is typically introduced by using resonant meta-atoms or, alternatively, meta-atoms acting as truncated waveguide segments with engineered dispersion. In the case of a structure supporting a single resonance, a simple study of the dynamics of the mode amplitude shows that the total rate of loss $\gamma$, due to absorption and radiation, determines both the full-width-at-half-maximum bandwidth of the resonance $\Delta \omega = 2\gamma$ and the mode lifetime $\tau=1/\gamma$, the latter of which corresponds to the maximum delay achievable through the interaction with the cavity. The delay-bandwidth product is therefore, $\Delta T\Delta\omega=2$, for any single-mode resonant cavity and it can be easily verified that, for multi-resonator systems, the product scales with the number of resonators . Furthermore, more general limits on the delay-bandwidth product can be derived for generic 1D systems, applicable also to non-resonant waveguiding structures. A particularly general expression for this limit was derived by Miller in : $\Delta T\Delta\omega\leq \kappa = L\omega_c\eta/2\sqrt{3}c$, where $L$ is the length of the structure, $\omega_c$ is the central frequency, and $\eta$ is the maximum contrast in relative permittivity at any frequency within the considered bandwidth and any point within the structure. While the derivation of this result is rather involved, the bound can be intuitively understood, in the case of dielectric waveguides, considering the dispersion diagram (band diagram) of the waveguide modes: since the guided-mode bands are confined between the light lines defined by the permittivities of the dielectric waveguide and the surrounding material, reducing the group velocity (i.e., increasing the group delay) by flattening a band between these two lines necessarily implies a reduction of the bandwidth over which the desired group velocity can be achieved. Applied to our case of interest, these bounds on the delay-bandwidth product allow limiting the maximum bandwidth over which a meta-atom is able to implement a certain group delay, and, in turn, the maximum operating bandwidth of the entire dispersion-engineered achromatic metalens. For further details on the relevance of the delay-bandwidth product for metalenses and metasurfaces, see Refs. .

Based on the above considerations, the maximum frequency range for achromatic operation for a metalens with a FOV equal to $2\alpha_0$ can then be calculated by finding from Eq. () the maximum required group delay within the angular range $-\alpha_0\leq\alpha\leq\alpha_0$, and substituting it in the delay-bandwidth bound,

$$\label{eq:bandwidth}
\Delta\omega\leq\frac{\kappa}{\Delta T_{\max}}.$$

The group delay range that the metalens should provide widens for larger lenses, as larger delays need to be implemented at the center ($r=0$) to compensate for the additional time taken by a pulse arriving to the focal spot from the edges ($r=R$). The required delay range also widens for larger FOV (wider range of angles of incidence), since, intuitively, an even greater delay needs to be compensated across the metalens as the angle of incidence increases. However, this is true only up to a point. Indeed, the range of required group delays across a finite metalens does not monotonically widens as the angle of incidence increases owing to the non-monotonic increase of phase range with angle that we discussed in the previous section (see Figs. (d),(e)). Taking this into account, the greatest overall delay that the metasurface must be able to implement can be calculated as $\Delta T_{\max}=t_g(r=0,\alpha=0)-t_g(r=R,\alpha=\alpha^*)$, where $\alpha^*$ is the angle, within the range $-\alpha_0\leq\alpha\leq\alpha_0$, for which the group-delay range across the metalens is widest (essentially this is the difference between the maximum and the minimum in the contour plots in Fig. (d,e), for any angle $\alpha$ and position $r$, when these plots are interpreted as a representation of the normalized delay). Since there is no simple expression for $\Delta T_{\max}$, we evaluate it numerically and then use it in Eq. () to determine the bandwidth limit for two different choices of focal position $r_0$. The results are plotted in Fig. .

As a function of the numerical aperture (NA) of the metalens, Figs. (a) and (b) show a clear trend of narrower achievable bandwidths for metalenses with larger NA, which is expected, since a metalens with larger NA has a larger delay to compensate between the center and the edges, consistent with . Here, we only plotted the results for moderately large NA because, for smaller values, the bandwidth bound for any FOV approaches the normal-incidence bound derived in . Most importantly, it can also be seen from these plots that the bandwidth limit shrinks for metalenses with larger FOV. However, surprisingly, this trend saturates for a certain FOV, depending on the NA and the choice of focal spot position, as seen in Figs. (c) and (d). This implies, for example, that a dispersion engineered metalens with a ultra wide FOV=180 deg could, in principle, have the same spectral bandwidth as a metalens with a relatively narrower FOV=60 deg, for a fixed $NA=0.7n_b$. This result is a consequence of the previously mentioned observation that for a finite metalens and a fixed FOV, the widest group-delay range that the metalens should implement is not always for light incident at the maximum oblique angle, implying that beyond a certain FOV (dependent on NA), the maximum allowed bandwidth does not further decrease. Furthermore, this trend is observed for both choices of focal spot position, as seen in Figs. (c) and (d).

# Discussion

In the recent literature, moderately broadband achromatic metalenses have been reported with remarkably large FOVs, as well as sufficiently low aberrations resulting in a high Strehl ratio . In particular, a complete 180 deg FOV over a relatively wide fractional bandwidth of 0.2 has been reported in . However, the NA in this design is limited to $0.24$ for which our bounds are less relevant since, as seen in Figs. (a) and (b), an appreciable decrease in bandwidth as a function of FOV is only seen for metalenses with a relatively large NA. In this context, it would be interesting to see how future wide-FOV metalens designs with larger NA compare with our bounds.

In relation to the bandwidth bounds derived above, it is also instructive to consider structures that can potentially break some of the assumptions in the derivation and thereby overcome the bound. Indeed, the bandwidth bounds derived here apply to the spatial phase profile given by Eq. (), which is specific to the case of diffraction-limited focusing on a flat focal plane. For other imaging system designs, such as focusing on curved focal surfaces, the resultant bound will have to be modified depending on the specific spatial phase profile of the considered metalens and the desired imaging functionality. In addition, relaxing the constraint on diffraction-limited resolution and Strehl ratio would imply that a certain level of error in the implemented phase/group delay profile would be acceptable, which would therefore relax the resulting bandwidth bounds, as also demonstrated in Ref. .

Moreover, since the considered limits to the delay-bandwidth product strictly only apply to 1D structures (the meta-atoms modeled as 1D delay lines), the derived bound will not apply to systems that allow additional group delay due to transverse propagation. For instance, systems consisting of cascaded metasurfaces could take advantage of the lateral dimension between metasurfaces to achieve larger group delays and potentially overcome the bandwidth bounds derived in this paper (however, a generalized version of these bounds should apply also to this type of systems, and its derivation will be the subject of future work). The same argument also applies to wide-FOV imaging systems constructed using bulk metamaterials, or thick volumetric structures obtained, for example, through all-area inverse design . More broadly, we note that the analogy between delay lines and meta-atoms is valid only if the metasurface response is based on the *local* interaction between fields and meta-atoms at a specific point on the surface. Differently from achromatic metalenses, achromatic diffractive lenses achieve broadband operation through energy redistribution among different diffraction orders for different frequencies, a mechanism that involves the overall surface structure rather a local effect, as discussed in . For this reason, achromatic diffractive lenses, unlike dispersion-engineered metalenses, are not limited by delay-bandwidth considerations, but suffer from other limitations especially in terms of focusing power . This general discussion also hints at the large potential of nonlocal metasurfaces – an emerging class of metasurfaces specifically designed to exhibit strong nonlocal effects – to potentially achieve performance metrics that are unattainable with more conventional designs.

To conclude, in this paper we have generalized our previous work on metalens bandwidth bounds to dispersion-engineered achromatic metalenses with a wide FOV. Single element (meta)lenses with large spectral and angular bandwidth are potentially of immense importance for a wide range of applications, including high-performance smartphone imaging, optical projection, endoscopic optical imaging, and augmented/virtual reality. In this context, the bandwidth bounds derived in this paper may provide important information and insights toward the design of next-generation metalenses with, simultaneously, wide FOV, high resolution, and broad spectral bandwidth.

*Acknowledgment –* We acknowledge support from the National Science Foundation (NSF) with Grant No. 1741694, and the Air Force Office of Scientific Research with Grant No. FA9550-19-1-0043.


author: Antonio Capizzi , Salvatore Distefano ,Manuel Mazzara
bibliography: biblio.bib
title: From DevOps to DevDataOps: Data Management in DevOps processes

# Introduction

DevOps is an approach for software development and (IT) system operation combining best practices from both such domains to improve the overall quality of the software-system while reducing costs and shortening time-to-market. Its effectiveness is demonstrated by the quite widely adoption of DevOps approaches in business contexts, where there is a big demand of specific professionals such as DevOps engineers as well as data scientists, just partially, minimally covered by current offer.

The DevOps philosophy can be generalized as a way, a good practice for improving a generic product or service development and operation, by connecting these through a feedback from operation to development. An important feature of DevOps is the automation of such a process: *continuous delivery* (CD) enables organizations to deliver new features quickly and incrementally by implementing a flow of changes into the production via an automated "assembly line" - the *continuous delivery pipeline*. This is coupled with *continuous integration* (CI) that aims at automating the software/product integration process of codes, modules and parts, thus identifying a CI/CD pipeline.

The tools adopted to implement this high degree of automation in the DevOps process identifies a *toolchain*. DevOps toolchain tools are usually encapsulated into different, independent containers deployed into physical or virtual servers (typically on Cloud), and then managed by specific scripts and/or tools (e.g. Jenkins), able to orchestrate and coordinate them automatically.

Such DevOps principles have been therefore either specialized to some specific software/application domains (security - SecOps, SecDevOps, DevSecOps , system administration - SysOps , Web - WebOps or WebDevOps ) or even adopted, rethought and adapted in other contexts such as artificial intelligence (AIOps ) and machine learning (MLOps, DeepOps ), and data management (DataOps ). The latter, DataOps, aims at mainly organizing data management according to DevOps principles and best practices. To this end, DataOps introduces the concept of *dataflow pipeline* and toolchain, to be deployed in containerized (Cloud) environment providing feedback on performance and QoS of the overall data management process, used to real-time tune the pipeline to actual operational needs and requirements.

As discussed above, the DevOps pipeline automation involves in the toolchain different tools, each continuosly generating messages, logs and data including artifacts. To achieve DevOps aims and goals, such data has to be properly managed, collected, processed and stored to provide insights from operations to the development stages. DevOps data management could therefore be quite challenging, due to the large amount of data to be considered as well as its variety, variability and similar metrics usually identified as V properties in the BigData community, to which we have to refer to. BigData approaches, indeed, could be a good solution to consider in the management of a DevOps process and toolchain.

In light of these considerations, in this paper we focus on DevOps data management, proposing to adopt BigData approaches and solutions. More specifically, the main goal of this paper is to explore the convergence between DevOps and DataOps approaches, defining a possible (big)dataflow pipeline for DevOps processes and toolchains and organizing it following a DataOps process, towards DevDataOps. This way, we investigate on the adoption of DataOps, mainly implementing a BigData pipeline and toolchain, in DevOps contexts, i.e. for improving the development and operation of a software architecture.

To this extent, Section describes the DevOps and DataOps processes and toolchains. Section discusses about DevOps artifacts and data in the BigData context. Then, Section proposes a DevOps (big) dataflow pipeline and related implementation in the DataOps philosophy. Section summarises the key aspects of the proposed approach and future work.

# DevOps and DataOps

## The DevOps process and toolchain

DevOps consists of a set of practices to promote collaboration between the developers, IT professionals (in particular sysadmin, i.e. who works on IT operations) and quality assurance personnel. DevOps is implemented via a set of software tools that enable the management of an environment in which software can be built, tested and released quickly, frequently, and a more reliable manner. In addition to *continuous delivery* (CD), which aims at developing "small" software releases in reasonably short cycles, *continuous integration* (CI) stands as a key concept in DevOps approaches. A typical example of CI consists of continuously integrating changes made by developers into a repository, then a project build is automatically executed, and if the build the modifications are integrated into the code through CD and published in the production environment.

A DevOps process is usually composed of different stages and phases, which can be periodically reiterated for proper development and operation of the software architecture, in an evolutionary fashion able to caught new requirements, features and behaviors arising from operation. This way, a DevOps process belogs to the category of agile process, not plan driver, where the number of development cycle is unknown a-priori. Consequently, the amount of data generated by a DevOps process is usually unpredictable and could be really high. There is no standard definition of a DevOps process, but several different versions and implementations have been provided by the related community. Among them, main DevOps stages can be summarized below.

-   *Plan*: activity planning and task scheduling for the current release. This step is usually dealt with by project managers in collaboration with the team and exploiting project management tools such as Trello, Asana, Clarizen, Jira, Azure DevOps, to name a few.

-   *Code*: code development and code review. Developers are the most closely involved in this activity using IDE and tools for source code management such as GitHub, Artfactory, CodeClimate, etc.

-   *Build*: is when source code is converted into a stand-alone form that can be run on a computer system. In this activity are involved various professional figures, mainly developers and sysadmins. The tools used in this phase are: CI tools (for example Jenkins, TravisCI), build tools (for example Maven) etc.

-   *Test*: in this phase the software is tested by the quality assurance staff using tools for (automatic) testing and similar. Examples of such kind of tools are JUnit, Jmeter, Selenium, etc.

-   *Release*: triggered when a new version of software is ready to be released to end users. In this activity, various professionals of the development team are involved, primarily developers and sysadmins. Release management tools (such as Spinnaker) or similar support such an activity.

-   *Deploy*: it deals with the installation and execution of the new software release in the production environment and infrastructure. At this stage, the collaboration between developers and syadmins is mandatory. The tools used for deployment depend on the target infrastructure (physical or virtual nodes, Cloud, etc.) as well as on the adopted system software (OS, virtualization, containerization, middleware, compilers, libraries), thus identifying a wide set of possible options for deployment management (VmWare, Virtualbox, Docker, LXD, Kubertenes, AWS CodeDeploy, ElasticBox etc.).

-   *Operate*: is the activity that maintains and adapts the infrastructure in which the software is running. This activity is mainly driven by sysadmins, supported by configuration management tools (such as Ansible, Puppet, Chef), security tools (such as Sonarqube, Fortify SCA, Veracode), database management tools (such as Flyway, MongoDB), recovery tools (PowerShell, Ravello), etc.

-   *Monitor*: in this activity the software in production is monitored by mainly sysadmins, operators and others managing the project. The tools used are: tools that monitor the performance of the service, tools that analyze the logs (for example Logstash, Nagios, Zabbix), tools that analyze the end user experience (Zenoss).

One of the main objectives of DevOps is to mitigate issues in production, which is done by reducing the gap among development and testing environments to the production one. To this purpose, several tools as the one mentioned above are usually used and combined into a set identified as the "DevOps Toolchain", which can be considered as a scaffold built around the development project. To compose a Toolchain, in general, there are no fixed rules, it is necessary to follow the DevOps principles and best practices to choose the tools according to the project characteristics. For a small project 3 or 4 tools might be enough, while in a larger project 10 or more tools might be necessary. A minimal (CI/CD) DevOps toolchain might include, at least, some version control tool (e.g. Git), automation tools (e.g. Jenkins), package managers (e.g. NPM) and test tools (e.g. JUnit). DevOps infrastructures are typically fully implemented on Cloud platforms. It is a good practice in DevOps to build the entire infrastructure using containers to minimize portability issues. For that, containerization technologies such as Docker, LXD or similar are adopted, sometimes coupled by tools for containers orchestration (e.g. Kubernetes or Swarm).

## DataOps

DataOps is a new approach that aims to improve quality and responsiveness of data analytics life-cycle . This approach is based on DevOps rules, in particular DataOps aims to bring DevOps benefits to data analytics, adopting Agile rules and Lean concepts. When the volume of data is larger and larger, the purpose of DataOps is to improve the life cycle of analytics by taking advantage of DevOps principles such as communication between teams (data scientists, ETL, analysts, etc.), cooperation, automation, integration, etc. To achieve this it is necessary to apply a set of human practices and dedicated tools. With DataOps a new professional figure called "DataOps Engineer" was born, to deal with the automation and orchestration of the process. A large DataOps community issued a Manifesto[^1] that contains 18 rules, the mission and best practices to apply DataOps. However, this is the most concrete activity behind DataOps, that is still mainly a set of rules, concepts and ideas to be applied to data management. There is, indeed, a lack of examples, dataflow pipelines, toolchians and standard process for DataOps.

However, despite the DataOps approach is still mainly abstract, it can be mostly summarized with the DataOps principles detailed in the Manifesto, some implementations of the DataOps idea start to be defined and fixed. For example, a DataOps process can be broadly organized into three steps [^2]:

-   *Build* - In this step, the data is taken from a source point (e.g. a database, a log file, etc.), transformed by applying one or more actions, and then written to a destination point. The flow executed by these actions is called "dataflow or DataOps pipeline". In the Build phase you can also have multiple pipelines connected to each other.

-   *Execute* - at this stage the build pipelines are put into production in a running environment e.g. clusters, datacenters, Clouds. It is important, for a company adopting DataOps, to be able to use the existing infrastructure to run the pipeline, in order to avoid incurring in additional costs.

-   *Operate* - When this step is reached, the system is running on an environment, it is necessary to monitor it and react to any change (for example when larger volumes of data arrive and it is required to scale the infrastructure to cope with burst). One approach used in DataOps (borrowed from Agile) is to start with small instances and increase their resources when the demand grows.

# DevOps data

An outcome from the complex pipeline involved in a DevOps project is the generation of a large amount of data, considering the process artefacts and the log files generated in each stage. Examples of activities that generate considerable data on the project cycle include changes made by developers; the application building and its corresponding entries on the compilation and dependencies of the project; the execution of automatic tests; software usage by end-users after release into the production. Furthermore, the number of cycle of a DevOps process is usually unknown, and typically lasting years, so reaching hundreds, or even thousands of releases each generating a considerable amount of data that should be adequately preserved and managed for gaining insights and the process.

Artifacts and data produced in a DevOps process are quite large and widely different. These can include software artifacts (code, documentation, test, executable, prototypes) and other information generated by the DevOps toolchain (logs, configuration files, traces, ...). More specifically, based on the above DevOps reference process, the data associated to each stage is reported below.

-   *Plan*: planning artifacts and data are software design blueprint, requirement documentations (UML or similar, if any), project environment information including user stories, tasks, activities, backlog, and statistics.

-   *Code*: development artifacts include codes, versions, prototypes and related info such as lines of code, version differences and relevant parameters.

-   *Build*: mainly executable files, packages, logs and metrics that contain information about builds and may indicate compilation errors, warnings, successes, failures etc.

-   *Test*: code for automatic tests, logs from automatic test tools that indicates unit tests failed or passed, system tests results, or documentation written by Quality Assurance staff about verification on software in development.

-   *Release*: documentation about releases (for example new features introduced), new final version of executable files or packages, logs and metrics from release orchestration tools.

-   *Deploy*: configuration files, scripts and logs originating from Deploy tools that may contain errors or warnings.

-   *Operate*: data generated by the software, logs from the tools involved in this stage and system logs from (physical or virtual) servers.

-   *Monitor*: logs, metrics and other information from monitoring tools, the data retrieved in this phase is important to obtain a feedback from users.

It could be worth to invest in a data management system for a DevOps process, where the high volume of generated data is not only properly collected and stored but also managed, filtered, aggregated and possibly made available for further processing, to gain insights on the overall process to achieve essential DevOps aims and goals. This calls for proper data management techniques, providing mechanisms for collection, aggregation, storage filtering, aggregation, fusion, archival, mining and feature extraction, local and global analytics preferably in an automated manner , to improve the DevOps pipeline . For example, historical data can be analyzed to estimate a probabilistic measure of the success of a new release, or for identifying potential source of bugs (root-cause analysis) or even to prevent them.

From a data/information-oriented perspective therefore, a DevOps process can be considered as a data-intensive process, in the sense that it could generate large amount of data. To this purpose, DevOps data management issues and challenges can be framed into the BigData context. Considering the reference DevOps process and toolchain described in Section , it could be interesting to characterize such a process in BigData terms. To this purpose, we refer to well-known and widely used Bigdata metrics: the "Vs". BigData V properties are usually used in the community to categorize an application, and range in number from original 3 (Volume, Velocity, Variety) to 10 or even more. Shortly, *volume* is probably the best known characteristic of big data, quantifying the amount of data generated; *velocity* refers to the speed at which data is being generated, produced, created, or refreshed; *variety* is related to the "structuredness" of data: we don't only have to handle structured data (logs, traces, DB) but also semistructured and mostly unstructured data (images, multimedia files, social media updates) as well; *variability* refers to inconsistencies in the data, to the multitude of data dimensions resulting from multiple disparate data types and sources and to the inconsistent speed at which big data is loaded into DB; *veracity* is the confidence or trust in the data, mainly referring to the provenance or reliability of the data source, its context, and how meaningful it is to the analysis based on it; *validity* refers to how accurate and correct the data is for its intended use; *vulnerability* is concerned with big data security, privacy and confidentiality; *volatility* refers to data "lifetime", i.e. the amount of time needed for data to be considered irrelevant, historic, or not useful any longer; *visualization* faces technical challenges due to limitations of in-memory technology and poor scalability, functionality, and response time to represent big data (billion data points) such as data clustering or using tree maps, sunbursts, parallel coordinates, circular network diagrams, or cone trees; *value* is the property to be derived from the data through processing and analytics. In the DevOps context, the process data value is exploited to support decision making in development. This way, Table reports the characterization of a DevOps pipeline from a BigData perspective, expressed in terms of V metric values ranges for a "mid-size" DevOps reference process.

```latex
\begin{adjustbox}{width=\textwidth}
\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|}
%|p{1cm}|p{0.9cm}|p{0.9cm}|p{0.9cm}|p{0.9cm}|p{0.9cm}|p{0.9cm}|p{1.6cm}||c|}
\hline
\textbf{\textit{Stage/Vs}} &\textit{Volume} & \textit{Velocity} & \textit{Variety} & \textit{Variability} & \textit{Veracity} & \textit{Validity} & \textit{Vulnerability}& \textit{Volatility} & \textit{Visualization}& \textit{Value} \\ \hline \hline
\textbf{Plan} & 10KB-1GB& Week& UnStr. & Medium/High & High & Low &   Low &Week/Days &Poor&High\\ \hline
\textbf{Code} & 1-100MB & Hours & SemiStr. & High & High & High & Medium & Hours &Poor&High\\ \hline
\textbf{Build} & 1-10GB & Hours & SemiStr & Medium & Low & Low & Medium & Hours &High &High \\ \hline
\textbf{Test}& 10KB-1GB & Minutes & Str & Medium & High & High & Medium & Days & High &Medium\\ \hline
\textbf{Release}& 1-10GB & Week & UnStr & High & Medium & Medium & Medium & Week/Month & Medium & High\\ \hline
\textbf{Deploy}& 1-100MB & Week & UnStr & High & Medium & Medium & Medium & Week/Month & Medium & High\\ \hline
\textbf{Operate} & 10KB-1GB & Hours & SemiStr & High & High & High & Medium & Hours & Medium & High\\ \hline
\textbf{Monitor}& 10KB-1GB & Seconds/Minutes & SemiStr & High & High & High & High & Hours & High & High\\ \hline
\end{tabular}
\end{adjustbox}
```

# DevDataOps

## DevOps Dataflow Pipeline

The data generated by a DevOps pipeline is therefore quite complex and heterogeneous, and consequently quite hard to manage and maintain . The DevOps data life-cycle and workflow can be decomposed into different stages and steps identifying the dataflow pipeline shown in Fig. . This could be considered as a quite generic DataOps pipeline that can be generally applied to any DevOps process, after adaptation, represented in Fig. by the conditional diamonds modeling the presence or absence of a specific step in the DevOps dataflow-DataOps pipeline.

-   **Generation**: Each module of the DevOps pipeline generates a log stream reporting its operation through a specific monitoring process.

-   **Collection**: The generated logs are collected, reordered according to their timestamps, and grouped altogether to provide a snapshot of the whole DevOps process for each time interval.

-   **Filtering**: Logs are then filtered to remove outliers, replicas, or observations that may contain errors or are undesirable for analysis. Logs filters are usually based on temporal statistics, i.e. based on previously logs average or similar statistical moments.

-   **Stream processing, analysis and alerting**: The processing of single logs streams is locally performed in nearly real-time to identify potential flaws, defects or errors in a particular DevOps pipeline stage. In such cases, warning, error messages or activities are triggered by the alerting tool.

-   **Aggregation**: The logs are aggregated and expressed in a summary form for statistical analysis. The main goal of the aggregation is to compress the volume of data.

-   **Delivery**: The logs are made available to end-users and applications. This data is typically transmitted through networks using related protocols to physical and virtual servers.

-   **Preprocessing, ordering, grouping, cleansing, and fusion**: The logs are reordered according to their timestamp and grouped to provide a snapshot of the DevOps process at each time interval. Next, the logs have to be cleaned, that is, having redundancies removed and being integrated with different sources into a unified schema before storage. The schema integration has to provide an abstract definition of a consistent way to access the logs without having to customize access for each log source format. Still during the preprocessing stage, logs undergo through a fusion process aiming to integrate multiple sources to produce a more consistent and accurate information.

-   **Storage, update and archiving**: This phase aims the efficient storage and organization as well as a continuous update of logs as they become available. Archiving refers to the long-term offline storage of logs. The core of the centralized storage is the deployment of structures that adapt to the various data types and the frequency of the capture (e.g. relational database management systems).

-   **Processing and analytics**: Ongoing retrieval and analysis operations on stored and archived logs, mainly offline for root cause analysis, to identify any correlation among stages, predict behaviors and support decisions. Analytics is the discovery, interpretation, and communication of meaningful patterns in data in the logs and can be performed at different levels with different objectives: *descriptive* (what happened), *diagnostic* (why something happened), *predictive* (what is likely to happen) and *prescriptive* (what action to take).

## DataOps implementation

The DevOps dataflow pipeline should be then implemented according to a DataOps approach, as reported below

#### Build -

To implement the DevOps dataflow pipeline of Fig. in a DataOps fashion, following the process described in Section , we have to start with building the toolchain, thus identifying the tools associated with each of the pipeline step. In this case, the real benefit of adopting BigData solutions in DataOps is clearly manifested: this way, the DevOps dataflow pipeline of Fig. can be just considered as a BigData workflow to be deployed exploiting a tool among the plethora available to manage BigData workflow (Hadoop, Spark, Storm, Flink, Samza, NiFi, Kafka, NodRed, Crosser.io, to name but a few). Mainly belonging to the Apache (big) family, they allows to define and manage BigData workflows composed of different tasks or processes, highly customizable and configurable, then linked through specific mechanisms and tools able to enforce the workflow topology, or even to further parallelize and optimize it.

#### Execute

Once built as a BigData workflow, the DataOps toolchain needs to be deployed and executed. At this stage, therefore, automation tools such as Jenkins and deployment tools such as Docker or Jupiter could be used to support and further automate the BigData one, by for example containerising tasks or connecting them with tools (monitoring) external to the BigData workflow. Usually, the target deployment infrastructure for a DataOps toolchain is the Cloud, public as Microsoft Azure and Amazon EC2, or private such as those managed locally by OpenStack or similar middleware.

#### Operate

The operation stage is mainly tasked at providing a feedback on the DataOps toolchain to the DataOps engineers that have to tune it based on this feedback. Both the process and the underlying infrastructure running the toolchains have to be monitored. Metric of interest to be benchmarked in this case could be system parameters (CPU, memory, storage utilization), process non functional properties (response time, reliability, availability, energy consumption), or even specific V properties (volume, velocity, variety, etc, see Section ) In this step, tools for monitoring such as Prometheus or Nagios, and for managing the infrastructure such as Chef, Puppet etc. can be exploited.

# Conclusions

DevOps is a modern approach to software development aiming at accelerating the build lifecycle via automation. Google, Amazon and many other companies can now release software multiple times per day making the old concept of "release" obsolete. Data is at the centre of all the DevOps process and requires BigData solutions to be managed. Despite of the growing importance of DevOps practices in software development, management of the data generated by the toolchain is still undervalued, if not entirely neglected.

In this paper we investigated this cutting-edge aspect of software development identifying related issues, challenges and potential solutions. Solutions do not need to be entirely new since large literature has been already published in the field of BigData. This emerging field of research is often referred to as *DataOps*. While DevOps was created to serve the needs of software developers, DataOps users are often data scientists or analysts.

The current work has just started scratching the surface of such a complex subject, and in the limited space could not explore all the detailed aspects of analytics. One of the project on which our team is working at the moment is the analysis of data generated by the DevOps toolchain in order to identify anomalies in the incoming releases . The same idea can be applied to Microservices monitoring, and this is exactly our next goal towards NoOps.

[^1]: https://www.dataopsmanifesto.org/

[^2]: https://dzone.com/articles/dataops-applying-devops-to-data-continuous-dataflo