author: Max Pellert,$^{1,2,3,4\ast}$ Hannah Metzler,$^{2,3,4,5}$ Michael Matzenberger,$^{6}$ David Garcia$^{2,3,4}$

$^{1}$Sony Computer Science Laboratories,
6, Rue Amyot, 75005 Paris, France
$^{2}$Institute of Interactive Systems and Data Science,
Graz University of Technology,
Inffeldgasse 16C, 8010 Graz, Austria
$^{3}$Complexity Science Hub Vienna,
Josefstädter Straße 39, 1080 Vienna, Austria
$^{4}$Section for Science of Complex Systems, Center for Medical Statistics,
Informatics and Intelligent Systems, Medical University of Vienna,
Spitalgasse 23, 1090 Vienna, Austria
$^{5}$Institute of Globally Distributed Open Research and Education
$^{6}$Der Standard,
Vordere Zollamtsstraße 13, 1030 Vienna, Austria

$^\ast$To whom correspondence should be addressed; E-mail: email@example.com.
bibliography: scibib.bib
title: Validating daily social media macroscopes of emotions

> **Measuring sentiment in social media text has become an important practice in studying emotions at the macroscopic level. However, this approach can suffer from methodological issues like sampling biases and measurement errors. To date, it has not been validated if social media sentiment can actually measure the temporal dynamics of mood and emotions aggregated at the level of communities. We ran a large-scale survey at an online newspaper to gather daily mood self-reports from its users, and compare these with aggregated results of sentiment analysis of user discussions. We find strong correlations between text analysis results and levels of self-reported feelings, as well as between inter-day changes of both measurements. We replicate these results using sentiment data from Twitter. We show that a combination of supervised text analysis methods based on novel deep learning architectures and unsupervised dictionary-based methods have high agreement with the time series of aggregated mood measured with self-reports. Our findings indicate that macro level dynamics of feelings expressed on an online platform can be tracked with social media text, especially in situations of high mood variability.**

# Introduction

User generated text from social media has become an important data source to analyze expressed mood and emotions at large scales and high temporal resolutions, for example to study seasonal mood oscillations , emotional responses to traumatic events , the effect of pollution on happiness , and the role of climate change in suicide and depression . Despite these promising applications, using social media text to measure emotion aggregates can suffer a series of methodological issues typical of studies of this kind of *found data* . Common validity threats are measurement error in sentiment analysis tools and the performative behavior of social media users due to platform effects or community norms. Sampling biases can generate a mismatch between users that produce text and a target group that might include silent individuals.

The validation of sentiment analysis methods has focused on micro level measurement accuracy at the individual post level . Recent work has assessed the measurement validity also at the individual person level, using historical records of text from a user. This has revealed low to moderate correlations between aggregates of sentiment produced by an individual over a period of time and emotion questionnaires . At the group level, static measurements of social media sentiment are only moderately correlated with affective well-being and life-satisfaction across regions . These earlier findings highlight the limits of static aggregations of sentiment to measure concepts like life satisfaction that are only slowly changing over time. However, it is still an open question if analyses of social media text can shed light on *faster* phenomena, for example core emotional experiences, when we stick to aggregating individual signals to a community of interest and observe variation over time.

Here, we address this research gap by testing whether social media text sentiment tracks the macro level dynamics of emotions with daily resolution in an online community. We study the convergence validity of two approaches to study *emotions* at scale: *sentiment* aggregates from social media text and *mood* self-report frequencies in a survey. For 20 days, we collected 268,128 emotion self-reports through a survey in an Austrian online newspaper. During the same period, we retrieved text data from user discussions on the same platform, including 452,013 posts in our analysis using our pre-existing Austrian social media monitor . To replicate our results with a second dataset, we conducted a pre-registered analysis of 635,185 tweets by Austrian Twitter users. We applied two off-the-shelf German sentiment analysis tools on the text data: a state-of-the-art supervised tool based on deep learning (German Sentiment, GS, ) and a popular dictionary method based on expert word lists (Linguistic Inquiry and Word Count, LIWC, ). Our results strongly support the assumption that social media sentiment can reflect both mean levels and changes of self-reported emotions in explicit daily surveys. We additionally analyze positive and negative components of the sentiment analysis methods to provide further methodological insight on how to measure time series of experienced mood in online communities.

# Results

We measured the time series of experienced mood as the fraction of self-reported positive feelings over the total of self-reports in a day. We coupled this with a daily sentiment aggregate based on text in the platform's forum (derstandard.at), namely the average of the GS and LIWC scores (see Sentiment analysis and Text data in Materials and Methods for more details).

Figure shows the time series of the fraction of responses that report a positive mood and the text sentiment aggregate from the Der Standard forum. The Pearson correlation coefficient between both measurements is $0.93$ ($[0.82,0.97],p<10^{-8}$), indicating a very strong positive correlation with daily resolution over a period of almost three weeks. The regression line in the scatter plot on Panel B in Figure confirms the relationship. A linear model with Heteroskedasticity and Autocorrelation Consistent (HAC) estimates shows the same robust effect. The model has an adjusted R-squared of $0.852$, with a coefficient $\hat{\beta}=0.597$ ($[0.465,0.728],p<10^{-7}$) for the unscaled average of sentiment aggregates. The text sentiment aggregate can explain 85% of the variance in the daily proportion of positive mood.

We additionally tested if *changes* in the text sentiment aggregate can approximate daily changes in the proportion of positive mood in the survey compared to the previous day. A similar regression model as before yields a coefficient of $\hat{\beta}=0.533$ ($[0.390,0.675],p<10^{-16}$) for changes in the text sentiment aggregate and an adjusted R-squared of $0.704$ (Panel C of Figure ). This model has a non-significant intercept of $0.002$ ($[-0.002,0.005],p=0.29$) showing that, in addition to explaining 70% of the variance in emotion changes at the macro level, the model's prediction of trend in mood changes is not significantly biased.

To test the robustness of our results as well as their generalizability to a different platform, we pre-registered a replication of our analysis using 515,187 tweets by Austrian Twitter users in the survey period instead of Der Standard forum posts (<https://aspredicted.org/blind.php?x=vb3gp2)> (see Methods for more details on sample size and selection criteria). The correlation coefficient between the survey and sentiment on Twitter is positive and significant ($0.63$ $[0.26,0.84],p<0.003$), confirming our pre-registered hypothesis and the robustness and generalizability of the results. Although it is somewhat lower than the correlation of the survey with text sentiment from the Der Standard forum, a coefficient above 0.6 is still sizeable, especially given that the survey and the postings now come from different platforms. Following our pre-registration, we filtered out accounts tagged as "organisational" by the aggregation service Brandwatch (formerly known as Crimson Hexagon) and accounts with less than 100 followers or more than 5000 followers. If we relax this criterion to include accounts with up to 100000 followers as in our previous study , the correlation increases to $0.71$ ($[0.39, 0.88],p<0.0005$). This suggests that influential accounts are also relevant to calculate sentiment aggregates, as central individuals in the Twitter social network might be serving as early sensors of sentiment shifts .

Beyond our pre-registered hypothesis, we found that the Twitter sentiment signal is lagged by a day compared to the mood survey. Figure S2 shows the data with a shift of one day in comparison to no shift. Correcting this by shifting by one day yields a correlation coefficient of $0.90 ([0.75,0.96],p<10^{-6})$. We see one explanation for this: The newspaper articles and discussions of their contents likely capture immediate reactions to events, while reaching the wider audience of Twitter takes longer. Panel A of Figure shows that the survey and Twitter time series closely track each other, and the regression line in Panel B confirms this (coefficient $\hat{\beta}=0.516$ ($[0.379,0.654],p<10^{-16}$ with an adjusted R-squared of $0.791$). Again, changes of both variables (Panel C) also have a strong relationship, indicated by a regression coefficient of $\hat{\beta}=0.557$ ($[0.296,0.819],p<10^{-16}$), an adjusted R-squared of $0.501$ and a non-significant intercept of $-0.00051$ ($[-0.009,0.008],p=0.90$). For the remaining analysis, we build on this model with a lag of one day to understand the best case of how Twitter sentiment can explain the survey.

We further explored which components of sentiment analysis are the most informative when estimating the daily proportion of self-reported positive feelings. Table shows the correlation of positive feelings with the aggregated values (positive minus negative emotions, averaged across the LIWC and GS measure) as well as the positive and negative components of both sentiment analysis methods separately. All variables were rescaled through a Z-transformation for both Der Standard and Twitter postings. The positive component of GS has a high correlation with the proportion of self-reported positive mood. The positive component of LIWC also has a positive, but somewhat lower coefficient for both Der Standard and Twitter. Additionally, the LIWC and the GS positive components are strongly correlated with each other on both platforms (Der Standard: $\rho=0.94$ $[0.85,0.98],p<10^{-8}$, Twitter: $\rho=0.90$ $[0.75,0.96],p<10^{-6}$), indicating convergent validity of the two methodologically distinct measures of positive emotions from text when aggregating as daily frequencies of emotional expressions. The analysis of LIWC scores shows inconsistencies: The negative component of LIWC does not correlate with the proportion of positive mood in the survey for Der Standard. Yet, LIWC negative is informative for Twitter, with a significant negative correlation coefficient. Overall, comparing machine-learning and dictionary-based methods shows that the supervised classifier shows more consistent performance and generally higher point estimates. Yet, confidence intervals overlap, and negative LIWC beats GS on Twitter data. Combining both methods for Der Standard adds a small increase to the already strong correlations of the supervised classifier alone. Taken together, it thus seems that both methods contribute unique variation for explaining self-reported feelings.

|  | Der Standard (No shift) | Twitter (Shift 1) | Twitter (No shift) |
|---:|:---|:---|:---|
| LIWC+GS | 0.93 \[0.82,0.97\] | 0.90 \[0.75,0.96\] | 0.71 \[0.39,0.88\] |
| LIWC | 0.74 \[0.44,0.89\] | 0.85 \[0.65,0.94\] | 0.66 \[0.31,0.85\] |
| LIWC pos | 0.81 \[0.56,0.92\] | 0.80 \[0.56,0.92\] | 0.60 \[0.22,0.83\] |
| LIWC neg | 0.03 \[-0.42,0.46\] | -0.74 \[-0.89,-0.43\] | -0.63 \[-0.84,-0.26\] |
| GS | 0.91 \[0.78,0.96\] | 0.91 \[0.79,0.96\] | 0.73 \[0.43,0.89\] |
| GS pos | 0.89 \[0.75,0.96\] | 0.91 \[0.79,0.97\] | 0.80 \[0.54,0.92\] |
| GS neg | -0.57 \[-0.81,-0.18\] | -0.39 \[-0.71,0.06\] | -0.17 \[-0.57,0.3\] |

Correlation of positive mood in the survey with text sentiment measures on both platforms (Der Standard and Twitter). The table presents sentiment aggregates (positive minus negative emotions), as well as positive and negative components separately. LIWC+GS indicates the average across both sentiment analysis methods, all other lines present aggregates or components separately for each method. Shift 1 denotes a shift of one day, where survey values precede Twitter values.

The strong relationship that we found between the signals of the survey, Der Standard postings and tweets opens up the possibility to measure emotion aggregates through social media text when survey data is not available. This raises the question how these three different affective measures correlate with external events that drive the emotions of a community. In the following, we test if our social media text measures provide comparable correlations with the number of new COVID-19 cases to self-reports in surveys. The survey period in November 2020 falls within the build-up of the plateau of COVID-19 cases in the second wave of the pandemic in autumn 2020 in Austria, providing an ideal time frame to test this hypothesis. The importance of the topic in public discussion makes new COVID-19 cases a relevant external variable that previous research has linked to emotional experiences in the population as a whole . We retrieved COVID-19 case data from Our World in Data for the period that overlaps with the survey. The survey and both aggregate Twitter sentiment measures (with and without shift) correlate significantly with the number of new cases for the corresponding day (Table ). Der Standard sentiment shows a weaker correlation and is not significant. Figure S3 confirms the relationships with scatter plots. Furthermore, we tested if this relationship with new COVID cases significantly differs when correlating survey feelings compared to text sentiment measures (Table S1 and Figure S4 in SI): The correlation obtained with both sentiment analysis methods for Twitter data does not significantly differ from the correlation with survey data.

|                              | New Cases             |
|-----------------------------:|:----------------------|
|  Twitter (Aggregate Shift 1) | -0.60 \[-0.82,-0.21\] |
| Twitter (Aggregate No Shift) | -0.57 \[-0.81,-0.17\] |
|                       Survey | -0.53 \[-0.79,-0.12\] |
|     Der Standard (Aggregate) | -0.33 \[-0.68,0.13\]  |

Correlation of survey, aggregate Twitter sentiment and aggregate Der Standard sentiment with the number of new COVID-19 cases. Figure S3 shows scatter plots for each of the variables and new COVID-19 cases.

We built on the strong correlation of sentiment of both platforms with the survey to study their relationship also outside of the survey time frame: Figure shows the time series of the two platform's sentiment signal before, during and after the survey. For the period between 2020-09-15 until 2021-12-30, we find a significant positive correlation of $\rho=0.53$ $[0.38,0.65],p<10^{-16}$. This additional analysis shows that the relationship of expressed emotions on the two platforms is stable also over longer time periods.

# Discussion

This study compared online newspaper readers' self-reported feelings with sentiment analysed in postings on two social media platforms. The results show that the sentiment contained in text of postings in the online discussion forum on the newspaper's website tracks the daily frequency of self-reported feelings in the survey. These results generalize across social media platforms, as a pre-registered replication shows similar correlations when using Twitter text instead of postings in the newspaper forum. Despite the methodological challenges in studying affective states with social media text, this provides evidence that the aggregate of sentiment analysis of social media text can be used to measure macro level mood. We find strong relationships with both levels of emotions and inter-day changes, showing that social media sentiment indeed tracks macro-level mood dynamics with daily resolution. This differs from previous studies reporting only low positive (or even negative) correlations between self-reported well-being indicators and long-term dictionary-based positive sentiment aggregates across US regions . In contrast, we find that dictionary as well as machine-learning based methods track short-term affective states, as opposed to more long-term concepts such as life satisfaction. Additionally, we find that adding indicators based on unsupervised dictionary methods increases the already high agreement of supervised machine-learning methods with the survey measurement of macro level emotions. This suggests that dictionaries generated by experts, while being less exhaustive than the large models of supervised methods, may include terms that are not discovered or not attributed adequate importance in the training phase of supervised models.

Other methodological issues of social media research remain unsolved, for example if social media sentiment measures the emotions of the wider population beyond an online community. However, all readers of the online newspaper Der Standard were prompted to anonymously fill in the survey. To not only consider active users of the forum with a registered account lowers the barrier for participation substantially. Our results provide first evidence that postings by active users can also reflect emotions of silent individuals in an online community to a very high degree. A potential influence on this relationship points in the direction of emotional contagion . Emotions can spread between registered forum user through their postings to unregistered users. *Digital* emotional contagion was indeed observed on social media both in studies of randoms sample of users as well as of users that displayed synchronized emotional sharing in the aftermath of a terrorist attack .

Beyond this, we show that our findings generalise to the online ecosystem of Twitter users, who most likely did not participate in the survey on Der Standard. With Der Standard as well as Twitter, we study two different online ecosystems whose users are based in the same geographical area and have similar demographic characteristics. There could potentially be a certain overlap in their user bases, but precise estimates are currently not possible (for privacy reasons, Der Standard users and survey respondents cannot be individually identified). Another limitation concerns the relatively short time period during which the survey was online, which was determined by administrative decisions at Der Standard. Future research should look for opportunities to distribute or access more long-term surveys, to test if these results extend to longer time periods. To address the question of longer term stability with the data that we have available, we compared the two text sentiment signals over an extended time period. We showed that their strong relationship is also present for several months before, during and after the survey period (Figure ).

When comparing the components of sentiment analysis methods, we find that positive sentiment is generally more informative than negative sentiment when analysing daily data from these German-speaking social media platforms. The negative affect dictionary measure does not significantly correlate in the Der Standard dataset while performing better, but still worse than positive affect, on Twitter. In contrast, in previous Twitter studies in English, LIWC negative affect signals were more informative than positive affect signals when studying well-being across regions . A concurrent study tracks weekly emotions in the UK and also finds stronger correlations with negative emotions measured with English LIWC . Our case of the German language analysis on Der Standard differs from those results, pointing to a potentially missing methodological link that connects which kind of sentiment captures which emotions or sentiments on social media text. A word shift graph (Figure S14) shows the higher prevalence of Austrian German dialect words on Der Standard as the biggest difference between the two platforms' text corpora during the survey period. We could assume that users use dialect words more often to express negative than positive affect (to swear or to express general discontent for example). This may explain the worse performance for dictionary methods such as German negative LIWC, and suggests explicitly including such dialect expressions (with no standard spelling) in the dictionary is warranted. GS, on the other hand, is trained on large amounts of "in-the-wild" texts from the internet and may already have encountered such non-standard expressions, or be able to infer their meaning from the context. Still, we also noticed generally weaker correlations for negative than positive sentiment with the GS method both on Der Standard and Twitter. A possible explanation for this discrepancy is an asymmetry in measurement error when detecting positive versus negative sentiment, which has been shown for English to be potentially different across various social media and especially challenging in political discussions . As emotional expression has a tendency to be positive , the large-scale training corpora in German for supervised methods like GS might have substantially more positive than negative text to learn from. In line with the current view in NLP research of "more is better" , a possible avenue to improve supervised sentiment analysis methods is to include additional negative texts or to generate balanced samples with respect to sentiment to improve negative sentiment detection.

Our results do not imply that all sentiment analyses on any social media platform will reflect macro level emotions. However, we show that social media data can reflect macro emotions, in particular for short-term emotional states, and that this can be validated against survey data. Aggregates of social media text analysis can serve as *macroscopes* which combine measurements that may be noisy at the level of individuals or posts, but, when aggregated across thousands of posts per day, can provide a valid signal that strongly correlates with the results of standard social scientific methods like surveys.

# Methods

Part of this analysis was pre-registered at <https://aspredicted.org/blind.php?x=vb3gp2>. Specifically, we pre-registered the methodology we previously developed for analysing sentiment in text from Der Standard, and then tested the robustness of the results by repeating the same analysis with text from Twitter. Figures S2-S6 provide information on demographics of users on Der Standard and Austrian Twitter. Users of the two platforms tend to be more often male, younger, more highly educated and more often from Vienna or Upper Austria than respondents of a representative survey in Austria . We average over waves 4-6 of that survey, the waves in which questions about Der Standard and Twitter were included. Of a total of 3002 respondents, 533 ($\sim 18\%$) report having a Twitter account and 200 ($\sim 7\%$) report having an account on Der Standard.

## Survey data

The survey was displayed after the text of all articles in the Austrian online newspaper derstandard.at between November 11th and 30th, 2020 (for an example see Figure S1). Der Standard is one of the major newspapers in Austria and its online community is highly active, with almost 57 million visits in November 2020. The headline of the survey was "How was your last day" ("Wie war der letzte Tag?") and the question displayed to respondents was "If you think back to the previous day, do you have a positive or a negative feeling?" ("Wenn Sie an den gestrigen Tag denken, haben Sie ein positives oder negatives Gefühl?"). Respondents had the following choices: "good" ("Gut"), "somewhat good" ("Eher gut"), "somewhat bad" ("Eher schlecht") and "bad" ("Schlecht"). This retrospective assessment is known as the day reconstruction method that was shown to reduce errors and biases of recall . In comparison to experience sampling methods that rely on repeatedly probing in real-time, the day reconstruction method is non-disruptive, places less burden on respondents, and provides an assessment of the experience of whole days instead of momentary snapshots. It has been used in research to study for example the experience of pain , the relationship of socio-economic status to the prevalence of a number of common illnesses , the influence of age on psychological well-being and weekly affect patterns . We used the proportion of "good" or "somewhat good" responses from among all responses in a day as our independent variable, measuring an aggregate of mood per day. In total, we collected $268,128$ survey responses.

## Sentiment analysis

We applied the supervised GS classifier through the pytorch implementation distributed in the Hugging Face Hub (<https://huggingface.co/oliverguhr/german-sentiment-bert>). The underlying BERT model was trained on a diverse corpus to capture different types of expressions including social media text, reviews, Wikipedia, and newspaper articles in German. GS adds a sequence classifier head on top of the language model that is pretrained in a supervised fashion using three classes ("positive", "neutral", and "negative").

For the unsupervised approach, we use the German adaptation of the LIWC dictionaries , in particular the word lists for positive emotions and for negative emotions. For efficiency reasons, we ran our own analyses based on a version of the LIWC emotion dictionaries with small modifications to avoid systematic error on Twitter as in . Specifically, we excluded COVID-19 related words (e.g. "treatment", "heal"), and words that negatively correlate with happiness and well-being (love", "good", "LOL", "better", "well" and "like") from the positive emotions dictionary. We tokenize the raw text of each post and compare the tokens to the LIWC dictionary categories, calculating a proportion of matching terms over all tokens in each post.

To be able to combine different sentiment analysis methods, we rescaled measurements against baseline means calculated over a period preceding our analysis. We used the data corresponding to the first Austrian COVID-19 lockdown as a baseline (March 16th to April 20th 2020), since the period covered by the survey also corresponds to a lockdown in Austria. We rescaled daily sentiment aggregates by subtracting the baseline mean and dividing by it . To construct an aggregate of emotions comparable to the survey, we calculated the aggregate as the rescaled measure of positive emotions minus the rescaled measure of negative emotions. Our aggregate sentiment measure is the arithmetic mean of both methods GS and LIWC. As the survey question is targeted at feelings experienced on the previous day, responses can be influenced by feelings experienced on the day of the question and also two days before when a user responds just after midnight. To take this into account, we calculated all our sentiment aggregates on social media text over rolling windows of three days. We chose on purpose only one well-established dictionary-based method (LIWC) and one innovative, out-of-the-box method based on deep learning (GS). The pre-registered part of our analysis (<https://aspredicted.org/blind.php?x=vb3gp2>) demonstrates rigidly that we did not employ post-hoc additional methods out of the large pool available to influence our results .

## Statistical analysis

We use the cocor R package to statistically assess if two correlations (between sentiment aggregates or components and the survey) are statistically significantly different. Additionally, we perform bootstrap sampling of the differences between correlation coefficients and report the results in Figure S4. We fit models of our sentiment measures and the survey for both platforms as well as models of the changes of both sets of variables with "lm" in R . We use HAC correction in the R package "sandwich" to provide a robust assessment of the informativeness of Twitter and Der Standard signals when autocorrelation and heteroscedasticity might be present. $\beta$ coefficients that we report of our models and their confidence intervals are HAC corrected.

## Text data

Between March 6th and December 30th, 2020, our Austrian social media emotions monitor retrieved $4,161,820$ posts in German from the forum on derstandard.at . All retrieved posts and all survey responses were considered in our analysis, i.e. there was no exclusion criterion. Our dataset includes $11,237$ unique users that published $635,185$ posts on derstandard in the subperiod that overlaps with the survey. For calculating baseline values, we used $1,021,978$ posts by $15,871$ users in the period between March 16th and April 20th, 2020. In the same period of March 6th and December 31st, 2020 we collected $5,886,805$ tweets from Twitter. We used information available about the number of followers to exclude users with more than 100,000 followers. This criterion and the shift of one day for the Twitter data are the only additional analyses with changes not included in the pre-registration <https://aspredicted.org/blind.php?x=vb3gp2> that follows the same methodology as for text data from derstandard.at. We used the archive of a data aggregation service (Brandwatch, formerly known as Crimson Hexagon) to retrieve all tweet IDs that were posted in Austria during the full period. The rehydration procedure that retrieves the full tweet object from the IDs makes sure that user's decision to remove their tweets from public display are respected. For the subperiod of the survey, our Twitter data set includes $11,237$ unique users with $635,185$ tweets on Twitter (same baseline period as for Der Standard using $743,003$ tweets by $11,082$ users). To investigate possible differences between user postings on the two platforms, we compared them using word clouds on the days of two important events in Figure S12 & S13. We find the focus of topics of discussions to be comparable. Similarly, comparing the two full text corpora for the entire survey period reveals no surprising differences in word frequencies (Figure S14).

## Data Statement

All data used in this research was either publicly available archival data (Twitter and Der Standard postings) or available to us as anonymized aggregated counts (survey data). The survey did not collect any individual or personally identifiable data. All data retrieval and analysis protocols comply with the regulations for ethical scientific practice at Graz University of Technology. We publish R scripts in a GitHub repository at <https://github.com/maxpel/SocialMediaMacroscopes> to replicate all figures and tables from this manuscript. We include daily aggregates of our measures for both platforms but do not redistribute text, as it could include personally identifying information and could be de-anonymized through search methods. For Der Standard forums, posts are publicly accessible on the newspaper site and can be retrieved respecting the rules of the platform. All posts that remain publicly available on Twitter can be retrieved for academic research by their IDs which are available in the repository.

# Acknowledgements

This work has been supported by the Vienna Science and Technology Fund (WWTF) through the project "Emotional Well-Being in the Digital Society" (Grant No. VRG16-005) and through project COV20-027. We thank Thomas Niederkrotenthaler, Vibrant Emotional Health and the Vienna Science and Technology Fund for providing the funding to access Brandwatch. We also thank Thomas Niederkrotenthaler for providing demographic data from the representative Austrian survey, which we used to describe Austrian users of derstandard.at and Twitter. The authors declare that they have no competing interests.

# Author contributions statement

D.G., M.P. conceived the research, M.M collected survey data, D.G., M.P. collected social media data, M.P. developed text analysis methods and performed data analysis, D.G., M.P., H.M. created preregistration, M.P., D.G. performed the statistical analysis. M.P., D.G. and H.M. wrote the manuscript. All authors reviewed and approved the manuscript.

# Supplementary Information

Table shows that for Der Standard, correlations of LIWC and aggregate sentiment with COVID-19 cases are significantly lower compared to the survey. In contrast, the correlation of GS sentiment alone is not significantly different compared to the survey. For Twitter, none of the correlations with COVID-19 cases differs between the sentiment measures and the survey. Overall, the signal from the survey is thus not more correlated with new cases than Twitter sentiment. Yet, when using data from Der Standard, only one of the sentiment methods (GS) correlates as strongly with cases as the survey, suggesting that the LIWC dictionary could be improved for this particular data source.

|                                               | p value | Difference (higher) |
|----------------------------------------------:|--------:|:--------------------|
|      Twitter (Aggregate Sentiment) vs. Survey |   0.501 | 0.06 (Twitter)      |
|                     Twitter (LIWC) vs. Survey |   0.617 | 0.05 (Twitter LIWC) |
|                       Twitter (GS) vs. Survey |   0.644 | 0.04 (Twitter GS)   |
| Der Standard (Aggregate Sentiment) vs. Survey |   0.015 | -0.2 (Survey)       |
|                Der Standard (LIWC) vs. Survey |   0.023 | -0.36 (Survey)      |
|                  Der Standard (GS) vs. Survey |   0.087 | -0.16 (Survey)      |

Significance tests comparing correlations with the number of new COVID-19 cases between aggregate sentiment on both platforms and the survey. We test the difference in correlations of sentiment measures with new COVID-19 cases compared to the survey with new COVID-19 cases.


**Canada and the SKA from 2020 – 2030**

Kristine Spekkens (RMC; Canadian SKA Science Director), Cynthia Chiang (McGill) , Roland Kothes (NRC), Erik Rosolowsky (Alberta), Michael Rupen (NRC), Samar Safi-Harb (Manitoba), Jonathan Sievers (McGill), Greg Sivakoff (Alberta), Ingrid Stairs (UBC), Nienke van der Marel (NRC), Bob Abraham (Toronto), Rachel Alexandroff (Toronto), Norbert Bartel (York), Stefi Baum (Manitoba), Michael Bietenholz (York), Aaron Boley (UBC), Dick Bond (Toronto), Joanne Brown (Calgary), Toby Brown (McMaster), Gary Davis (SKAO), Jayanne English (Manitoba), Greg Fahlman (NRC), Laura Ferrarese (NRC), James Di Francesco (NRC), Bryan Gaensler (Toronto), Severin Gaudet (NRC), Vanessa Graber (McGill), Mark Halpern (UBC), Alex Hill (UBC), Julie Hlavacek-Larrondo (Montreal), Judith Irwin (Queen's), Doug Johnstone (NRC), Gilles Joncas (Laval), Vicky Kaspi (McGill), JJ Kavelaars (NRC), Adrian Liu (McGill), Brenda Matthews (NRC), Jordan Mirocha (McGill), Raul Monsalve (McGill), Cherry Ng (Toronto), Chris O'Dea (Manitoba), Ue-Li Pen (CITA), Rene Plume (Calgary), Tim Robishaw (NRC), Sarah Sadavoy (Queen's), Viraj Sanghai (Dalhousie), Paul Scholz (NRC), Luc Simard (NRC), Richard Shaw (UBC), Saurabh Singh (McGill), Kris Sigurdson (UBC), Kendrick Smith (Perimeter), David Stevens (MDA), Jeroen Stil (Calgary), Sean Tulin (York), Cameron van Eck (Toronto), Jasper Wall (UBC), Jennifer West (Dunlap), Tyrone Woods (NRC), Dallas Wulf (McGill)

**Executive Summary**

The Square Kilometre Array (SKA), an exciting new world observatory that will enable transformational science at metre and centimetre wavelengths for years to come, is rapidly becoming reality. The SKA will be built in two phases, with the first phase (SKA1) representing  10% of the full facility (SKA2). The SKA1 Design Baseline development is almost complete, and construction is set to begin early in the next decade. When constructed, it will be the largest and most powerful general-purpose radio telescope operating from 50 MHz – 15 GHz for years to come. Scientific and technological participation in the SKA has been identified as a top priority for the Canadian astronomical community for almost twenty years. This 2020 Canadian Long-Range Planning process (LRP2020) white paper advocates for Canada's continued scientific and technological participation in the SKA project, focussing on Canadian prospects for SKA1 from 2020 – 2030.

SKA1 is poised to make fundamental advances across a broad range of fields by virtue of its combination of sensitivity, angular resolution, imaging quality and frequency coverage. SKA1 scientific goals align well with the strengths of Canadian researchers. Canada is a world leader in studies of pulsars, cosmic magnetism and transients, as well as in low-frequency cosmology. Our multi-wavelength expertise in galaxy evolution, multi-messenger astronomy and planetary system formation –- in which radio observations play a critical role –- is also a key strength. The Canadian community therefore has the potential to carry out important PI science with SKA1, as well as to play world-leading roles in a number of the transformational Key Science Projects (KSPs) that are anticipated to take up the majority of available telescope time. An examination of the KSP leadership opportunities afforded by a decade of full operations implies that a 6% participation in the SKA1 Design Baseline is well-matched to Canadian scientific capacity and ambitions.

Canada is a leader in technological development for the SKA through effective partnerships between universities, the National Research Council (NRC) and industry. Our key SKA1 technological capabilities include the design and fabrication of correlators and beamformers, digitisers, low-noise amplifiers, signal processing, and monitor & control. These technologies provide a suite of possible in-kind contributions to offset construction costs for good return on the capital investment required to participate in SKA1 at a level commensurate with our scientific ambitions. Canada also has the computing platform and archive development expertise to make important contributions to the SKA Regional Centre (SRC) network that will deliver global SKA1 scientific computing capability. A Canadian SRC would leverage our national compute strength to provide processing, storage, and user support tailored to Canadian SKA1 needs while also fulfilling our SKA1 participation requirements.

Canadian contributions to the SKA now span two decades, marked by scientific and technological leadership that persists today within a vibrant metre and centimetre-wave radio community. SKA1 is happening now. Canada at last has the opportunity to reap the scientific benefits of our contributions, while an early commitment to construction would maximize our impact on this phase and our technological benefits as well.

LRP2020 will determine the future of the SKA in Canada for the next decade and beyond. We make the following recommendations:

**Canada and the SKA from 2020 – 2030**

** This white paper submitted for the 2020 Canadian Long-Range Planning process (LRP2020) presents the prospects for Canada and the Square Kilometre Array (SKA) from 2020 – 2030, focussing on the first phase of the project (SKA1) scheduled to begin construction early in the next decade. SKA1 will make transformational advances in our understanding of the Universe across a wide range of fields, and Canadians are poised to play leadership roles in several. Canadian key SKA technologies will ensure a good return on capital investment in addition to strong scientific returns, positioning Canadian astronomy for future opportunities well beyond 2030. We therefore advocate for Canada's continued scientific and technological engagement in the SKA from 2020 – 2030 through participation in the construction and operations phases of SKA1.**

# SKA1 Overview

The Square Kilometre Array (SKA) is one of the most ambitious astronomy projects on the horizon today, with broad scientific goals and challenging technical requirements. The SKA will be built in two phases, with the first phase (SKA1) representing $\sim$`<!-- -->`{=html}10% of the full facility (SKA2). When constructed, SKA1 will be the largest and most powerful general-purpose radio telescope operating from 50$\,$MHz – 15$\,$GHz for years to come.

The SKA1 Design Baseline is now mature, and represents the construction and operations benchmark for the next decade. SKA1 will consist of two facilities: SKA1-Low (50$\,$MHz – 350$\,$MHz) located in Australia, and SKA1-Mid (350$\,$MHz – $15+$ GHz) located in South Africa (Fig. ). The SKA1-Low Design Baseline comprises 512 stations each with 256 antennas, providing continuous frequency coverage from 50$\,$MHz – 350$\,$MHz and a maximum interferometric baseline of 65$\,$km (Dewdney+ 16). Averaged over the LOFAR band, SKA1-Low will have 1.25 times the LOFAR resolution, 8 times its sensitivity and 135 times its survey speed (see Fig. ). The SKA1-Mid Design Baseline comprises $133 \times 15\,$m dishes as well as the $64 \times 13.5\,$ dishes from the MeerKAT SKA Pathfinder working in concert from 350$\,$MHz – $15+$ GHz, with a maximum interferometric baseline of $150\,$km (Dewdney+ 16). The frequency range spanned by SKA1-Mid is divided into several bands, of which 3 are part of the Design Baseline: Band 1 will contiguously span $350\,$MHz – $1.05\,$GHz, Band 2 will contiguously span $950\,$MHz – $1.76\,$GHz, and Band 5 will span $4.6\,$GHz – $15.3\,$GHz in $2 \times 2.5\,$GHz sub-bands. When averaged over the overlapping (J)VLA bands, the SKA1-Mid Design Baseline will have 4 times the resolution of the (J)VLA in its most extended ("A") configuration, 4 times its point-source sensitivity and 60 times its survey speed (see Fig. ).

SKA1 is poised to make fundamental advances across a broad range of fields by virtue of its combination of sensitivity, angular resolution, imaging quality and frequency coverage. In particular, SKA1 will be a superb survey instrument (see Fig. ). While the operations model for SKA1 will be finalized during the construction phase, the majority of the available observing time, $50\% - 75\%$, is expected to be earmarked for the execution of large programs called Key Science Projects (KSPs) by participating countries, compared to $25\% - 45\%$ for principal-investigator (PI) programs by participating countries and $\lesssim 5\%$ open time. The scientific legacy of SKA1 is therefore likely to lie in the suite of KSPs ultimately carried out.

With a mature Design Baseline and the critical design review (CDR) process for the individual design elements and the integrated system nearly complete (SKAO+ 19), the construction timeline for SKA1 is becoming more concrete. A representation of that timeline which focusses on scientific milestones is shown in Fig. . The timeline is anchored by an anticipated construction start in Q2 2021 and a nominal construction duration of 7 years. Science commissioning will start as soon as the first dish or station is on site in 2023, while science verification with antenna assemblies that are scientifically competitive with current facilities (64 SKA1-Low stations, 64 SKA1-Mid dishes) is anticipated in 2026. Shared-risk observations with the full complement SKA1-Mid and SKA1-Low arrays are expected to begin in late 2027. Full SKA1 operations are expected in 2028, with PI projects scheduled towards the end of that year and KSPs on the telescope in late 2029. The SKA1 construction and ramp-up to full operations will therefore be complete in the next decade.

There are 3 basic elements to the cost of SKA1 (excluding science compute; see below) in the next decade: 1) the cost of procuring and constructing the facilities in Australia and South Africa, 2) the cost of construction activities at SKA Headquarters in the UK, and 3) the cost of early operations and business-enabling functions ramping up to full operations. This white paper will follow the historical project convention of defining "construction" as the first element, and "operations" as the second two. The most recent estimated construction cost of the Design Baseline is €$\,$`<!-- -->`{=html}914M (2017 Euros; Césarsky 19b), while operations costs through the end of 2030 are comparable to the construction cost (Césarsky 19a). It is important to note that the Design Baseline and operations costing will soon evolve, with a bottom-up Cost Book using the same methodology as other large projects (e.g. the TMT) anticipated in Q2 2020. This costing will also define the final "Deployment Baseline" of SKA1, which corresponds to as much of the Design Baseline as can be afforded for a construction cap of €$\,$`<!-- -->`{=html}691M (2017 Euros; Womersley 13, Césarsky 19b). Since the well-defined Design Baseline is the construction benchmark, and since the scalable nature of interferometers makes it relatively straightforward to recover from any cost-saving cuts, the ambitions for the Canadian scientific and technological community for SKA1 in this white paper reference the Design Baseline.

The raw data rates, processing speeds and calibrated data volumes implied by the SKA1 Design Baseline are enormous, leading to the adoption of a tiered model for data and science support. Similar to the model adopted by CERN, a network of SKA Regional Centres (SRCs) will handle the global science processing, archive and user support needs for SKA1. In this white paper, the anticipated cost for Canadian SRC contributions is considered separately from construction and operations costs (§).

SKA Design and pre-construction activities in the last decade have been overseen by the SKA Organisation (SKAO), a UK-based company which currently comprises 13 participating countries: Australia, Canada, China, France, Germany, India, Italy, New Zealand, South Africa, Spain, Sweden, The Netherlands, and the UK. In the SKA1 construction and operations phases, stewardship of the project will transition to an intergovernmental organisation (IGO) called the SKA Observatory. An IGO, which is the governance structure that underlies CERN and ESO, differs fundamentally from a company in that it provides sovereign protection to the organization and its employees. The 7 countries that signed the IGO Treaty Convention and Final Record in March 2019 (Australia, China, Italy, South Africa, The Netherlands, and the UK) are the Founding Members of the IGO, which becomes active once the Convention is ratified by 5 of them. The Netherlands became the first country to ratify the Convention in August 2019, and the IGO is on pace to come into force by Q3 2020.

The most significant governance milestone in the history of the SKA project is the upcoming transition from the SKAO and its Board of Directors to the IGO and its Council, which is currently anticipated for Q4 2020 (Césarsky 19b). The last major goals of the SKAO are to deliver detailed costing, construction and operations proposals to the IGO Council for their approval in Q3 2020. The Council Preparatory Task Force (CPTF) represents the interests of the IGO until the Convention is ratified, and the focus of CPTF activities is the development of a funding schedule, the establishment of principles governing Associate IGO Membership, and the finalisation of key documentation such as procurement strategy and IP policy (Césarsky 19a). Since funding commitments towards SKA1 construction and operations were decoupled from treaty negotiations early in the process, the IGO will exist before financial commitments are in place. The first major task of the IGO Council at its inaugural meeting will be to approve a budget for the IGO's first year of operations (anticipated to be 2021; Césarsky 19b).

# Canada and the SKA

Canada has a 20+ year history of significant scientific and technological contributions to the SKA initiative, and Canadian governance leadership was instrumental in its early days (Ekers 12). A detailed account of the history and current status of Canada and the SKA is given in the final report submitted to the LRP 2020 panel (Spekkens+ 19); this section focusses on science, technology and governance in Canada pertaining to its future participation in SKA1 construction and operations.

The science goals of SKA1 are among the broadest of any current or planned facility worldwide (SKAO+ 15), and are represented by the themes and questions given in Fig. . They result from a community-based exercise to produce Highest-Priority Science Objectives (HPSOs; Braun+ 14) that informed the SKA1 Level 0 Science Requirements (Braun+ 15), as well as emerging fields such as fast radio bursts and gravitational waves. Fundamentally answering many of these questions is expected to require large allocations of observing time – $5000$ to $10,000\,$hours – in order to exploit SKA1's survey capabilities (Braun+ 15; see also Fig.  and §). It is therefore reasonable to expect that the KSPs that will define the scientific legacy of SKA1 (§) will focus on these themes.

The SKA1 scientific goals embodied by the themes and questions in Fig.  align well with the strengths of Canadian researchers. Canadians are members of all 11 SKA Science Working Groups (SWGs), and have co-chaired 4 of them in the past 5 years. In particular, Canada is a world leader in studies of pulsars, cosmic magnetism and transients, as well as in low-frequency cosmology. Our multi-wavelength expertise in galaxy evolution, multi-messenger astronomy and planetary system formation – in which radio observations play a critical role – is also a key strength. Details regarding the particular advances that SKA1 is expected to make in these fields as well as the capacity for Canadian leadership in each one are summarized in Boxes 1 and 3 at the end of this white paper, as well as in several others submitted for LRP 2020 (e.g. Kaspi+ 19; Fonseca+ 19; Liu+ 19a,b; Ruan+ 19; Sadavoy+ 19; Stairs+ 19; West+ 19). The Canadian community therefore has the potential to carry out important PI science with SKA1, as well as to play world-leading roles in a number of transformational KSPs.

The high prioritization of SKA R&D in previous Long-Range Plans and reviews has helped forge and maintain effective partnerships between government, academia and industry for Canadian SKA1 technology development (Rupen+ 19; Spekkens+ 19). Canadian leadership in SKA1-related technologies includes the design and fabrication of correlators and beamformers for SKA1-Mid, low-noise amplifiers and digitisers for SKA1-Mid Bands 1 and 2, and signal processing and monitor & control for SKA1-Mid. Notably, Canada led the Central Signal Processor element consortium that passed CDR with no action - the only consortium to have received this high rating. The technologies above provide a suite of possible in-kind contributions to offset SKA1 construction costs for good return on capital investment. Canada also has the computing platform and archive development expertise to make important contributions to the SKA Regional Centre (SRC) network that will deliver global SKA1 scientific computing capability. Further details regarding Canadian technological leadership and return on investment for SKA1 are given in Boxes 3, 5 and 6 of this white paper.

Canada currently contributes to the governance of the SKA project through its membership in the SKA Organisation (SKAO), with the National Research Council (NRC) as the adhering organisation (see Box 4). However, the anticipated transition in global project leadership from the SKAO to the intergovernmental organisation (IGO) for construction and operations (Césarsky+19b; §) implies that the SKAO's governance role in SKA1 is becoming increasingly limited. Canadian participation in the IGO is complicated by laws stipulating that governmental approval must be obtained before treaty negotiations can be entered. NRC has been directed by the Canadian government to explore IGO membership options, and Canada is now an Observer country to the Council Preparatory Task Force (CPTF). Bilateral discussions with the CPTF regarding possible SKA1 participation mechanisms for Canada are ongoing.

# Canadian Participation in SKA1: Recommendations

SKA1 is on the verge of construction (§), and its scientific goals and technical challenges align well with Canadian strengths (§). Considerations for Canada's future participation in SKA1 and recommendations for 2020 – 2030 are below.

Several lines of reasoning suggest that a participation level in the SKA1 Design Baseline of $6\% \pm 2\%$ is appropriate for the Canadian community. Focussing on science, the total number of KSP leadership positions over a decade has been calculated given current estimates of the SKA1 operations model, a distribution of KSP durations commensurate with SKA1 scientific goals and a KSP management structure similar to that of current SKA Pathfinder surveys (c.f. Appendix A of Spekkens+ 19). These numbers are preliminary since telescope operations and KSP structure have yet to be finalized, but they are the most sophisticated ones currently available across the project and therefore a reasonable benchmark. This model suggests that a 6% participation level affords leadership of $\sim$ 4 medium/large (ie. 5000–10000 hours) KSPs, chairship of $\sim$`<!-- -->`{=html}40 KSP working groups (WGs), and $\sim300$ KSP team memberships over the course of a decade. These numbers are broadly consistent with the expertise and ambitions from §: Canadians could lead a KSP in each of pulsars, magnetism, transients and cosmology, lead $\sim$`<!-- -->`{=html}5 KSP WGs in each of pulsars, magnetism, transients, cosmology, galaxies, GWs and planets, and have a few team memberships in all KSPs project-wide.

If Canada does participate in the SKA1 Design Baseline at the 6% level, then the latest available costing and schedule imply that construction funding of order (€$\,918\mathrm{M} \times 0.06 \times 1.46\,\mathrm{CAD/}$€$\,\sim$) \$80M$\,$CAD (2017 CAD) will be required in the next decade, as well as commensurate operations funding for a total estimated construction and operations cost of \$160M CAD (2017 CAD) from 2021 – 2030. This funding includes contributions to a staged observatory development fund that grows to a project-wide €$\,20\mathrm{M}$/year (2017 €) by the onset of full operations.

In addition to construction and operations funding, Canada will need to contribute to the global SKA Regional Centre (SRC) network that will supply the requisite processing, storage and user support to scientifically exploit SKA1 (see §). The estimated Canadian SRC cost in the next decade during construction and ramp up to full operations has been calculated from the bottom up (c.f. Appendix B of Spekkens+ 19), and is the most sophisticated currently available across the project: it includes PI and KSP networking, processing and online storage, near-line storage for the SKA1 archive in compliance with global SRC guidelines (Bolton+ 18), 5-year processing refresh and continuous storage refresh, and staffing to provide user support for both observers and archive users.

For a 6% Canadian participation level in the SKA1 Design Baseline, the estimates described above predict that the SRC cost will be \$45M CAD (2017 CAD) from 2021 – 2030, where the bulk of the cost stems from on-line storage, followed by support staff, processing and networking. The first hardware would be purchased and minimal processing/storage carried out in 2022, with processing/storage/support capacity ramping up to handle full SKA1 Design Baseline operations by 2028. A Canadian SRC (as opposed to a model in which science compute is purchased elsewhere) would leverage Canadian leadership in scientific computing platform and archive development to provide processing, storage, and user support tailored to Canadian SKA1 needs. SRC capabilities are part of the broader digital research infrastructure in astronomy that will be required in the next decade (Kavelaars+ 19).

SKA1 is happening, and Canada at last has the opportunity to reap the scientific benefits of decades-long technical and governance contributions by participating in the construction and operations phases. In order to do so, a mechanism for participating in the IGO that will soon take over the project is needed. SKA1 construction tender and procurement will take place soon after the construction phase begins in 2021 (§); a participation agreement finalized early in the next decade would maximize Canadian opportunities to secure construction contracts as well as to inform policy and process during the construction phase.

It is possible for Canada to accede to the IGO if governmental approval is obtained; alternatively, Canada could participate in SKA1 construction and operations via some form of Associate Membership or other agreement to be negotiated with the IGO. The IGO Treaty Convention and Final Record do not distinguish between the scientific or technological access and leadership rights of Full Members and those of countries that participate through Associate Membership or some other model. It is therefore possible for Canada to negotiate for full scientific and technological rights if an alternative to Full IGO Membership is sought. Given the significant scientific leadership potential of the Canadian community as well as the potential for key technical contributions that deliver return on SKA1 construction funding investment (§), this approach aligns well with Canadian ambitions.

# Responses to Panel Questions


abstract: Researchers have proposed various activation functions. These activation functions help the deep network to learn non-linear behavior with a significant effect on training dynamics and task performance. The performance of these activations also depends on the initial state of the weight parameters, i.e., different initial state leads to a difference in the performance of a network. In this paper, we have proposed a cooperative initialization for training the deep network using ReLU activation function to improve the network performance. Our approach uses multiple activation functions in the initial few epochs for the update of all sets of weight parameters while training the network. These activation functions cooperate to overcome their drawbacks in the update of weight parameters, which in effect learn better "feature representation" and boost the network performance later. Cooperative initialization based training also helps in reducing the overfitting problem and does not increase the number of parameters, inference (test) time in the final model while improving the performance. Experiments show that our approach outperforms various baselines and, at the same time, performs well over various tasks such as classification and detection. The Top-1 classification accuracy of the model trained using our approach improves by 2.8% for VGG-16 and 2.1% for ResNet-56 on CIFAR-100 dataset.
author: Pravendra Singh Munender Varshney Vinay P. Namboodiri
Department of Computer Science and Engineering, IIT Kanpur, India
`{psingh, munender, email@example.com`
bibliography: egbib.bib
title: Cooperative Initialization based Deep Neural Network Training

# Introduction

Deep neural networks (DNNs) are state-of-the-art models, responsible for transforming research in the area of vision, language and speech . Various works have been proposed for efficient deep learning. These deep network at core performs a linear transformation followed by a non-linear operation using an activation function. The activation function is the one, which is responsible for nonlinear behaviour and the learning capabilities of the network. These activations are non-linear continuous functions which may also possess non-differentiability . Researchers have proposed many activation functions which can be classified into saturated and non-saturated activation functions .

The saturated activation belongs to a category, in which the learning process gets slow down due to the very small gradient near-saturated output. These activation functions are experimentally proved to be less effective for training a deep network. The key reason for the failure is the (vanishing/exploding) gradient problems, which mostly occurs due to saturated output in an activation function. This problem is efficiently tackled by using non-saturated activation function, like ReLU . In particular, the derivative of ReLU is one for the positive inputs; hence, the gradient cannot vanish. In contrast, all the negative values are mapped to zero, which restricts the flow of information in DNNs for these negative values. ReLU gets saturated exactly at zero, which makes ReLU fragile at the time of training, and the neuron can die forever. For example, the flow of large gradients through ReLU may update weight parameters in a way that may deactivate neurons for all data points. This problem is known as dying ReLU, which implies that the gradients flow through the neuron will forever be zero from that point. Due to this, the gradient-based optimization algorithm will not be able to update the weights of that neuron unit. Also, training the network on a high learning rate may shoot the number of "dead" neurons in the network as much as 40% of the network (i.e., neurons that never activate across the entire training dataset). So there is a need to set the learning rate properly to reduce the issues.

To resolve these potential problems originated by the hard zero mappings in the ReLU units, various generalizations of ReLU such as Leaky ReLU , and PReLU have been proposed. Both Leaky ReLU and PReLU are same as ReLU except for the case of negative inputs in which a small constant slope for Leaky ReLU and a learnable slope for PReLU are used. Similarly, Exponential Linear Units (ELU) is also proposed, which resolves the bias shift from the succeeding layers. ELU gives an exponential value corresponding to negative inputs, which force the mean output of the activation function to reach towards zero. Although ELU is not backed by concrete theory, ELU shows competitive results. Besides all these, Softplus is approximately similar to ReLU, except at zero, where softplus is differentiable and smooth. Softplus is also differentiable everywhere and saturates less, which gives an edge over ReLU. In practice, there is no non-linear activation function that outperforms all the time over all models, datasets, and problems.

In this paper, we propose an approach in which multiple non-linear activation functions are exploited using a cooperative strategy to overcome their drawbacks. We have used multiple activation functions in the initial epochs of training the deep network. The aggregation of gradient from all the activation function gives a *regularization effect* for the gradient flow corresponding to the whole range of inputs (negative values also). This results in regularizing the update of weight parameters, which is a very crucial step in the initial stage. In the next stage, we train the network with only one activation function (standard network) corresponding to each layer of the network. The proposed approach has experimented extensively on different architectures such as ResNet, and VGG-16 models over CIFAR-10, CIFAR-100, and ImageNet datasets. We also experimented with object detection task using SSD-300 on the PASCAL VOC dataset.

Major contributions of this paper are as follows:

-   We have shown experimentally that using multiple activation functions in the initial few epochs of the training process benefits the update of the full set of weight parameters, which results in substantial performance improvement later.

-   We have shown empirically that a mixture of non-linear activation functions results in significant improvement in the performance as compared to the individual non-linear activation function.

-   We have shown that Cooperative Initialization based training also help in reducing overfitting problem.

-   Our proposed approach does not increase the number of parameters and inference (test) time in the final model while improving the performance.

# Previous Work

The first activation function is a step function originally used in the perceptron model . Researchers in the same direction have also proposed many other saturated activation functions such as sigmoid, softmax, and tanh . The ReLU activation further replaces these activation functions, owing to the outstanding performance on deep neural networks . ReLU had escalated the convergence and resolved the vanish gradient problem, normally occurred in saturated activation functions. These activation functions have accelerated the efforts of the research community in solving various vision problems. Several attempts have been made to develop a more efficient network by developing better activation functions, which can resolve the problems arising in the above activation functions. Some variants of ReLU have been proposed such as RReLU, PReLU, leaky ReLU, and others .

To resolve the issues of mapping, all negative input to zero (dying ReLU) in the ReLU activation function, Leaky ReLU has been proposed. This mapping causes an information loss (dead neuron), which is resolved by defining a linear function corresponding to negative input, having a small predefined constant slope, to leak some information . However, Leaky ReLU does not give any notable results on performance experimentally. Further, a parametric rectified linear unit (PReLU) has been proposed which uses a learnable slope parameter instead of a constant slope as in Leaky ReLU for negative inputs. PReLU gives better performance than ReLU in many cases. On the second thought, the slope parameter can be randomly sampled from a uniform distribution as used in Randomized Leaky Rectified Linear Unit (RReLU) which reduces the risk of overfitting in the training phase. ELU is also same as ReLU for positive input while it behaves similarly to saturated exponential function for negative inputs. Further, Parametric ELU (PELU) is a scaled version of ELU having a learnable scaling parameter . Most of the above discussed ReLU variants are based on the experiments over the negative inputs of ReLU.

There are few other works in which new activation functions are proposed, such as Maxout (maximum over K affine functions), Softplus, and Adaptive Piecewise Linear (APL) . The APL consist of many non-differentiable points that scale linearly with the number of hinge functions. This will increase the model complexity and affect the parameter updates during back-propagation . In the same manner, Maxout take maximum over multiple feature maps. Softplus is the smooth approximation of ReLU, which is differentiable everywhere.

In this work, we have focused on leveraging benefits from the multiple non-linear activation functions simultaneously. To the best of our knowledge, this is the first work that considers a mixture of non-linear activation functions in the initial few epochs. We have also presented an ablation study and feature visualization to support the proposed approach.

# Cooperative Network Design (Phase-1)

Nowadays, deep networks consist of huge depth due to the presence of multiple convolutional layers. These convolutional layers are followed by an activation function, which operates on the feature maps (output) of these layers. The training process will update weight parameters using back-propagation. This update solely depends on the behavior of the activation function, i.e., ReLU never updates the weight parameter corresponding to negative inputs. Restricting these weights from an update in the initial phase of training may hurt the performance of the deep network later.

To overcome these issues in the network, researchers have proposed various activation functions, which have their advantages and disadvantages. In this work, we have proposed a *cooperative design* having multiple activation functions, helping each other in the initial update of parameters in such a way that all set of weight parameters get an opportunity to contribute to the performance. These activation functions cooperate to overcome their drawbacks and improve the update process for all sets of weight parameters in the initial epochs of training.

In Figure , we have shown a block diagram of a layer, where the convolution operation is same as of standard CNN, but we have used k activation functions instead of one activation function. The input feature maps for the given layer are represented as $FM_i$ in Figure , which are then convolved using convolutional filters shown in green color. The output feature maps $FM_o$ generated by convolution operation are passed to each activation function, which operates over each element of the given feature maps. This results in the generation of k different sets of feature maps corresponding to each activation function, as shown in Figure . The Final output Feature Maps ($FFM_o$) are the result of the weighted average of these k feature maps as given by the following equation:

$$FFM_o = \beta_1 F_1(FM_o) + \beta_2 F_2(FM_o) + ... + \beta_k F_k(FM_o) ;$$ where $F_1, F_2, ...  F_k$ are the activation functions applied on the $FM_o$ feature map. The final output feature maps ($FFM_o$) are the weighted average of outputs from these activation function when applied to input feature maps (shown in red color). The corresponding weights for averaging are $\beta_1, \beta_2, ..., \beta_k$ where $1,2, ..., k$ represents the k activation functions. We assume that each activation contributes equally in the update process (improvement) of the weight parameters by assigning equal weight to each parameter ($\beta_i = 1/k$).

We have presented an ablation study to shown empirically that even if we train the model completely by using a mixture of activation functions, it will results in substantial performance improvement as compare to the individual activation function.

# Standard Network Design (Phase-2)

In Phase-2, the network use only a single activation function, i.e., ReLU activation for each layer in the model, instead of the mixture of activation functions. In the training process of phase-1 design, our focus is to shift the model in a stable state where all sets of model parameters are in a better state than a randomly initialized state. The phase-1 network design is trained for a few epochs such that the mixture of activation functions will overcome the drawbacks of each other by cooperation in the update process of weight parameters. All the k activation functions will get different gradients and averaging them give a *regularization effect*, which updates all sets of parameters uniformly without undermining others.

In Phase-2, We use only one activation function, so we opt to use only ReLU as our activation function. ReLU is a linear (identity) mapping for all positive values and zero values for all negative values. ReLU activation shows a sparse behavior as all negative inputs are mapped to zero. After getting into a better state from phase-1, we often desire to make feature maps sparse enough, which is easily possible with ReLU as it possesses many deactivated neurons, giving a regularized effect to the model. Sparsity results in concise models that often have better predictive power and fewer chances of overfitting/noise. In a sparse network, all neurons cannot be activated simultaneously in a model. Only those set of neurons get activated which are responsible for a particular aspect of the given task, e.g., there are a given set of neurons which gets activated for a face like structure in a human detecting task while the same set of neurons are inactive for other parts of the body. That's the reason standard ReLU seems to be less prone to overfitting vs. leaky ReLU with modern architectures. We have also presented an ablation study to empirically show that ReLU in Phase-2 will results in substantially improved performance as compare to other possible options. Hence ReLU is competing with all other non-linear activation functions in Phase-2.

# Training Method

The proposed training framework is divided into two phases corresponding to the two network design phases. For Phase-1 (Cooperative) training, we use 20% of epochs used in standard (Phase-2) training with a mixture of non-linear activation functions (having equal weight). We have presented an ablation on the number of epochs used in Phase-1 training to validate the choice mentioned above.

In the phase-1 training process, gradient calculation in the back-propagation algorithm is the aggregate of the gradient calculated for each activation function. This gives a regularized effect on the gradient and provides equal opportunity for all weight parameters to optimize. Phase-2 training is the same as the standard training of the model with only one non-linear activation function (ReLU). Please note that a mixture of activation functions is used in only Phase-1 training, while Phase-2 has only ReLU at every layer. Further details are provided in the experimental section.

# Experiments and Results

In this section, we have evaluated the performance of the proposed approach on classification and detection task. Our experimentation uses the state-of-art CNN architectures such as ResNet , and VGG-16 for various activation functions. All these models are trained over three standard benchmark datasets: CIFAR10, CIFAR100 and ImageNet dataset. We have also performed experiments using SSD on PASCAL VOC for object detection.

In these experiments, we have used four most prominently used non-linear activation functions (ReLU, PReLU, ELU, and SoftPlus). We have preferred PReLU to resolve the issue of dying ReLU; however, someone may also prefer Leaky ReLU. ELU has an exponential function for negative input, which is contrary to ReLU. This behavior of ELU pushes the mean to the neighborhood of zero, similar to the case of batch normalization . This shift of mean toward the vicinity of zero accelerate the training of network (fast convergence). ELU also guarantees more robustness towards the noise. These are the few reasons we have selected ELU in the mixture of activation functions. The behavior of ReLU and Softplus is almost similar, excluding near the periphery of zero, where the softplus is differentiable and smooth. Softplus has privilege over ReLU due to differentiability in the entire domain, and it saturates less.

The scope of activation functions depends on the problems. We have selected ReLU, PReLU, ELU, and SoftPlus, which are the most widely used non-linear activation functions in image classification and object detection problems.

In our experiments, baselines (using activation function ReLU/PReLU/ELU/SoftPlus) have been reproduced using a standard training procedure in PyTorch framework. We trained these models using 300 and 90 epochs for CIFAR and ImageNet dataset respectively.

## CIFAR 10 and CIFAR 100

The CIFAR-10 and CIFAR-100 are the datasets having tiny natural images. CIFAR10 datasets have 10 different image classes, while CIFAR-100 datasets have 100 classes. There are 50,000 training images and 10,000 test images, where all images are RGB images with a dimension of 32x32.

In the experiments with the CIFAR dataset, we perform standard data augmentation methods of random cropping to a size of $32\times32$ and random horizontal flipping. The optimization is performed using Stochastic Gradient Descent (SGD) algorithm with momentum $0.9$ and a minibatch size of $64$. In Phase-2 training, the initial learning rate is set to $0.1$, which is decreased by a factor of $5$ after every $50$ epoch. The models are trained from scratch for around $300$ epochs. For Phase-1 (Cooperative) training, we use only 20% of epochs used in Phase-2 training with PReLU, ReLU, ELU, and SoftPlus activation functions (having equal weight). The learning rate is set to $0.1$ and is decreased by a factor of $5$ after every $10$ epoch in Phase-1 (Cooperative) training. For evaluation, the validation images are used. The results on the CIFAR-10/100 datasets for all the architectures have been reproduced in the PyTorch framework.

The results are shown in Table , . We observe a consistent improvement in accuracy for VGG-16 and ResNet-56 over CIFAR. The model trained with our proposed two-phase training procedure not only outperforms ReLU significantly but also other non-linear activation functions such as PReLU, ELU, and SoftPlus, as shown in Table , .

```latex
\scalebox{.9}{
\begin{tabular}{lcc}
\toprule
Model & Activation Function & Accuracy(\%) \\
\midrule
VGG-16 (Baseline)& ReLU & 93.6 \\
VGG-16 (Baseline)& PReLU & 93.7\\
VGG-16 (Baseline)& SoftPlus & 90.5 \\
VGG-16 (Baseline)& ELU & 92.3 \\
\textbf{VGG-16 (Ours)} & \textbf{Mix (ReLU)} & \textbf{94.2} \\
\midrule
ResNet-56 (Baseline)& ReLU & 93.5 \\
ResNet-56 (Baseline)& PReLU & 94.0 \\
ResNet-56 (Baseline)& SoftPlus & 92.0 \\
ResNet-56 (Baseline)& ELU & 92.0 \\
\textbf{ResNet-56 (Ours)} & \textbf{Mix (ReLU)} & \textbf{ 94.4} \\
\bottomrule
\end{tabular}}
```

```latex
\scalebox{.9}{
\begin{tabular}{lcc}
\toprule
Model & Activation Function & Accuracy(\%) \\
\midrule
VGG-16 (Baseline)& ReLU & 72.0 \\
VGG-16 (Baseline)& PReLU & 72.5 \\
VGG-16 (Baseline)& SoftPlus & 64.2 \\
VGG-16 (Baseline)& ELU & 66.9  \\
\textbf{VGG-16 (Ours)} & \textbf{Mix (ReLU)} & \textbf{74.0} \\
\midrule
ResNet-56 (Baseline)& ReLU & 71.6 \\
ResNet-56 (Baseline)& PReLU & 71.9 \\
ResNet-56 (Baseline)& SoftPlus & 69.3 \\
ResNet-56 (Baseline)& ELU & 69.5  \\
\textbf{ResNet-56 (Ours)} & \textbf{Mix (ReLU)} & \textbf{73.1} \\
\bottomrule
\end{tabular}}
```

```latex
\scalebox{.9}{
\begin{tabular}{lcc}
\toprule
Model & Activation Function & Accuracy(\%) \\
\midrule
AlexNet (Baseline) & ReLU & 56.6 \\
AlexNet (Baseline) & PReLU & 56.9 \\
AlexNet (Baseline) & SoftPlus & 55.2 \\
AlexNet (Baseline) & ELU & 56.6 \\
\textbf{AlexNet (Ours)} & \textbf{Mix (ReLU)} & \textbf{57.2} \\
\midrule
ResNet-18 (Baseline) & ReLU & 69.8 \\
ResNet-18 (Baseline) & PReLU & 69.1 \\
ResNet-18 (Baseline) & SoftPlus & 68.8 \\
ResNet-18 (Baseline) & ELU & 68.2  \\
\textbf{ResNet-18 (Ours)} & \textbf{Mix (ReLU)} & \textbf{70.8} \\

\bottomrule
\end{tabular}}
```

```latex
\scalebox{.9}{
\begin{tabular}{lcc}
\toprule
\textbf{Class} & \textbf{SSD (Baseline) AP} &\textbf{SSD Mix (ReLU) AP} \\
\midrule
\textbf{aero} & 80.40 &  82.53\\
\textbf{bike} & 82.95 & 82.54\\
\textbf{bird} &  74.62 & 77.02\\
\textbf{boat} & 71.61 & 72.45\\
\textbf{bottle} & 50.49 & 51.36\\
\textbf{bus} & 86.04 & 85.57\\
\textbf{car} & 86.55 & 86.28\\
\textbf{cat} & 88.02 & 86.91\\
\textbf{chair} & 60.88 & 63.23\\
\textbf{cow} & 83.10 & 81.58\\
\textbf{table} & 77.87 & 78.35\\
\textbf{dog} & 85.55 & 84.06\\
\textbf{horse} & 86.68 & 87.79\\
\textbf{mbike} & 84.14 & 85.85\\
\textbf{person} &  78.26 & 79.29 \\
\textbf{plant} & 50.44 & 52.82\\
\textbf{sheep} & 74.28 & 78.13\\
\textbf{sofa} &80.03 & 80.72\\
\textbf{train} & 85.88 & 87.28\\
\textbf{tv} & 75.49 & 77.15\\
\midrule
\textbf{mAP} & 77.16 &  \textbf{78.05} \\
\bottomrule
\end{tabular}}
```

## ImageNet

ImageNet dataset contains 1000 classes, each category roughly having 1000 images. The dataset contains about 1.2 million training images, 50,000 validation images, and 100,000 test images (with no labels). The training is performed on training data, whereas all evaluations are performed on the validation set.

For ImageNet experiments, we perform standard data augmentation methods of random cropping to a size of $224\times224$ and random horizontal flipping. For optimization, Stochastic Gradient Descent (SGD) is used with momentum $0.9$ and a minibatch size of $256$. For Phase-2 training, the initial learning rate is set to $0.1$, which is decreased by a factor of $10$ after every $30$ epoch. The models are trained for around $90$ epochs. The evaluation is done over validation images are subjected to center cropping of size $224\times224$. For Phase-1 (Cooperative) training, we use 20% of epochs used in Phase-2 training with PReLU, ReLU, ELU, and SoftPlus activation functions (having equal weight). The learning rate is set to $0.1$ and is decreased by a factor of $10$ after every $5$ epoch in Phase-1 (Cooperative) training.

The results are shown in Table . We have consistent improvement in accuracy for AlexNet , ResNet-18 over ImageNet dataset. The model trained with our proposed two-phase training procedure not only significantly outperforms ReLU but also other non-linear activation function such as PReLU, ELU, and SoftPlus (Table ).

## PASCAL VOC

We have performed experiments on the SSD model over PASCAL VOC dataset to validate our proposed approach for the object detection task. In this experiment, we follow the same experimental setting and training schedule, as described in for Phase-2 training. For Phase-1 (Cooperative) training, we have used 20% iterations of Phase-2 training with PReLU, ReLU, ELU, and SoftPlus activation functions (having equal weight). The SSD detection model is a feed-forward convolutional network that produces a collection of fixed-size bounding boxes and predicts classification scores to represent the presence of object class instances in these boxes, followed by a non-maximum suppression which produces the final detections.

As shown in Table , our proposed approach is not limited to classification but also works well on object detection task. We have a significant improvement (approx. 1%) in mAP as compare to baseline by using our training procedure.

```latex
\scalebox{.82}{
\begin{tabular}{lcc}
\toprule
Model & Activation Function & Accuracy(\%) \\
\midrule
ResNet-56 (Baseline)& ReLU & 93.5 \\
ResNet-56 (Baseline)& PReLU & 94.0 \\
ResNet-56 (Baseline)& SoftPlus & 92.0 \\
ResNet-56 (Baseline)& ELU & 92.0 \\
\midrule
ResNet-56 (Baseline-TPT)& ReLU & 93.6 \\
ResNet-56 (Baseline-TPT)& PReLU & 94.0 \\
ResNet-56 (Baseline-TPT)& SoftPlus & 92.0 \\
ResNet-56 (Baseline-TPT)& ELU & 92.1 \\
\midrule
ResNet-56 & WNLA & 40.0 \\
\midrule
ResNet-56 & Mixture &  94.3\\
ResNet-56 & Mix (PReLU) & 94.1 \\
ResNet-56 & Mix (SoftPlus) &  93.1\\
ResNet-56 & Mix (ELU) &  93.2\\
\midrule
\textbf{ResNet-56 (Ours)} & \textbf{Mix (ReLU)} & \textbf{ 94.4} \\
\bottomrule
\end{tabular}}
```

```latex
\scalebox{.85}{
\begin{tabular}{lcc}
\toprule
\#Epochs in Phase-1  & Activation Function & Accuracy(\%) \\
\midrule
10\% & Mix (ReLU) &  94.03 \\
\textbf{20\%} & Mix (ReLU) &  \textbf{94.40} \\
30\% & Mix (ReLU) &  94.40 \\
40\% & Mix (ReLU) &  94.42 \\
\bottomrule
\end{tabular}}
```

# Ablation Studies

As shown in Table , if we train a model without any non-linear activation function (WNLA), the performance of the model degrades massively since ResNet-56 is a deep CNN model which is very hard to optimize without any non-linear activation function. There is a performance boost in the case of Mix (SoftPlus) and Mix (ELU) from their respective baselines, but the overall performance scores are significantly lower than Mix (ReLU). The key reason for the significant difference in the performance can be due to the sparse behavior of (ReLU) activation function in Phase-2 training. This sparsity is often desirable in the deep network due to better predictive power and less overfitting/noise.

Although Mixture and Mix (ReLU) have similar performance, Mix (ReLU) should be preferred because of the following reasons:

-   Mixture will occupy more feature maps memory as compare to Mix (ReLU) at run time because separate feature maps will be generated for every non-linear activation function. Mix (ReLU) will not increase in feature maps memory at a run time (GPU Memory).

-   Mixture will add some delay at the inference time as compare to Mix (ReLU) due to extra calculations performed by multiple activation functions.

Hence, Mix (ReLU) is the most suitable combination while considering all the other possibilities. Our proposed approach uses Two-Phase Training (TPT), where Phase-1 training uses 20% of Phase-2 epochs. Therefore, one may think that performance improvement is due to more training epochs (20%). Hence we also train baselines with the same Two-Phase Training schedule (Baseline-TPT), except in Phase-1, only a single activation function is used. As shown in Table , Baseline-TPT has a similar performance as the baseline. From Table , we can conclude that performance improvement is not due to more training epochs (20%) in Two-Phase Training (TPT) but because of *cooperative initialization* in Phase-1.

We also conduct an ablation to decide the number of Epochs in Phase-1 training. For Phase-1 (Cooperative) training, we use 10-40% of epochs used in standard (Phase-2) training. As shown in Table , 20% of Phase-2 epochs in Phase-1 (Cooperative) training is the most suitable choice because it gives significant performance improvement with only 20% increase in overall training time.

# Visualizing Last Layer Features on MNIST

In the Figure , we are plotting t-SNE plots for LeNet-like network on MNIST dataset to visualize the features learned for various non linear activation functions. The LeNet-like network contains two convolutional layers and one fully-connected layer. The convolutional layers have 5x5 kernel size while the first and second convolutional layer consists of twenty and thirty number of filters respectively.

The t-Distributed Stochastic Neighbor Embedding (t-SNE) is a dimensionality reduction technique that is heavily used to visualize the high-level features learned by CNN. The t-SNE is a commonly used technique to visualize feature representations in high-dimensional data into a space of two or three dimensions. From Figure , we can visualize the two-dimensional embeddings of the last layer. The features corresponding to Mix (Relu) are more separable than remaining other embeddings. The features corresponding to Mix (ReLU) are well separated and discriminated enough, as shown using corresponding representative points in Figure .

# Analysis

This section focused on the analysis of performance gain from two different perspectives. The first perspective focus on investigating the convergence of Mix (ReLU) and ReLU on the ResNet-56 architecture. The convergence speed of Mix (ReLU) is much faster as compared to ReLU, which can be inferred in Figure-. The second perspective for investigation is based on over-fitting aspects of the models where Mix (ReLU) is more robust compared to ReLU based on the empirical results.

The investigation is performed on the CIFAR-100 dataset using the ResNet-56 model. We used standard data augmentation techniques such as random horizontal flipping and random cropping to size $32\times32$. The optimization is performed using Stochastic Gradient Descent (SGD) with momentum set to $0.9$ and weight decay as 0.0001. The minibatch size of $64$ is selected to perform experiments. The initial learning rate is taken as $0.1$, which is then decreased by a factor of $5$ after every $50$ epoch. The models are trained from scratch for around $300$ epochs.

## Effect of using Mix (ReLU) on convergence

```latex
\begin{table*}[t]\centering
    % \renewcommand{\arraystretch}{1.2}
        \addtolength{\tabcolsep}{2pt}
        \begin{tabular}{|l| c| c|c|}
            \hline
            Method & Activation Function & Train Accuracy(\%)  & Test Accuracy(\%) \\
            \hline\hline
            ResNet-56 Baseline (100\% of training set) & ReLU & 99.4  & 71.6 \\
            ResNet-56 (100\% of training set) & \textbf{Mix (ReLU)} & 99.6  & \textbf{73.1} \\
            \hline\hline
            ResNet-56 Baseline (25\% of training set) & ReLU &  99.9 &   51.3 \\
            ResNet-56 (25\% of training set) & \textbf{Mix (ReLU)} &  99.9 &  \textbf{60.8} \\
            \hline
        \end{tabular}
    % \vspace{5pt}
    \caption{The table shows the results for ResNet-56 ReLU and ResNet-56 Mix (ReLU) over CIFAR-100 dataset in different setups.}
    \vspace{-5pt}
    \label{tab:overfitting}
\end{table*}
```

We analyzed the convergence rate of Mix (ReLU) based model, and we found that the convergence using Mix (ReLU) is slightly better compared to ReLU, which can be inferred from their respective curves in Figure . The two graphs of cross-entropy losses vs. the number of epochs for training and test set in Figure-, shows that the dropping rate of cross-entropy losses is quite higher as compared to the loss corresponding to ReLU on the training set in the experimental results shown in Figure-.

## Effect of Mix (ReLU) on over fitting

Our method utilize multiple activation functions in the initial few training epochs of the deep network. The gradient from these multiple activation functions gets accumulated and gave a *regularization effect* to the gradient flow in the model corresponding to the complete range of input data (negative and positive ). This aggregation of gradients also helps in the regularization of weight parameters while updating, which in effect knock-down the chances of over-fitting issues. In support of our hypothesis, we have performed some experiments in two different scenarios. In the first scenario, experiments are performed out over the complete dataset (100 % of training data), while the second scenario of experiments is performed on only 25 % of the training data.

The first scenario of experiments is performed on the CIFAR-100 dataset using ResNet-56 architecture. The first experiment with only ReLU achieved an accuracy of 71.6% as presented in Table- while the second experiment uses Mix (ReLU) and achieve 73.1% accuracy. From Table-, we can infer that Mix (ReLU) is more robust to overfitting as the difference between test and training accuracy is 26.5, which is lesser than the difference between test and training accuracy of ReLU (27.8). In these experiments, the difference between test and training accuracy is not that much significant for ReLU and Mix (ReLU), hence in the second scenario, we have chosen only 25% samples from the training images of CIFAR-100 dataset to perform various experiments.

The second scenario of experiments is performed to train the ResNet-56 ReLU model using only 25% train samples, and we achieve accuracy of 99.9% and 51.3% corresponding to train and test data respectively. The ResNet-56 Mix (ReLU), on the other hand, achieves 99.9% and 60.8% of training and test accuracy respectively.

From Figure-, We can conclude that the Mix (ReLU) is less prone to overfitting issues, as the difference between test and training accuracy is 39.1 while this difference is quite higher than that of ReLU, i.e., 48.6.

# Conclusion

We propose a Cooperative Initialization for training deep networks to improve the performance. We have shown experimentally that a mixture of the non-linear activation function is beneficial for CNN in the initial phase of training, where we start from random initialization. Initially, multiple activation functions regularize the gradient flow corresponding to the positive and negative input of activation functions, thereby improving the update of weight parameters, which is very crucial at the initial stage. Our experimental results show that the proposed approach improves the performances of state-of-the-art networks. Our proposed approach also helps in reducing the overfitting problem and does not increase the number of parameters, inference (test) time in the final model while improving the performance. Therefore, cooperative initialization is a promising approach to improve the feature representation and performance of deep networks.