# Abstract

Despite the increasing number of women graduating in mathematics, a systemic gender imbalance persists and is signified by a pronounced gender gap in the distribution of active researchers and professors. Especially at the level of university faculty, women mathematicians continue being drastically underrepresented, decades after the first affirmative action measures have been put into place. A solid publication record is of paramount importance for securing permanent positions. Thus, the question arises whether the publication patterns of men and women mathematicians differ in a significant way. Making use of the zbMATH database, one of the most comprehensive metadata sources on mathematical publications, we analyze the scholarly output of $\sim\!150,000$ mathematicians from the past four decades whose gender we algorithmically inferred. We focus on development over time, collaboration through coautorships, presumed journal quality and distribution of research topics—factors known to have a strong impact on job perspectives. We report significant differences between genders which may put women at a disadvantage when pursuing an academic career in mathematics.

# Introduction

The overall number of women in Science, Technology, Engineering and Mathematics (STEM) has undoubtedly been increasing in the past decades . There is a growing presence of women on all academic levels: more female students, university graduates, PhDs, and professors. However, gender inequities persist, in particular towards higher academic ranks. Women are still significantly underrepresented when it comes to permanent academic positions such as tenure-track and university faculty posts.

Providing comprehensive figures and statistics on women in mathematics is not trivial, due to scattered data among countries, national differences in academic careers, and the difficulty to exactly define what "mathematical research" entails, among others. Among the available surveys, Hobbs and Koomen  is the most complete report on women in mathematical research in Europe and shows a comparison between data taken in 1993 and 2005. While the proportion of female mathematicians had increased in the 12-year-long span in almost all countries, women are still underrepresented, especially in northern and western Europe. More specifically, the London Mathematical Society's report on Women in Mathematics from 2013 surveyed the university mathematics departments in the UK and states a 29% female representation at the lecturer and a mere 6% at the professor level . For the faculty positions in the U.S., the Conference Board of Mathematical Sciences (CBMS) 2015 study  reports an increase in percentages of women among tenured and tenure-eligible faculty, with a relatively low level of 11% women in tenured faculty of doctoral university departments. On the other hand, the share of doctorates granted to women remained almost constant at around 30% between 2000 and 2010 . For a list of more descriptive statistics see .

Scholarly publications are (still) the major means of communicating research results. A solid publication record is a key factor in achieving and sustaining a successful academic career in every discipline. The relentless pressure to promptly and steadily publish in the hope of sustaining one's career has lead to the coining of the phrase "publish or perish". The central role of scholarly productivity when pursuing a professorial post bears the question whether men and women in mathematics differ regarding their publication patterns, and whether these differences have changed over time.

Due to the difficulty in obtaining complete records, many studies on gender effects in academia are restricted to small amounts of data, mostly from surveys, or limited to a few journals or countries. While small-data studies might suffer from lack of generality and the danger of "anecdata", they do typically deal with cleaner, less ambiguous data sources. In particular, surveys based on self-reported data have built-in authorship disambiguation and can acquire gender assignments from the respondents. Large-scale publication databases instead often suffer from missing information, which then needs to be inferred. In this article, we analyze publication metadata from zbMATH, an online bibliographic database comprising more than 3 million mathematical publications classified by research topics and enhanced with author profiles. Based on the authors' first names, we attribute a gender to $\sim\!150,000$ mathematicians from the past four decades. To the best of our knowledge, this is the most comprehensive large-scale study of this kind in the field of mathematics.

Our analysis addresses four dimensions which are crucial for understanding the impact of publication patterns on the gender gap in mathematics: (1) research activity over time, (2) choice of journals and their perceived quality, (3) collaboration through coauthorship, and (4) distribution across specialisation fields.

The growing presence of women in mathematics in particular and in STEM disciplines in general has often led to the belief that achieving a balanced gender ratio is merely a matter of time. Yet, the increasing number of women mathematicians entering graduate school will only translate into an overall increase of women in academia if they move on towards higher scholarly ranks, which motivates our longitudinal analysis. For active researchers in mathematics, journal articles are the predominant publication form, and the perceived quality of a journal is an important criterion when judging the excellence of a researcher's publication record. Said quality is often measured by means of journal rankings, which are highly controversial. Nevertheless, they are widely in use, often relieving evaluators of the onus of reading and judging an article by its content. For authors pondering which journal to submit to, or for university administrators, research funders, or governmental bodies evaluating publication lists, journal rankings are presumed to be helpful, thus we employ them in our study. A further career determinant worth analyzing is the interaction with other researchers: a prospering network advances the chance of citations by peers, facilites building collaborations and can help securing a university position. On the other hand, the share of single-authored publications in mathematics is, in contrast to many experimental sciences, very high. Such publications act as proof of one's individual capabilities, and are thus particularly valuable at an early career stage. They show that one is not "completely dependent on senior people for ideas, guidance, techniques, \[…\] Hence, one is ready for a faculty position" . Last but not least, mathematical topics differ in terms of general popularity, preferences of journal editors, the overall community, and particular gatekeepers. Some topics bear higher chances for a scientific career in mathematics than others, and popular journals reflect a certain bias for particular mathematical fields .

Our analysis shows a higher dropout rate of women as authors of mathematics papers, despite an upwards trend in the share of women mathematicians over the last four decades. Women publish less articles than men during their careers and write less articles alone but establish networks of comparable size. Their work is significantly less published in journals of high presumed quality, measured in terms of the Thomson Reuter's journal impact factor (JIF) as the most prominent journal ranking schema and the manually compiled Australian ERA ranking. In three highly regarded journals, Annals of Mathematics, Inventiones Mathematicae, and Journal of the AMS, a significant underrepresentation of women persists without notable uptrend over the last four decades. While the presence of women is shown to strongly fluctuate across mathematical fields, the choice of research fields is found not to be responsible for the substantial underrepresentation of women in said three journals.

## Related work

Several studies have looked at publication patterns in the context of gender. Frietsch et al.  studied women's share among patent applications and scientific publications. Using two large data sets from the 1990s and 2000s, the authors estimate the average proportion of women in mathematics publications of 14 countries in 2005 as 16.5% in total, being lowest in Germany (11.7%) and highest in Italy (19.6%). Bentley  studied data from two international surveys in 1993 and 2007 involving academic staff in Australian public universities. While gender differences in publication productivity decreased significantly within the 14-year time period, in 2007 women published on average only 76% of what men published.

West et al.  analyzed JSTOR data, a digital library corpus spanning several centuries of publications in sciences and humanities. Across all disciplines, the authors found 21.9% women; for mathematics, the estimated percentage of female authors is significantly lower at only 10.6% (= 6,134 persons). The authors showed that authorships by women are not evenly distributed over the dimensions time, field/subfield, coauthorship, and authorship position, the latter being particularly relevant for disciplines such as molecular biology or medicine. A recent, large-scale study based on Thomson Reuters' Web of Science (WoS) unveiled the persistence of women's underrepresentation in Russian science between 1973 and 2012 across many disciplines, particularly in mathematics and physics . Using fractional authorships computed by weighting an authorship as a reciprocal of the total number of (gender-identified) authors of the respective publication, the authors show disparities between women and men regarding research output, productivity, collaboration and scientific impact. Employing similar methods, the analysis of all articles published between 2008 and 2012 and indexed in WoS by Larivière et al.  confirms the persistence of gender imbalances in research output worldwide.

Collaboration, e.g. in the form of joint research, publishing or grant proposals, is considered a strong indicator for productivity and integration within the research community, and hence has been the subject of various studies. For various career stages of mathematicians, a positive correlation between the number of distinct coauthors and the overall number of publications has been observed . The understanding of the structural factors that affect men's and women's collaboration could lead to the development of policies to address differences in power and resource dynamics . A similar argument is made regarding interdisciplinary research and the possibility to use it to foster career success. Bosquet and Combes  use publication records of a group of French economists to evaluate the impact of various factors on two measures of article quality defined via journal quality and Google Scholar citation numbers, respectively. Among others, they show a high impact of the coauthor network size with respect to both quality measures. Effects of topical diversity and specialisation are considered as well, showing disparities between single subfields regarding the number and average quality of publications, as well as the number of citations.

According to Garg and Kumar , female and male Indian researchers in life sciences differ with respect to team work and choice of journals: women tend to work in smaller teams with less international collaborative papers, and their publications are more often published in domestic journals with lower impact factors. Using a set of high-quality Spanish journals across various fields of science, Mauleón et al.  analyzed the gender distribution over time of authors and editorial board members as representatives of active and leading scientists, respectively. While the share of female authors grew over the years, only a minimal effect was visible in editorial board membership. Restricting to eight international mathematics journals, Mauleón and Bordons  calculated the share of women in 2004 to be 4% in editorial boards and 10% of all authors. The authors could not discover any relation between gender and the journals' impact factors. They showed only slight differences in the collaborative behaviour and a slight disadvantage for women regarding citation metrics.

Finally, questions related to productivity, amount of scholarly output, and merit are the topic of multiple studies. For a discussion on why some faculty members in political science publish more than others, see Hesli and Lee . The construction of the concept of "academic excellence" is analyzed in Brink and Benschop . Recently, in a detailed discussion on productivity based on arXiv publications, Pierson  expresses hope that the gender productivity gap is closing, albeit recognizing that there is still a long way to go until gender parity is accomplished. Her analysis of almost 1 million arXiv articles on STEM disciplines supports some of the findings that we present in this paper.

# Materials and methods

## The zbMATH corpus

zbMATH (formerly Zentralblatt MATH) is an abstracting and reviewing service in mathematics, produced by FIZ Karlsruhe – Leibniz Institute for Information Infrastructure and coedited by the European Mathematical Society (EMS) and the Heidelberg Academy of Sciences and Humanities. Along with MathSciNet, zbMATH is one of the main services that provide bibliographic information (e.g. author names, title, source), abstracts and expert reviews as well as aggregated data (e.g. keywords) on mathematical publications to the scientific community.

#### Author profiles

The process of attributing authorship of bibliographic records to specific individual researchers, known as "author name disambiguation", is essential to perform analyses on publication data. This is by no means a straightforward task, due to the fact that personal names are not sufficiently distinct to unambiguously identify each individual within the large number of researchers active today and in the past. The problem is exacerbated by the inconsistent way in which author names appear in publications.

Author name disambiguation, and the more general problem of word sense disambiguation, is an open issue in computational linguistics and natural language processing. A taxonomy of automatic author name disambiguation methods can be found in . zbMATH provides author profiles for its indexed publications. They are constructed following a multifaceted approach based on automatic, manual, and collaborative methods. Initially, the assignment of a publication to an author is performed algorithmically, using mainly the name string and coauthorship information; some authorship assignments are post-processed manually by the zbMATH staff, and, recently, users can contribute corrections through an online form .

Name-string-based authorship attribution suffers from various well-known issues such as confusion of authors with frequent names (John Miller), missing name parts (middle names), variability through transliteration (Chebyshev vs. Tschebyscheff), or name changes throughout a researcher's life (after a change of civil state). In zbMATH, an ambiguous authorship is initially attributed to all possible candidates, e.g. an article authored by "P. Smith" will appear in the profiles of "Paul Smith" and "Patricia Smith". Hence an author profile in zbMATH can contain wrongly assigned articles, or publications can be missing. A combination of both issues is possible as well. Some types of assignment error are more frequent in relation to certain regions; for instance, authors with Chinese names are often assigned to ambiguous profiles due to common surnames and transliteration problems. Due to this latter factor, authors from the former Soviet countries frequently suffer from profile splitting.

Cases such as "P. Smith" referred above, when the author name string does not suffice to unequivocally assign an author, are marked as ambiguous and are not to be trusted. We excluded such authorships from our analyses.

#### Mathematics Subject Classification (MSC)

The Mathematics Subject Classification (MSC) is a universally accepted classification schema in mathematics jointly produced and maintained by MathSciNet and zbMATH. It is a tree-like, three-level alphanumerical scheme to label publications according to their subject matter. For instance, "35" stands for the area of "Partial Differential Equations", "35J" refers to "Elliptic Equations and Systems", and "35J75" is the classification for "Singular Elliptic Equations". The MSC exists since 1970; it is revised every 10 years, with MSC 2010 being the current version . All entries in zbMATH since 1970, and even some older data, are classified by its editorial board according to the MSC. The first code of an article is its primary classification, supposedly reflecting its most important content; subsequent codes are secondary.

Articles classified with first-level MSC codes between 03 and 65 belong to mathematics subfields such as Logic, Discrete Mathematics, Algebra, Analysis, Geometry, Topology, Probability Theory, Statistics, and Numerical Mathematics. Those with codes between 68 and 94 fall into mathematical applications related to Computer Science, Physics, or Economics. The areas 00 and 01 refer to general topics and history of mathematics; 97 is Mathematical Education.

#### Focus on core mathematicians

The zbMATH corpus includes publications in which mathematical concepts are applied to scientific subjects such as quantum theory or computational biology. Hence, the zbMATH author records necessarily contain individuals whose primary research is not in mathematics. Their publications usually appear in interdisciplinary journals. For our analysis we have decided to only consider authors as mathematicians if they have published at least one article in a "core mathematics journal". To extract such journals, we resort to the MSC labels applied to a journal's articles to identify core mathematics publications, using the following heuristics: a journal belongs in the group of core mathematics journals if at least 90% of its articles with primary classification between 03 and 94 fall within the range 03–65. Our corpus consists of publications drawn from 4064 journals. By including the rather applied areas of Statistics (62) and Numerical Mathematics (65) we retain a comparatively large number of 1,050 core mathematics journals, representing 26% of all serials indexed by zbMATH with publications after 1970.

## Guessing an author's gender

In general, the gender of a person cannot be inferred programmatically from the name string—hence the use of a database of annotated names for gender assignment is often unavoidable. Several studies have made use of annotated data from varied sources: West et al.  used a two-gender data set provided by the US Social Security Administration containing the top 1,000 annual girl and boy baby names since 1888; Frietsch et al.  worked with a first name database comprising 8,291 names from six European countries. This database was created as part of a project carried out by the European commission in 2002, in which first name assignments were collected from various data sources, then post-processed in order to achieve a high quality level, and finally tested . Implementations of gender guessing procedures include the Python library *Gender Detector*  and the R package *gender* . A benchmark comparison of gender detection algorithms can be found at .

We tried to attribute a gender to all zbMATH author profiles. The zbMATH corpus contains no supplementary metadata on authors' genders. We resorted to the annotated collection of first names described in . The dictionary file contains a list of more than 42,000 names with gender assignment ; at the time of the last update in November 2008 the list was assumed to cover the vast majority of names in all European and some non-European countries like China, India, Japan, USA. According to the author Jörg Michael, for some countries such as Turkey or Korea all names have been independently classified by native speakers, indicating a high quality of the data. Yet the zbMATH data contains no information on the country of origin of the authors, hence we could not use this additional facet. Every name in the dictionary is labelled as "unknown", "female" or "male", with additional information on the certainty levels of the assignment.

On top of this algorithmic assignment, we have performed a manual check of the most prolific profiles. Specifically, we have reviewed the gender assignment for the top (in number of publications) 300 profiles classified as men and 500 women. These typically correspond to well-known mathematicians, whose gender we were able to assess. We found no instances of women mathematicians being incorrectly classified and a $\sim\!5\%$ error in men that were incorrectly labelled as women. We manually changed the gender of the latter, resulting on cleaner and more trustworthy gender data. We observed that these errors were more frequent for names from India, Pakistan and Japan. We conclude that our study overrepresents the amount of women by at most $\sim\!5\%$.

We point out that troughout the text, we speak of men and women as if those categories were real. Gender binary is a concept that lacks both complexity and suitability. But categories are constructs and results of societal action. Almost all databases and publications connecting names with gender operate within this binary schema. Since we do not have access to detailed author profiles and data on how authors identify when it comes to gender, we use the name as an indicator of the perceived gender of each author. Thus, our categorization of mathematicians into men and women is following societal and academic action and is due to pragmatic and substantive reasons.

## Data overview

Our data set contains information on all publications by authors we identified as "core mathematicians" as described in subsection *Focus on core mathematicians*. For every such publication we extracted the authors, their order of appearance, the title, the publication year, all MSC classification codes, and the journal name together with ISSN and e-ISSN codes, if the publication appeared in a journal. We additionally tagged authorships with a value indicating the quality of the authorship assignment (manually curated, algorithmically unambiguous, and ambiguous). The MSC classification exists since 1970. We restricted our analyses to authorships published since then to ensure consistency when addressing our key questions.

The corpus consists of 2,245,205 document entities corresponding to 3,433,553 instances of authorship, yielding 1.53 authors per article. The amount of ambiguous authorships which could not be assigned by zbMATH to a unique author amounts to 226,500 ($6.6\%$). We excluded these authorships from the analyses since we cannot decide which attribution might be correct.

We applied a gender assignment algorithm as described in  to 262,146 author identities, the remaining 47,319 ($\sim\!15\%$) authors were listed without a first name. We were able to assign a gender to $\sim\!55\%$ of all profiles with a real first name; among those, 27,596 ($\sim\!19\%$) authors were classified as women and 116,657 as men.

We were able to guess the gender for $61.4\%$ of all (non-ambiguous) instances of authorship. Fig  displays the percentage of all authorships that could be labelled as women since 1970. It is undeniable that the amount of women that publish in mathematics has been increasing—their share has tripled over the past four decades.

As shown in Table , profiles of women are on average slightly less ambiguous than those of men; profiles with a high level of ambiguity are significantly more frequent among those authors we could not classify as women or men. One explanation for the latter observation is the fact that our gender assignment algorithm requires the first name (which is missing for $15.3\%$ of authors in our corpus), and the name string is also the main information used for authorship assignment by zbMATH.

```latex
\begin{tabularx}{\textwidth}{ X X X X X X }
    \toprule
     & \multicolumn{5}{ c }{Ratio of non-ambiguous publications per profile} \\
     \cmidrule(r){2-6}
     & 100\% & 90\% & 80\% & 70\% & 60\%\\
     \midrule
    all authors & 49\% & 52\% & 55\% & 58\% & 65\% \\
    women & 71\% & 73\% & 75\% & 77\% & 84\% \\
    men & 62\% & 67\% & 71\% & 73\% & 81\% \\
    \bottomrule
\end{tabularx}
```

# Results

## Cohort analysis

A fundamental metric to understand the gender gap among scientists is the distribution of researchers that successfully sustain a career in academia over time. We analyse this aspect with a cohort analysis of authors since 1970. A cohort is a group of individuals that share a common characteristic over a defined period of time. Comparing the development of cohorts allows for longitudinal studies that unveil patterns over large periods of time. In our study, we assign authors to a given cohort according to the year of first recorded publication, which acts as an approximation for the beginning of their scientific career. We then follow the publication records of all authors in every cohort and select those individuals that continue publishing at least 5 and 10 years, respectively, after their first paper. We roughly assume researchers to be at the postdoctoral stage 5 years after their first publication and to have secured a permanent position at professorial level another 5 years later.

#### Women mathematicians have tripled their number since 1970

The total number of authors that continue publishing 5 and 10 years after their first paper appears in the upper panel of Fig . Despite a brief stagnation in the 1980s, the number of mathematicians at our estimate of professorial level (10 years) has increased from $\sim\!1500$ in the 1970s to $\sim\!1800$ per year in the 1990s. Note that the decreasing number of authors after the 1990s reflects the fact that data for the most recent cohorts is partially incomplete: members of the early 2000s cohorts may well be still active today, eventhough they do not have a publication in the early 2010s. Therefore we regard the figures from the 2000s as non-conclusive.

Whereas a mere $\sim\!6\!-\!8\%$ of all authors in the early 1970s were women, their percentage has tripled within the following three decades (Fig ). Roughly 20% of all authors with careers lasting at least 10 years that started publishing in the late 1990s are women. However, this is still far from the percentage of women that pursue mathematics education at the undergraduate level.

#### Women publish less than men at the beginning of their careers

An important metric that reveals differences between genders is the amount of publications. A shorter publication record at the time of applying for an academic position diminishes the chances of success. Our findings confirm the phenomenom known as productivity gap, agreeing with those of Pierson . We have analyzed the average amount of publications of men and women during the 5 and 10 years after their first article appeared; we find that men routinely surpass women. Fig  illustrates this fact, showing the excess of publications by men per cohort year. Despite the visible trend towards a narrower productivity gap, men are shown to publish on average 9% and 13% more papers than women within their first 5 and 10 years as authors, respectively.

#### Women leave academia at a higher rate than men

In Fig  we evaluate the percentage of men and women that stop publishing at some point between 5 and 10 years after their first paper. This we interpret as them having discontinued their research careers. The results clearly show an overall increase in the dropout rate for both genders over the past decades. This phenomenon reflects the fact that an increasing number of doctorates and postdoctoral scholars find it impossible to secure a permanent position in science, thus being forced to leave academia even after a comparatively large publishing trajectory of at least 5 years.

On top of that, women end their research career in this period at a higher rate than men, as can be seen in Fig . A larger comparative drop in the years that roughly correspond to the postdoctoral stage may indicate that women either lack support to reach the stable professorial ranks, or do not wish or are not able to continue holding temporary positions in a period of life often coincident with family-planning decisions. This evidences the existence of a leaky pipeline, understood as a continuous loss of women in academia as they climb the career ladder, that prevents the number of women in mathematics from increasing as fast as the rates of women enrolling at the graduate level would suggest.

## Perception of quality

We analyze two of the most prominent journal rankings in mathematics, the Australian ERA list and the journal impact factor. ERA (Excellence in Research for Australia) is an initiative of the Australian Research Council to measure the quality of Australian research. ERA compiled a ranked journal list with ranks reaching from A\* (with the expectation that almost every article in these journals will be of very high quality) over A (most articles of very high quality) and B (only some articles of very high quality) to C (very high quality not expected) and "not ranked". We use the ERA 2010 ranked journal list which is the most recent list of this kind. It comprises 20,712 journals of all academic fields. Since 2012 ERA does not rank the journals anymore and publishes instead an unranked but more complex list of journals.

The journal impact factor (JIF) is calculated based on Thomson Reuter's Journal Citation Reports. It is defined as the average number of article citations in a journal in the last two years. There is widespread criticism of the impact factor, as there are many means to manipulate it . Based on different publication and citation practices in various academic fields, it is difficult to compare the impact factor across disciplines. We refer to the 2015 edition of Journal Citation Reports (which uses publication data from 2013/2014) and investigate impact factors for 7 subject categories: "Mathematics", "Mathematics, Applied", "Mathematics, Interdisciplinary Applications", "Computer Science, Theory and Methods", "Logic", "Physics, Mathematical", "Statistics and Probability".

We identify the ERA rank and JIF by ISSN, which leads to 1,768 matched journals in the ERA list and 728 journals with a JIF (663 journals appear in both indexes). We investigate publications from 2004–2013. Note that we refer to the most recent impact factor from 2015 which in most cases is not the impact factor of the journal at the submission or publication date. We further restrict our sample to publications with conclusive gender attribution and in journals from the ERA list or with an impact factor. We do not limit the analysis to core mathematics journals (see subsection *Focus on core mathematicians*) after our tests showed no major differences with respect to the full analysis.

For both rankings we investigate how the percentage of women among the authors per rank deviates from the average in the complete data set. We use the ranks A\*, A, B, and C as assigned by ERA. Since the impact factor is not discrete, we calculate percentiles of the journals in our list, the highest (lowest) containing the 25% journals with the highest (lowest) impact factors. We end up with four groups of publications for each ranking. In both cases, the number of articles per group differs. Furthermore, we examine authorships of single-authored publications versus all authorships (including single- and multi-authored) and quote separate results.

#### High-ranked mathematics journals publish less articles authored by women

We uncover significant differences between female and male authors in both these rankings. There are fewer women amongst the authors of top-ranked journals than could have been expected from the overall distribution of men and women in our samples. Our analysis of the ERA ranked journal list shows that only $8.5\%$ women are authors of single-authored publications in A\* journals, a deviation of $-21.3\%$ from the average, while there is a deviation of $+20.9\%$ in the low-ranked category C. The results for all single- and multi-authored papers are similar, since the A\* journals show $18.9\%$ less authorships by women while there is an excess of $16.6\%$ with respect to the average in C journals. The results are displayed in Fig .

For the percentiles of the JIF we obtain a significantly smaller share of female authors in the top 25% journals with $17.4\%$ and $7.8\%$ less women in the single-authored and all-authorships samples, respectively, when compared to the corresponding overall shares. Details are shown in Fig .

#### Women are drastically underrepresented in three top-ranked journals

Female authors publish significantly less in the top journals according to the two ranking mechanisms we examined. To look further at the peak of what is perceived as good journal quality, we analyze the gender distribution of authors in three highly regarded serials: "Annals of Mathematics", "Inventiones Mathematicae", and "Journal of the AMS". Our aim is to shed light on the percentage of female authorships and its development over the last decades. Fig  illustrates that the share of female authors is significantly lower than in the overall sample of mathematical journals, but also lower than in the top categories of the ERA list and the JIF. Even more remarkable: no upward trend is discernible: in the Journal of the AMS, for instance, we do not identify a single female author in the last years (2012: 42 men/23 unknown; 2013: 43 men/16 unknown; 2014: 20 men/9 unknown). Note that these numbers include all authorships. Limiting the analysis to single-authored publications results in very low absolute numbers, but following a similar tendency.

## Collaboration through coauthorship

Often, the idea of mathematics is associated with the image of a solitary researcher, whose work requires only pen and paper and hence can be performed in pure isolation. It is therefore no surprise that more than 95% of all publications of "core mathematicians" in zbMATH before 1950 were single-authored; in our corpus they still comprise almost 60% of all records. Despite the strategical relevance of single-authored publications, a strong shift towards collaborative authoring is evident. This trend is supported by our data: among the publications that appeared after 2000, only 39% had been authored by a single person. On the other hand, almost 25% of the publications were authored by more than 2 authors compared to e.g. less than 8% in the period 1970–2000, indicating a growing interest in building networks. We investigate coauthorship patterns in terms of two metrics: share of single-authored publications per author and number of distinct coauthors ("network size"). In contrast to other studies, we do not analyze the positions of coauthors with respect to gender, since in mathematics, authors are usually listed in alphabetical order (see, e.g., ).

We examine the records of mathematicians that started publishing in 1970 or later for whom we were able to successfully assign a gender (25,050 women and 77,792 men). Since the odds of collaborating with different coauthors increase with publication record length, we segment authors according to their number of publications. We sample the data as to include the same number of female and male authors per total number of publications. The sample size is thus determined by the maximum number of authors in the women's group.

#### Women publish less single-authored papers

On average, men write 38% of their scientific records as single authors, in contrast to 29% among women. This trend remains unchanged when looking at groups according to the total number of publications, in segments of at most 5, 10, 20, 50, or more than 100. Fig  illustrates this conclusion, showing a statistically significant gap between men and women in the percentage of single-authored publications across all segments. A similar tendency has also been reported in  for UK-authored journal articles in Social Sciences, or in  for the field of Information Systems; in the later study, the percentage of male authors with a single-authored publication was twice as large as for their female counterparts.

Sooner or later, many mathematicians publish an article on their own. In our data set, this is the case for 52% of the women; in the men's sample this holds true for 61%. The single-authored paper is usually not the author's first publication, in particular in the case of female mathematicians. Women write their first publication most often together with coauthors: only 33% of their first publications are single authored, for men it is 43%.

#### Women's and men's coauthor networks are of similar size

Although women mathematicians have significantly less single-authored publications than their male counterparts, they do not collaborate with a considerably larger number of peers. On average, women have 3.6 distinct coauthors, 0.2 more than men. Fig  shows the network size for men and women in more detail, split according to the length of publication record. The dark and light green bars display the average network sizes of women and men, respectively, with the enclosed triangles representing the medians. The mean network size of women is slightly larger in each of the groups; however, the gap decreases with a growing number of publications from factor 1.15 in the group of authors with at most 5 publications (i.e. the network size of women in this group is on average 1.15 times larger than that of men) to 1.02 in the two groups of authors with more than 50 publications. The median values for women and men are equal in the first two and the fourth group; for authors with 11-20 publications in total, women have a median of 6 compared to a median of 5 for men. In the last two groups, the median value for men is larger; among authors with more than 100 publications the group of men exhibits a median of 42 versus 38 for women.

Both measures suggest that women and men collaborate with a similar number of peers. However, when measuring the network size with respect to the time line instead of the total number of publications, then Fig  suggests that the networks of women are even smaller than those of men, at least when looking at an early career stage, since women publish around 10% less articles than men within the first 5 years of their research .

Bozeman and Gaughan  statistically analysed collaboration patterns based on surveys of STEM researchers in the United States. In contrast to the authors' expectations, women had a slightly (but not significantly) larger collaboration network. However, the study did not include information on publications, and hence could not elaborate on the relation between the number of collaborators and single-authored publications.

It is a common assumption that a larger coauthor network is indicative of the visibility of an author's research. For instance, Bentley  shows for Australian university research that international collaborators were one of the main factors associated with increased publication productivity among both men and women. Our findings support the conclusion that men and women collaborate with similarly-sized peer groups. However, women seem to do so at the expense of their own single-authored publications, which might undermine their profiles as fully independent researchers.

## Topical distribution

We study the distribution of authors and publications in mathematical fields as represented by the MSC 2010 (see subsection *The zbMATH corpus*). We focus on the first and second MSC levels; the first level represents mathematical fields such as "Algebraic Topology" or "Dynamical Systems", whereas the second level allows for a more granular view of related mathematical communitites. We restricted to the primary MSC code assigned to an article since this usually captures the main content of the research.

#### Women's distribution across research fields is not homogeneous

Women publish in all of the 63 mathematical fields represented by the first hierarchy level of the MSC; out of the 528 topic-based subfields at the second MSC-level, there are only three in which none of the women in our data set have published either alone or with a coauthor. By restricting our search to single-authored publications, we find 13 second-level classes in which women have not yet published, such as "$K$-theory in number theory" (19F) or "Function theory on the disk" (30J).

However, women are not equally represented across mathematical areas. Using hierarchical maps as displayed in Fig  we visualize the share of women resp. women's publications across the MSC classes on level 1 and 2: each rectangle represents a mathematical class, its size is proportional to the number of authors resp. publications in that field, and the darker the color, the larger their share.

This confirms the results by West et al.  who showed an uneven distribution based on publications from JSTOR. Unfortunately, we cannot compare the results in detail since West et al. did not use a classification schema such as the MSC but instead employed an unsupervised hierarchical clustering approach to assign publications to topics.

In subsection *Perception of quality* we showed that the presence of women in three of the most renowned mathematical journals—the "Annals of Mathematics", "Inventiones Mathematicae" and the "Journal of the American Mathematical Society"—is low, and reveals no significant improvement over the last three decades. The three journals, despite aiming to cover all fields in mathematics, display a bias towards certain fields. For instance, the five areas "Algebraic Geometry (14)", "Number Theory (11)", "Differential Geometry (53)", "Dynamical Systems and Ergodic Theory (37)" and "Group Theory and Generalizations (20)", already cover more than 50% of all publications in these journals. On the second MSC level we see a similar dominance of certain subfields, leading to a coverage of more than a third of all publications by only 12 classes which are shown in Fig . To investigate whether the selection of research areas might be responsible for the underrepresentation of women in these journals, we checked for correlation between a particular field's presence in the three journals and the women's share in this field. It turns out that women are well represented in the top 12 level-2 classes ($\sim\!9\%$ above the average when considering all publications and $\sim\!13\%$ when restricting to single-authored publications). When looking at publications in those fields instead of authors, the numbers even out ($\sim\!-3\%$ when looking at all publications and $\sim\!4\%$ when restricting to single-authored publications). Most level-1 fields with strong dominance in the three journals show an average or even higher presence of women, with the traditional field of "Number Theory" being the most significant negative exception.

#### Women's distribution of research topics is narrower than men's

As shown in subsection *Perception of quality*, women and men collaborate with a comparable number of peers when measuring against groups of authors with a similarly-sized publication record. Taking the point in time as scale instead of the total number of publications (see Fig ), the networks of women are smaller than those of men. We see a similar pattern when considering the variation of topics: as shown in Fig , publications of women spread across less MSC classes than those of men, while keeping the overall number of publications per author fixed. Their stronger focus on a smaller set of research topics could potentially be another drawback for women in academia, since it reduces the already low number of possible faculty positions suitable for applying to.

# Discussion

Our study is based on comprehensive data from publications in all areas of mathematics research between 1970 and 2013. Backed by a statistically significant data corpus and scanning a rather broad set of determinants for scholarly productivity, we audit the status of women mathematicians in the publication landscape and identify research questions which would provide additional substantial insights.

We have split all authors in our data set in cohorts according to the year of their first publication and have investigated their progression 5 and 10 years into their careers. We have looked into variations in publication patterns between men and women regarding presumed quality of the journals they publish in, their coauthorship networks, and the topical distribution of their scholarly output. We report statistically significant differences among genders in all the aforestated aspects.

The share of women mathematicians who have been active for at least 5 to 10 years has roughly tripled since the 1970s (see Fig ). However, women abandon their academic careers within a decade of having started them at a much higher rate than men (see Fig ). This indicates the existence of a "leaky pipeline" that slows down the inclusion of women in the higher ranks of academia, and it invalidates the hypothesis that, given the steady increase of women entering mathematics at the undergraduate level, their incorporation to higher academic ranks should simply follow suit. Furthermore, the so-called productivity gap, stating that women produce less publications within a fixed period of time, is clearly visible in every cohort. However, it has been narrowing down drastically since 1970 with a visibly positive trend in recent years.

The perceived quality of the journal in which an article is published is one of the most common means of evaluating a researcher's academic output. There is legitimate and compelling criticism of this approach. Beyond all objections to particular shortcomings of these measurements, what they all have in common is the impossibility of providing any information on individual articles. While we believe that journal rankings are no proper means to evaluate scientists and their work, they are still widely adopted to measure research quality. With respect to such a measurement scheme, women are extremely underrepresented in top-ranked journals. A negative correlation exists between journal rank, measured using the ERA ranking or the journal impact factor, and the percentage of women among its authors, a pattern that remains fairly stable over time. A further investigation of three highly renowned mathematics journals shows an unusually low percentage of published women, with almost no improvement over time. Several explanations may be proposed for this fact, from biased editorial boards and reviewers to lack of strategic publication choices on the part of women. Using our data, we were able to exclude the choice of the research field as possible reason, since women are well represented in the fields favoured by the considered journals. Certainly, it would be worth to further investigate if and why women simply submit less articles to prominent journals or whether their papers are more often rejected. Nonetheless, our findings uncover a specific phenomenon correlating journal rank and scarcity of published women, and the reasons for these differences need to be studied in more detail.

With respect to collaboration through coauthorship, our results indicate that men publish far more single-authored papers than women. Since a solid record of single-authored publications seemingly shows one's ability of independence and creativity and is hence associated with "scientific autonomy", the disparity could have a negative effect on women's career perspectives, in particular at an early stage. Considering that their network is not significantly larger, this suggests that women regularly collaborate with the same researchers and begs the question of *with whom* women and men collaborate. Are the most frequent coauthors the doctoral advisor and close collaborators? Likewise, an analysis of the coauthors' gender could help understand whether gender induces distinctive publication dynamics, e.g in an all-female versus a mixed-gender collaboration. Additionally, it would be interesting to explore if factors such as the academic position or the country/internationality of the collaborators have an effect on the collaboration patterns.

It seems sensible to consider the aforementioned questions also in a field-focused setting since the presence of women strongly deviates across research area, as shown in Section *Topical distribution*. It would also be interesting to explore in more detail which factors motivate women to enter a certain research area. What constitutes an inviting "atmosphere" within a field, what is the importance of role models, mentoring, etc.? How does this relate with the age and interdisciplinarity of a field? Do new and emerging areas produce a different mind-set and are hence more attractive for women scientists?

We note that our analysis is robust with respect to the counting scheme. In certain cases, e.g. when papers authored by men have a larger number of authors than papers by women, it might be more suitable to analyse distributions of fractional authorships where an authorship is weighted reciprocally to the overall number of authors of the paper . For our data set, however, both full and fractional counting lead to very similar results. This is explained by the dominance of single-authored publications in mathematics and the relative balance of women's and men's network sizes. Since women in our data set have, on average, more coauthors than men, the gender gap is even slightly stronger in terms of fractional authorships.

Our findings support the thesis that gender-related publication patterns exist and are one of the factors that lead to an underrepresentation of women in mathematics. Our analysis cannot give a definite answer to the main motivating question, i.e. *why* the gender gap in mathematics. The differences in research productivity according to gender are by no means trivial to measure or explain. Likewise, research productivity alone does not fully translate into career success, however defined. Various studies, including those cited in Section *Related work*, have looked at the factors that might contribute to a higher or lower research output, including demographics, family-related circumstances, working environment, etc. Furthermore, academic excellence is an inherently gendered concept , with several aspects contributing to its fabrication. Even if the publication and collaboration patterns of men and women showed no significant differences, a completely gender-blind approach to promotion in mathematics academic positions is unfortunately not to be expected. Hence a thorough discussion on the topic must necessarily expand towards further aspects. Perhaps a first one is the question of who or what is setting the pace and norms of scholarly publishing. In the work of Brink and Benschop  the concept of "masculine career path" (p. 513) is put forward: every researcher (man or woman) whose publishing record deviates from the traditional trajectory, maybe due to career interruptions or periods of part-time work, is at dire disadvantage for promotion. A similar remark is found in Bentley , which mentions the institutionalization of male dominance by associating merit primarily with scholarly output, an activity for which men receive greater patronage and support. The lack of encouragement that many female doctoral students perceive when it comes to applying for funding or to collaborate on research projects already sets back their careers. Women are forced to either "catch up" later or, or are driven out of the field altogether at a higher rate than men, as evidenced by our cohort analysis. The fact that in many cases the number of publications is the main factor in defining the quality of research constitutes a drawback for the numerous women with nonlinear career trajectories or less research time.

Using one of the largest metadata sources on mathematical publications we have shown that the collective actions of all actors in mathematics publishing result in significant differences between two gender groups regarding key factors of academic careers. A more thorough and detailed analysis of single aspects would clearly help to explain the observed disparities between women and men. For this, additional data such as affiliations, acceptance and rejection statistics from journals, or the country of origin would be very helpful; the latter could also be used for an improvement of the gender assignment algorithm which constitutes the base of our study. In fact, we would prefer not to use such an assignment schema at all but ask the authors instead how they identify. Our analyses have unveiled significant differences which may already foster individual takes on publishing, hence career decisions, and inspire institutional measure. We believe that the gained insights will deliver input for further discussions within the community for potential collective and/or individual strategies to overcome the current situation.

# Acknowledgements

We gratefully acknowledge the support of FIZ Karlsruhe for granting us access to their publication data. We thank the European Women in Mathematics network and specially its former convenor Marie-Françoise Roy for feedback in the first stages of this project and useful comments on an earlier paper draft.


abstract: Current daily paper releases are becoming increasingly large and areas of research are growing in diversity. This makes it harder for scientists to keep up to date with current state of the art and identify relevant work within their lines of interest. The goal of this article is to address this problem using Machine Learning techniques. We model a scientific paper to be built as a combination of different scientific knowledge from diverse topics into a new problem. In light of this, we implement the unsupervised Machine Learning technique of Latent Dirichlet Allocation (LDA) on the corpus of papers in a given field to: *i)* define and extract underlying topics in the corpus; *ii)* get the topics weight vector for each paper in the corpus; and *iii)* get the topics weight vector for new papers. By registering papers preferred by a user, we build a user vector of weights using the information of the vectors of the selected papers. Hence, by performing an inner product between the user vector and each paper in the daily Arxiv release, we can sort the papers according to the user preference on the underlying topics.
  .
  We have created the website `IArxiv.org` where users can read sorted daily Arxiv releases (and more) while the algorithm learns each users preference, yielding a more accurate sorting every day. Current `IArxiv.org` version runs on Arxiv categories `astro-ph, gr-qc, hep-ph` and `hep-th` and we plan to extend to others. We propose several new useful and relevant implementations to be additionally developed as well as new Machine Learning techniques beyond LDA to further improve the accuracy of this new tool.
bibliography: biblio.bib

**Intelligent Arxiv:
Sort daily papers by learning users topics preference**
**Ezequiel Alvarez$^{(a)\dagger}$, Federico Lamagna$^{(b,a)\ddag}$, Cesar Miquel$^{(c)\star}$ Manuel Szewc$^{(a)\diamond}$**
*$^{(a)}$ International Center for Advanced Studies (ICAS) and CONICET, UNSAM,
Campus Miguelete, 25 de Mayo y Francia, CP1650, San Martín, Buenos Aires, Argentina*
*$^{(b)}$ Centro Atómico Bariloche, Instituto Balseiro and CONICET
Av. Bustillo 9500, CP8400, S. C. de Bariloche, Argentina*
*$^{(c)}$ Easytech, Parana 759, piso 3, CP1048, Buenos Aires, Aregntina*

# Introduction

Machine Learning techniques are playing a crucial role in changing our every day experience with the World. During the last years there has been a huge growth in the amount and in the class of applications of the different techniques and subfields within Machine Learning. In particular, Natural Processing Language has been greatly developed by needs in the internet and communication industry. The concept of *Topics Model* has been extremely useful not only to understand and index a corpus of documents but also to learn compelling features about it. This same technique has been adapted and used in other disciplines such as Bioinformatics , Chemistry , Health care and Physics , among others. Along this work we pursue the objective of applying Topics Model to scientific literature and to use its outcome to create a new tool for sorting papers according to each user topics preference.

There exist a variety of algorithms applied to scientific papers and scientific literature. Among them we can mention `Arxitics.com` which allows to share voting, reviews and comments to provide an enhanced interface for reading and discussing the Arxiv; `Scirate.com` which sorts papers according to ratings from the community, Ref.  which mines scientific articles for recommending them to users based on abstract content using a personal collection of references, `CiteULike` which allowed users to share preferences on scientific papers, `Mendeley.com` which is a complete desktop service for sorting and archiving bibliography and generating bibliography for given articles, among other services, and Ref.  which uses collaborative filtering within a framework of topic modeling to recommend articles to users. Some of these and other cases use rating algorithm, hand-made functions, and/or Machine Learning techniques to provide scientists with a better access to bibliography. However, at current knowledge of the authors, there is not yet an available algorithm that learns from each user personal preferences and sorts the scientific papers accordingly for each user, which is the main goal that drives the content of this work.

Our departure point to tackle this problem is that in many cases the creation of a new scientific paper can be modeled as putting together scientific knowledge from different topics into a new problem. This understanding of a scientific paper corresponds to the modeling of documents within a corpus in which each document is a specific mixture of topics in given proportions. The Latent Dirichlet Allocation (LDA) algorithm is an unsupervised Machine Learning framework to address the topics and weights extraction in this kind of corpus. On the other hand, and solely for the purposes of this article, we may also model the interests of scientists through these given topics: scientists have a weight on each one of these topics that represents their interest on them. Given this modeling, it is suitable to start our enterprise of classifying papers and Arxiv readers based on the LDA algorithm.

This work is divided as follows. In Section 2 we briefly describe the Topics Model and the Latent Dirichlet Allocation tool as an unsupervised Machine Learning algorithm to extract abstract topics from a given corpus of documents. In Section 3 we apply Latent Dirichlet Allocation to a selected group of Arxiv categories and to each category itself. We show how the study of these corpora through this tool provides a new mechanism to understand their constitution and a new tool for classifying them. In Section 4 we present and discuss the website `IArxiv.org` which allows Arxiv readers to access the bibliography sorted according to their topics preference. In Section 5 we discuss the prospects for future implementations to `IArxiv.org`. Section 6 contains the conclusions of this work.

# Topics Model in Latent Dirichlet Allocation (LDA)

Topic Modeling, or Topics Model, is a general framework of statistical models which aims to infer abstract "topics" from a given corpus of unlabeled documents. These abstract topics can be thought of as the generators of the corpus and can thus be assumed to encode all the information necessary about the corpus. The topics can be used to label the documents with some criteria or to perform a dimensional reduction from a large corpus to a relatively small number of topics over which to conduct different operations, in a similar way to other unsupervised clustering techniques such as Principal Component Analysis (PCA).

While in PCA the clustering works with correlations to find the principal variables that encode the variance of the data, topic modeling aims to find clusters of words with semantic meaning (although this is not guaranteed as we are using abstract topics). With these criteria in mind, we focus on probabilistic topic modeling where we assume a generative model that encodes the semantic structure of the corpus. These generative models can capture better inter- and intra-documents statistical structure than non-generative models such as Term Frequency times Inverse Document Frequency (TF.IDF) and are thus better suited for semantic clustering. From the generative model we can derive a posterior using Bayes theorem which, although intractable, can be approximated by algorithms such as Gibbs Sampling or Variational Bayes. The estimated posterior then provides the semantic information from which we can obtain the topic distributions over the vocabulary and each document distribution over the topics. One example of this, and the one we focus on this work, is the Latent Dirichlet Allocation (LDA) as detailed in Ref. .

In LDA we assume a mixed membership model with a fixed amount $K$ of topics, in which all of the $D$ documents are composed of every topic and every topic contains all the $N$ words in the vocabulary. To account for this, we assume each document $d$ has a multinomial probability distribution $\theta_d$ over the topic space and each topic $k$ has a multinomial distribution $\beta_{k}$ over the vocabulary. In turn, these probability distributions are sampled from two Dirichlet Distributions with hyperparameters $\alpha$ and $\eta$ respectively. In this context, $\theta$, $\beta$, $\alpha$ and $\eta$ play the role of latent variables which generate the corpus but are not directly observable. The Dirichlet distribution is the conjugate prior of the multinomial distribution, which allows the Bayesian inference to keep the multinomial distribution shape, albeit with different probabilities assigned to each category.

The procedure to generate the corpus is then the following:

1.  For each topic $k=1,...,K$ sample a $\beta_k$ multinomial encoding the word proportions from the Dirichlet distribution $\text{Dir}(\eta)$.

2.  For each document $d=1,...,D$, sample a $\theta_d$ multinomial encoding the topic proportions from the Dirichlet distribution $\text{Dir}(\alpha)$.

3.  For each word $n=1,...,N$, sample a topic $Z_{dn}$ from the multinomial distribution, $\theta_d$.

4.  Sample a word $w_{dn}$ from the topic distribution $\beta_{Z_{dn}}$.

The corpus can then be simplified to two sets of distributions $\theta_{d}$ and $\beta_{k}$ sampled from two Dirichlet distributions, with each document generated by a set of two coin tosses per word, one to select a topic according to $\theta_{d}$ and one to select a word according to $\beta_{k}$ Further details can be found elsewhere .

The procedure can also be encoded and understood in a plate diagram, where a circle represents variables of the model. If the circle is shaded, then the variable is observable while if it is unshaded, then the variable is latent and has to be inferred. The arrows indicate the direction of the sampling while the plates denote repetition over a number of steps indicated in the bottom right-hand corner. The plate diagram for LDA is shown in Fig. .

While the generative model computes the probability of obtaining a word from a given set of variables $p(w,\theta,\beta,Z | \alpha, \eta )$, we are interested in the posterior probability $p(\theta, \beta, Z | w, \alpha, \eta )$ which allows to infer the topics and its distributions over the documents from the words. The posterior can be estimated by several means. We base our work on Online Variational Bayes as stated in Ref.  because it provides a relatively fast and efficient posterior estimation for large batches of documents.

All text processing and posterior estimation can be performed using the `Gensim` package , which implements the Online Variational Bayes method. We use this package to obtain the probability distributions and the topic proportions in each paper. We do not infer $\alpha$ and $\eta$ but instead we keep them as priors from which we obtain the topic proportions in the documents $\theta$, the topic distributions over the vocabulary $\beta$ and the topic chosen to generate each word $Z$. As discussed in Ref. , $\alpha$ and $\eta$ can also be updated. However, due to the large number of documents and topics, we are largely independent of the priors and we do not extract useful information from this update.

LDA performance has to be evaluated by taking into account convergence and validity of the obtained topics. While the former can be evaluated straightforwardly by measuring how the topics change with time, the latter is more arbitrary. The metrics we considered were perplexity and topic coherence. Perplexity is a measure of the models' ability to predict previously unseen data , which can also account for LDA convergence as each new document should not produce a noticeable change in the topic structure once a good model is obtained. Topic coherence, on the other hand, is a measure of how the top words inside each topic share semantic similarity and can be used to evaluate the goodness of the models' semantic clustering.

# LDA on the Arxiv

In this section we apply the LDA algorithm to the Arxiv database. We use the Bulk Data Access tool to download data from papers in categories `astro-ph, gr-qc, hep-ph` and `hep-th` of the last 8–10 years. We download paper id, title, abstract, submission date, authors, and the categories it belongs to. We use both title and abstract contents for the topics modeling. A starting point is to examine the Arxiv categories themselves, to study how well the LDA can find these categories as topics and how it performs as a classifier. Another, more ambitious task is to run an LDA model in each of these categories to find a topic structure in them, and analyze if it can help sort through the evergrowing number of articles in the scientific community. A large number of tasks can be performed with the use of this tool, for instance examine author affinity, or paper affinity, or even as a way of finding useful articles previously unknown to the user. Also it can help to study the evolution of certain themes within the community, or to see representative articles of a particular subject.

Before running LDA on the corpus of documents, a first processing has to be done on the text. This involves building the dictionary, and mapping the text into its bag-of-words form: this means that the order of the words in a text is no longer considered relevant, and texts become collections of words. The processing of the text for building the dictionary consists in two steps: word lemmatization and the removal of *stop words*. Word lemmatization, or stemming, refers to the process of keeping the stem of the words, and thus identifying words of the same origin. For instance, words "running, ran, run, runs, runner" become the same one "run", and thus preventing the proliferation of unique words with similar meaning in the dictionary . All punctuation and special characters are removed from the text, along with capitalization. The term stop words refers to those words that are the most common in natural language and are present in every text, like articles and prepositions. These words that carry syntactic (grammatical) information are not relevant on the topic structure of documents. Techwords that would otherwise be removed or modified by the preprocessing but carry precise meaning are manually kept in the text. A few examples of these are "1-d", "2-d", or particle names like "e+", or for instance "AdS", which would be modified to "ad". A list of these words is made by inspecting individual abstracts and the effect of the preprocessing on them. After this is done, an initial dictionary is created. However, the extremes of the word distribution are also removed. This implies words that are far too common, or extremely uncommon. We remove words that are present in less than 50 documents, and those that are present in more than 90% of the corpus. After this is done, each document becomes a list of observed frequencies of words from the dictionary. The remaining dictionary size is around 2500 words.

A first exercise in the use of LDA is to check if it can classify between actual Arxiv categories. For this task we take samples of "pure" (that is, without cross-list) papers from each of the four categories `astro-ph, gr-qc, hep-ph` and `hep-th`, and we run an LDA model with four topics. For each category, we plot the topic proportions over its papers as a histogram, this can be seen in Fig.

We see that for each category there is a topic that has a large proportion over the documents, and the rest of the topics have much smaller values. This means that these topic weights can be used for classification. To see how well this classifier can tag we plot the receiver operating characteristics (ROC) curves, one for each of the categories. The idea is that by using a threshold in the value of the weight, we can calculate the true positive rate and the false positive rate of one class against all others. We then plot the true positive rate against the false positive rate, for varying values of this threshold. A measure of the efficiency of the classifier is the area under the ROC curve, an area of 1 means an optimal classifier, whereas an area of 0.5 implies a random classifier. In Fig. () we show these ROC curves for each of the four categories.

The areas under each of the curves show how good is the classifier and, in this case, all ROC curves have areas of around .95 and higher. This indicates that the *unsupervised* LDA algorithm has found topics that are in correspondence with the *defined* categories in the Arxiv database. This is an interesting result, as topics found by LDA are not required to have human interpretability. To examine topics further, we can analyze the top words occurring in each of the topics, as depicted in Table .

```latex
\resizebox{\textwidth}{!}{%
\begin{tabular}{ | c | c |c | c |c | c |c | c |c | c |c | }
    \hline
    hep-th &  \thead{theori \\ 0.030} & \thead{field \\ 0.016} & \thead{gaug \\ 0.014} & \thead{gener\\ 0.009} & \thead{model\\0.009} & \thead{function\\0.009} & \thead{space\\0.009} & \thead{dimension\\0.009} & \thead{string\\0.008} & \thead{symmetri\\0.008} \\
    \hline
    hep-ph &  \thead{model \\ 0.021} & \thead{mass \\ 0.015} & \thead{neutrino \\ 0.011 }& \thead{decay \\ 0.010} & \thead{higg \\ 0.010} &\thead{quark \\ 0.009} & \thead{effect \\ 0.007} & \thead{product\\ 0.007} & \thead{dark\\ 0.007} & \thead{matter \\ 0.007} \\
    \hline
    astro-ph & \thead{star \\ 0.014} & \thead{observ \\ 0.013} & \thead{galaxi \\ 0.013} & \thead{mass \\ 0.010} & \thead{model \\ 0.008} & \thead{data \\ 0.006} & \thead{high \\ 0.006} & \thead{emiss \\ 0.006} & \thead{time \\ 0.006} & \thead{stellar \\ 0.005} \\
    \hline
    gr-qc & \thead{black \\ 0.019} &  \thead{ hole \\ 0.018} &  \thead{ field \\ 0.015} & \thead{model \\ 0.012} & \thead{equat \\ 0.011} & \thead{solut \\ 0.011} & \thead{ gravit \\ 0.011} &  \thead{graviti \\ 0.010} & \thead{energi \\ 0.010} & \thead{gener \\ 0.010} \\
    \hline
    \end{tabular}}
```

We can see how the topics are different between themselves and how they can be interpreted. The fact that the algorithm was able to do this separation of topics in a completely unsupervised manner shows the strength of topic modeling.

To compare with more typical Machine Learning algorithms, we also study a multi layer perceptron for the task of classifying the documents. In this case, the learning is supervised, as the labels have to be used for the calculation of the loss function. We used documents in their bag of words form as input for the neural network. That is, the input layer has a dimension equal to the size of the vocabulary, with each neuron corresponding to a different word, and its activation value corresponding to the number of times it appears in the document. These neurons are then normalized, to work with values between 0 and 1. We work with different architectures, changing the number and size of hidden layers. The final layer has 4 neurons, equal to the number of classes. We used `categorical crossentropy` as the loss function, and a `softmax` activation for this final layer. We implement these architectures by using the Keras package for Python 3. We find that already with a single hidden layer of 5 neurons the classification is as good as LDA, with ROCs having areas of about .95 and higher. With two hidden layers, of 5 and 50, the areas go upwards of .97 for all four topics. We see that the unsupervised classifier using LDA has a performance as good as a supervised NN classifier. For cases where tags are available there is usually no point in using unsupervised learning, however, inside each Arxiv category, there is a latent structure that is not made readily available by tags, and where an unsupervised Machine Learning algorithm like LDA is very useful.

Taking this idea into account, and using the fact that some Arxiv categories do have subcategories available, we focus on `astro-ph` and run an LDA model inside it. From 2009 onwards, this Arxiv category is split into 6 subcategories. We then take those papers, focusing once again on those without any cross-lists. We get a reasonably good clustering of the papers into 6 distinct topics, as can be seen in the histograms in Fig.

Once again as a measure of the classification capabilities of the model, we plot ROC curves for one-vs-all classification. Except the `astro-ph.SR` (Solar and Stellar) category, which has some non-negligible proportion of `astro-ph.GA` (Galaxies) and `astro-ph.EP` (Earth and Planetary) has an area under the ROC curve of about 0.7, the rest of the ROC curves have areas above 0.90. One can visualize the topics by seeing the most frequent words in each topic in Table .

```latex
\resizebox{\textwidth}{!}{%
\begin{tabular}{ | c | c |c | c |c | c |c | c |c | c |c | }
  \hline
    CO & \thead{model \\ 0.018} & \thead{galaxi \\  0.016} & \thead{redshift \\  0.014} & \thead{cluster \\  0.012} & \thead{measur \\  0.012} & \thead{cosmolog \\  0.011} & \thead{scale \\  0.011} & \thead{dark \\  0.010} & \thead{paramet \\  0.010} & \thead{data \\  0.009} \\
 \hline
 GA & \thead{star \\ 0.028} & \thead{galaxi \\  0.024} & \thead{mass \\  0.021} & \thead{ga \\  0.014} & \thead{stellar \\  0.012} & \thead{format \\  0.011} & \thead{line \\  0.010} & \thead{disk \\  0.009} & \thead{observ \\  0.009} & \thead{dust \\  0.009}  \\
 \hline
 IM & \thead{data \\ 0.016} & \thead{telescop \\  0.015} & \thead{imag \\  0.015} & \thead{observ \\  0.010} & \thead{instrument \\  0.008} & \thead{high \\  0.008} & \thead{optic \\  0.007} & \thead{present \\  0.007} & \thead{base \\  0.007} & \thead{survey \\  0.007}  \\
 \hline
 EP & \thead{planet \\ 0.027} & \thead{star \\  0.021} & \thead{orbit \\  0.020} & \thead{period \\  0.014} & \thead{mass \\  0.011} & \thead{observ \\  0.011} & \thead{binari \\  0.011} & \thead{model \\  0.008} & \thead{transit \\  0.008} & \thead{earth \\  0.007}  \\
 \hline
 HE & \thead{ray \\ 0.023} & \thead{emiss \\  0.019} & \thead{observ \\  0.018} & \thead{energi \\  0.016} & \thead{sourc \\  0.016} & \thead{gamma \\  0.016} & \thead{x-ray \\  0.015} & \thead{radio \\  0.014} & \thead{time \\  0.011} & \thead{detect \\  0.010}  \\
 \hline
 SR & \thead{magnet \\ 0.023} & \thead{field \\  0.019} & \thead{solar \\  0.013} & \thead{model \\  0.013} & \thead{observ \\  0.011} & \thead{simul \\  0.008} & \thead{region \\  0.008} & \thead{flare \\  0.007} & \thead{rotat \\  0.007} & \thead{structur \\  0.007} \\
 \hline
    \end{tabular}}
```

We conclude from this exercise that the division into subcategories is slightly less sharp than the division of the whole Arxiv into categories, as the subcategories themselves might share more in common and have higher correlation between themselves. In any case, as LDA is a mixed-membership model, it is designed to work better when the documents are composed of several topics simultaneously, otherwise for a single topic per document one could resort to simpler clustering algorithms. This property of LDA, along with the fact that some categories are not subdivided motivates us to use this topic modeling to explore the underlying structure of the papers within these categories.

Taking for example `hep-ph`, we can run LDA with 40 topics, and then look at the topics themselves to see if they have interpretability. We select and show some of those topics in Table . We can see how the model has captured distinct topics that are readily interpretable. Here we included four different ones and label them according to the subjects they are about.

```latex
\resizebox{\textwidth}{!}{%
\begin{tabular}{ | c | c |c | c |c | c |c | c |c | c | c | }
  \hline
 lhc & \thead{lhc \\ 0.066} & \thead{search \\  0.043} & \thead{collid \\  0.042} & \thead{tev \\  0.033} & \thead{model \\  0.018} & \thead{energi \\  0.017} & \thead{decay \\  0.016} & \thead{mass \\  0.016} & \thead{signal \\  0.016} & \thead{hadron \\  0.015} \\
 \hline
 higgs & \thead{higg \\ 0.136} & \thead{model \\  0.058} & \thead{boson \\  0.058} & \thead{coupl \\  0.033} & \thead{standard \\  0.028} & \thead{scalar \\  0.027} & \thead{electroweak \\  0.021} & \thead{sm \\  0.019} & \thead{mass \\  0.017} & \thead{h \\  0.017} \\
 \hline
 lattice qcd & \thead{quark \\ 0.166} & \thead{qcd \\  0.064} & \thead{mass \\  0.055} & \thead{lattic \\  0.051} & \thead{gluon \\  0.029} & \thead{heavi \\  0.024} & \thead{light \\  0.023} & \thead{result \\  0.021} & \thead{hadron \\  0.019} & \thead{flavor \\  0.018} \\
 \hline
 dark matter & \thead{dark \\ 0.156} & \thead{matter \\  0.148} & \thead{dm \\  0.026} & \thead{particl \\  0.024} & \thead{model \\  0.023} & \thead{interact \\  0.019} & \thead{detect \\  0.016} & \thead{annihil \\  0.015} & \thead{direct \\  0.015} & \thead{relic \\  0.014} \\
 \hline
    \end{tabular}}
```

As a remark on the LDA Topics Model power, it is interesting to notice that, for instance, among the 40 topics in `hep-ph` there are four that have the word "qcd" among the 10 most likely ones. Looking at the rest of the accompanying words, these four topics can be interpreted as corresponding to "lattice qcd", "precision qcd", "qcd effective theories" and "qcd phase transition", as it can be seen in Table . Similar conclusions are found with other keywords in this and in other categories.

```latex
\resizebox{\textwidth}{!}{%
\begin{tabular}{ | c | c |c | c |c | c |c | c |c | c | c | }
  \hline
 lattice & \thead{ quark \\ 0.166} & \thead{ qcd \\   0.064} & \thead{ mass \\   0.055} & \thead{ lattic \\   0.051} & \thead{ gluon \\   0.029} & \thead{ heavi \\   0.024} & \thead{ light \\   0.023} & \thead{ result \\   0.021} & \thead{ hadron \\   0.019} & \thead{ flavor \\   0.018} \\
 \hline
  precision & \thead{ jet \\  0.046} & \thead{ order \\   0.040} & \thead{ correct \\   0.039} & \thead{ lead \\   0.036} & \thead{ qcd \\   0.033} & \thead{ nlo \\   0.020} & \thead{ calcul \\   0.018} & \thead{ parton \\   0.017} & \thead{ soft \\   0.017} & \thead{ result \\   0.017} \\
 \hline
  effective theory & \thead{ meson \\  0.101} & \thead{ vector \\   0.067} & \thead{ eta \\   0.061} & \thead{ rule \\   0.048} & \thead{ sum \\   0.043} & \thead{ rho \\   0.041} & \thead{ qcd \\   0.035} & \thead{ scalar \\   0.025} & \thead{ light \\   0.025} & \thead{ pseudoscalar \\   0.024} \\
 \hline
  phase transition & \thead{ phase \\  0.063} & \thead{ temperatur \\   0.054} & \thead{ transit \\   0.038} & \thead{ model \\   0.029} & \thead{ potenti \\   0.026} & \thead{ finit \\   0.025} & \thead{ qcd \\   0.022} & \thead{ critic \\   0.021} & \thead{ chemic \\   0.019} & \thead{ thermal \\   0.014} \\
 \hline
\end{tabular}}
```

We then conclude that Topics Model can help provide a way to sort through documents in a deeper way than simply looking at individual keywords. As topics involve several words at the same time, the model can capture co-occurrence of keywords within topics, which is important for sorting papers according to personal preference in this space.

As mentioned in section , we consider different metrics to measure the goodness of the LDA Topics Model, in a sense allowing us to perform a scan over the hyperparameters and tune them to their optimal values. In particular, we plot both perplexity and topic coherence in Fig. . Both plots have been computed using the `CoherenceModel.get_coherence()` and `log_perplexity()` functions from the `Gensim` Package , respectively. As it can be seen from these results, specially from the perplexity plot, a good model convergence requires above $\sim 30$ topics and above $\sim 50$ passes and iterations. In our LDA models we have used number of topics ranging from 30 to 60 and passes and iterations above 100.

Another way to measure the model for different $N_{topics}$ values is by looking at what we coined as *pizza-plots*, which contain visual information on some of the model features. These plots are defined as follows. A circle centered in the origin is divided into slices and each topic is identified with one of these slices. Each document in the corpus is a point that is located in the slice corresponding to its main topic. The distance to the center of the circle for each point is defined as $\sum_i \left( w^d_i - \frac{1}{N_t} \right)^2$, where $w^d_i$ is the weight of document $d$ in topic $i$. This distance indicates the inhomogeneity of the document in the topics: the closer to the center is the point, the more homogeneous in the topics is the document. For the sake of a better visualization, the angle of each point within each slice is random. Fig.  contains four pizza plots for $N_t=2,\ 4,\ 15$ and $40$ topics. As it can be seen in the plots, as $N_t$ increases one obtains a clearer center (no documents uniformly spread in topics) and a clearer crust (no monotopic documents). One can also see that as the number of topics increases, the documents get more asymmetrically distributed in the topics, as expected in these models.

Using the above results and conclusions, we have constructed four unsupervised LDA Topics Model with each one of the categories `astro-ph, gr-qc, hep-ph` and `hep-th` with number of topics ranging from 30 to 60 and using more than 100 passes and iterations in each construction. These LDA Topics Models provide us with a vector of weights on each topic for each paper in the corpus and allow us to extract a new vector of weights for each new paper in each category. These tools are the key for sorting papers for each user according to next Section discussion.

# IArxiv.org

To allow users to navigate the corpus of papers from `Arxiv.org` according to their preferences we built a website where they can register and view the listings from four of the Arxiv categories: `astro-ph, gr-qc, hep-ph` and `hep-th`. When a user starts browsing the papers we register each user preference by saving which papers the user is accessing. In this section we describe how the site works and what technologies we used to build it.

Once user have followed a standard registration process they can access their user preferences to indicate which categories they are interested in reading. The main page listing allows the user to filter the papers by date and displays a listing of title, abstract and authors. The papers are sorted according to the users preference as follows. Each user has initially a vector (per Arxiv category the user is following) which has null components for each topic. As the user opens the different papers the site records which paper was opened and uses this information to update each user vector as explained below. To sort a set of papers for a given user, the system computes the scalar product of the user vector with each one of the papers in the set and sorts them accordingly to their results, which is a set of real numbers. Users can read sorted papers either through the daily Arxiv release, or can also choose specific slices of days. The latter being very convenient for after a period of not reading the Arxiv.

The website is mobile friendly so that users can easily navigate it using their cellphones and the interface is minimal to reduce clutter and improve usability.

Current users vector computation is performed as follows. We add the paper vector to the user vector for each paper in which the user is author, weighting as more important recent works. We also add a paper vector to the user vector for each click that opens a paper and for each click that expands an abstract. In these last operations we weight as more important opening the PDF than expanding an abstract. The system is designed to give more weight to recent papers as time goes by.

In terms of technology, the website is separated into a frontend web application developed using the ReactJS (`https://reactjs.org/`) JavaScript framework and a backend server written in Golang (`https://golang.org/`). All the information is stored on a PostgreSQL database that is accessed through the Golang backend. On a nightly basis a Python process reads all the new papers from the Arxiv and adds them into the database. The process computes the paper vectors. The database model is relatively straightforward: it has several tables to store users, papers, authors, paper vectors and user vectors. The information is normalized to reduce redundancy. Both backend and frontend are deployed using Docker (`https://www.docker.com/`) into an EC2 instance in Amazon Web Services. This infrastructure will allow us to scale easily if we have an increase of traffic.

# Discussion and future prospects

`IArxiv.org` is currently running and providing sorted daily Arxiv papers according to each user preference, as described above. However, having built this new tool using Machine Learning techniques over the Arxiv opens a new landscape of possibilities that could still improve the interaction between scientists and bibliography, and scientists among them, beyond the expected.

In a first phase for future upgrades we plan the following add-ons. We will implement an optional daily IArxiv e-mail, in which users receive an e-mail with the papers sorted according to their preferences. This e-mail should arrive as soon as the papers are available at Arxiv. We also plan to include in the website a system that facilitates users to read many categories simultaneously meanwhile controlling the amount of papers selected in the release. This will consist in an optional system of weighting and thresholds to be applied to each category.

Once the above goals are accomplished and implemented, we propose the following objectives, lines of research and work.

A direct and simple improvement to be performed at `IArxiv.org` is to extend its scope to the remaining Arxiv categories. This duty, however, requires a set of scientists in these categories to test its correct functioning during a reasonable period of time before opening it for the general public.

It is also planned to go beyond the LDA technique in the sorting of papers. A compelling possibility is to construct an algorithm that recognizes authors affinity. Bringing in this way a slight tunable increase to the score of papers in which –for instance– some of the authors have been previous co-authors of the IArxiv user. Other tools to be studied and developed may also assist to recognize authors affinity.

There is also an interesting possible implementation in which `IArxiv.org` can assist to find related bibliography to a given paper. Within the LDA framework this is easily done by finding those papers in the corpus whose vectors have a large inner product with the desired paper. We are currently testing this technique and considering implementing additional clustering techniques to provide a better outcome of this product. Once this tool is consolidated, we will also implement an original variation consisting in the following: users can provide a new (not necessarily existing) title and abstract, and the system will return a set of papers in the corpus which are related by different mechanisms of clustering.

There are many other features which could be developed as byproducts of the above and/or from new tools. As for instance, given two users who would be interested in starting a collaboration, the `IArxiv.org` system could provide –as a guideline– a set of papers in the corpus in which each paper contains simultaneously a large fraction of the topics preferred by the users. Analogously, users could wonder in which affiliations they may find best affinity to their preferred topics. Another potential interesting use is for seminars and workshops, where speakers may share their IArxiv topics vector and the scientists planning to attend may have a prior knowledge of their affinity to the speaker by requiring the scalar product of their vectors to the speaker vector through `IArxiv.org`.

As part of `IArxiv.org` maintenance we plan to rebuild the LDA models periodically in order to include new lines of research and experimental updates. In such cases we can rebuild all users vectors by re-applying their preferences within the new model topics.

# Conclusions

We have studied the application of Topics Model to the corpus of Arxiv papers through the Latent Dirichlet Allocation unsupervised Machine Learning technique. This has converted each paper into a topics weight vector which can be used to classify papers according to their distribution in topics. As a byproduct of this study we have created the website `IArxiv.org` which allows users to read daily Arxiv papers sorted according to their topics preference.

We have studied the Arxiv categories `astro-ph, gr-qc, hep-ph` and `hep-th` through their title and abstracts. In a first analysis we have merged all categories into one corpus to run a LDA Model with four topics on it. We have observed that this technique produces an excellent *unsupervised* separation of the four *defined* topics. Motivated by this result, in a following step we have studied each category by separated. We have analyzed variables as perplexity and coherence and determined that construction of good LDA models using `Gensim` package would require number of topics ranging between $\sim$ 30–60, and no less than $\sim$`<!-- -->`{=html}50–100 passes and iterations. We have also defined new plots to visualize the document distribution along the topics and verified that this number of topics provides a solid Topics Model for the studied corpora.

To create the website `IArxiv.org` we provide each user with a new null vector of topics and we add to it weighted vectors of papers in which the user is author and papers that the user clicks to read in IArxiv web interface. We have designed the system to provide more weight to more recent preferred papers. This user vector contains therefore the user distribution of interest along the topics that describe the corpus. To sort a daily Arxiv release of papers for a specific user, we compute the scalar product of the user vector with all the vectors of the papers in the release and sort them according to the outcome of this product.

We have discussed future prospects for the `IArxiv.org` system. In a next update we will provide the option of receiving a daily e-mail and introduce weight and threshold tools to facilitate users to read many categories simultaneously. We also plan to create tools which would allow users to find relevant bibliography within the corpus of papers according to each specific topic preference. We propose to study different Machine Learning techniques beyond LDA to further improve the accuracy of this tool, among other possible upgrades.

This novel system may find different applications in the scientific community. As for instance scientific journals may find `IArxiv.org` useful to classify and distribute new papers to referees, if the latter have their user vector defined.

This new Machine Learning tool on the Arxiv provides an outstanding landscape of new tools and services for scientists. We are open to new ideas and to scientists and institutions wishing to collaborate and/or partner in constructing a new bibliography engine for the benefit of everybody.

# Acknowledgments

We thank and congratulate `Arxiv.org` for the outstanding duty of providing an excellent bibliography server and for sharing its content to Bulk Data Access through `export.arxiv.org`. We thank D. de Florian for very valuable contributions and discussions on this project since it started. We thank R. Tito D'Agnolo, J. Thaler, A. Szynkman, E. Roulet, J. Kamenik, B. Dillon and D. Faroughy for fruitful conversations about the subject of this work. We thank all colleagues who have used the `IArxiv.org` website and provided crucial feedback for improving it. We thank and congratulate A. Cerqueiras for scripting IArxiv's front- and back-end. This project was conceived in Workshop Voyages Beyond the SM III.