A Survey of Computational Tools in Solar Physics
Monica G. Bobra Stuart J. Mumford Russell J. Hewett Steven D. Christe Kevin Reardon Sabrina Savage Jack Ireland Tiago M. D. Pereira Bin Chen David Pérez-Suárez
astro-ph.IM physics.ed-ph






addressref=aff1]M.G.Monica G. Bobra0000-0002-5662-9604
addressref=aff2,aff3,corref,email=stuart.mumford@sheffield.ac.uk, ]S.Stuart Mumford0000-0003-4217-4642
addressref=aff6]R.Russell J. Hewett0000-0001-8944-4705
addressref=aff4]S.D.Steven D. Christe0000-0001-6127-795X
addressref=aff7]K.Kevin Reardon0000-0001-8016-0001
addressref=aff8]S.Sabrina Savage0000-0002-6172-0517
addressref=aff4]J.Jack Ireland0000-0002-2019-8881
addressref=aff9,aff10]T.M.D.Tiago M. D. Pereira0000-0003-4747-4329
addressref=aff11]B.Bin Chen0000-0002-0660-3350
addressref=aff5]D.David Pérez-Suárez0000-0003-0784-6909

[id=aff1]W.W. Hansen Experimental Physics Laboratory, Stanford University, Stanford, CA 94305, USA
[id=aff2]SP^2RC, School of Mathematics and Statistics, The University of Sheffield, Sheffield, UK
[id=aff3]Aperio Software Ltd., Headingley Enterprise and Arts Centre, Bennett Road, Leeds, UK
[id=aff6]Department of Mathematics, Virginia Polytechnic Institute and State University, Blacksburg, VA 24061-0123, USA
[id=aff4]NASA Goddard Space Flight Center, Greenbelt, MD 20771, USA
[id=aff7]National Solar Observatory, Boulder, CO 80303, USA
[id=aff8]NASA Marshall Space Flight Center, Huntsville, AL 35812, USA
[id=aff9]Institute of Theoretical Astrophysics, University of Oslo, Oslo, Norway
[id=aff10]Rosseland Centre for Solar Physics, University of Oslo, Oslo, Norway
[id=aff11]Center for Solar-Terrestrial Research, New Jersey Institute of Technology, Newark, NJ 07102-1982, USA
[id=aff5]University College London, London, UK

Bobra M.G. et al.
A Survey of Computational Tools in Solar Physics


The SunPy Project developed a 13-question survey to understand the software and hardware usage of the solar physics community. 364 members of the solar physics community, across 35 countries, responded to our survey. We found that 99±0.5% of respondents use software in their research and 66% use the Python scientific software stack. Students are twice as likely as faculty, staff scientists, and researchers to use Python rather than Interactive Data Language (IDL). In this respect, the astrophysics and solar physics communities differ widely: 78% of solar physics faculty, staff scientists, and researchers in our sample uses IDL, compared with 44% of astrophysics faculty and scientists sampled by <cit.>. 63±4% of respondents have not taken any computer-science courses at an undergraduate or graduate level. We also found that most respondents utilize consumer hardware to run software for solar-physics research. Although 82% of respondents work with data from space-based or ground-based missions, some of which (e.g. the Solar Dynamics Observatory and Daniel K. Inouye Solar Telescope) produce terabytes of data a day, 14% use a regional or national cluster, 5% use a commercial cloud provider, and 29% use exclusively a laptop or desktop. Finally, we found that 73±4% of respondents cite scientific software in their research, although only 42±3% do so routinely.








§ INTRODUCTION

 
The SunPy Project <cit.> facilitates and promotes the use and development of community-led, free, and open source[According to the Open Source Initiative, stewards of the Open Source Definition (available at ), open source software consists of source code under an open source license. Open source licenses “allow modifications and derived works, and must allow them to be distributed under the same terms as the license of the original software.” In addition, open source software must not discriminate against persons, groups, or fields and the associated licenses must be non-specific, non-restrictive, and technology-neutral.] data-analysis software for solar physics based on the scientific Python environment. To better understand the software and hardware preferences of the solar-physics community, the Project developed a 13-question survey (reproduced in Appendix <ref>) and disseminated it internationally[The UK Solar Physics, European Physical Society's Solar Physics Division, American Astronomical Society's Solar Physics Division, the solar-physics subdivision of the Astronomical Society of Japan, the Astronomical Society of India, and the Brazilian Astronomical Society organizations advertised the survey to their members. The SunPy Project also advertised the survey on the  Twitter account and sent it to the  and  e-mail lists, both of which are public.] over a six-month period between 7 February 2019 and 28 July 2019. 

Many of the survey questions were similar (and in some cases, identical) to those posed by <cit.> in an informal survey of 1142 members of the astrophysics community. The SunPy Project did this deliberately to compare software preferences between the solar and astrophysics communities.

This article presents the survey results, derived from analyzing 364 responses from community members across 35 countries.
All of the survey responses, along with the code <cit.> to analyze these data and produce the figures in this article, are publicly available at .



§ DEMOGRAPHICS

 

Since the SunPy Project relies largely on volunteer efforts, we chose to construct and disseminate this survey ourselves (instead of going through a formal channel such as the Statistical Research Center at the American Institute of Physics). As a result, we recognize that this survey may suffer from coverage error.

Our survey garnered 368 responses. Most of the survey respondents fit into one of four career stages: 56% (n=205) described themselves as a faculty member, staff scientist, or researcher, 15% (n=53) as a postdoc, 23% (n=84) as an undergraduate or graduate student, and 6% (n=22) as a software or instrument developer. This adds up to n=364. Four respondents did not fit into any career stage, and we dropped their responses from our analysis.

Community members across 35 countries[See the analysis code, available at , for a full list.] responded to our survey. About three-quarters of the respondents came from the US, UK, Germany, India, and Japan. Together, these five countries include about 1150 solar physicists[The Solar Physics Division of the American Astronomical Society includes 521 members (private communication, S. Savage, 28 January 2020). The UK Solar Physics community estimates “over 150 scientists" on its website, , as of 27 January 2020. The European Solar Physics Division counts 222 members (private communication, T.M.D. Pereira, 28 January 2020). The Astronomical Society of India includes approximately 100 solar physicists (private communication, D. Banerjee, 30 January 2020). The communication newsletter for the solar-physics subdivision of the Astronomical Society of Japan, called Renraku-kai, counts about 150 subscribers (private communication, K. Hayashi, 27 January 2020).]; therefore, our survey sampled roughly a quarter of the solar-physics community. Our results are based on the assumption that our sample is representative of the solar-physics community overall.

We asked respondents to identify all of the areas of research relevant to their career. Most respondents identified multiple sub-disciplines of expertise. We found that 76% (n=275) work with space-based observational data, 46% (n=169) work with ground-based observational data, and 26% (n=93) work on building instruments. A vast majority of respondents, 82%, work with ground-based or space-based data. 29% (n=105) identified theory as a relevant sub-discipline, and 47% (n=171) identified numerical simulations. 
 
Most of the survey respondents (82%) chose to answer an optional question about whether they self-identified as an underrepresented minority; 16% of this subset (13% of the total sample) said yes. 79% of respondents chose to answer another optional question about whether they self-identified as a underrepresented gender identity; 11% of this subset (9% of the total sample) said yes.



§ SOFTWARE TOOLS



In our survey of the solar-physics community, we found that 99±0.5% of respondents use software in their research[While three respondents did apparently indicate that they do not use software in their research, their further answers on the survey about software package usage suggest that those might have been erroneous responses.]. In a survey of the astrophysics community, <cit.> found that 100% of respondents use software in their research.

We asked users to list all of the scientific software tools, including programming languages, software development tools, and data-analysis frameworks, that they utilized within the last year. We summarized their responses in Figure <ref>. We found that 66% of respondents use the Python scientific software stack and 73% use IDL[Where relevant, we supplied our counting error for non-demographic software and hardware related questions (Questions 6 – 12). For Question 6, we report √(3)/364, or 0.5%, as the percentage error in the number of no responses. Since this question required respondents to pick one response from a binary choice, we apply that same uncertainty to the yes responses. For Questions 7, 8, 10, and 11, which required respondents to pick only one response from a list of options, we quantified the percent error in each response simply by applying the square-root rule for counting experiments <cit.>. For Questions 9 and 12, which allowed respondents to select as many options as they liked, we do not calculate a percent error.]. Overall, respondents listed 42 different software tools and the average respondent used five tools in the past year.

We observe a stark contrast in usage between the two primary data-analysis languages in solar-physics research, Python and IDL, when viewed  by respondent career stage.  The earlier the career stage, the greater the percentage of Python users: 59% of faculty, staff scientists, and researchers, 75% of postdocs, and 79% of students use Python. The earlier the career stage, the fewer IDL users: 78% of faculty, staff scientists, and researchers, 75% of postdocs, and 60% of students use IDL. 

Of course, these tools are not necessarily used in isolation – about half (45%) of respondents use both Python and IDL. Figure <ref> shows that 28% of respondents use IDL exclusively (in other words, they use IDL and do not use Python), while 21% use Python exclusively. The ratio of exclusive IDL users to exclusive Python users is roughly 2:1 for faculty, staff, and research scientists and the opposite, 1:2, for students.

Figure 10 of <cit.> shows that Python is not only the most popular programming language within their sample of the astrophysics community, but it is also the most popular within every individual career category. Our survey results show that Python is the most popular programming language only among students; IDL and Python are at parity for postdocs, and IDL is more popular than Python for faculty, staff scientists, researchers, software developers, and instrument developers. In this respect, the astrophysics and solar-physics communities differ widely: 78% of solar-physics faculty, staff scientists, and researchers in our sample use IDL[The use of IDL by the solar-physics community may be explained partly by how instrument teams provide their data.  Many instrument teams provide data that have been calibrated to a low level, plus software that allows the data to be further calibrated for scientific use.  The advantage of this model of scientific-data provision is that as knowledge of the instrument improves over time, the software can be updated to provide better high-level science-ready data products.  A side-effect of this model of scientific-data provision is that scientific use of the data requires use of a particular package/language. Since many instrument teams chose to take advantage of the significant functionality provided by the SolarSoftWare (SSW: ) package, much of the software required to create higher-level data products is written in the primary language of SSW: IDL.  Hence the model of scientific-data provision may explain why IDL is used by a significant proportion of respondents.], compared with 44% of astrophysics faculty and scientists sampled by <cit.>.

The two groups of respondents share the same statistics, however, when it comes to writing software. In both the astrophysics and solar-physics communities, roughly a third of respondents write their own software most of the time (see Figure <ref> of this article and Figure 3 of ). Furthermore, about 90% of respondents in both communities often or occasionally write their own software (see the same figures).









§ EDUCATION AND TRAINING

 
Although 99±0.5% of respondents use software in their research and 91±5% often or occasionally write their own software, 63±4% of respondents have not had any formal training (e.g. computer-science courses) at an undergraduate or graduate level. We found that people who write mostly their own software are no better trained than everyone else: 44±6% of people who write their own software reported “a lot (e.g. computer science courses)" of formal training, compared with 37±3% overall. We also found that students today are twice as likely to have a lot of formal training in programming compared with faculty, researchers, and staff scientists (see Figure <ref>). The amount of training does not vary with area of expertise; each sub-discipline shows roughly the same amount of formal training as the general population (37±3%).





§ HARDWARE TOOLS

 
We also found that most respondents utilize consumer hardware to run software for solar-physics research. Although 82% of respondents work with space-based or ground-based data, and some of these missions (e.g. the Solar Dynamics Observatory and Daniel K. Inouye Solar Telescope) produce terabytes of data per day, 14% use a regional or national cluster[We recognize that some countries, such as the United States, require citizenship or permanent residence status to use these clusters.] and 5% use a commercial cloud provider (see Figure <ref>). 29% use exclusively a laptop or desktop. The community puts considerable effort into maintaining clusters and workstations, with 40% of respondents using a shared workstation, 51% using a local cluster, and 96% using a laptop or desktop.

These percentages vary significantly by sub-discipline. A larger percentage of respondents in the numerical simulations and theory sub-disciplines use local clusters (63% and 60%, respectively, compared with 51% overall) and regional or national clusters (26% and 26%, respectively, compared with 14% overall).



 



§ CITING SCIENTIFIC SOFTWARE

 
Figure <ref> shows that 73±4% of respondents cite scientific software in their research, although only 42±3% do so routinely. Roughly a quarter (27±3%) never cite scientific software in their research. When asked why, about half (53±8%) responded that they do not know how to appropriately cite scientific software (see Figure <ref>); we note that only 4±1% of respondents do not think software belongs in citations.







§ DISCUSSION

 

Scientific software is an indispensable component of the modern scientific research workflow <cit.>. Virtually all of the solar-physics community uses software in their research. Based on this fact, we find three of the statistics presented in this article worrisome. First, similar to the astrophysics community[<cit.> found that only 8±1% of the astrophysics community received substantial training; however, their question did not define “a lot" or “a little".], a significant fraction of the solar-physics community (63±4% of respondents) have not taken any computer-science courses at an undergraduate or graduate level. Second, most of the solar-physics community (82% of respondents) works with space-based or ground-based facilities, several of which produce terabyte- or petabyte-sized data sets, and nearly a third of the community (29% of respondents) uses exclusively a laptop or desktop to run software for solar-physics research. It is unclear whether the computing power offered by laptops and desktops limit the type of scientific endeavors in solar-physics. Finally, less than half of the community (42±3% of respondents) routinely cites scientific software in their research.

The United States <cit.> report entitled Software Policy Options for NASA Earth and Space Sciences recognizes the lack of education in software development among scientists. The report recommends initiating and sponsoring “programs to educate and train researchers in open source best practices," suggesting topics such as “export controls, licensing and intellectual property, workflows, and software development.” This includes sponsoring community members to attend conferences about open source software development, such as Python in Astronomy () or Scientific Computing with Python (), take online courses about software development, available on learning platforms such as Coursera () and edX (), join workshops like those led by The Carpentries (), and develop training programs, such as the Large Synoptic Survey Telescope's Data Science Fellowship program (). Our findings in Section <ref> show that the solar physics community could benefit immensely from education and training in open source software.

The Ford Foundation's report, entitled Roads and Bridges: The Unseen Labor Behind Our Digital Infrastructure <cit.>, also suggests “expanding the pool of contributors so that more people, and more types of people, can build and sustain public software together." Increasing the diversity of the talent pool, which is still lacking in the solar-physics community, will help sustain a long-term future for open source software in solar-physics.

However, maximizing the scientific return of large data sets, such as those produced by the Solar Dynamics Observatory and the Daniel K. Inouye Solar Telescope, requires both skill in software development and computational resources. The United States <cit.> report entitled Progress Toward Implementation of the 2013 Decadal Survey for Solar and Space Physics: A Midterm Assessment and the Kavli Foundation series of workshops called Petabytes To Science <cit.> recommend adopting science platforms, which co-locate both data and computational resources required to analyze these data. In this paradigm, users run software in an external computing environment where the data lives, instead of moving the data to a desktop or laptop where the software lives. The astrophysics community already developed several science platforms, such as the ASTRO Data Lab (), run by the NSF’s National Optical-Infrared Astronomy Research Laboratory. We encourage the solar-physics community to fund the development of science platforms so that scientists are not restricted by the computational power of consumer hardware for analyses involving terabytes of data.

Finally, we recognize that software development, and hardware development, takes a vast amount of time. This time is rarely recognized by the academic community, which largely rewards publications. Therefore, we encourage the community to publish scientific software (by submitting articles that describe research software to refereed journals and archiving this software in publicly available digital repositories; see ), cite scientific software (see Appendix <ref> about how to cite scientific software), and count scientific software as a co-equal research artifact when considering career evaluation. This has two benefits: it gives academic credit and career recognition to those who write software and it makes it easier to reproduce studies in solar-physics.

Some of the earliest advocates for scientific reproducibility, <cit.> and <cit.>, suggested that a journal article “about computational science in a scientific publication is not the scholarship itself, it is merely advertising of the scholarship.” The actual scholarship, they argue, is the code and development environment used to generate the results. Preserving these elements of scholarship require tools like version control, which create snapshots of software or data as they change over time. At the moment, less than half the community (44% of respondents) uses version control[We found that 44% of respondents selected the option “Github (or similar)" in Question 9. However, we realize this option is ambiguous. In retrospect, we should have provided “Git, Github, or similar" instead of “Github (or similar)" as an option in Question 9.]. The United States <cit.> report entitled Reproducibility and Replicability in Science recommends that “researchers should convey clear, specific, and complete information about any computational methods and data products that support their published results in order to enable other researchers to repeat the analysis," including the data, study methods, and computational environment.

Scientists make a critical choice when selecting a computational environment, because the quality of our tools informs the quality of our research. A large fraction of the community uses the Python scientific-software stack (66% of respondents). This number will only grow over time, since Python is the most popular programming language among students in the solar-physics community (79% of students who took our survey use Python).

There are a number of reasons why the Python scientific-software stack is growing in prominence both in the solar-physics community and many other scientific disciplines[The number of contributors to the SunPy codebase grew by an average rate of one per month since 2011 <cit.>. According to the 2019 Stack Overflow developer survey, Python is the fastest-growing major programming language today (see ); furthermore, most universities use Python to teach computer science <cit.>.]. Interoperability between many packages for numerical methods, plotting, astronomy, statistics, and computing (e.g. ) allows researchers to write code with relative speed and ease. The rise of more than fifty packages in heliophysics alone (see )  enables interdisciplinary analysis across traditionally isolated fields. The open-development model[An open-development model goes beyond providing open source software, it also includes making project-level decisions in publicly-visible and accessible spaces, such as mailing lists, and inviting input from the user and developer communities <cit.>.], adopted by most of the scientific Python ecosystem, improves the longevity of software since anyone can contribute to the codebase and no single institution or person controls the software.

For these reasons, the United States <cit.> report entitled Software Policy Options for NASA Earth and Space Sciences recommends that the “NASA Science Mission Directorate should explicitly recognize the scientific value of open source software and incentivize its development and support, with the goal that open source science software becomes routine scientific practice.” As the SunPy Advisory Board, we endorse this recommendation not only for the NASA Science Mission Directorate but for scientific funding agencies worldwide. 


We would like to thank Ivelina Momcheva, Erik Tollerud, and Nabil Freij for their help and guidance on this project, the numerous professional astronomy societies who helped publicize this survey, and everyone who took this survey.




§ DISCLOSURE OF POTENTIAL CONFLICTS OF INTEREST
 The authors, all members of the SunPy Advisory Board serving in a volunteer, unpaid, capacity, declare that they have no conflicts of interest.




§ SURVEY QUESTIONS


The full contents of the survey, distributed as a Google Form, appear below. All the responses to the Question 13, an optional question which solicited general, free-form comments, are publicly available at .



  * Which of these areas of solar physics do you work in? Check all that apply.
    
    
  □ Observational (Space-Based)
    
  □ Observational (Ground-Based)
    
  □ Numerical Simulations
    
  □ Theory
    
  □ Instrumentation
    
 

  * How would you describe the stage of your career?
    
    
   Undergraduate student
    
   Graduate student
    
   Postdoc
    
   Faculty, Staff Scientist, Researcher
    
   Software Developer
    
   Instrument Developer
    
   Retired
    
   My role is something other than solar physics or software development
    
   Other (Respondents can enter their own description.)
    


  * What country is your institution in? (Respondents check appropriate country from a list of options.)


  * Do you self-identify as one or more underrepresented minorities in solar physics? This question is optional.
    
    
   Yes
    
   No
    


  * Do you self-identify as a unrepresented gender identity in Solar Physics? This question is optional.
    
    
   Yes
    
   No
    


  * Do you use software in your research?
    
    
   Yes
    
   No
    


  * Have you had formal training in programming?
    
    
   Yes, a lot (e.g. CS courses at an undergraduate or graduate level)
    
   Yes, a little (e.g. online classes, books, workshops)
    
   No
    


  * Which of the following statements is most applicable to you?
    
    
   I write mostly my own software.
    
   I mostly use software written by others.
    
   Somewhere in between.
    


  * Which of the following have you personally utilized in your work within the last year? Check all that apply.
    
    
  □ IDL
    
  □ SolarSoft
    
  □ Python
    
  □ SunPy
    
  □ Shell Scripting
    
  □ C
    
  □ C++
    
  □ Fortran
    
  □ IRAF
    
  □ Perl
    
  □ Javascript
    
  □ Julia
    
  □ MATLAB
    
  □ Java
    
  □ R
    
  □ SQL
    
  □ Ruby
    
  □ HTML/CSS
    
  □ Spreadsheets (e.g. Excel)
    
  □ Mathematica
    
  □ MPI
    
  □ Github (or similar)
    
  □ Other (Respondents can enter their own description.)
    


  * Have you cited software papers in your published research?
    
    
   Yes
    
   Sometimes
    
   No
    


  * If ‘No’ for the previous question: Why haven’t you cited software in your research?
    
    
   I am not sure how to appropriately cite software
    
   I do not think it is necessary
    
   I do not think software belongs in citations
    


  * On which of these have you run software for solar-physics research?
    
    
  □ Laptop / Desktop computer
    
  □ Shared workstation
    
  □ Local Cluster
    
  □ Regional or National Cluster
    
  □ GPU
    
  □ Commercial cloud
    


  * Do you have any comments? (This is a free form response; comments are not required. Please feel free to give us feedback about topics like: version control, collaborative coding platforms such as Github, standard or best practices in coding, operating systems, text editors, or your personal experience with writing code and releasing software, or general thoughts about SunPy).




§ CITING SCIENTIFIC SOFTWARE


To cite scientific software, please follow these two steps:



  * Cite the refereed journal article describing the research software. To find this article, visit the website for a software package and look for citation instructions. For example, the SunPy website includes dedicated citation instructions and an associated BibTex entry.


  * Cite the software archive. Publicly available digital repositories, such as Zenodo, issue a Digital Object Identifier (DOI) for archived software. (Some institutions also provide digital repositories as part of their library system.) Generally, open source software projects in the Python scientific stack will archive their software every time that they release a new version. For example, the SunPy Github page includes the Zenodo DOI for the most recent release (as of this writing, v1.1.1); clicking on it leads to the Zenodo deposit, which provides an associated BibTex entry.


Many projects release multiple versions of software per year, but they only write a refereed journal article once in a while (for example, the SunPy Project published an article about the v1.0 release, but they will not publish an article about the v1.1.1 release). Therefore, creating reproducible results requires citing both the journal article and the software archive. Here is an example: This research used version 1.1.1 <cit.> of the SunPy open source software package <cit.>.



spr-mp-sola
  

 
