abstract: Due to the ever-increasing number of security breaches, practitioners are motivated to produce more secure software. In the United States, the White House Office released a memorandum on Executive Order (EO) 14028 that mandates organizations provide self-attestation of the use of secure software development practices. The OpenSSF Scorecard project allows practitioners to measure the use of software security practices automatically. However, little research has been done to determine whether the use of security practices improves package security, particularly which security practices have the biggest impact on security outcomes. ***The goal of this study is to assist practitioners and researchers in making informed decisions on which security practices to adopt through the development of models between software security practice scores and security vulnerability counts.***
  .
  To that end, we developed five supervised machine learning models for npm and PyPI packages using the OpenSSF Scorecard security practices scores and aggregate security scores as predictors and the number of externally-reported vulnerabilities as a target variable. Our models found that four security practices (*Maintained, Code Review, Branch Protection, and Security Policy)* were the most important practices influencing vulnerability count. However, we had low $R^2$ (ranging from 9% to 12%) when we tested the models to predict vulnerability counts. Additionally, we observed that the number of reported vulnerabilities increased rather than reduced as the aggregate security score of the packages increased. Both findings indicate that additional factors may influence the package vulnerability count. Other factors, such as the scarcity of vulnerability data, time to implicate security practices vs. time to detect vulnerabilities, and the need for more adequate scripts to detect security practices, may impede the data-driven studies to indicate that a practice can aid in the reduction of externally-reported vulnerabilities. We suggest that vulnerability count and security score data be refined such that these measures may be used to provide actionable guidance on security practices.
author:
bibliography: arxiv.bib
title: Do Software Security Practices Yield Fewer Vulnerabilities?

# Introduction

A 2022 report from Synopsys  assessed the reliance of the software industry on open-source software (OSS) and estimated that 97% of applications use OSS. The continuous reliance on OSS comes with a risk of supply chain attack. Sonatype has recorded an average 700% jump in supply chain attacks , as measured by the number of newly-published malicious packages in open-source repositories. Software developers largely did not anticipate how the software supply chain would become a deliberate attack vector. Many stakeholders have started to recognize this urgent concern; most prominently, Executive Order 14028 , issued May 12, 2021, calls for identifying practices that enhance software security and mandating the use of software security practices.

As organizations seek to address escalating security risks and comply with regulations, a myriad of activities are available to improve software security. The National Institute of Standards and Technology (NIST) issued guidance  on software development practices that enhance the security of the software supply chain. The Open Source Security Foundation (OpenSSF), a cross-industry organization hosted at the Linux Foundation, provides the vehicle for collaboration on tools, services, training, infrastructure, and resources for securing open-source projects. The OpenSSF Scorecard  project computes automated scores of package's security practices to help developers make better decisions about security when consuming open source projects.

While NIST's guidelines and OpenSSF projects provide guidelines and comprehensive lists of security practices, challenges arise in validating whether these practices improve security. Organizations may not have the resources to adopt a full suite of these practices. They would like to understand the key drivers of success, especially which of many possible software security practices to undertake first. A data-driven study on the relationship between the use of software security practices and vulnerability count could aid in this understanding. ***The goal of this study is to assist practitioners and researchers in making informed decisions on which security practices to adopt through the development of models between software security practice scores and security vulnerability counts.***

To that end, this study is focused on the relationship between publicly-available data of security practice use and externally-reported vulnerability count. This relationship may be used to provide actionable recommendations to practitioners on security practices. Our work addresses the following **research questions**:

-   **RQ1 (Security Practices)**: Which Scorecard security practices are most important to understand the relationship between security practices and vulnerability counts in regression models?

-   **RQ2 (Security Outcome)**: Do packages with higher aggregate security scores have fewer vulnerabilities?

The OpenSSF Scorecard project  uses security practice metrics and auto-generates a "security score" for each practice. The tool also computes an aggregate "security score" which is a weighted average of the individual score. In this study, we leverage the security practice score data provided by the OpenSSF Scorecard project and vulnerability information from security advisory databases to evaluate the relationship between security practices and vulnerability count. To answer RQ1, we used the "feature importance" technique in four regression models to understand the importance of each security practice in interpreting the relationship between security practices and vulnerability count. For RQ2, we built our fifth regression model to evaluate the statistical association of a package's aggregate security scores with package vulnerability count and to understand whether implementing security practices assists in secure coding with fewer vulnerabilities.

This study can assist practitioners in making informed choices regarding their security practices, particularly on which practices have the most impact or whether specific security practices have improved security outcomes. This study makes the following contributions:

-   A proposed model to identify important security practices to enhance software security.

-   A proposed model to understand how higher aggregate security scores affect security outcomes.

-   Evaluation of the proposed model and security practices using open source datasets.

-   The list of challenges impeding the model's performance

This paper is organized as follows: Section includes the related concept used in our study, section discussed the step taken by the software industry to improve software supply chain security. Section describes the data collection process, and Section describes the data analysis to measure model performance. We close with a discussion and limitations of our findings (Section ).

# Background

In this section, we discuss the "OpenSSF Scorecard" tool and the two most frequently used terms in our study: "Security Outcome" and "Package".

## **OpenSSF Scorecard**

The OpenSSF Scorecard  is an automated tool that runs on source code hosted by GitHub to monitor the security health of the packages. The Scorecard automatically computes a normalized integer score between 0 to 10 for each of the 18 security practices. For each GitHub repository, an individual score for each practice and an aggregate score for all practices are returned by the tool. Each security practice is assigned one of four risk levels: **" Critical"** risk-weight 10; **" High"** risk-weight 7.5; **" Medium"** risk-weight 5; and **" Low"** risk-weight 2.5. The aggregate score is a weight-based average of the individual scores, weighted by risk.

Apart from the scores between 0-10, the Scorecard tool also assigns a score of -1 to the project. The notation -1 indicates that Scorecard could not get conclusive evidence of implementing practices, or perhaps an internal error occurred due to a runtime error in Scorecard. The inconclusive outcome is graded as -1 instead of 0. Since a value of 0 will affect the package's aggregate score, Scorecard assigned a value of -1 to avoid the penalty of failing a check.

## **Security Outcome**

The Security Outcomes represent indications of the failure or achievement of security associated with the software throughout the software's life cycle. Counting vulnerabilities is the most common means of measuring security in software . In this study, we used package vulnerability counts as a security outcome.

## **Package**

A **Package Registry** such as npm and PyPI stores packages, the metadata associated with them, and the configurations needed to install it, as well as keeps track of the versions of packages . The registry offers a way to upload packages and APIs for the client to call packages. Anyone can download and reuse the packages from the registry. A **Package** is a reusable piece of software that can be called from a global registry (npm, PyPI) as a dependency to build another package . Each package may or may not depend on other packages. However, publishing a package in a registry allows consumers to import and reuse these packages.

# Related Work

This section highlights prior related studies, different security guidelines, frameworks, and tools on package security health and security practices. While we only studied the Scorecard tool's security metrics, we further discussed other security frameworks and tools to encourage future research to evaluate which of these recommended security practices help to improve security outcomes. Because many practices are recommended for software security, but none of these guidelines () demonstrated which of these practices improved security and why we should adopt them. Practitioners will comply with these guidelines if data-driven research studies on security practice measurement and feedback loops validate the improvement in security outcomes.

## **Research Studies**

Zahan et al.  conducted a measurement study on the **OpenSSF Scorecard** security practices in the context of the npm and PyPI ecosystems to identify the ecosystem's security practices trends and gaps. The study analyzed Scorecard scores for 767,389 npm packages and 191,158 PyPI packages. The authors observed gaps in both ecosystems in practicing **Code-Review, Maintained, License, Branch-Protection**, and **Security-Policy** practices in the GitHub repository. Then, the rules specified by the Scorecard tool for Dependency-Update-Tool, Fuzzing, CII-Best-Practices, Signed-Releases, and Packaging security checks, were weakly adopted in npm and PyPI due to Scorecard's inherent reliance on other systems. The study revealed that 13 Scorecard metrics could be mapped back to the NIST SSDF framework's  security practices. Although this research showed a gap in both ecosystems to adopt different security practices, the authors did not verify whether adopting these practices would improve the npm and PyPI ecosystems' security outcomes.

On October 18, 2022, Sonatype released the **8th annual State of the Software Supply Chain report** . As part of project quality metrics identification, they built several machine learning classification models on Scorecard security practices to understand how well a model based on security practices could correctly identify projects with known vulnerabilities. Their classification models were based on 12,786 Java projects, and they found **Code-Review, Binary-Artifacts, Pinned Dependencies, Branch Protection** as important practices in their classification model.

Our study focuses on understanding whether good security practices improve security outcomes, whereas Sonatype research built models to identify vulnerable projects using Scorecard data. However, study  showed that not all Scorecard metrics apply to npm and PyPI ecosystems. From the Sonatype annual report, we could not verify whether all the metrics applied to Java projects or whether the study removed noisy metrics from the model's independent variables.

## **Guidelines and Frameworks:**

The **Software Component Verification Standard (SCVS)**  by OWASP, is a framework to develop a common set of activities, controls, and security practices that can help in identifying and reducing risk in a software supply chain. There are $6$ control families that contain $87$ controls for different aspects of security verification or processes. The SCVS has three verification levels, where higher levels include additional controls.

The **Building Security In Maturity Model (BSIMM)** is the result of a multiyear study of real-world software security initiatives (SSIs). Each year, various firms in different industry verticals use the BSIMM to manage their SSI improvements because the BSIMM report provides a clear picture of actual practices used by organizations across the security landscape . The BSIMM is organized as a set of 122 activities, which consist of four domains distributed into 12 practices.

The **CNCF Technical Advisory Group (TAG)**  published a series of recommended best practices, tooling recommendations, and design considerations that can reduce the likelihood and overall impact of a successful supply chain attack. They discuss the $5$ stage of software supply chain security—securing code, materials, building pipelines, artifacts, and deployments. The framework proposed $54$ practices with associated risk factors to provide a holistic, end-to-end guide for organizations and teams.

In response to Section 4 of the President's Executive Order (EO) on "Improving the Nation's Cybersecurity (14028)" , the U.S. National Institute of Standards and Technology (NIST) improvised the **Secure Software Development Framework (SSDF)** . The framework does not introduce new practices or define new terminology. Instead, the framework describes a set of high-level practices based on established standards  of secure software development practices.

```latex
\begin{table*} \renewcommand{\arraystretch}{1.2}
\caption{\centering{Scorecard Security Practices used in ~\cite{Zahan2022} }} \label{tab:scorecards}
\centering
\begin{tabular}{| p{10pt} || p{95pt} || p{280pt} ||p{40pt}| }\hline
No. & Security Practices & Description & Risk Label \\
\hline\hline
%
1& Dangerous-Workflow & This check determines dangerous code patterns in the project's GitHub Action workflows. These patterns include logging GitHub context and secrets, untrusted code checkouts, or script injection with untrusted context variables.  & Critical\\\hline

2& Branch-Protection & This check determines whether a project's branches are protected with GitHub's branch protection settings. This check has five tiers of scoring. Each tier must be fully satisfied to acquire points at the next tier.  & High\\\hline

3& Code-Review & This check determines whether the project practices code review before merging pull requests. The check first detects whether Branch-Protection is enabled with at least one required reviewer. If this fails, the check determines whether the most recent commits are Github-approved reviews or whether the merger differs from the committer. Note that this check is not applicable if the package has one maintainer. & High\\\hline

4& Binary-Artifacts &This check determines whether the project has generated executable (binary) artifacts in the source repository since Binary-Artifacts cannot be reviewed, allowing possible obsolete or maliciously subverted executables. &High\\\hline

5& Dependency-Update-Tool & This check determines whether the dependency updates tools like dependabot or renovatebot are enabled or not. A project that uses other tools will receive a low score on this test.  & High\\\hline

6& Maintained & This check determines whether the project is actively maintained. If there is activity on commit and issues from users who are collaborators, members, or owners, the project receives the highest and partial score. & High\\\hline

7& Signed-Releases & This check determines whether the project signs release artifacts, currently limited to GitHub repositories only. This check looks for the following filenames in the project's last five releases: *.minisig, *.asc (pgp), *.sig, *.sign. & High\\\hline

8& Token-Permissions & This check determines whether the project's automated workflow tokens are set to read-only. If the permissions definitions in each workflow's yaml file are set as read-only at the top level, and the required write permissions are declared at the run-level, the project gets the highest score.  If the top-level permissions are not defined, one point is reduced.  &High\\\hline

9& Vulnerabilities & This check determines whether the project has unfixed vulnerabilities using the OSV (Open Source Vulnerabilities) database.  &High\\\hline

10& Pinned-Dependencies &This check determines if the project has pinned its dependencies. The check works by looking for unpinned dependencies in Dockerfiles, shell scripts, and GitHub workflows. & Medium\\\hline

11& Packaging & This check determines if the project is published as a package. The check looks for GitHub packaging workflows and language-specific GitHub Actions that upload the package to a corresponding hub. & Medium\\\hline

12& Fuzzing & This check determines if the project uses fuzzing by checking the repository name in the OSS-Fuzz project list. & Medium\\\hline

%

13& Security-Policy & This check determines if the project has published a security policy  by looking for a file named SECURITY.md. & Medium\\\hline

%

%

14& License & This check determines if the project has published a license by checking standard locations for a file named according to common license conventions. & Low \\\hline

15& CII-Best-Practices & This check determines whether the project has a CII Best Practices Badge to ensure that the project uses a set of security-focused best practices. & Low \\\hline
\end{tabular}
\end{table*}
```

**Supply Chain Levels for Software Artifacts (SLSA)**  is a security framework established by industry consensus, a set of standards and controls to prevent tampering, enhance the integrity, and also a set of security rules that may be adopted incrementally. SLSA has four levels, with SLSA 4 representing the ideal end state. The lower levels represent incremental milestones with corresponding incremental integrity guarantees.

In August 2022, the Cybersecurity and Infrastructure Security Agency (CISA) and the National Security Agency (NSA) released a recommended practice guide on **Securing the Software Supply Chain for Developers** . The report advocated specific frameworks like SLSA and SSDF. The framework is a roadmap to building a healthy software development environment that includes a Software Bill of Materials (SBOM) for describing software components, active monitoring for vulnerabilities and software supply chain attacks, and a secure build environment and development team.

Microsoft released the **Open Source Software (OSS) Secure Supply Chain (SSC) Framework** , which is a security assurance and risk reduction process focused on securing how developers consume open source software. The framework provides security guidance and tools throughout the developer's inner-loop and outer-loop processes that have played a critical role in defending and preventing supply chain attacks through the consumption of open-source software across Microsoft. In September 2022, the OpenSSF released the **npm Best Practices Guide** to help JavaScript and TypeScript developers reduce the security risks associated with using open-source dependencies.

## **Tools**

**Open Source Insights (OSI)** or Deps.dev  is a Google-developed and hosted tool to aid practitioners in grabbing information about the source code location, package metadata, licenses, releases, and vulnerabilities of open-source products. OSI scans millions of open-source packages from different ecosystems, constructs dependency graphs, and annotates the metadata in a dashboard. Apart from the package's metadata, the OSI dashboard also shows statistics about a package's direct or transitive dependencies. On the OSI website, users can view the vulnerability mapping of a package as well as the vulnerability mapping with associated dependencies. In addition to the package metadata, OSI has also incorporated Scorecard security practices to help understand package security practices.

**In-toto:** A framework that holistically enforces the integrity of a software supply chain by gathering cryptographically verifiable information about the chain itself . `in-toto` grants the end user the ability to verify the software's supply chain from the project's inception to its deployment. To achieve this, `in-toto` requires a project owner to declare and sign a layout of how the supply chain's steps must be carried out and by whom. The concerned parties will document their actions and produce a cryptographically signed declaration for each step as they are completed. The link metadata recorded from each step can be verified to ensure that each step was completed correctly. Both SLSA  and CNCF Technical Advisory Group (TAG)  recommend in-toto as a framework to secure the integrity of software supply chains.

# Study Design and Data Collection

In this section, we discuss our study design and the data collection process that was used in our research. Section highlights our high-level overview of the research workflow. Section covers how we obtained Scorecard security practices' scores for each package, and Section discusses how we collected vulnerability count data from Open Source Vulnerabilities (OSV) and Snyk databases.

## **Study Design**

To answer our two research questions, we leverage regression models where we used the security scores of package repositories' as independent variables and security outcomes in terms of package vulnerabilities count as the dependent variable. Figure shows an overview of the research workflow.

RQ1 aims to identify the most important practices that explain the relationship between security practices and security outcomes. To that, we utilized the 15 security practice score as independent variables generated by the Scorecard tool. Next, our RQ2 verifies whether packages with higher aggregate security scores have fewer vulnerabilities. Here, we leverage each repository's aggregate security score as an independent variable, which is generated by Scorecard by measuring the weight-based average of the individual security practice's scores, weighted by risk. For both RQ1 and RQ2, we used the security outcomes in terms of vulnerability counts as the dependent variable.

For RQ1, we classified a security practice as an important metric if it is identified as important in more than one model. To that, we analyzed 4 different regression model responses. For RQ2, we developed our fifth regression model to test our hypothesis.

## **Scorecard Security Practices data**

We obtained Scorecard security practices data from  . (see Section ). The data includes the 15 security practices scores and aggregate security score of 767,389 npm packages and 191,158 PyPI packages. The dataset contained mapping between the package name, the GitHub repository, the 15 security practices, and the aggregate security score. Each practice score ranges between $-1$ to $10$. We describe the OpenSSF Scorecard project in Sections . The 15 security practices used in our study are discussed in Table . More details of these checks can be found in the Scorecard Project GitHub repository .

The Scorecard tool has 18 security practices. However, the OpenSSF Scorecard team publishes the security score data for 15 of these practices. The Scorecard team took out CI-Test, SAST, and Contributor practices to scale the weekly job since computing these practices is API intensive.  The security scores for the other three practices can be obtained by running Scorecard from the command line interface (CLI). Our study can be replicated for 15 security practices by collecting weekly data from the Google BigQuery dataset published by the Scorecard team.

```latex
\begin{table*}[!t]\centering  \renewcommand{\arraystretch}{1.2} \caption{Unique vulnerabilities count collected from OSV and Snyk} \label{tab:vul_details}
\footnotesize
\begin{tabular}{|c||c||cc||cc||cc||c|c|} \hline
Ecosystem & Database  & Github\_Package & GitHub\_Vul\_Count & CVE & Non-CVE & Total\_GitHub\_Package & Total\_GitHub\_Vul \\\hline\hline
\multirow{2}{*}{PyPI} & Snyk  & 523 & 1189 & 830 & 359 & \multirow{2}{*}{589} & \multirow{2}{*}{2261} \\
 & OSV  & 352 & 1407 & 1374 & 33 &  &  \\\hline
\multirow{2}{*}{npm} & Snyk  & 1689 & 2838 & 1888 & 950 & \multirow{2}{*}{1833} & \multirow{2}{*}{3655} \\
 & OSV  & 1311 & 1883 & 1559 & 324 &  & \\\hline
\end{tabular}
\end{table*}
```

## **Advisory Data**

This section provides background on the database used in this study and advisory data collection processes to obtain the vulnerability count.

### **Database**

We used the OpenSSF Open Source Vulnerabilities (OSV)  and Snyk.io vulnerability  databases in this study. Both databases have a similar structure in terms of the patch, Common Vulnerabilities and Exposures (CVE), and Common Weakness Enumeration (CWE) information and they track vulnerabilities across multiple ecosystem's databases. In September 2022, we acquired vulnerability data from these two databases for both ecosystems.

**OSV** provides both human and machine-readable data formats to describe vulnerabilities and maps to open-source package versions or commit hashes. OSV is an aggregator of vulnerabilities from different security advisories, e.g., GitHub Security Advisories , PyPI Advisory Database , Go Vulnerability Database , Rust Advisory Database , OSS-Fuzz (mostly C/C++) , and Global Security Database . These databases have adopted the OpenSSF OSV format to facilitate use in tooling and aggregation. **Snyk.io** is another platform that provides security reports containing vulnerability information for different package ecosystems, including npm, PyPI.

### **Vulnerability Data Collection**

As Scorecard only works for Github repositories, our vulnerability collection was limited to packages with GitHub repositories. Hence, each database may have more vulnerabilities than we presented in this study. Table contains the detailed counts of vulnerabilities.

The  data contained 767,389 npm and 191,158 packages with GitHub repositories. We extract the vulnerability data from both databases using the package name. For each package, we collected vulnerability details in terms of CVE, CWE and patch link from both databases. OSV exported data to a Google Cloud Storage (GCS) bucket. The bucket contains a zip of all vulnerabilities JSON file for each ecosystem at `gs://osv-vulnerabilities/<ECOSYSTEM>/all.zip`. However, we had to build a scraper for Snyk to extract data from the Snyk website.

To create the dataset, we counted the number of unique vulnerabilities found for each package. First, we merged both databases and applied our filter conditions. During this stage, we eliminated vulnerabilities if we found i) duplicate package-to-CVE mappings or duplicate patches for the Non-CVE vulnerabilities; ii) Non-CVE vulnerabilities without patch links; iii) Vulnerabilities with Non-CVE, NON-CWE and No Patch link. Table shows the unique count for each row. After filtering and merging both advisory databases, we found 589 PyPI packages with 2261 vulnerabilities and 1,833 npm packages with 3,655 vulnerabilities. In total, we have vulnerability counts of 2,422 packages as a dependent variable.

At this stage, we mapped 2,422 packages with the Zahan et al.  dataset to collect the Scorecard security practice's values. As a result, 2,422 packages with vulnerability count and security score are used as our final dataset, and we did not consider any other packages from  study.

## **Dependents and Downloads Data**

The number of dependents and downloads reflects the importance of a project based on how many other projects use it. To determine whether the packages in our dataset are utilized by other open-source packages, we obtained dependents and downloads data. We collected direct-dependent and transitive-dependent information from the Open Source Insights (OSI) API (Section ). Then, we collected 12 months of download statistics of npm packages from the study  and for PyPI packages from the Google BigQuery public dataset.

# Data Analysis

This section discusses our data analysis to answer our research questions. Sections and discuss pre-processing and regression models to answer our research question.

## **RQ1: Important Security Practices**

In this section, we discuss the analysis approach for RQ1: *Which Scorecard security practices are most important to understand the relationship between security practices and vulnerability counts in regression models?*

### **Data Pre-processing**

In this section, we discuss the data pre-processing of missing, inconsistent, and noisy data found in Scorecard data due to internal and external errors. For example, we eliminated predictors and packages from our dataset if Scorecard generated a score of $-1$. The score $-1$ depicts that the tool could not obtain concrete evidence that practice had been implemented or if an internal error had occurred due to a runtime error. Therefore, we utilized the following exclusion criteria for security practices and packages to acquire consistent security scores.

-   **Practice Exclusion**: We removed security practices if a practice has more than 10% packages with $-1$ score. Signed-releases (91%), Packaging (94%) and Fuzzing (16%) practices had more than 10% packages with a score of $-1$. We also removed "Vulnerability" metrics as a predictor. The metric checks whether a package has open or unfixed vulnerabilities in the OSV database. Since part of our dependent variable was collected from OSV, we removed this metric to avoid introducing bias. Although we did not find any package in our dataset where a package had open vulnerabilities, it may occur in any future packages. To standardize our study for replication, we removed the vulnerability metric from our dataset. At this stage, we removed 4 practices and have 11 practices as predictors.

-   **Package Exclusion**: We found Branch Protection (6%), and Code-Review (0.08%) had less than 10% of packages with a score $-1$. Therefore, we only removed those packages instead of removing the practice itself. At this stage, we removed 150 packages from the dataset.

To protect the quality of the data that can directly affect how well our ML model learns, we are removing these practices. Given that Scorecard will compute the accurate security status of these practices in the future, practitioners can elect them when constructing our model. Conversely, leveraging different vulnerability databases other than OSV would allow us to use vulnerability practice as one of the predictors. However, the lack of vulnerability data compelled us to gather vulnerabilities from several sources.

Then, we used `qq-plot`  to determine outliers in vulnerability count data. Four packages had vulnerability counts (36, 77, 158, 217) far from the reference lines and were considered outliers. Since we will use regression models, extreme outliers will affect the line of regression. Hence, we removed those packages from our dataset as an outlier in the dependent variable. We utilized 2268 packages and 11 security practices in our model.

### **Correlation Analysis**

To evaluate the normality of the dataset, the Henze-Zirkler multivariate normality  test was utilized. Results indicated that the dataset was not normally distributed. We expected non-normality, given that the independent variables were discrete variables. At this stage, we measure the correlation to know the association or the absence of a relationship between two variables in our dataset. With evidence of non-normality, we deployed the Spearman rank correlation  tests.

A value close to +1 and -1 indicated a strong positive or negative correlation, respectively. A correlation coefficient close to 0 indicates a weak correlation between the variables . Since values above $\pm 0.3$ start to indicate a moderate positive (negative) linear relationship , we consider values between $\pm 0.3$ to $\pm 1$ as a limit. Figure shows the heatmap matrix for the Spearman rank correlation test.

The Spearman test showed the strongest correlation between Pinned-Dependencies and Token-Permission(0.84), followed by Code-Review and Maintained (0.51), Branch-Protection showed a moderate correlation with Code-Review (0.50), and Maintained (0.44) and Security Policy (0.3). Negative correlations were observed between Maintained and Pinned-Dependencies (-0.61), Maintained and Token-Permission (-0.59), Code-Review and Pinned-Dependencies (-0.46), and Code-Review and Token-Permissions (-0.46).

Token Permission looks at the permission of the GitHub workflow files, and Pinned Dependencies look for the presence of unpinned dependencies in those files, indicating why they might have a strong positive relationship. Both the Code-Review and Maintained variables showed a moderate positive relationship. In contrast, an increase in Maintained comes with a decrease in Pinned-Dependencies and Token-Permission practices. Token-Permission and Pinned-Dependencies do not require active maintenance directly, which might explain the inverse relationship. However, code review could be part of active maintenance, hence the positive relationship. Branch-Protection enables a formal approval process for code changes and pairs well with Maintained and Code-Review.

### **Feature Selection**

For the selection of best features, we used the `SelectKBest`  library from the $sklearn$, which shows the feature importance based on $K$ highest scores. F-regression was selected as the scoring function, which performs a univariate linear regression test and returns F-statistics and p-values. All of the independent predictors were considered to identify the most important features (K="all"). The SelectKBest analysis identified nine predictors as the most important features: Maintained, Security-Policy, Pinned-Dependencies, Code-Review, Branch Protection, Token-Permission, License, CII-Best-Practices and Dependency-Update-Tool. Each of them had a high F-score and statistically significant p-values (p$<$`<!-- -->`{=html}0.01). A reduced dataset was prepared from the statistical significance measure with the selected components as independent variables and the total vulnerability count as the dependent variable.

```latex
\begin{table*}[h]\centering \renewcommand{\arraystretch}{1.3}
\caption {Model performance and important features} \label{tab:R2}
%
\begin{tabular}{| p{25pt} || p{25pt} |p{30pt} ||p{280pt} |} \hline
\textbf{Models} & \textbf{${R^2}$} & \textbf{Adj. ${R^2}$}  & \textbf{Important Features} (Ranked)\\\hline\hline
OLS & 0.09 & 0.08 & Security Policy, Pinned Dependencies, Maintained \\\hline
DT & 0.11 & 0.10 & Security Policy, Pinned Dependencies, Branch Protection, Maintained, Code Review \\\hline
GB & 0.12 & 0.10 & Code Review, Security Policy, Pinned Dependencies, Branch Protection, Maintained \\\hline
RF & -0.008 & -0.022 & Code Review, Pinned Dependencies, Branch Protection, Maintained, Security Policy \\\hline
\end{tabular}
\end{table*}
```

### **Model Performance:**

We measured the performance of each model with the following metrics:

-   $\mathbf{R^2}$ is the statistical measure representing the proportion of the variance for a dependent variable explained by an independent variable or variables in a regression model. R-squared explains to what extent one variable's variance explains the second variable's variance or "percent of variance explained by the model" . It supports values between 0 and 1, with greater values suggesting more variability explained by the model. A high $R^2$ value suggests good explanatory power but not predictive capability .

-   The **adjusted** $\mathbf{ R^2}$ is a variant of R-squared that accounts for predictors that are not significant in a regression model. By considering the degrees of freedom of the independent variables and the sample population, adjusted $R^2$ demonstrates how well the data fit a curve or line and explains any bias in the $R^2$ value. A model's adjusted R-squared will decrease if insignificant variables are added and will increase when significant variables are added. Adjusted $R^2$ will always be less than or equal to $R^2$.

### **Regression Model**

Four supervised machine learning models were used to evaluate the importance of each feature variable in interpreting the relationship between security practices and vulnerability count. Feature Importance techniques in machine learning models assign the score of input features for a given model based on their importance to predict the output.

We first applied **ordinary least square (OLS)** to the dataset. Then, tree-based regression models were selected due to the non-normal distribution of the dataset. Since non-parametric tests do not require the data to follow the normal distribution , we applied three tree-based regressor models in this study: **Decision tree (DT), Random Forest (RF)** and **Gradient Boosting (GB)**. Following a split of the dataset into a 70/30 training/test split, all models were fit on the training dataset. Then, we used grid search techniques to optimize each model's hyper-parameters. To minimize bias and over-fitting, ten-fold cross-validation was used during model training.

Table provides the performance comparison of four regression models. The dataset contains nine statistically significant variables suggested by the `SelectKBest` tool. Compared to all models, the Gradient Boosting (GB) model showed the highest ${R^2}$ and adjusted ${R^2}$ values, followed by DT, OLS and RF. However, the inferior performance of each model indicates that models (Table ) were unable to explain the variability above 12%. The OLS model found the highest coefficients for Security Policy, Pinned Dependencies, and Maintained. The DT model included Branch Protection and Code-Review as important features, along with Security Policy, Pinned Dependencies, and Maintained. The GB model ranked the same practices as DT as important practices. However, in the case of ranking, Code-Review was the most important metric in GB, whereas DT ranked Code-Review last. The RF model showed the lowest prediction performance of all four models, with a negative $R^2$ score.

The performance of four models indicates that the currently-available independent security practices are inadequate to explain the variability of vulnerability counts. In Section , we discuss factors that might influence the model's prediction performance.

## **RQ2: Aggregate Security Score and Security Outcomes**

In this section, we describe our analysis for RQ2: *Do packages with higher aggregate security scores have fewer vulnerabilities?* We look for the statistical association of a package's aggregate score with package vulnerability count to understand if implementing security practices assists in secure coding with fewer vulnerabilities. We hypothesize that- *packages with higher security scores promote better security practices and contain fewer externally-reported vulnerabilities than packages with lower security scores*. We mentioned in Section  that Scorecard generates an aggregate score for each package, which is a weighted average of the individual checks, weighted by risk. To that end, we used the vulnerability count data of 2,422 packages (Section ) as a target variable and the aggregate score of those packages as independent variables to answer RQ2.

We removed six packages as outliers, identified by `qq-plot`. We perform the normality check to verify the normal distribution of the independent variable or aggregate score. We found the data was normally distributed to proceed with the linear regression model. The model, however, rejected our hypothesis by showing a significant positive relationship between aggregate score and vulnerability count. The linear model (p-value $<$ 0.001, $R^2=0.04$, adj. $R^2=0.03$) had a 0.5-unit increase in vulnerability count for every unit increase in aggregate security score.

# Discussion

This section discusses the implications of our findings. We also addresses the other factors (section ) that might impact the effectiveness of the use of Scorecard scores to identify the security practice.

## **Important security practices**

To explore the findings related to RQ1, we used a wide range of security practices, multiple statistical learners, and prediction models to help practitioners to identify important security practices for improving package security. Table showed the performance of four models.

Since we were interested in understanding the relationships between the variables, a low R-squared does not always negate the importance of any significant variables where there might be multiple unknown factors. Even with a low R-squared, statistically significant coefficients represent the mean change in the dependent variable given a one-unit shift in the independent variable . Hence, using statistically significant variables as predictors indicates there is a relationship between these predictors and the vulnerability count in the current dataset. Our models explain the past data in a statistically significant manner, but its predictive ability is less than 12% to extrapolate beyond the available data.

**Security Policy, Pinned Dependencies, Maintained, Code Review** and **Branch Protection** were the most important practices to explain our models. These five practices were statistically significant and came as an important feature in at least three models. Even if we discounted the RF model because of negative $R^2$, Security Policy, Pinned Dependencies, and Maintained came as an important practice in OLS, DT, GB models. Code Review and Branch Protection were important in DT and GB models. Our findings align with the Sonatype report , where they found **Code-Review, Pinned Dependencies, Branch Protection**, and **Binary-Artifacts** as important practices for Java projects. Binary-Artifacts were not important in our model for npm and PyPI ecosystems. Sonatype used 12k Java Projects, which is six times bigger than our dataset. We need more research to validate whether it is because of the size of the dataset or because Binary-Artifacts is a crucial practice for Java projects.

From correlation analysis (Section  ) we have observed that Pinned Dependencies is negatively co-related with four of these important practices (Security Policy, Maintained, Code Review, and Branch Protection). Pinned-Dependency prevents auto-updating a dependency to a new version without reviewing the differences between the two versions, which may include an insecure component. Zahan et al.  showed that "Pinned-Dependencies" has high false positives in the Scorecard's current computing process, which might explain the inverse relationship. Scorecard looks for the presence of unpinned dependencies in Dockerfiles, shell scripts, and GitHub workflows. However, the tool does not check `package.json`, `package-lock.json` or `setup.py` files where npm and PyPI package developers lock their dependencies. Hence, "Pinned-Dependencies" scoring only involved GitHub-specific workflows, limiting the metric's generalizability for npm and PyPI ecosystems. To verify the negative association of "Pinned-Dependencies" with other practices, we will need precise data from Scorecard tool that detects all pinned or unpinned dependencies in different ecosystems. Due to the necessity for additional research to establish the importance of "Pinned-Dependencies" by gathering reliable data, we have thus withdrawn "Pinned-Dependencies" as a suggested important practice.

Scorecard defined Maintained, Code Review, and Branch Protection as high-risk practices and Security Policy as a medium-risk practice. Zahan et al.  showed that both ecosystems have a gap in practicing Code-Review, Maintained, Branch-Protection, and Security-Policy practices. Although the current security trend shows practitioners do not employ these practices, our study indicates that practitioners adopting these practices will improve package security.

## **Aggregate Security Score and Externally-reported Vulnerabilities:**

Our model suggests that packages with more indicators of good security practices also had more reported vulnerabilities. Many factors are believed to increase the vulnerability count of packages. A possible explanation of the increase in reported vulnerability count while increasing in security score could be that the selected packages are used frequently by other open-source packages. Since more clients utilize these packages, there is a higher likelihood that the package will be tested or attacked, and there will be more reported vulnerabilities. The average stats (Direct Dependents: 427, Transitive Dependents: 2,758, Downloads: 72M) of packages in our dataset revealed that other open-source packages frequently used these packages. The number of developers may also impact the number of vulnerabilities. For instance, Meneely et al.  empirically showed that projects with more developers have more vulnerabilities. However, from OSV and Snyk databases, we can not collect such metadata to evaluate the relationship of package vulnerabilities with package popularity and developer community.

Without exploring and controlling for confounding variables, such as package popularity, the findings are unlikely to provide certainty that the good security practices of these packages helped to find more vulnerabilities. We need to conduct a more thorough investigation to confirm the claim, taking into account the time it took to find and fix the vulnerabilities, the practices that were used to find them, and whether or not third-party users found the vulnerabilities while using the package rather than the package's owner. The scarcity of vulnerability data could be another reason influencing our model. We utilized externally-reported reported vulnerabilities and not those found prior to release and by the development team.

# Challenges in available data

In this section, we discussed the challenges we found in explaining our findings. To improve model performance, future research replicating our study should consider these issues.

## **Scarcity of Vulnerability Data**

Our findings raise the concern of scarce vulnerability data in OSS communities. We need more studies and accurate vulnerability data to evaluate the relationship between security practices and vulnerabilities as security outcomes, especially to comprehend how security practices influence vulnerability detection and prevention. A package may have more vulnerabilities than we found in OSV and Snyk vulnerability databases (Section  ). Dumidu et al.  showed that a large number of vulnerabilities that affected the Linux kernel (39.4%) and the MySQL database server (62.23%) were vulnerabilities found in bug reports but not listed in common vulnerability databases, such as the NVD. Elder et al.  also identified 1,047 vulnerabilities in OpenMRS applications using five vulnerability detection techniques and tools. However, at the time of their study, OpenMRS had 11 reported CVE vulnerabilities. These studies show that vulnerability data that we acquire from OSV and Snyk databases hardly contain the accurate number of vulnerabilities as security outcomes.

Additionally, even within our dataset, we removed four packages as outliers because the distance from other packages in terms of vulnerability count was high. Out of 2,272 packages, 2,268 had an average vulnerability count of 2.07; these four packages had an average of 122 vulnerabilities. Even if these four packages had real-world vulnerabilities that were more prevalent than those of other packages, we were unable to use the data to build models because of their low percentage (0.2%) in the dataset.

## **Security Score Data**

Scorecards are designed to provide users of open-source projects with information that may be used to assess whether a dependency is secure. Although Scorecard collects raw/granular data to determine the normalized score, the public API only releases the security score of a package. A model using more granular data may have different outcomes from what we observed in this study. Studies show that defect prediction and vulnerability prediction can be improved using more granular data .

The Zahan et al.  analysis also demonstrated that not all security practices scores accurately reflect the state of package security practices. For example, Dangerous-Workflow, Binary-Artifacts, Pinned-Dependencies, and Token-Permissions looked for the existence of specific properties in GitHub workflows. However, Scorecard does not verify the presence of GitHub workflows in a repository. Hence, if a package does not have GitHub workflows, the tool will grade that package as a score of 10 as part of good security practices.

Then, the standards specified by the Scorecard tool for Dependency-Update-Tool, Fuzzing, and CII-Best-Practices were weakly adopted in npm and PyPI. One possible explanation behind weak adoption could be that these practices have inherent reliance on other system to measure package security. We cannot, however, avoid the possibility that packages may use different tools or techniques to improve security than the one defined by Scorecard. For example, 99% of packages did not pass the CII-Best-Practices check because the maintainer did not use the CII Best Practices program to self-certify how they follow different best practices. Therefore, the lack of consensus on standardizing Scorecard metrics for different ecosystems is a challenge that Scorecard and the community should overcome to improve the security measures.

## **Time**

The multivariate nature of security practices significantly complicates its quantification and measurement . To validate security practice and security outcomes relationships, we need to evaluate the time when security practices were implemented and the time when vulnerabilities were injected, identified, and fixed. The data for time to remediate an externally-reported vulnerability can be collected from the vulnerabilities database. At the time of the study, Scorecard only measures security practices from the last 90 days' activities, which does not verify the security practices before 90 days. For better clarity, we need to do a controlled study by recording time of security practices implication in a packages and detection of vulnerabilities. The scorecard team run a weekly Scorecards scan and publish the results in a BigQuery public dataset. OSV and Snyk also record the time of vulnerability detection and fix. Researchers could gain valuable insight from our proposed model by conducting a long-term control study on package security practice data and tracking vulnerabilities.

## **Unpredictability**

Security metrics are a hard problem, especially in predicting vulnerabilities or assessing the effectiveness of counter measures . We should consider that the software security field has an inherently greater amount of unexpected variation. Due to a lack of precision and unknowable variables, industries are still exploring how to adopt the most effective security measures to eliminate the most number of vulnerabilities efficiently .

For instance, on July 20, 2022, Checkmarx security researchers warned people about a new supply chain attack strategy that uses fake commit metadata to give the appearance of legitimacy to malicious GitHub repositories . Threat actors may modify the metadata in GitHub repositories to improve their reputation and increase the likelihood that application developers will choose them as reliable package owners. The researchers found it is possible to manipulate commit metadata, including timestamps, to make a repository appear older than it is or to show that credible contributors have been actively involved in its maintenance. This attack is an example of how frequently the software industry encounters new security threats. Attacks and attackers are unpredictable, as other recent supply chain attacks have also shown . Such unpredictability in software security often makes it challenging to interpret how security practices have affected software's security outcomes.

# Threats to Validity

To our knowledge, our study is the first peer-reviewed paper to look at how software security practices impact security outcomes using OpenSSF Scorecard data. However, as stated by Basili et al.  , drawing general conclusions from empirical studies in software engineering is difficult. Because the software security process depends on a potentially large number of relevant context variables, we only used the OpenSSF Scorecard tool to measure software security practices. Our study findings are dependent on the ruleset defined by the tool. For this reason, we cannot assume a priori that the results of our study can be generalized to all ecosystems.

Since Scorecard only operated on GitHub repositories at the time of the study, GitHub was the only platform we looked at. Scorecard has four practices (Dangerous-Workflow, Binary-Artifacts, Pinned-Dependencies, and Token-Permissions) that are GitHub workflow specific. Hence, our findings could vary if we used dataset outside of GitHub repositories.

# Conclusion

In this study, we looked into how multiple security practices and their aggregate score affect the security outcome of software. First, our models identified Maintained, Code Review, Branch Protection and Security Policy as the most important feature practices in our dataset. However, low $R^2$ and `adj` $R^2$ values raise concerns about model performance beyond the available data. Second, packages with increased good security practices also had increased reported vulnerability counts, demonstrating the need for confounding variable control and more vulnerability data to ascertain whether the relationship is explained by other factors. In the end, we explain our findings by discussing challenges related to scarce vulnerability data and security scores that might affect the model's performance.

Future work should focus on collecting more granular-level data for security practices. The Scorecard team can contribute by releasing the non-normalized, granular data that they collect to determine the security score. Then, unsupervised machine learning can aid in identifying inherent trends in data to improve the model's performance. Additionally, we need more large-scale focus studies to identify accurate vulnerability data in software components for modeling and understand how security practices impact security outcomes.

# ACKNOWLEDGMENTS

This work was supported and funded by Cisco and National Science Foundation Grant No. 2207008. Any opinions expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. We thank the OpenSSF Scorecard Team and Google GOSST for encouraging us to conduct this research. We also thank the NCSU Realsearch group for valuable feedback. In particular, we thank Parth Kanakiya for his assistance in generating vulnerability data.


abstract: We investigate the collective dynamics of bi-stable elements connected in different network topologies, ranging from rings and small-world networks, to scale-free networks and stars. We estimate the dynamical robustness of such networks by introducing a variant of the concept of multi-node basin stability, which allows us to gauge the global stability of the dynamics of the network in response to local perturbations affecting a certain class of nodes of a system. We show that perturbing nodes with high closeness and betweeness-centrality significantly reduces the capacity of the system to return to the desired state. This effect is very pronounced for a star network which has one hub node with significantly different closeness/betweeness-centrality than all the peripheral nodes. In such a network, perturbation of the single hub node has the capacity to destroy the collective state. On the other hand, even when a majority of the peripheral nodes are strongly perturbed, the hub manages to restore the system to its original state, demonstrating the drastic effect of the centrality of the perturbed node on the dynamics of the network. Further, we explore explore Random Scale-Free Networks of bi-stable dynamical elements. We exploit the difference in the distribution of betweeness centralities, closeness centralities and degrees of the nodes in Random Scale-Free Networks with $m=1$ and $m=2$, to probe which centrality property most inluences the robustness of the collective dynamics in these heterogeneous networks. Significantly, we find clear evidence that the betweeness centrality of the perturbed node is more crucial for dynamical robustness, than closeness centrality or degree of the node. This result is important in deciding which nodes to safeguard in order to maintain the collective state of this network against targetted localized attacks.
address: Indian Institute of Science Education and Research (IISER) Mohali, SAS Nagar, Sector 81, Mohali 140 306, Punjab, India
author: Pranay Deep Rungta, Chandrakala Meena and Sudeshna Sinha[^1]
title: Identifying nodal properties that are crucial for the dynamical robustness of multi-stable networks

Collective spatiotemporal patterns emerging in dynamical networks are determined by the interplay of the dynamics of the nodes and the nature of the interactions among the nodes. So it is of utmost relevance to ascertain what properties of the nodes of the network impact collective dynamics. This also helps to address the important reverse question: perturbation of what class of nodes in the network have the most significant effect on the resilience of the network? Understanding this will allow us determine which nodes render the network most susceptible to external influences. Alternately, it will suggest which nodes to protect more stringently from perturbations in order to protect the dynamical robustness of the entire network. As a test-bed for understanding this we consider the collective dynamics of a group of coupled bi-stable elements. Bi-stable systems are relevant in a variety of fields, ranging from relaxation oscillators and multivibrators, to light switches and Schmitt triggers. Further it is of utmost importance in digital electronics, where binary data is stored using bi-stable elements.

Specifically, in this work we will explore bi-stable elements, connected in different network topologies, ranging from regular rings to random scale-free and star networks. We focus on the response of this network to localized perturbations on a sub-set of nodes. The central question we will investigate here is the following: *what characteristics of the nodes (if any) significantly affect the global stabilty?* So we will search for discernable patterns amongst the nodes that aid the maintenance of the stability of the collective dynamics of the network on one hand, and the nodes that rapidly destroy it on the other. In particular, we consider three properties of the nodes: degree, betweeness centrality and closeness centrality. Since these features of a node determine the efficiency of information transfer originating from it, or through it, they are expected to influence the propagation of perturbations emanating from the node.

Normalized degree of a node $i$ in an undirected network is given by the number of neighbors that are directly connected to the node scaled by the total number of nodes $N$, and is denoted by $k_i$. So a high degree node indicates that there is direct contact with a larger set of nodes. Normalized betweeness centrality of a node $i$ is given as: $$b_i= \frac{2}{(N-1)(N-2)} \sum_{s,t\in I}\frac{\sigma(s,t|i)}{\sigma(s,t)}$$ where $I$ is the set of all nodes, $\sigma(s,t)$ is the number of shortest paths between nodes $s$ and $t$ and $\sigma(s,t|i)$ is the number of shortest paths passing through the node $i$. So if node $i$ has high betweeness-centrality, it implies that it lies on many shortest paths, and thus there is high probability that a communication from $s$ to $t$ will go through it. Normalized Closeness Centrality is defined as: $$c_i=\frac{N-1}{\sum_{j} d(j,i)}$$ where $d(j,i)$ is the shortest path between node $i$ and node $j$ in the graph. Namely, it is the inverse of the average length of of the shortest path between the node and all other nodes in the network. So high closeness centrality indicates short communication path to other nodes in the network, as there are minimal number of steps to reach other nodes.

In this work we will explore the extent to which the features of the nodes given above influence the recovery of a network from large localized perturbations. In order to gauge the global stability and robustness of the collective state of this network, we will *introduce a variant of the recent framework of multi-node basin stability* . In general, the basin stability of a particular attractor of a multi-stable dynamical system is given by the fraction of perturbed states that return to the basin of the attraction of the dynamical state under consideration. In our variant of this measure, we consider an initial state where all the bi-elements in the network are in the same well, and we will refer to this as a *synchronized state*. So a synchronized state here does not imply complete synchronization. Rather it implies a collective state where the states of the nodes are confined to the neigbourhood of the same attracting stable state, i.e. lies within the basin of attraction of one of the two attracting states. We then perturb a specific number of nodes of a prescribed type, with the perturbations chosen randomly from a given subset of the state space. The multi-node basin stability (BS) is then defined as the fraction of such perturbed states that manage to revert back to the original state from these localized perturbations. Namely, multi-node BS reflects the fraction of the volume of the state space of a sub-set of nodes that belong to the basin of attraction of the synchronized state. So the importance of multi-node BS stems from the fact that it determines the probability of the system to remain in the basin of attraction of the synchronized state when random perturbations affect a specific number of nodes. This allows us to extract the contributions of individual nodes to the overall stability of the collective behaviour of the dynamical network. Further, since one perturbs subsets of nodes with certain specified features, our variant of the concept of multi-node BS will suggest which nodal properties make the network more vulnerable to attack.

Specifically we consider the system of $N$ diffusively coupled bi-stable elements, whose dynamics is given as:

$$\dot{x_i} = F(x_i) + C \frac{1}{K_i} \sum_j (x_j - x_i) = F(x_i) + C (\langle x_i^{nbhd} \rangle - x_i)
\label{main}$$

where $i$ is the node index ($i = 1, \dots N$) and $C$ is the coupling constant reflecting the strength of coupling. The set of $K_i$ neighbours of node $i$ depends on the topology of the underlying connectivity, and this form of coupling is equivalent to each site evolving diffusively under the influence of a "local mean field" generated by the coupling neighbourhood of each site $i$, $\langle x_i^{nbhd} \rangle = \frac{1}{K_i} \sum_j x_j$, where $j$ is the node index of the neighbours of the $i^{th}$ node, with $K_i$ being the total number of neighbours of the node.

The function $F(x)$ gives rise to a double well potential, with two stable states $x^*_-$ and $x^*_+$. For instance one can choose $$F(x) = x - x^3$$ yielding two stable steady states $x^*_{\pm}$ at $+1$ and $-1$, separated by an unstable steady state at $0$. Note that the synchronized state here is a fixed point, either $x^*_-$ and $x^*_+$, for all the nodes, i.e. $x_i$ is equal to $x^*_-$ or $x^*_+$, for all $i$.

We first investigate the two limiting network cases: (i) *Ring*, where all nodes have the same degree, closeness and betweeness centrality, and a (ii) *Star network*, where the central (hub) node has the maximum normalized degree ($k_{hub} = 1$), betweeness centrality ($b_{hub} \sim 1$), and closeness centrality ($c_{hub} = 1$), while the rest of the nodes, namely the peripheral nodes ("leaves") have very low degree ($k_{peri} \sim 0$ for large networks), betweeness centrality ($b_{peri}=0$) and closeness centrality $c_{peri} \sim 0.5$. So on one hand we have the Ring which is completely homogeneous, and on the other hand we have the Star network where the difference in degree, closeness and betweeness centrality of the hub and the peripheral nodes is extremely large. Exploring these limiting cases allows us to gain understanding of the robustness of the network to large perturbations affecting nodes with different properties.

As indicated eralier, to gauge the effect of different nodal features on the robustness of the dynamical state of the network, we do the following: we first consider a network close to a stable synchronized state, namely one where the states $x_i$ of all the nodes $i$ have a small spread in values centered around $x^*_-$ or $x^*_+$, i.e. all elements are confined to the same well. We then give a *large perturbation* to a small fraction of nodes, denoted by $f$. This strong perturbation typically kicks the state of the perturbed nodes to the basin of attraction of the other well. We then ascertain whether all the elements return to their original wells after this perturbation, i.e. if the perturbed system recovers completely to the initial state. We repeat this "experiment" over a large sample of perturbed nodes and perturbation strengths, and find the fraction of times the system manages to revert to the original state. This measure of global stability is then a variant of multi-node Basin Stability and it is indicative of the robustness of the collective state to perturbations localized at particular nodes of a certain type in the network.
**Dynamics of a Ring of Bistable Systems**
We first investigate the spatiotemporal evolution of a ring of bistable elements, all of whose states are confined to the same well, other than a few nodes that experience a large perturbation which pushes their state to the basin of the other well. We find that even when the fraction of perturbed nodes is very small, these perturbed nodes are unable to return to the original well. That is, the elements in the Ring are unable to drag the few perturbed nodes back to the well of the majority of the elements, suggesting that the Ring is not robust against such localized perturbations.

Next we attempt to discern the effect of coupling on the robustness of the dynamics. Fig. (a) shows the multi-node basin stability for this system, as the coupling strength is increased in the range $0$ to $2$, for clusters of perturbed nodes with $f$ ranging from $0.01$ to $0.08$. It is evident from the basin stability of the system, that there is a *sharp transition* from zero basin stability, namely the situation where *no* perturbed state returns to the original state, to basin stability close to one, namely where *all* sampled perturbed states return to the original state. This indicates that the *system recovers from large localized perturbations more readily if it is strongly coupled*. Further, the figure also demonstrates the extreme sensitivity of basin stability to the number of nodes being perturbed. We find that the system fails to return to the original state, even at very high coupling strengths, when more than 5% of the nodes experience perturbations. For instance, Fig. (a) shows the case of a single perturbed node (i.e. $f=0.01$), where the entire network recovers for coupling strengths stronger than approximately $0.2$. In contrast, for $f=0.08$, where a cluster of $8$ nodes are perturbed in the Ring of $100$ elements, there is zero basin stability in the entire coupling range. So a Ring loses its ability to return to the original state rapidly with increasing number of perturbed nodes.

For very small $f$, the entire network returns to its original state, and basin stability is close to $1$. On increasing $f$ one observes that there exists a minimum fraction, which we denote as $f_{crit}$, after which the basin stability sharply declines from $1$. So $f_{crit}$ indicates the minimum fraction of nodes one typically needs to perturb in order to destroy the collective state where all elements are in the same well. We find that high coupling strengths increase $f_{crit}$. For instance, $f_{crit} \approx 0.02$ for $C=0.5$ and $f_{crit} \approx 0.04$ for $C=1$. So for stronger coupling, the bulk of the elements are capable of pulling the perturbed nodes back to original well, increasing the resilience of the network.

Due to the structure of the ring, the stability of the system with respect to localized perturbations depends on whether the perturbed nodes are contiguous and occur in a cluster (cf. the case in Fig. (a)) or randomly spread over the ring, where the locations of the perturbed nodes are uncorrelated. Fig (b) shows the multi-node basin stability when nodes perturbed are chosen randomly for different values of coupling. We observe that the system is more stable here, as compared to the case when nodes are perturbed in cluster, namely *perturbations at random locations in a ring allows the system to recover its original dynamical more readily than perturbations on a cluster of contiguous nodes*.

Further, the inset in Fig.  (a) shows the dependence of multi-node basin stability on the fraction of perturbed nodes $f$ for different system sizes $N$. We find that $f_{crit} \sim \frac{C}{N}$, implying that $f_{crit} \rightarrow 0$ as system size $N \rightarrow \infty$. This indicates that in a very large Ring, even the smallest finite fraction of perturbed nodes can disturb the Ring from its original steady state. So one can conclude that the synchronized state in the Ring is very susceptible to destruction, as only very few nodes in the system need to be perturbed in order to push the system out of the original state. Namely, in a ring of bi-stable elements, *the collective state where all elements are in the same well, is a very fragile state.*
**Dynamics of a Star Network of Bistable Systems**
Now we study the spatiotemporal evolution of bistable elements connected in a star configuration. Here the central hub node has the maximum degree, betweenness and closeness centrality, while the rest of the nodes, namely the peripheral leaf nodes have very low degree, betweeness and closeness centrality. Namely, in this network the difference in degree, closeness and betweeness centrality of the hub and the peripheral nodes is extremely large. So this network offers a good test-bed to investigate the correlation between specific properties of a node and the resilience of the network to large localized perturbations at such nodes.

Figs. a-b display the dynamics for two illustrative cases. In Fig. (a), *only the hub node is perturbed* in the star network consisting of $100$ elements. We notice, that this *single* perturbed node pulls *all* the other nodes of the network away from its original state. So the star network is *extremely vulnerable to perturbations at the hub*, and cannot typically recover from disturbances to the state of the hub, even if all the other nodes are unperturbed. On the other hand, Fig. (b), shows what ensues when a *large number of peripheral nodes are perturbed*. Now, even when as many as $90$ nodes are perturbed, namely 90% of the network experiences a disturbance in its state, the entire network still manages to recover to its original state. This dramatic difference in the outcome of perturbations clearly illustrates how sensitively the robustness of a dynamical state depends on the degree, closeness and betweeness centrality of the perturbed node.

Next we examine the multi-node basin stability of the network, for fraction $f$ of perturbed nodes ranging from $1/N$ (namely single node in Fig. (a) ) to $f \sim 1$, namely the case where nearly all nodes in the system are perturbed. As evident from Fig.  (b), when only the peripheral nodes are perturbed, even for values of $f$ as high as $0.7$, there is no discernable difference in the basin stability, which remains close to $1$. This implies that even when more than half the nodes in the network are perturbed the entire system almost always recovers to the original state. In contrast, in Fig. (a) shows the single-node basin stability for the case of the hub node being perturbed, where the basin stability is clearly drastically reduced and approaches zero very quickly. It is clear that just a *single* node is enough to destroy the stability of the network, if that node has very high degree, closeness and betweeness centrality, such as the hub node. These quantitative results are consistent with the qualitative spatiotemporal patterns observed in Fig. .

Further Fig. (b) shows the decline in Multinode Basin Stability with increasing fraction of perturbed nodes $f$. Interestingly, when only the peripheral nodes are perturbed, Basin Stability is close to one even when a very large fraction of nodes in the system are perturbed, and $f_{crit} \approx 0.7$. So when only peripheral nodes are perturbed, the perturbed nodes manage to return to their original state, *even when a majority of nodes in the system have been pushed to the basin of attraction of the other state*. However when the perturbed nodes include the central hub node, the basin stabilty declines rapidly with increasing $f$, and $f_{crit} \approx 1/N$. So our analysis reveals how significant the degree, closeness and betweeness of nodes are in determining the resilience of the network. In fact, very clearly, the *hub node holds the key to the maintenenance of the collective state*.

We can rationalize the above results from dynamics of the coupled system as follows: the influence of the neighbours on a particular node is through the local mean field generated by the nieghbouring nodes. Now it is clear if a node has few neighbours the influence of a perturbation on its neighbour will be very large. On the other hand, if a node is connected to many other nodes, as is typically true of nodes with high degree or betweenness centrality, such as a hub in a star network, the influence of peturbations on a node in its neighbourhood is scaled by a factor of $1/k$, where $k$ is the number of neighbours of the node. This implies that the effect of the peripheral nodes on the hub is much smaller than the hub on the periphery. Namely, the hub affects all peripheral nodes strongly with the coupling term being of $O(1)$. However a perturbation on a peripheral node affects the hub only through a coupling of $O(1/N)$, which is vanishingly small for large networks. Further the peripheral nodes do not affect each other directly, but only through perturbations propagating to the hub, while the hub simultaneously affects all peripheral nodes.
**Dynamics of a Random Scale-Free Network of Bistable Systems**
We will now go on to explore Random Scale-Free (RSF) Networks of bi-stable dynamical elements. In particular, we construct these networks via the Barabasi-Albert preferential attachment algorithm, with the number of links of each new node denoted by parameter $m$ . The network is characterised by a fat-tailed degree distribution. Specifically we will display results for networks of size $N=100$, with $m=1$ and $m=2$. Figs. a-b shows two contrasting representative cases where (a) twenty nodes with highest betweeness centrality are perturbed, and (b) twenty nodes with the lowest betweeness centrality were perturbed. It is observed that perturbation on nodes with high betweeness centrality destabilises the entire network, and the perturbed nodes rapidly drag all the other nodes to a different well. This is evident from the switched colors of the asymptotic state in Fig. a. On the other hand, when the perturbed nodes have low betweeness centrality, the network recovers quickly from the perturbation and reverts to the original well, as clearly seen in Fig.b. These completely different outcomes occur even though the *number of perturbed nodes is the same* in both cases, thereby clearly illustrating that nodes with high betweeness-centrality have much stronger influence on the global stability of the system.

We will now present the dependence of the global stability of the collective dynamics on different centrality measures in this heterogeneous network, quantitatively, through multi-node basin stability measures. In particular, in order to explore the correlation between a given centrality measure of the nodes and the resilience of the system, we will estimate the multi-node basin stability under perturbations on sub-sets of nodes with increasing (or decreasing) values of the centrality under consideration. That is, we order the nodes according to the centrality we are probing, and consider the effect of perturbations on fraction $f$ of nodes with the highest (or lowest) centrality.

The influence of perturbations on nodes with the highest and lowest betweeness, closeness and degree centrality in a Random Scale-Free network are displayed in Figs. a-c. The broad trends are similar for all three centrality measures, and it is clearly evident that when nodes with the highest betweeness, closeness and degree centrality are perturbed, multi-node basin stability falls drastically. On the other hand, perturbing the same number of nodes of low centrality leaves the basin stability virtually unchanged. Further, when nodes of low centrality are perturbed, for sufficiently high coupling strengths, the network almost always recovers to its original state, yielding a basin stability of $1$. So one can conclude that perturbing nodes with high betweeness, closeness and degree centrality destroys the synchronized state readily, while perturbing nodes of low centrality allows the perturbed nodes to return to the original state, thereby restoring the sychronized state. For reference, Fig.  also shows the basin stability of a network where the perturbed nodes are randomly chosen, corresponding to *random attacks* on a subset of nodes. Clearly, a *targetted attack* on nodes with high centrality can destroy the collective dynamics much more efficiently than random attacks.

Now we investigate which centrality measure is most crucial in determining the global robustness of collective behaviour in the network. We do this through the following numerical experiment: we compare the basin stability of the collective dynamics of Random Scale-Free networks with $m=1$ and $m=2$. Interestingly, the distribution of the betweeness centrality, closeness centrality and degree of the nodes in RSF networks with $m=1$ and $m=2$ are significantly different, as evident in Fig.  (top panels). It is clear that for $m=1$ the distribution of the degree and closeness centrality of the nodes in the network is shifted towards lower $k$ and $c$ values as compared to RSF networks with $m=2$, while the distribution of betweeness centrality shifts towards higher values in RSF networks with $m=1$ vis-a-vis the distribution of the betweeness centrality in RSF networks with $m=2$. So in RSF networks with $m=1$ the nodes with the highest betweeness centrality typically have significantly higher $b$ than in RSF networks with $m=2$ of the same size. On the other hand, since the tail of the probability distribution of degree and closeness centrality of the nodes in a RSF network with $m=2$ extends further than that in a RSF network with $m=1$, the nodes with the highest degree and closeness centrality typically have lower $k$ and $c$ in RSF networks with $m=1$ compared to RSF networks with $m=2$. So these networks can potentially *provide a test-bed for determining which of the centrality properties most crucially influence dynamical robustness*. Note that it was not possible to use the Ring to probe this issue, as all nodes there have identical centrality propoerties. Nor did the Star network offer a system where one could distinguish between the effects of different centrality measures on dynamical robustness, as the nodes there split into two classes, the single hub and the periphery, with all the peripheral nodes having identical betweeness, closeness and degree. However, one can compare the response of Random Scale-Free networks with different $m$ to probe which nodal property renders a heterogeneous network most vulnerable to large localized perturbations.

Fig.  (bottom panels) displays the dependence of the multi-node Basin stability on the fraction of perturbed nodes $f$ in the Random Scale-Free network with $m=1$ and $m=2$. As number of nodes perturbed increases, the multi-node basin stability falls significantly for RSF networks with $m=1$, while RSF networks with $m=2$ remains robust up to a critical fraction $f_{crit}$ of perturbed nodes, with $f_{crit} \sim 0.2$. One can rationalize this, by noting the difference in the typical values of betweeness centrality at the highest end in the RSF network with $m=1$ and $m=2$. For instance, if one considers $10 \%$ of nodes with the highest betweeness centrality in these networks of size $N=100$, typically $b$ lies between $0.1$ to $0.9$ for $m=1$ and between $0.01$ and $0.5$ for $m=2$. So the marked difference in the sensitivity of the global stability to perturbations in Random Scale-Free networks with $m=1$ and $m=2$ stems from the higher betweeness centrality of the nodes in the former network.

Now when nodes of the highest closeness centrality and degree are perturbed we observe the same trend as above. This occurs inspite of the tail of the distribution of closeness centrality and degree extending to higher values for RSF networks with $m=2$ as compared to RSF networks with $m=1$, implying that the nodes with highest degree and closeness centrality for the $m=2$ case will have a larger value of $k$ and $c$, as compared to the $m=1$ case. So one may have expected that the RSF network with $m=2$ would be less stable than the RSF network with $m=1$. However, the observations are contrary to this expectation and this surprising result stems from the following: the set of nodes with the highest betweeness centrality, closeness centrality and degree, overlap to a very large extent. So for instance, for $f=0.1$ in a network of size $N=100$, the set of $10$ nodes with the highest betweeness centralities, is practically the same as the set of nodes with the highest closenes centralities and highest degrees. However, in the RSF network with $m=1$ these nodes have higher betweeness centrality, while having lower closeness centrality and degree, than the corresponding set in the RSF network with $m=2$. Now higher betweeness centrality should inhibit stability, while lower closeness and degree should aid the stability of the collective state. So the comparative influence of these two opposing trends will determine the comparative global stability of these two classes of networks. If the betweeness centrality of the perturbed nodes is more crucial for stability, the multi-node Basin Stability of the network with $m=1$ will go to zero faster than the network with $m=2$. On the other hand if closeness centrality (and/or degree) of the perturbed nodes dictates global stability rather than betweeness centrality, the network with $m=2$ will lose global stability faster than the one with $m=1$. Now, since we find that network with $m=1$ always loses stability faster than the network with $m=2$, we can conclude that the *effect of betweeness centrality on the global stability is more dominant than the effect of the closeness centrailty and degree of the perturbed nodes*.

Lastly, we study the effect of system size on multi-node basin stability, perturbing nodes in decreasing order of betweeness centrality. Figs. a-b shows the results for networks sizes ranging from $50$ to $200$. We have found an appropriate finite-size scaling that allows data collapse (cf. Fig.  insets), and this indicates the value $f_{crit}$ in the limit of large network size. We observe that a Random Scale-Free network with $m=1$ yields $f_{crit} \rightarrow 0$ (i.e. the smallest fraction of perturbed nodes destroy the collective state), while $f_{crit} \sim 0.2$ for the case of $m=2$. So a RSF network with $m=2$ is more robust to localized perturbations than a a RSF network with $m=1$, as in the $m=2$ case, even when nearly $20 \%$ of the nodes of the highest betweeness centrality are perturbed the entire network still manages to return to the original state. This compelling difference again arises due to the fact that the highest betweeness centrality found in the RSF network with $m=1$ is significantly higher on an average than that in RSF networks of the same size with $m=2$. This again corroborates the results in Fig. , and highlights the profound influence of betweeness centrality on global stability.
**Robustness of the phenomena:**
In order to ascertain the generality of our observations, we have considered different nonlinear functions $F(x)$ in Eq.(1). For example, we explored a system of considerable biological interest, namely, a system of coupled synthetic gene networks. We used the quantitative model, developed in , describing the regulation of the operator region of $\lambda$ phase, whose promoter region consists of three operator sites. The chemical reactions describing this network, given by suitable re-scaling yields $$F_{gene}(x) = \frac{m (1 + x^2 + \alpha \sigma_1  x^4)}
                {1 + x^2 + \sigma_1 x^4 + \sigma_1 \sigma_2 x^6} - \gamma_x x$$ where $x$ is the concentration of the repressor. The non linearity in this $F_{gene} (x)$ leads to a double well potential, and different $\gamma$ introduces varying degrees of asymmetry in the potential. We studied a system of coupled genetic oscillators given by: $\dot{x_i} = F_{gene}(x_i) + C (\langle x_i^{nbhd} \rangle -x_i)$, where $C$ is the coupling strength and $\langle x_i^{nbhd} \rangle$ is the local mean field generated by the set of neighbours of site $i$.

Further we studied different networks of a piece-wise linear bi-stable system, that can be realised efficiently in electronic circuits , given by: $$F(x) = - \alpha x + \beta \ g(x)$$ with the piecewise-linear function $g (x)=x$ when $x^*_{l} \le x \le x^*_{u}$, $g(x) = x^*_{l}$ when $x < x^*_{l}$ and $g(x)= x^*_{u}$ when $x > x^*_{u}$, where $x^*_{u}$ and $x^*_{l}$ are the upper and lower thresholds respectively.

We simulated the coupled dynamics of these two bi-stable systems for different network topologies as well. We find that the qualitative trends in both these bi-stable systems is similar to that described above, indicating the generality of the central results presented here.
**Conclusions:**
In summary, we have investigated the collective dynamics of bi-stable elements connected in different network topologies, ranging from rings and small-world networks, to scale-free networks and stars. We estimated the dynamical robustness of such networks by introducing a variant of the concept of multi-node basin stability which allowed us to gauge the global stability of the dynamics of the network in response to local perturbations affecting particular nodes of a system. We show that perturbing nodes with high closeness and betweeness-centrality significantly reduces the capacity of the system to return to the desired stable state. This effect is very pronounced for a star network which has one hub node with significantly different closeness/betweeness-centrality than the peripheral nodes. Considering such a network with all nodes in one well, if one perturbs the hub to another well, this *single* perturbed node drags the entire system to its well, thereby preventing the network from recovering its dynamical state. In contrast, even when *all* peripheral nodes are kicked to the other well, the hub manages to restore the entire system back to the original well. Lastly we explore explore Random Scale-Free Networks of bi-stable dynamical elements. Since the distribution of betweeness centralities, closeness centralities and degrees of the nodes is significantly different for Random Scale-Free Networks with $m=1$ and $m=2$, these networks have the potential to provide a test-bed for determining which of these centrality properties most inluences the robustness of the collective dynamics. The comparison between the global stability of these two classes of networks provides clear indications that the *betweeness centrality of the perturbed node is more crucial for dynamical robustness, than closeness centrality or degree of the node*. This result is important in deciding which nodes to safeguard in order to maintain the collective state of this network against targetted localized attacks.

[^1]: e-mail: email@example.com