 is replaced by --cassandra.tls.enabled.}\\ 
        \hline 
        Action required (9) & Action required to mitigate breaking changes. &
         \makecell{\\
\begin{sparkline}{7}
\definecolor{sparkspikecolor}{named}{black}
\sparkspike.05 0
\definecolor{sparkspikecolor}{named}{black}
\sparkspike.18.67 
\definecolor{sparkspikecolor}{named}{black}
\sparkspike.31.33 
\definecolor{sparkspikecolor}{named}{black}
\sparkspike.44.33 
\definecolor{sparkspikecolor}{named}{black}
\sparkspike.57.33 
\definecolor{sparkspikecolor}{named}{black}
\sparkspike.70 1.3 
\definecolor{sparkspikecolor}{named}{black}
\sparkspike.83 0 
\end{sparkline}}
        & \textit{Those affected must upgrade to a newer Docker version.} \\
        \hline
        \makecell[l]{Code change \\reference (9)} & \makecell[l]{Link to commit, pull request\\ inducing breaking change.} & 
         \makecell{\\
\begin{sparkline}{7}
\definecolor{sparkspikecolor}{named}{black}
\sparkspike.05.33
\definecolor{sparkspikecolor}{named}{black}
\sparkspike.18 1.67 
\definecolor{sparkspikecolor}{named}{black}
\sparkspike.31.33 
\definecolor{sparkspikecolor}{named}{black}
\sparkspike.44.33 
\definecolor{sparkspikecolor}{named}{black}
\sparkspike.57 0 
\definecolor{sparkspikecolor}{named}{black}
\sparkspike.70 0 
\definecolor{sparkspikecolor}{named}{black}
\sparkspike.83 0 
\end{sparkline}}
        & \textit{GPU metrics provided by kubelet are now disabled by default (\#95184).} \\
        \hline
        \makecell[l]{Affected \\configuration (7)} & \makecell[l]{Dropping support; configuration\\ change.} &
         \makecell{\\
\begin{sparkline}{7}
\definecolor{sparkspikecolor}{named}{black}
\sparkspike.05 0
\definecolor{sparkspikecolor}{named}{black}
\sparkspike.18 1.28 
\definecolor{sparkspikecolor}{named}{black}
\sparkspike.31 0 
\definecolor{sparkspikecolor}{named}{black}
\sparkspike.44.43 
\definecolor{sparkspikecolor}{named}{black}
\sparkspike.57 0 
\definecolor{sparkspikecolor}{named}{black}
\sparkspike.70.86 
\definecolor{sparkspikecolor}{named}{black}
\sparkspike.83.43 
\end{sparkline}}
        & \textit{Support for the CoreOS OS distribution has been removed.} \\
        \hline
        \multicolumn{4}{l}{1. Explanation and sub-codes of the category used during open coding.} \\
\multicolumn{4}{l}{2. Histogram of relative frequency for each category, plotted in the order: Composer, Go, Maven, npm, NuGet, pip, RubyGems.}\\
\multicolumn{4}{l}{3. Excerpt from release notes exemplifying the corresponding category of information.}
    \end{tabular} 
\end{table*}



\subsection{Results}

Table \ref{tab:manualsample} shows how many security releases have a release note, and if the release note mentions the security fix, unrelated changes, and breaking changes. We identified eight categories of information
that are documented in the release notes
for security fixes and five categories of information for breaking changes. Table \ref{tab:codes} lists the categories of information on the security fixes and breaking changes along with the sub-codes used during open coding, relative frequency over the seven ecosystems, and an example for each of the categories.


Our agreement rate for these categories
ranged from 0.32 to 1.00, as shown in Table \ref{tab:kappa}. In the case of binary coding on the mention of unrelated changes, the agreement rate was 0.82. According to the interpretation of Cohen's Kappa~\cite{viera2005understanding}, the two coders had fair to perfect agreement over the different cases. Below, we discuss our findings for each of the sub-questions of RQ2.

\subsubsection{How many of the security releases contain a release note? What are the sources to find the release notes?}
We could locate a release note for 363 (72.7\%) cases in our data set. We found changelog to be the most common source (201),
followed by GitHub release note (93). In 45 cases, we located the release note on the package homepage website. In 23 cases, we considered the tag message as the release note. In 1 case, we found a separate security notice for the release. In 9 of these cases, 
we noticed the security release listed by Snyk was incorrect, as we were able to locate the fix information for the corresponding advisory in a different release.

\begin{table}[]
    \centering
    \caption{Agreement rate for categories emerged from open coding for RQ2 analysis*}
    \label{tab:kappa}
    \begin{tabular}{lr}
\hline
 Code (count)                      &    $kappa$ \\
\hline
\multicolumn{2}{c}{Categories for security fix}\\
\hline
 Security notice (198)          &    0.60  \\
 Fix reference (181)             &    0.53 \\
 Vulnerability description (150) &    0.67 \\
 Advisory reference (121)       &    0.81 \\
 Affected component (79) &    0.39 \\
 Affected version (33)  &    0.41 \\
 Exploit (26)                   &    0.32 \\
 Affected configuration (3)    &    0.5  \\
 \hline
 \multicolumn{2}{c}{Categories for breaking change}\\
\hline
 Breaking change notice (22)    &    0.83 \\
 Affected API (20)       &    0.92 \\
 Action required (9)           &    0.87 \\
 Code change reference (9)     &    1    \\
 Affected configuration (7)    &    0.72 \\
\hline
\multicolumn{2}{c}{\makecell{*In cases where the codes appeared less\\ frequently, Cohen's kappa penalizes more even for\\ small disagreements due to the unbalanced dataset}}
\end{tabular}
\end{table}


\subsubsection{How many of the release notes mention the security fix? What information regarding the security fix is present in the release notes?}
\label{sec:secfixres}

We identified the mention of security fix for the corresponding advisory in 307 cases (61.5\%). If we disregard the releases without a release note, we find that the release notes document the security fix 84.5\% of the time. We find that Python packages are most likely to mention the security fix in the release notes (82.8\%), while Maven packages are the least likely to do so (40.8\%). 

One possible reason for not mentioning security fixes in the release notes is that the omission is intentional, and the fix may be announced in a private channel. Another possible explanation is the security aspect of the fix was not recognized by the package maintainers at the time of the release.  

For the release notes where the security fix was mentioned, we identified eight categories of information to be documented about the security fix. Table \ref{tab:codes} lists the identified categories. Below, we explain the categories in the order of their frequencies. A single release note can contain multiple information categories, therefore, the total frequency of all the categories is greater than 100\%. 

\begin{enumerate}
    \item \textbf{Security notice} (39.6\%) refers to cases where the release note notifies the reader about a security fix through an explicit mention of the keyword `security' or `vulnerability'. While an explicit signal can be helpful for the client projects, we observe that release notes can be long and readers may miss the signal if not properly highlighted. 
    
    \item \textbf{Fix reference} (36.2\%) refers to cases where the release note either points to the fix commits or summarizes the fix. However, when the security is not explicitly mentioned, the readers may fail to understand that the fix addresses a potential security issue
    and not just a general bug.
    
    \item \textbf{Vulnerability description} (30.1\%) refers to cases where the release note describes the vulnerability, such as the vulnerability type and the explanation of potential risk.
    
    \item \textbf