\section*{Introduction}

Semiconductor avalanche photodiodes (APD's) are versatile for weak light detection, with applications from remote ranging\cite{Wehr1999,Schreiber1999}, quantum communication \cite{yuan2018} and fluorescence lifetime imaging\cite{Damalakiene2016} to optical time-domain reflectometry \cite{Healey1984,Eraerds2010}.
For practical fiber quantum key distribution (QKD), InGaAs/InP APD's are the detector of choice because they are compact and low cost, and allow cryogenic-free or even room-temperature operation \cite{yuan2018}.
However, they suffer from spurious afterpulsing arising from carrier trapping by defects in the multiplication layer, especially at high detection efficiencies \cite{Comandar2015,Tada2020}.
To minimise afterpulsing,  an APD can be biased on for a sub-nanosecond duration only when a photon arrival is expected.
In doing so, charge per avalanche can be reduced to the order of 10~fC \cite{yuan2010,Restelli2013,Namekata2006}, corresponding to a transient current of less than 0.1~mA.
Such weak avalanches have to be discriminated through use of a readout circuit that removes the strong capacitive response to the applied gates. Gated InGaAs detectors are capable of counting photons at up to 60\% efficiencies \cite{Fang2020} and 1~GHz rate \cite{patel2012} and with photon number resolution \cite{kardynal2008}.  Thanks to this success, gating approach has been applied to traditionally free-running Si devices for performance enhancement \cite{Thomas2010,wayne2021}.

Existing readout circuits include band stop \cite{Namekata2006,Namekata09,Tada2020} or low-pass\cite{walenta2012,He2017,Fang2020} filtering under sine-wave gating\cite{Namekata2006}, self-differencing \cite{Yuan2007,Comandar2015}, and transient reference cancellation \cite{Restelli2013,Liang2019}. While simple for implementation, frequency filtering distorts the avalanche signals due to its rejection of a sizeable portion of frequency components, thus increasing time jitter and temporal errors in photon registrations \cite{walenta2012}.  Self-differencing \cite{Yuan2007} and reference cancellation methods \cite{Restelli2013} are able to maintain avalanche signal fidelity but may suffer operational complexities.
The former requires wideband performance for the entire circuitry and thus inconvenience an adjustable delayline \cite{yuan2010} for frequency alignment, while the latter can be unstable because the transient reference is derived separately from the capacitive response.

Here we propose and experimentally demonstrate a simple, low-distortion ultra-narrowband interference circuit (UNIC)
that can suppress the capacitive response for a 1.25~GHz gated InGaAs/InP APD single photon detector.
The circuit is an asymmetric radio-frequency (RF) interferometer. One of its arms contains a narrow band pass filter (BPF) based on surface acoustic wave resonator (SAW) to retrieve the fundamental wave of the gating signal.
The filtered wave then interferes destructively with the same frequency component transmitted via the other arm through a coupling module, thereby eliminating the capacitive response.
This interference occurs over the narrow band, so it can provide a broad
and continuous pass band in frequency domain to maintain the avalanche signal with little distortion.
This allows to achieve ultra-low afterpulsing probabilities and excellent jitter performance at high detection efficiencies from two InGaAs APD's that exhibit capacitive responses of very different amplitudes.

\section*{Detector characterisation setup}

\begin{figure*}[htb]
\centering
\includegraphics[width=.6\linewidth]{Figure1.pdf}
\caption{
\textbf{(a)} Single-photon characterisation setup for 1.25~GHz sinusoidally gated InGaAs/InP APDs using UNICs for avalanche impulse readout;
\textbf{(b)} Histogram of the photon detection events measured by the characterisation setup \textbf{(a)} on an InGaAs APD detector that was regulated at a temperature of 30~$^\circ$C. The photon detection peak exhibits a 30~dB width of 650~ps.
AMP: amplifier; APD: avalanche photodiode; BSF: band stop filter;  DISC: discriminator;
SG: signal generator; TDC: time-to-digital converter; UNIC: ultra-narrowband interference circuit; VOA: variable optical attenuator.
}
\label{fig:001}
\newpage
\end{figure*}

Figure~\ref{fig:001}(a) shows our single photon characterisation setup for InGaAs APDs. A 1550~nm passively mode-locked laser serves as the light source and provides stable short pulses of 0.5~ps duration at a repetition rate of 10~MHz.
The laser output power is monitored by an optical power meter (EXFO FTB-1750) and its pulse intensity is set by a variable optical attenuator (VOA, EXFO FTB-3500) to 0.1 photon/pulse at the fiber input of APD under test. It provides a 10~MHz reference to a signal generator (SG) for synthesising a 1.25~GHz sinusoidal wave with up to 27 V voltage swing.
In combination of a suitable DC bias, this AC signal periodically gates the APD above its breakdown voltage ($60 - 70$~V) to achieve single photon sensitivity.
The APD output is processed by the readout module consisting of two identical 1.25~GHz UNIC's, one 2.5~GHz band stop filter (BSF), and three RF amplifiers (AMPs). Amplification of the raw APD signalis is useful as it prevents weak avalanche signals from falling below thermal noise by attenuation of the first UNIC.
The readout signal is discriminated by a discriminator for avalanches before feeding to a time-digital-converter (TDC) with a dead time of
2~ns for time-resolved photon counting.
Figure~\ref{fig:001}\textbf{(b)} is a typical histogram obtained with this setup.

APD under test is temperature-regulated using their integrated thermal-electric cooler, which is driven by a temperature controller (Thorlabs TED200C).
A source-measure unit (Keithley 2635B) provides the DC bias and simultaneously monitors the current flowing through the APD.  In characterising the maximum count rate, we replace the 10~MHz laser with a continuous-wave distributed feedback laser (DFB) laser, the output of which is carved into 1.25~GHz, 50~ps pulse train using an intensity modulator.
We use a high speed digital oscilloscope to record the detector output and extract the count rate through digital discrimination in software. The oscilloscope method is carefully calibrated at low count rate regimes to be consistent with the hardware discriminated result using the photon counter (Stanford Research SR400).

The setup is able to measure dark count probability, afterpulsing probability, detection efficiency, maximum count rate, avalanche charge and time jitter. With no performance screening, two fiber-pigtailed APDs from different manufacturers were used in this study, named APD\#1 and APD\#2 respectively.
\section*{Ultra-narrowband interference circuit (UNIC)}

\begin{figure}[ht]
\centering
\includegraphics[width=.8\linewidth]{Figure2.pdf}
\caption{\textbf{(a)} Schematic for ultranarrow interference circuit (UNIC); \textbf{(b)}  Transmission spectrum of a heroic UNIC PCB; Inset: Magnified view for region of 1.24 -- 1.26~GHz. \textbf{(c)} Raw capacitive responses from APD\#1 (top) and APD\#2 (bottom) under identical 27.0~V V$_{p-p}$ gating; \textbf{(d)} Recovered avalanche impulses.  ATT: attenuator; SAW BPF: surface acoustic wave band pass filter.
}
\label{fig:002}
\end{figure}

With sub-nanosecond gating,  a photon induced avalanche is an impulse and has a wide spectrum.  On the other hand, the capacitive response is periodic and has its most energy concentrated at the gating frequency or its higher harmonics.  This spectral difference allows frequency-dependent signal processing to remove the capacitive response and keep the wide-band impulses intact.
Figure~\ref{fig:002}\textbf{(a)} shows a circuit diagram of UNIC. It is an RF interferometer containing two couplers of 9:1 power splitting ratio, a $\pi$-resistive  attenuator (ATT) and surface acoustic wave  band pass filter.
Two of the ports are terminated by 50~$\Omega$ resistors.
The SAW BPF features a central frequency of 1.25~GHz, 20-dB passband of 35~MHz, insertion loss of 3~dB, and group delay of 34~ns. It filters out the fundamental wave of the gating frequency, which then interferes with the APD signal transmitted through the other arm.
The attenuation and differential delay are set to enable destructive interference for the 1.25~GHz frequency component at the UNIC output port. The UNIC differential delay ($\Delta t$) meets the condition below
\begin{equation}
    \Delta t = T_g^{SAW} + \delta t
             = (N+1/2)/f_g,
\end{equation}
\noindent where $T_g^{SAW}$ is the group delay of the SAW BPF, $\delta t$ the delay caused by the track length difference between two arms, $f_g = 1.25$~GHz the APD gating frequency, and $N$ is an integer number.  For a compact circuit, we choose $\delta t$ to be less than the half-wave of the gating signal. With the SAW device used, $N = 42$ and $\delta t = 155$~ps. The resulting UNIC unit has a small footprint of $38 \times 15$~mm$^2$ on printed circuit boards (PCBs).

The large $T_g^{SAW}$ brings two additional benefits. Firstly, it substantially increases the PCB manufacturing tolerance, as a 0.5~mm deviation in the RF track length will just alter the circuit central frequency by less than $10^{-4}$.  This eliminates the requirement of an adjustable delayline which is required in a self-differencing circuit for precise frequency alignment. Secondly, it helps to produce an ultra-narrow band rejection at its designed frequency.  Figure~\ref{fig:002}(b) shows the measured transmission spectrum  (S21 parameter) of our heroic UNIC PCB, and its inset expands the frequency section of 1.24 -- 1.26~GHz to show the narrowness of the insertion loss dip in the close proximity of the resonance frequency of 1.25~GHz.  The dip of the heroic (typical) PCB features a loss of -95~dB (-80~dB), representing a suppression of 80~dB (65~dB) as compared with the background loss for other frequencies under 2~GHz.
The dip has a 30~dB linewidth of merely 30~kHz, thus ensuring crucial suppression of the APD gating signal without overly distorting the avalanche signals.
The background loss of about 14~dB is caused mainly by the 9:1 couplers and can be reduced in future with more balanced splitters.

Cascading two UNIC's enables a stable 100~dB suppression of the primary gating frequency and thus provides sufficient performance redundancy. Their attenuation to the avalanche signal is compensated by using RF amplifiers (Fig.~\ref{fig:001}\textbf{(a)}). Second order harmonics (2.5~GHz) is suppressed by a band stop filter of conventional LC  design.
Figure~\ref{fig:002}\textbf{(c)} shows raw outputs from two different APD's under identical sinusoidal gating. Their respective capacitive responses are measured to be 0.42~V and 1.75~V.  Despite their 4 times  differences,  UNIC's can successfully reject the sinusoidal responses and retrieve avalanches with excellent signal-to-background ratio, as shown in Fig.~\ref{fig:002}\textbf{(d)}.  For APD\#2, we just adjusted the gain of the first AMP to avoid saturation and distortion.

\section*{Results and discussion}

Time-resolved photon counting allows precise extraction of the net photon detection efficiency ($\eta_{net}$) and the afterpulsing probability ($P_A$), which is defined as the ratio of the total afterpulses per photon counting event.
Figure~\ref{fig:001}\textbf{(b)} shows a histogram of avalanche events measured for APD\#1 under 10~MHz pulsed excitation of 0.1~photon/pulse.
The illuminated peak has a full-width of 1/1000 maximum (30~dB width) of just 650~ps,  which is shorter than the gating period of 800~ps and thus allows low-error clock number assignment that is essential for high speed QKD. The count at non-illuminated gates arise from detector dark count and afterpulses and is more than 3 orders of magnitude lower than that of the illuminated gate.
We extract quantities of $P_I$ and $P_{NI}$, \textit{i.e.}, the respective counting probabilities for each illuminated and non-illuminated gate. With a separate measurement of the detector dark count probability ($P_D$), we calculate the afterpulsing probability using the standard method \cite{Yuan2007,Namekata09},
\begin{equation}
	P_A  = \frac{(P_{NI} - P_{D}) \cdot R}{P_I - P_{NI}},
\end{equation}
\noindent where $R = 125$ here is the ratio of the  gating frequency (1.25~GHz) to the laser illumination (10~MHz).
Excluding the dark and afterpulse count probabilities, the net single photon detection efficiency is given by~\cite{Comandar2015}
\begin{equation}
\eta_{net} = \frac{1}{\mu}\mathrm{ln}\frac{1-P_{NI}}{1-P_{I}},
\end{equation}
\noindent where $\mu$ is the average incident photon number per illumination pulse.

\begin{figure}[t]
\centering
\includegraphics[width=.8\linewidth]{Figure3.pdf}
\caption{Dark count probability (top) and afterpulse probability (bottom) as a function of photon detection efficiency of \textbf{(a)} APD\#1  and \textbf{(b)} APD\#2 measured for several different temperatures.
}
\label{fig:003}
\newpage
\end{figure}

Figure~\ref{fig:003} shows the characterisation results for APD\#1 and APD\#2.
We fixed the amplitude of the 1.25~GHz sinusoidal signal at 27.0~V, and measured the relevant parameters as a function of the applied direct current (DC) bias, but for clarity the results are plotted as a function of the net detection efficiency ($\eta_{net}$). Each device was measured at several different temperatures, while APD\#2 could reach only a narrower temperature range due to its cooler compatibility with the temperature control driver.
Qualitatively, two devices behave similarly.
Both dark count and afterpulsing probabilities increase with photon detection efficiency, and exhibit opposite dependencies on temperature.  For both APDs at $\eta_\mathrm{net} = 30~\%$, the afterpulsing probabilities are less than 2.3~\% at their lowest measurement temperatures with corresponding dark count probabilities of $1.25 \times 10^{-6}$ and $1.6 \times 10^{-6}$ for APD\#1 (-30~$^\circ$C) and APD\#2 (-20~$^\circ$C), respectively.
Moreover, our UNIC-APDs can offer record low afterpulsing probabilities, as summarised for APD\#1 in Figure~\ref{fig:004}.  At -30~$^\circ$C, APD\#1 is able to achieve 5~\% and 21.2~\% detection efficiencies at 0.5~\% and 1.0~\% afterpulsing probabilities.  At these afterpulsing probabilities, the maximum detection efficiency increases with temperature and reaches 25.3~\% and 34.2~\% at 30~$^\circ$C. At 5.9~\% $P_A$, APD\#2 has a detection efficiency of 50~\% efficiency at 30~$^\circ$C and dark count probability of $1.1 \times 10^{-4}$.

\begin{figure}[t]
\centering
\includegraphics[width=.8\linewidth]{Figure4.pdf}
\caption{Temperature dependencies of photon detection efficiency for APD\#1 at the given afterpulsing probabilities of 0.5 \% (blue) and 1 \% (red).}
\label{fig:004}
\newpage
\end{figure}

\begin{figure}[b]
\centering
\includegraphics[width=.8\linewidth]{Figure5.pdf}
\caption{Maximum count rate (blue) and photoncurrent (red) \textit{vs } incident flux  for APD\#1.
}
\label{fig:005}
\newpage
\end{figure}

The maximum count rate is a crucial parameter for a number of applications, for example, high bit rate QKD \cite{yuan2018} and rapid phase tracking in twin-field QKD \cite{lucamarini18,zhou22}.
To determine their maximum count rates, we used a DFB laser transmitting at 1.25~GHz as the illumination source and measure the count rate as a function of photon flux.
Figure~\ref{fig:005} shows an exemplar result obtained from APD\#1 at a temperature of
30~$^\circ$C with its detection efficiency set to 25.3~\% in the low flux regime.  The detector maintains a linear dependence with incident flux for count rates exceeding  100~MHz, while a maximum count rate of 700~MHz is obtained at the few photons/pulse regime.
We attribute the high count rate to the UNIC's ability of removing the capacitive response and thus allowing discrimination of faint avalanches.  From the accompanying current measurement, we extract an avalanche charge of 38~fC, comparable to the best value of 35~fC \cite{yuan2010} obtained with the photocurrent measurement method. The ability to detect such weak avalanches ensures low afterpulsing probabilities in our UNIC-APDs.
APD\#2 was measured to have a similar avalanche charge as that of APD\#1.  When setting its  efficiency to 50~\%,  APD\#2's avalanche charge rose to 65~fC due to stronger bias applied. Nevertheless, it was still able to achieve a maximum count rate of 600~MHz.

\begin{table*}[h]
		\caption{\label{tab:Comparison} Performance comparison of sub-nanosecond gated  InGaAs detectors using different types of readout circuits.}
        \setlength{\tabcolsep}{0.4mm}{
	\centering
	\begin{tabular}{ccccccc}
		\hline\hline
		& $ P_{\mathrm{A}}$(\%)& $\eta_\mathrm{net}$ (\%)&$P _{\mathrm{D}}$ ($ \mathrm{gate}^{\mathrm{-1}}$)&T ($^{\circ}$C) &   $f_g$ (GHz)& Readout Method
		\\
		\hline	\hline

		This work &  1.0  &21.2 & 5.4$\times10^{-7} $  & -30 &1.25 &	UNIC \\
        \hline

		He \textit{et al}\cite{He2017} &1.0  &20.7 & 7.6$\times10^{-7}$     &-30 &1.00  &low-pass filter +\\ & & & & & & variable width discriminator  \\
		\hline

		Tada \textit{et al}\cite{Tada2020} &1.8 &27.7 & 8$\times10^{-7}$   &-35 &1.27
        & band stop filter		\\
		\hline

		Fang \textit{et al}\cite{Fang2020}  &2.5 &20 & 1.1$\times10^{-6}$  &-30 &1.25 &low-pass filter  \\
        \hline

		Comandar \textit{et al}\cite{Comandar2015} &2.9 &20 &  1.0$\times10^{-6}$ &-30 &1.00 &self-differencing 	\\
		\hline

  	Liang \textit{et al}\cite{Liang2019} &4.5 &20 &  3.2$\times10^{-6}$   &-30          &1.25  &reference subtraction \\

   	\hline\hline

	\end{tabular}}

\end{table*}

Table~\ref{tab:Comparison} compares our results with those gigahertz-gated detectors equipped with different readout circuits. For impartiality,  we list just data measured at a fixed temperature of -30~$^\circ$C whenever possible.  Here, our UNIC-APD achieved an impressive 1\% afterpulsing probability at $\eta_\mathrm{net} =21.2$~\%, considerably outperforming most other methods among filtering\cite{Fang2020,Tada2020}, self-differencing \cite{Comandar2015} and reference subtraction \cite{Liang2019}.
In terms of detection efficiency, our result improves marginally over the previous best\cite{He2017}, but which was achieved with help of an uncommon variable width discriminator to mitigate signal distortion by excessive filtering.  We attribute the outstanding performance of our detectors to low-distortion signal processing by UNIC's.

It is useful to compare our UNIC-APDs with detectors deployed in QKD systems.  In the QKD system optimised for secure key rates (SKRs) \cite{yuan2018},  the room-temperature self-differencing detectors featured $f_g = 1$~GHz,  $\eta_\mathrm{net} = 31$ \%, $P_A = 4.4$\% and $P_D = 2.25 \times 10^{-4}$ and a SKR of 13.72~Mb/s over a 2~dB channel was obtained.  Our UNIC-APD could outperform in all these parameters. At 30~$^\circ$C and with $P_A =4.4$~\%, APD\#2 offers a higher efficiency of 49~\% efficiency and twice lower dark count probability of $9.4 \times 10^{-5}$, see Fig.~\ref{fig:003}\textbf{b}.  Combined with its high count capability, UNIC detectors are expected to allow a SKR exceeding 25~Mb/s over the same channel loss.  This provides an interesting technological path towards 100~Mb/s via wavelength multiplexing.

\section*{Conclusion}
To summarise, we have developed a novel approach of using UNICs for reading out avalanche signals from 1.25~GHz sinusoidally gated InGaAs APDs.
UNIC-APDs were characterised to exhibit excellent performance across the temperature range of $-30$ -- 30~$^\circ$C, and can offer \textgreater20~\% detection efficiency at an ultra low afterpulsing probability of 1~\%.   This performance, together with the circuit's compactness and manufacturing tolerance, will allow UNIC-APDs a considerable potential in QKD applications.

\begin{backmatter}
\bmsection{Disclosures}
The authors declare that there are no conflicts of interest related to this article.

\bmsection{Data availability}
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
\end{backmatter}


\section{Introduction}
Cricket is a bat and ball game played between two teams. This was found in early 16th century. There are different formats in cricket which are the tests which consists of two innings for each team and played in five days, ODI which gives each team 50 overs to bat and finishes in one day and T20 format which gives 20 overs each for team. T20 was introduced in 2005 and became much popular among various audience. The main reasons were the small duration of the match and the rules which are more favourable to batsmen compared to other formats. Indian Premier League (IPL) is such a cricket event where different teams based on different cities in India compete each other. IPL is based on the T20 format of cricket. IPL started in 2008 and have been a celebration among cricket followers around the globe. IPL has been earning lot of profit across the years.Players from different countries have been playing for these teams which increases the entertainment value as well as quality of cricket. In the latest IPL season(2020) BCCI earned a total profit of Rs 4000 crore. Along with increasing popularity of IPL, the analysis of players as well as teams across various conditions have became important among various sections.

Due to various factors like money, fan following, broadcasting, the entertainment value from matches is very important. This can be dependent on team selection, historical data of ground as well as the players, head to head to matches etc. This analysis can be used to analyse the effect of each player in a team combination with respect to opposition as well as the ground. Another similar use case of the analysis is in betting. We can find out fantasy points for each players from the model and provide good combinations to use in the betting platforms. As the entertainment value of the match highly depends on the batting performances, this analysis can help the pitch curators on making the decisions which will help them to create a bigger fan following as well in better marketing.

An inherent limitation to the current machine learning literature for sports data analysis is the lack of feature representations for players that contains necessary information on inter-player and inter-team relations. In this study we explore the use of vector embedding representations and the possible mediums to evolve them efficiently. Since the task has an inherent data scarcity (in comparison to the natural scenarios in which vector embeddings are used widely), we formulate a siamese-network\cite{schroff2015facenet} based representational learning framework that could effectively handle these limitations.

\section{Related Work}\label{Related Work}
Data analytics is used in various aspects of cricket such as run prediction, player performance evaluation, team management, strategy formation etc.. There is a great demand for those algorithms that can perform the above tasks. IPL is a cricket event which is short in format and a lot of money involved. In India the followers  of  cricket  are  also  followers of  statistical  records. Thus the analysis of a league like IPL becomes more
important. The following are some studies related to cricket which are reported in literature.

\cite{ref2}This article is concerned with the simulation of one-day cricket matches. Given that only a finite number of outcomes can occur on each ball that is bowled, a discrete generator on a finite set is developed where the outcome probabilities are estimated from historical data involving one-day international cricket matches. The simulator allows a team to easily investigate the results of making changes to the batting and bowling orders.

\cite{ref1}This paper presents a mathematical model that can be used for prediction  of the results of the matches prior to the match based on the knowledge of past matches, playing eleven and the toss result. In this work, three different models have
been constructed based on three approaches. Machine learning techniques have been utilized with advantage for this purpose. The outcome of a match is predicted  by taking a majority vote of these three models.

\cite{ref3}In this paper, prediction of the performance of players as how many runs will each batsman score and how many wickets will each bowler take for both the teams is performed. Both the problems are targeted as classification problems where number of runs and number of wickets are classified in different ranges. Random Forest turned out to be the most accurate classifier for both the data-sets with an accuracy of 90.74\% for predicting runs scored by a batsman and 92.25\% for predicting wickets taken by  a bowler.

\cite{lamsal2020predicting}A multivariate regression based solution is proposed to calculate points for each player in the league and the overall weight of a team is
computed based on the past performance of the players who have appeared most for the team. Multi-layer Perceptron model gave the highest  accuracy of 71.66\% in predicting the outcome of the match.

\cite{ref5}Proposed a novel Recurrent Neural Network model which can predict the win probability of a match at regular intervals given the ball-by-ball statistics.

\pagebreak

\section{The Proposed Method}\label{Method}

We split our overall methodology into three sub-components, initially we learn the Embedding representations of players independently of other features. Next we use the learned embeddings from the model along with other commonly available pre-match data features to predict the overall run-rate. Finally we include a separate branch in the network to analyse the corresponding pre-match pitch report to investigate it's impact on overall prediction metrics.

\subsection{Player Embedding Model}\label{player_emebed_Model}

The model composes of two separate embedding maps that represents the batting and bowling characteristics of each player. The embedding vectors are a normalized set of size 64. We mean-pool the embeddings from each map and pass the same through a fully connected layer with ReLU activation so as to obtain a latent representation of each team's characteristics. A joint representation of the batting and bowling team corresponding to an innings is obtained by concatenating the latent vectors so obtained from each branch. The joint representation so obtained is passed through a series of fully connected layers (with ReLU non-linearity) attached to a prediction or representational head. The player embedding model is trained independently so as to avoid induced bias from other data features.

\begin{figure}[!htb]
    \centering
    \includegraphics[width=\textwidth]{NN_Design__IPL____Embed.png}
    \caption{\textbf{Complete architecture of the player embedding model.} The final layer of the given model could change with the training setting used.}
    \label{fig:player_embed}
\end{figure}

\subsection{Prediction Model}\label{Pred_model}

Since player lineups aren't the only deciding factor in predicting match metrics, we investigate the impact of other commonly available match and timeline features on match metrics. In this section we build a prediction model that bases it's prediction on the analysis of joint representations formed from the latent vectors formed from commonly available pre match data and player representations. The bowling and batting embedding matrices used are obtained from the player embedding model \ref{player_emebed_Model}, the embedding representations are frozen during training. We increase the no of fully connected layers in each branch that processes the player embeddings as compared to the player embedding model, the same is done to evaluate the generality of the embedding representations formed.

\begin{figure}[!htb]
    \centering
    \includegraphics[width=1\textwidth, height=13cm]{NN_Design__IPL____Predict1.png}
    \caption{\textbf{Complete Architecture of the Prediction model}. The model is used as a medium for evaluating the effectiveness of the overall formulation of using embedding representations}
    \label{fig:prediction_model_1}
\end{figure}

\subsection{Prediction Model with pitch reports}\label{Pred_model_pitch}

Pitch reports are an essential source of pre-match information, that could prove vital in effective overall modelling . To evaluate the impact of the same we build a separate model which incorporates information from pitch reports into the joint representations formed. The pitch reports are prepossessed using a sentence vectorizer prior to model input.

\begin{figure}[!htb]
    \centering
    \includegraphics[width=1\textwidth, height=15cm]{NN_Design__IPL____Predict2.png}
    \caption{\textbf{Complete Architecture of the Prediction model with pitch reports}. A Joint representation is formed from the concatenation of latent vectors obtained from each of the different data branches.}
    \label{fig:prediction_model_2}
\end{figure}

\subsection{Representational Learning}

Apart from using the traditional cross-entropy loss to build the classifier, we experiment with a representational learning framework which generates meaningful data representations for classification by latent vector comparisons. The objective function used is the contrastive loss \cite{schroff2015facenet}, the standard form of which is as shown in Equation \ref{eq1}.

\begin{equation}
L(W, (Y, \vec{X_1}, \vec{X_2})^i) = (1 - Y)L_S(D_W^i) + Y L_D(D_W^i)
\label{eq1}
\end{equation}

\noindent
$Y$ is the binary label that indicates if two data samples are acquired from a common category or not and $D_W$ is the parameterized distance function as shown in Equation \ref{eq2}.
\begin{equation}
D_W(\vec{X_1}, \vec{X_2}) = \left\|G_W(\vec{X_1}) - G_W(\vec{X_2}) \right\|_2
\label{eq2}
\end{equation}

\noindent
For our purposes we choose $D_W$ as the euclidean norm, $L_S$ = $D_W^i$, and $L_D$ as the $min(m - D_W^i, m)$. Where $m$ is the margin parameter. The form of the final loss function is  shown in Equation \ref{eq3}.

\begin{equation}
L(W, (Y, \vec{X_1}, \vec{X_2})^i) = (1 - Y)(D_W^i) + Y (min(m - D_W^i, m))
\label{eq3}
\end{equation}

\section{Experiments and Results}\label{Results}
\subsection{Dataset} \label{Dataset}
The data which contains the team lineup, venue, date and run-rate was collected from the official IPL webpage~\cite{iplt20}. It includes the innings wise data from the year 2012 to 2019. sportskeeda.com~\cite{pitch1}, espncricinfo.com~\cite{pitch2} and cricbuzz.com~\cite{pitch3} were the sources for the pitch report data. K-Means clustering~\cite{cluster} was performed on the run rate data to to divide into three classes. Elbow method~\cite{elbow} was used to find the optimal number of classes. Further Hierarchical clustering was done to obtain better class distribution, splitting the majority class into two sub classes. The following figure \ref{fig:picture} shows the  average dispersion across different number of clusters.

\begin{figure}[!htb]
    \centering
    \includegraphics[width=0.8\textwidth, height=6cm]{elbow_curve.png}
    \caption{\textbf{The Elbow curve obtained for initial clustering}. The optimal initial no of clusters to be used can be infer-ed from the shown elbow curve and can be seen to be = '3'.}
    \label{fig:picture2}
\end{figure}

\noindent
Four classes were obtained after the hierarchical clustering of overall run rate per innings. The following pie diagram \ref{fig:picture} shows the individual class distributions and the centroid of respective clusters. The dataset so obtained has overall class balance.

\begin{figure}[!htb]
    \centering
    \includegraphics[width=0.7\textwidth, height=5cm]{dataset.png}
    \caption{\textbf{Final class distribution}. The centroid of each corresponding class is as marked in the figure}
    \label{fig:picture}
\end{figure}

\noindent
The testing set is randomly sampled each time during k-fold cross validation and confidence intervals estimations. The overall data distribution used in this study remains invariant.

\subsection{Experimental Setup}

A separate testing set was created by randomly sampling 10 data points from each run-rate class obtained post hierarchical clustering. We conduct our experiments as two separate settings, Firstly we train the player embedding model by basing it's objective to predict the relevant class using the standard cross entropy loss. The trained set of embeddings so obtained are used as inputs during the training of the prediction model, the objective function used for the same is also cross-entropy.

Next we train the embedding model using the contrastive loss to enforce meaningful representations, the embedding matrix so obtained is used to train a the prediction model whose objective is also the minimization of a contrastive loss. For evaluation we perform similarity analysis between the obtained representation of a test data point and the representation matrix corresponding to the train set.

Finally for both the settings given above, we add a separate branch of input in the prediction model so as to include the pre-match pitch report data. The pitch report is processed into a representational embedding using a pretrained sentence BERT model \cite{reimers2019sentence}. We thoroughly evaluate the model performance in both settings and the impact pitch-data has in predicting the overall run-rate class.

All experiments were conducted in the Pytorch framework with a CUDA backend on NVIDIA Tesla V100 GPU. The prediction model is trained in batch-sizes of 64 using the Adam optimizer. The learning rate used is 10\textsuperscript{-3}, the player and pitch embedding weights are frozen during the training of the prediction models.

\pagebreak

\subsection{Results}

\begin{figure}[!htb]
\begin{minipage}[b]{0.48\textwidth}
\includegraphics[width=\linewidth]{conf_sdl_pitch.png}\\
\subcaption{Confusion matrix with pitch data}\label{fig:a0}
\end{minipage}%
\begin{minipage}[b]{0.48\textwidth}
\includegraphics[width=\linewidth]{conf_sdl.png}\\
\subcaption{Confusion matrix without pitch data}\label{fig:b0}
\end{minipage}

\caption{The shown confusion matrices shown are obtained after training the prediction models till point of divergence using the categorical cross entropy loss.}\label{fig5}
\end{figure}

\begin{figure}[!htb]
\begin{minipage}[b]{0.48\textwidth}
\includegraphics[width=\linewidth]{conf_rep_pitch.png}\\
\subcaption{Confusion matrix with pitch data}\label{fig:a1}
\end{minipage}%
\begin{minipage}[b]{0.48\textwidth}
\includegraphics[width=\linewidth]{conf_rep.png}\\
\subcaption{Confusion matrix without pitch data}\label{fig:b1}
\end{minipage}

\caption{The shown confusion matrices shown are obtained after training the prediction models in the representational learning framework for 2000 epochs}\label{fig6}
\end{figure}

\noindent
The training setting that used cross-entropy loss acquired an accuracy of 22.5\% (CI -95\%) when pitch report data was not included (corresponding  confusion matrix shown in Figure \ref{fig:b0}), the model that based it's prediction with all of the prior's input and pitch report data achieved an accuracy of 30\% (CI -95\%) (corresponding  confusion matrix shown in Figure \ref{fig:a0}).
The representational learning framework showed a considerable increase in performance and achieved an accuracy of 90\% (CI -95\%) when pitch report data was not included (corresponding confusion matrix shown in Figure \ref{fig:b1}) and an accuracy of 95\% (CI -95\%) in the model setting that included pitch reports (corresponding confusion matrix shown in Figure \ref{fig:a1}).

\section{Discussion}\label{Discussion}
The setting that used cross-entropy loss achieved feeble accuracies as compared to the representation learning setting. We speculate the reason for the same being the inability of cross-entropy setting to learn meaningful vector embeddings for player feature representations. The model setting that included the pitch reports showed considerable increase in performance, the same maybe attributed by the similarities captured in the pitch embedding vectors. The cross-assessment of pitch properties could thus be speculated to be crucial for overall score-performance analysis. The prediction inaccuracies in intermediate classes could be associated with the lack of efficient input features to fully represent the match characteristics.

\section{Future Works}
Although the present work was focused entirely on efficient modeling of player representations we speculate that their might be overall bias to the current setting due to data scarcity. The semantic meanings of the player representations could be analysed and made robust by employing the same for better prediction tasks. We believe that the model architecture could be tuned further, maybe by introducing a convolutional mode of analysis for player embeddings and a relative weighted concatenation for joint representations.

\section{Conclusion}\label{Conclusion}
In this paper, we proposed a representational learning framework for optimal cricket data analysis. We observe that meaningful data representations could be obtained for players using the same. The performance of prediction models trained using cross entropy objective was proven to be feeble as compared to the proposed framework. We hope our work would act as a motivation for future research using  deep-representational framework and learnable input representations for sports analytics tasks.

\printbibliography

\section*{Code Availability}
The custom Python based code used in this study is available from the corresponding author upon reasonable request. Any commercial use including the distribution, sale, lease, license, or other transfer of the code to a third party, is prohibited.

\end{document}


\section{Introduction}
\label{sec:Intro}

The current 5G networks are designed to support a wide range of services, including \gls{embb}, \gls{urllc} and \gls{mmtc}, which in turn will support an important growth in the number of applications.
This upsurge in novel services and applications is expected to also happen with 6G \cite{6g_vision}.
This heterogeneity of wireless networks may represent a challenge to protocol design.
Therefore, protocols tailored to specific applications may perform better than general-purpose solutions \cite{pasandi2021}.

\Gls{ml} can be used to design protocols, boost the network capacity \cite{hoydis20216g} and reduce the efforts and costs for future standardization \cite{6g_standardization}.
It is possible to view a protocol as the language of a network, since the network nodes have to negotiate how to transmit data by exchanging messages.
Then, the idea of emerging a new protocol would be similar to emerging communication between the network nodes.
\Gls{rl} is one category of \gls{ml} that provides the means to reach this goal.

During the last years, research on how to emerge communication in order to achieve collaboration between multiple agents received a growing attention \cite{lazaridou2020emergent}.
This growth partly relies on recent advances in \gls{marl} for cooperative problems \cite{dafoe2020open}.
Learning to cooperate by leveraging communication is about teaching agents to either learn existing natural languages or to emerge a fully new communication protocol that would help them collaborate to solve a task.

\textbf{Contribution:}
Our proposal is to leverage cooperative \gls{marl} augmented with communication to allow a fully new \gls{mac} protocol to emerge.
The idea of learning a given protocol has already been addressed in a previous work \cite{alvaro2021}, but to the best of the authors' knowledge, there is no previous work on studying the emergence of a new \gls{mac} protocol (signalling included) with \gls{marl}.
In the future, this idea may be used to develop application-tailored protocols that could perform better than the human-designed ones.

This work is structured as follows.
In Section~\ref{sec:rel-work}, we briefly review the literature.
In Section~\ref{sec:background}, we give a short background overview of \gls{marl} and the algorithm used in this work.
Section~\ref{sec:system-model} describes the system model used and in Section~\ref{sec:proposed}, we present a new framework allowing the emergence of \gls{mac} protocols with \gls{marl}.
Finally, Section~\ref{sec:simulation} illustrates the performance of our algorithm with our numerical results, where we compare the proposed solution with a baseline.
The main conclusions are drawn in Section~\ref{sec:conclusion}.

\section{Related Work}
\label{sec:rel-work}

Several papers have applied \gls{rl} to the \gls{mac} layer, mostly to solve \gls{rrm} problems such as scheduling (\cite{leasch, rrs_deep_pointer}) and  dynamic spectrum access (\cite{rl_for_spectrum_access, wong2020dynamic}).

In \cite{alvaro2021}, \gls{marl} is used to learn a predefined protocol and a new channel access policy.
This is done by having a \gls{bs} which uses a predefined protocol while the \glspl{ue} are \gls{rl} agents.
The \glspl{ue} are trained to learn the signaling and how to access the channel without any prior knowledge.
This way, they can learn their own channel-access policy, while respecting the target signaling policy.
However, in this case the agents only learn to use an already known \gls{mac} signaling, rather than developing a new one.

In \cite{pasandi2021}, a framework to design a protocol is proposed by considering the different functions a \gls{mac} protocol must perform.
An \gls{rl} agent designs a protocol by selecting which building function to use according to the network conditions.
However, in this case, the \gls{rl} agent still has a prior knowledge due to the use of the predefined protocol functions.

In \cite{learning_phy, remote_rl}, cooperative \gls{marl} is used to emerge a coding scheme by joint learning of communication and cooperation to solve a task with the help of a noisy communication channel.
The proposition of both works is to emerge a coding scheme that is tailored to the application.
None of these works address the question of learning a new signaling protocol.

\section{Background on MARL}
\label{sec:background}

\Gls{rl} is an area of \gls{ml} that aims to find the best behavior for an agent interacting with a dynamic environment in order to maximize a notion of accumulated reward \cite{Bishop07}.
The goal of the \gls{rl} agent is to find the best policy, which is the mapping of the perceived states to the actions to be taken.
The action-value function $Q^{\pi}(s_t,a_t)$, also known as Q-function, is the overall expected reward for taking action $a_t$ in state $s_t$ and then following a policy $\pi$.

MARL is an extension of RL for to \gls{mas}, where multiple agents interact with a system, i.e the environment.
In this work, we use the \gls{dec-pomdp} formulation \cite{oliehoek2008optimal}, augmented with communication.
A \gls{dec-pomdp} for $n$ agents is defined by the global state space $\mathcal{S}$, an action space $\mathcal{A}_1 , \ldots , \mathcal{A}_n $, and an observation space  $\mathcal{O}_1 , \ldots , \mathcal{O}_n $ for each agent.
In \gls{dec-pomdp}, the agent observation does not fully describe the environment state.
All agents share the same reward and the action space of each agent is subdivided into one environment action space and a communication action space.
The communication action represents the message sent by an agent and it does not affect the environment directly, but it may be passed to other agents.
This formulation is shown in Fig.~\ref{fig:marl-scheme}, where $o_i$ represents the observation received by the $i^{\mathrm{th}}$ agent, $r$ represents the reward, $a_i$ and $c_i$ represent the environment and communication actions, respectively.
In this work, the agent internal state $x_i$ may comprise not only the agent's current observation, but also previous observations, actions and received messages.

\begin{figure}[tb]
    \vspace*{3pt}
    \centering
    \includegraphics[width=\columnwidth]{figs/marl-diag.pdf}
    \caption{Cooperative MA-RL scheme with communication,}
    \label{fig:marl-scheme}
\end{figure}

\Gls{marl} introduces some new challenges, such as partial observability and non-stationarity \cite{MARL_Challenges}.
In this work, we adopt the \gls{maddpg} algorithm \cite{lowe2017multi}, an extension of the \gls{ddpg} algorithm \cite{ddpg} to multi-agent problems with \gls{ctde}.
It addresses the non-stationarity problem by using a centralized critic.
Each agent has an actor network that depends only on its own agent's state in order to learn a decentralized policy $\mu_i$ with parameters $\theta_i$.
During the training, each agent has a centralized critic that receives the agent states and actions of all agents in order to learn a joint action value function $Q_i (x,a)$ with parameters $\varphi_i$, where $x = (x_1, x_2, \ldots, x_n)$ is a vector containing all the agents' states and $a = (a_1, a_2, \ldots, a_n)$ contains the actions taken by all of the agents.

The critic network parameters $\varphi$ are updated by minimizing the loss given by the temporal-difference error
\begin{equation}
\label{eq.:critic}
    L^i \coloneqq \E_{x, a, r, x^{\prime} \thicksim \mathcal{D} } \left[ y^i - Q_i (x, a_1, \ldots, a_n ; \varphi_i)  \right]
\end{equation}
\noindent where $\mathcal{D}$ denotes the experience replay buffer in which the transition tuples $ (x, a, r, x^{\prime})$ are stored, $Q^{\prime}$ and $\mu^{\prime}$ represent the target critic network and the value of the target actor network, with parameters $\theta^{\prime}$ and $\varphi^{\prime}$, respectively, and $y^i$ is the temporal-difference target, given by
\begin{equation}
\label{eq.:td-error}
    y^i \coloneqq r + \left. \gamma Q^{\prime}_i ( x^{\prime}, a^{\prime}_1,  \ldots, a^{\prime}_n; \varphi^{\prime}_i )  \right|_{a^{\prime}_k = \mu^{\prime}_k (x_k)}
\end{equation}
\noindent where $\gamma$ is the discount factor.
The actor network parameters $\theta$ are updated using the sampled policy gradient
\begin{equation}
\label{eq.:actor}
    \nabla_{\theta_i} J = \E_{x, a \thicksim \mathcal{D} } \left[ \nabla_{a_i} Q_i (x,a) \nabla_{\theta_i} \mu_i (x_i) \mid a_i=\mu_i (x_i) \right] .
\end{equation}
The target networks parameters are updated as
\begin{equation}
\label{eq.:target-critic}
    \varphi^{\prime}_i  \leftarrow \tau \varphi_i + (1-\tau) \varphi^{\prime}_i
\end{equation}
\begin{equation}
\label{eq.:target-actor}
    \theta^{\prime}_i  \leftarrow \tau \theta_i + (1-\tau) \theta^{\prime}_i
\end{equation}
\noindent where $\tau \in \left[ 0 , 1 \right]$ is the soft-update parameter.
Smaller values of $\tau$ lead to slow target network changes and are generally preferred \cite{ddpg}.

\section{System Model}
\label{sec:system-model}

We consider a single cell with a \gls{bs} serving \gls{not:nUE} \glspl{ue} operating according to a \gls{tdma} scheme, where each \gls{ue} needs to deliver \gls{not:total-packets} \glspl{sdu} to the \gls{bs}.
We assume that each \gls{mac} \gls{pdu} contains only one \gls{sdu}.
The network nodes can communicate, i.e. exchange information, using messages through the control channels.
In the rest of this paper, we use the expressions \gls{ue} and \gls{bs} to refer to the \gls{ue} \gls{mac} agent and the \gls{bs} \gls{mac} agent, respectively.

The channel for the uplink data transmission is modeled as a packet erasure channel, where a \gls{tb} is incorrectly received with a probability given by a \gls{tbler}.
The \glspl{dcm} and \glspl{ucm} are transmitted over the \gls{dl} and \gls{ul} control channels, which are assumed to be error free and without any contention or collision.

We assume that the sets of possible \gls{dl} and \gls{ul} control messages have cardinality \gls{not:dl-vocabulary-size} and \gls{not:ul-vocabulary-size}, respectively.
For example, the \glspl{ucm} in an \gls{ul} control vocabulary of size $\gls{not:dl-vocabulary-size} = 8$ would have a bitlength of $\log_2 {\gls{not:dl-vocabulary-size}}=3$.
This exchange is shown in Fig.~\ref{fig:sys-model}, where dashed lines indicate control information and solid lines indicate user data.

\begin{figure}[tb]
    \vspace*{3pt}
    \centering
    \includegraphics[width=0.98\columnwidth]{figs/system-model.pdf}
    \caption{System model example with two \glspl{ue}.}
    \label{fig:sys-model}
\end{figure}

Each \gls{ue} has a transmission buffer of capacity \gls{not:buffer-size} \gls{mac} \glspl{sdu} that starts empty.
At each time step \gls{not:time-step}, a new \gls{sdu} is added to the buffer with probability \gls{not:p-arrival}, until a maximum of \gls{not:total-packets} \glspl{sdu} have been generated for each \gls{ue}.

At each time step \gls{not:time-step}, the \gls{bs} can send one control message to each \gls{ue} and each \gls{ue} can send one control message to the \gls{bs} while being able to send data \glspl{pdu} through the \gls{ulsch}.
Furthermore, the \glspl{ue} can also delete a \gls{sdu} from the buffer at each time step.

The transmission task is considered finished once all \glspl{sdu} are received and all transmission buffers are empty.
We define the goodput $G$ (in \glspl{sdu}/TTIs) as the number of \gls{mac} \glspl{sdu} received by the \gls{bs} per unit of time, without considering the retransmissions:
\begin{equation}
    G = \frac{N_{\mathrm{RX}}} {N_{\mathrm{TTIs}}}
\end{equation}
\noindent where $N_{\mathrm{RX}}$ represents the number of \glspl{sdu} received and $N_{\mathrm{TTIs}}$ is the total time taken to finish the transmission task.
The delivery-rate $\Gamma_{\mathrm{RX}}$ is the percentage of \glspl{sdu} correctly received by the \gls{bs}:
\begin{equation}
    \label{eq:delivery-rate}
    \Gamma = \frac{N_{\mathrm{TX}}}{\gls{not:total-packets} \gls{not:nUE}} \text{.}
\end{equation}

\section{Emerging a MAC Protocol with MARL}
\label{sec:proposed}

\subsection{MARL Formulation}

We can formulate the problem defined above as a \gls{marl} cooperative task, where the \gls{mac} layers of the network nodes (\glspl{ue} and \gls{bs}) are \gls{rl} agents that need to learn how to communicate with each other to solve an uplink transmission task.
In addition, the \gls{ue} agents need to learn when to send data through the \gls{ulsch} and when to delete an \gls{sdu}, in other words, to learn how to correctly manage the buffer.

In order to decide how to act, an agent needs to consider the messages received from the other agents. %
In addition, the \glspl{ue} also take into account their buffer status when taking actions, while the \gls{bs} takes into account the state of the \gls{ulsch}, i.e idle, busy or collision-free reception.

We use the following notations:
\begin{itemize}

    \item $\gls{not:obs}^{\mathrm{u}}_{t}$: Observation received by the $u$\textsuperscript{th} \gls{ue} at time step \gls{not:time-step}.
    \item $\gls{not:obs}^{\mathrm{b}}_{t}$: Observation received by the \gls{bs} at time step \gls{not:time-step}.
    \item $\gls{not:ul-message}^{\mathrm{u}}_{t}$: The \gls{ucm} sent from the $u^{\mathrm{th}}$ \gls{ue} at time step $t$.
    \item $\gls{not:dl-message}^{\mathrm{u}}_{t}$: The \gls{dcm} sent to the $u^{\mathrm{th}}$ \gls{ue} at time step $t$.
    \item $\gls{not:ue-action}^{\mathrm{u}}_{t}$: Environment action of the $u^{\mathrm{th}}$ \gls{ue} at time step $t$.
    \item $\gls{not:state}^{\mathrm{u}}_{t}$: Agent internal state of the $u^{\mathrm{th}}$ \gls{ue} at time step $t$.
    \item $\gls{not:state}^{\mathrm{b}}_{t}$: Agent internal state of the \gls{bs} at time step $t$.

\end{itemize}

The observation $\gls{not:obs}^{\mathrm{u}}_{t} \in \left\{ 0, \dots, \gls{not:buffer-size} \right\} $ is a integer representing the number of \glspl{sdu} in the buffer of the \gls{ue} $u$ at that time $t$.
Similarly, the observation $\gls{not:obs}^{\mathrm{b}}_{t} $ received by the \gls{bs} is a discrete variable with $\gls{not:nUE} + 2$ possible states:
\begin{equation}\label{eq.:base-obs}
    \gls{not:obs}^{\mathrm{b}}_{t} = \begin{cases}
    0, \text{ if the \gls{ulsch} is idle} \\
    \mathrm{u}, \text{ } \parbox[t]{.33\textwidth}{ if the \gls{ulsch} is detected busy with a single \gls{pdu} from \gls{ue} $\mathrm{u}$, correctly decoded} \\
    \gls{not:nUE} + 1, \text{ non-decodable energy in the \gls{ulsch}} \\
    \end{cases}
\end{equation}
\noindent where $\mathrm{u} \in \left\{ 0, \dots, \gls{not:nUE} \right\}$.
The environment action $\gls{not:ue-action}^{\mathrm{u}}_{t}  \in \{0, 1, 2\}$ is interpreted as follows:
\begin{equation}
\label{eq.:action}
    \gls{not:ue-action}^{\mathrm{u}}_{t} = \begin{cases}
    0 \text{: do nothing} \\
    1 \text{: transmit the oldest \gls{sdu} in the buffer} \\
    2 \text{: delete the oldest \gls{sdu} in the buffer} \\
\end{cases}
\end{equation}

We assume the episode ends when all the \glspl{sdu} are correctly received by the \gls{bs} or when a maximum number of steps \gls{not:episode-duration} is reached.
The reward given at each time step is:
\begin{equation}
\label{eq.:reward}
    r_{t} = \begin{cases}
    +\gls{not:reward-signal}, \text{ if a new \gls{sdu} was received by the \gls{bs} } \\
    -\gls{not:reward-signal}, \text{ } \parbox[t]{.25\textwidth}{ if a \gls{ue} deleted a \gls{sdu} that has not been received by the \gls{bs}} \\
    -1, \text{ else,}
\end{cases}
\end{equation}
\noindent where \gls{not:reward-signal} is a positive integer.
This choice of reward is possible by leveraging the \gls{ctde}.
During the centralized training, a centralized reward system can be used to observe the buffers of the \gls{bs} and \glspl{ue} in order to assign the reward.

\subsection{Training Algorithm}
\label{sec:solution}

The proposed \gls{rl} solution is based on the \gls{maddpg} algorithm \cite{lowe2017multi}.
Each entity of the system has its own actor network which outputs the action of an agent given its state.
Each agent also has a centralized critic network which outputs the Q-value given the actions and states of all agents.
The critic networks are only used during the centralized training.

The actor and critic networks have the same architecture; a fully connected \gls{mlp} with two hidden layers, of $64$ neurons each.
The activation function of all hidden layers is the \gls{relu}.

The agent state at time step $t$ is a tuple comprising the most recent $k$ observations, actions and received messages:
\begin{itemize}
    \item \Gls{ue} $\mathrm{u}$: $\gls{not:state}^{\mathrm{u}}_{t} = ( \gls{not:obs}^{\mathrm{u}}_{t}, \ldots , \gls{not:obs}^{\mathrm{u}}_{t-k} , \gls{not:ue-action}^{\mathrm{u}}_{t}, \ldots, \gls{not:ue-action}^{\mathrm{u}}_{t-k}, \gls{not:ul-message}^{\mathrm{u}}_{t}, \ldots, \gls{not:ul-message}^{\mathrm{u}}_{t-k} , \\ \gls{not:dl-message}^{\mathrm{u}}_{t}, \ldots, \gls{not:dl-message}^{\mathrm{u}}_{t-k} ) $
    \item \Gls{bs}: $ \gls{not:state}^{\mathrm{b}}_{t} = ( \gls{not:obs}^{\mathrm{b}}_{t}, \ldots , \gls{not:obs}^{\mathrm{b}}_{t-k} , \mathbf{\gls{not:ul-message}}_{t}, \ldots, \mathbf{\gls{not:ul-message}}_{t-k} , \mathbf{\gls{not:dl-message}}_{t}, \ldots, \mathbf{\gls{not:dl-message}}_{t-k} ) $, with $ \mathbf{\gls{not:ul-message}}$ and $\mathbf{\gls{not:dl-message}}$ containing the messages from all the \glspl{ue}.
\end{itemize}

In order to improve training of our \gls{maddpg} solution, we make use of parameter sharing \cite{foerster2016learning} for similar network nodes, in this case the \glspl{ue}.
Similarly to the original work \cite{lowe2017multi}, we use the Gumbel-softmax \cite{gumbel-softmax} trick to soft-approximate the discrete actions to continuous ones.
The Gumbel-softmax reparameterization also works to balance exploration and exploitation.
The exploration-exploitation trade-off is controlled by the temperature factor $\zeta$.

After training finishes, we have successfully trained a population of $\gls{not:repetitions}=32$ protocols.
We then select the protocol that performed best during the last $\gls{not:episodes-eval}=500$ evaluation episodes.
This selection step can be seen as a "survival of the fittest" approach because only one protocol of the population of \gls{not:repetitions} is chosen going forward.

\section{Results}
\label{sec:simulation}
\subsection{Simulation Parameters}

\begin{figure*}[!tb]
    \centering
    \begin{subfigure}[t]{0.49\textwidth}
        \centering
        \includegraphics[width=\textwidth]{figs/goodput_1pkt.pdf}
        \caption{Number of SDUs: $\gls{not:total-packets}=1$}
        \label{fig:goodput-1-sdu}
    \end{subfigure}%
    ~
    \begin{subfigure}[t]{0.49\textwidth}
        \centering
        \includegraphics[width=\textwidth]{figs/goodput.pdf}
        \caption{Number of SDUs: $\gls{not:total-packets}=2$}
        \label{fig:goodput-2-sdu}
    \end{subfigure}
    \caption{Goodput comparison.}
    \label{fig:train-results-goodput}
\end{figure*}

For simplicity, we assess the performance of a system with one \gls{bs} and two \glspl{ue}.
The transmission buffer of each user starts empty and the \gls{sdu} arrival probability \gls{not:p-arrival} is $0.5$.

The system is trained for a fixed number of episodes \gls{not:episodes-train}.
At some points during the training, we evaluate the policy on a total of \gls{not:episodes-eval} evaluation episodes with disabled exploration and disabled learning in order to assess the current performance of the \gls{mac} protocol.
The set of evaluation episodes remain the same in order to effectively compare the performance on the same set.
At the end of the training procedure, we further evaluate the learned protocol by assessing its performance in \gls{not:episodes-test} episodes with exploration and learning disabled.
This whole procedure represents a single training repetition.
We evaluate a total of \gls{not:repetitions} repetitions, each with a different random seed.

A summary of the main simulation parameters is provided in Table~\ref{tab:sim-params}, while the parameters of the \gls{maddpg} and \gls{ddpg} algorithms are listed in Table~\ref{tab:rl-params}.

\begin{table}[tb]
\centering
\caption{Simulation Parameters}
\label{tab:sim-params}
\begin{tabularx}{0.98\columnwidth}{l c c}
\toprule
\textbf{Parameter}                      & \textbf{Symbol} 	             & \textbf{Value}           \\
\midrule
Number of \glspl{ue}                    & \gls{not:nUE}                 &  $2$                      \\
Size of transmission buffer             & \gls{not:buffer-size}         &  $5$                      \\
Number of \glspl{sdu} to transmit       & \gls{not:total-packets}       &  $\left[ 1, 2 \right]$    \\
\Gls{sdu} arrival probability           & \gls{not:p-arrival}           &  $0.5$                    \\
Transport block error rate              & \gls{tbler}                   &  $\left[ 10^{-1}, 10^{-2}, 10^{-3}, 10^{-4} \right]$               \\
\gls{dcm} vocabulary size               & \gls{not:dl-vocabulary-size}  &  $3$                 \\
\gls{ucm} vocabulary size               & \gls{not:ul-vocabulary-size}  &  $2$                 \\
Max. duration of episode (\glspl{tti})  & \gls{not:episode-duration}    &  $24$                \\
Reward function parameter               & \gls{not:reward-signal}       &  $3$                 \\
Number of training episodes             & \gls{not:episodes-train}      &  $300 \si{k}$        \\
Number of evaluation episodes           & \gls{not:episodes-eval}       &  $500$               \\
Number of test episodes                 & \gls{not:episodes-test}       &  $5000$              \\
Number of randomized repetitions        & \gls{not:repetitions}         &  $32$                \\
\bottomrule
\end{tabularx}
\end{table}

\subsection{Baseline Solutions}

We compare the proposed solution with a contention-free baseline.
We also compare the proposed solution to two simplified approaches where either the communication between agents is not permitted or the centralized critic is disabled, i.e the \gls{ddpg} algorithm.
The ablation comparison helps to evaluate if communication and the centralized critic are needed to solve this task.

In the contention-free protocol, the \gls{ue} sends an \gls{sr} if its transmission buffer is not empty and it only transmits if it has received a \gls{sg}.
Similarly, it only deletes a \gls{tb} from the transmission buffer after the reception of an \gls{ack}.
At each time step, the \gls{bs} receives zero or more \glspl{sr}.
It then chooses one of the requesters at random to transmit in the next time-step, sending a \gls{sg} to the selected \gls{ue}.
However, if the \gls{ue} had made a successful data transmission simultaneously with an \gls{sr}, the \gls{bs} will send an \gls{ack} to this \gls{ue} and its \gls{sr} is ignored.

\begin{table}[tb]
\centering
\caption{Training Algorithm Parameters}
\label{tab:rl-params}
\begin{tabularx}{0.8\columnwidth}{X c}
\toprule
\textbf{Parameter}                      & \textbf{Value} 	            \\
\midrule
Memory length                           & 3                             \\
Replay buffer size                      & $10^5$                        \\
Batch size                              & 1024                          \\
Number of neurons per hidden layer      & $\{ 64, 64 \} $      \\
Interval between updating policies      & 96                            \\
Optimizer algorithm                     & Adam \cite{kingma2014adam}    \\
Learning rate                           & $10^{-3}$                     \\
Discount factor                         & $0.99$                        \\
Policy regularizing factor              & $10^{-3}$                     \\
Gumbel-softmax temperature factor       & $1$                           \\
Target networks soft-update factor      & $10^{-3}$                     \\

\bottomrule
\end{tabularx}
\end{table}

\subsection{Results}

\begin{figure*}[!tb]
    \centering
    \begin{subfigure}[t]{0.49\textwidth}
        \centering
        \includegraphics[width=\textwidth]{figs/episode_duration.pdf}
        \caption{Episode duration in TTIs}
        \label{fig:episode-duration}
    \end{subfigure}%
    ~
    \begin{subfigure}[t]{0.49\textwidth}
        \centering
        \includegraphics[width=\textwidth]{figs/reception.pdf}
        \caption{Pct. of Packets received}
        \label{fig:pct-packets}
    \end{subfigure}
    \caption{Performance comparison for two SDUs.}
    \label{fig:train-results-deep}
\end{figure*}

We compare the \gls{maddpg} solution with three other solutions, the contention-free baseline, the \gls{maddpg} solution without communication, and the \gls{ddpg} version of the proposed solution, i.e. the proposed solution without the centralized critic.
For the \gls{rl} solutions, the solid lines in Figs.~\ref{fig:train-results-goodput} and~\ref{fig:train-results-deep} show the average performance in the evaluation episodes during the training and the shaded areas represent the $95 \%$ \gls{ci}.
The dashed lines show the average performance of the baseline.

In Fig.~\ref{fig:train-results-goodput}, we compare the performance in terms of goodput for the \gls{tbler} of $10^{-1}$.
Figures~\ref{fig:goodput-1-sdu} and~\ref{fig:goodput-2-sdu} show the results when the \glspl{ue} have to transmit one and two \glspl{sdu}, respectively.
After assessing the performance on the last \gls{not:episodes-eval} evaluation episodes, we select the best performing repetitions for each solution in terms of average goodput to compare using boxplots of the test episodes.

By comparing the \gls{rl} solutions in both cases, the \gls{maddpg} has the best performance and the ablation without communication has the worst performance overall.
In addition, the \gls{maddpg} shows a more stable performance during training, with less variation than both other \gls{rl} solutions.
The ablation without communication has the greatest variation of performance, demonstrated by the \glspl{ci} and by the boxplots, indicating that communication helps achieving a more robust solution.

In Fig.~\ref{fig:goodput-1-sdu}, both the \gls{maddpg} and \gls{ddpg} solutions outperform the contention-free baseline, whereas the ablation without communication fails to effectively solve the task in this case.
When we move from one \gls{sdu} to two \glspl{sdu}, in Fig.~\ref{fig:goodput-2-sdu}, the \gls{ddpg} solution does not outperform the baseline.
Moreover, the difference in performance between the \gls{maddpg} and the baseline is reduced.

To better understand the goodput results of Fig.~\ref{fig:goodput-2-sdu}, Fig.~\ref{fig:train-results-deep} shows the performance in terms of episode duration, Fig.~\ref{fig:episode-duration}, and of percentage of the total \glspl{sdu} received during the episode as defined in Eq.~\eqref{eq:delivery-rate}, Fig.~\ref{fig:pct-packets}.

As shown in Fig.~\ref{fig:pct-packets}, the \gls{ddpg} algorithm achieves a high performance in terms of delivery-rate, but it takes more time to solve the task, thus the lower performance in terms of goodput when compared with the \gls{maddpg} and the baseline.
Comparing the \gls{maddpg} with the contention-free solution in Fig.~\ref{fig:episode-duration}, the proposed solution achieves a better goodput by finishing the task in less \glspl{tti}.
The proposed solution has a delivery-rate lower than the contention-free baseline, although it is also very close to $100 \%$.

By applying "survival of the fittest" to pick the best protocol in terms of goodput, the delivery-rate difference becomes even lower than shown in Fig~\ref{fig:pct-packets}.
The best protocol produced by the proposed solution has an average delivery-rate on the test episodes of $\Gamma_{\textsc{maddpg}} = 99.973 \% $, whereas the average of the contention-free baseline is of $\Gamma_{\mathrm{base}} = 99.998 \% $

In Fig.~\ref{fig:goodput-bler}, we compare the proposed \gls{maddpg} framework with the contention-free baseline for different \glspl{tbler} and with each \gls{ue} having to transmit two \glspl{sdu}.
The performance is evaluated on \gls{not:episodes-test} test episodes by comparing the average goodput achieved.
For the \gls{maddpg} solution, we also show the $95\%$ \gls{ci} across randomized repetitions.
The proposed solution maintains a better performance than the baseline across the different \glspl{tbler}.
Moreover, the lowest difference in performance between the baseline and the proposed solution occurs when the \gls{tbler} is equal to $0.1$, showing that the proposed solution adapts well to lower \gls{tbler} regimes.

\begin{figure}[tb]
    \centering
    \includegraphics[width=0.98\columnwidth]{figs/goodput_per_bler.pdf}
    \caption{Performance in terms of goodput for different \glspl{tbler} with each \gls{ue} having to transmit two \glspl{sdu}.}
    \label{fig:goodput-bler}
\end{figure}

\section{Conclusions and Perspectives}
\label{sec:conclusion}

We have proposed a novel framework based on cooperative \gls{marl} augmented with communication, that provides us with the means to emerge a new protocol.
In essence, the agents have to learn the signaling policy, representing the control messages they exchange, and the channel-access policy, representing the \gls{phy} control of the agents.
Comparing with two ablations and a baseline, the results show that a solution capable to overcome the challenges in multi-agent systems is needed in order to emerge a protocol.
The results also indicate that enabling communication between agents is needed in order to solve a transmission task.
In addition, the results illustrate that the proposed solution can produce a protocol tailored to all \gls{tbler} regimes that outperforms a more general one.

Concerning future work, we highlight a study on the effect of the different parameters, such as the vocabulary sizes and \glspl{tbler}
Moreover, we envision a comparison with different \gls{marl} algorithms.
Finally, the application of this framework to a more complex system model is planned.

\section*{Acknowledgment}
\label{sec:ack}
\addcontentsline{toc}{section}{Acknowledgment}
The work of Mateus P. Mota is funded by Marie Sklodowska-Curie actions (MSCA-ITN-ETN 813999 WINDMILL).

\bibliographystyle{IEEEtran}