abstract: Contrast-enhanced computed tomography angiograms (CTAs) are widely used in cardiovascular imaging to obtain a non-invasive view of arterial structures. However, contrast agents are associated with complications at the injection site as well as renal toxicity leading to contrast-induced nephropathy (CIN) and renal failure. We hypothesised that the raw data acquired from a non-contrast CT contains sufficient information to differentiate blood and other soft tissue components. We utilised deep learning methods to define the subtleties between soft tissue components in order to simulate contrast enhanced CTAs without contrast agents. Twenty-six patients with paired non-contrast and CTA images were randomly selected from an approved clinical study. Non-contrast axial slices within the AAA from 10 patients (n = 100) were sampled for the underlying Hounsfield unit (HU) distribution at the lumen, intra-luminal thrombus and interface locations. Sampling of HUs in these regions revealed significant differences between all regions (p\<0.001 for all comparisons), confirming the intrinsic differences in the radiomic signatures between these regions. To generate a large training dataset, paired axial slices from the training set (n=13) were augmented to produce a total of 23,551 2-D images. We trained a 2-D Cycle Generative Adversarial Network (cycleGAN) for this non-contrast to contrast (NC2C) transformation task. The accuracy of the cycleGAN output was assessed by comparison to the contrast image. This pipeline is able to differentiate between visually incoherent soft tissue regions in non-contrast CT images. The CTAs generated from the non-contrast images bear strong resemblance to the ground truth. Here we describe a novel application of Generative Adversarial Network for CT image processing. This is poised to disrupt clinical pathways requiring contrast enhanced CT imaging.
author: Anirudh Chandrashekar$^{1*}$, Ashok Handa$^{1*}$, Natesh Shivakumar$^{1}$, Pierfrancesco Lapolla$^{1}$,
Vicente Grau$^{2**}$ and Regent Lee$^{1**}$ [^1] [^2] [^3]
bibliography: NC2C.bib
title:  **A Deep learning Approach to Generate Contrast-Enhanced Computerised Tomography Angiography without the Use of Intravenous Contrast Agents**

# INTRODUCTION

Since its introduction in the 1970s, computed tomography (CT) has become widely used in medical imaging to obtain a comprehensive and non-invasive view of internal structures and has revolutionized diagnostic decision making. Especially in the field of surgery, CT imaging decreased the need for emergency procedures from 13% to 5% and has significantly decreased the need for exploratory surgical procedures . Furthermore, its incorporation into clinical practice has optimized hospital workflow by decreasing the number of patients requiring inpatient care . In the NHS alone, 6 million CT scans were performed in 2018-2019 . The basic principle of CT technology involves transmitting ionizing radiation, or x-rays, through a region-of-interest (ROI). The transmitted rays are then incident on an electronic detector to create a 'cut' through the object. Both the radiation source and the detector rotate around the object to obtain multiple 'slices or 'cuts'. These projections are then used to reconstruct a 3-D representation of the ROI. Recent advancements have allowed for faster acquisition times, less intrinsic movement artefact and the ability to capture a greater area at a higher resolution during a single acquisition period .

As the x-rays pass through the patient, the rays are attenuated depending on the density of tissue it passes through. The variations in physical density of different objects translates to differences in attenuation and subsequent radio- densities (measured in Hounsfield Units, HU) on a CT scan. The higher the attenuation, the brighter CT image (ex. Bone and calcification) and the lower the attenuation, the darker the CT image (ex. Air). Therefore, the intrinsic contrast of the image is generated based on the differences in attenuation between adjacent tissues .

**Figure 1** gives an example of an axial CT slice through the abdomen. Bony spine (orange arrow) is distinctly more radio-opaque compared to psoas muscle (blue arrow) or the abdominal aorta (red arrow). However, the densities between the abdominal aorta and psoas muscles appear similar here.

Where treatment of an artery is being considered, a detailed view of the arterial anatomy is required. In the example of abdominal aortic aneurysms (AAA, abnormal ballooning of the abdominal aorta), an intra-luminal thrombus (ILT) adherent to the aortic wall within the enlarging aneurysmal sac is present in 95% of cases . Given the similarities in density between the blood lumen, ILT and the complex blood-thrombus interface, these regions cannot be readily distinguished using a conventional CT image.Clear visualization of these regions can only be achieved by the injection of an intravenous (IV) contrast agent. The primary purpose of IV contrast is to enhance luminal density and increase both the attenuation and intrinsic contrast between the vascular tree and surrounding soft tissues. This optimizes the visualization of the vasculature and generates a CT angiogram (CTA) .

Although CTAs may provide unique insight into the structure of the vascular tree, it is associated with several disadvantages . CTAs are contraindicated in patients with iodine allergies as most agents are iodine-based. Furthermore, administration of contrast agents requires needle insertion. This causes additional discomfort and has been associated with complications including inadvertent arterial puncture by needle, and contrast leak from veins causing skin irritation/damage . Additionally, contrast agents are nephrotoxic and have up to 12% incidence of acute kidney injury (contrast-induced nephropathy) following use . This is especially a problem within the elderly population, who either have decreasing baseline renal function or concomitant chronic kidney disease. In these high-risk patients, there is a recognised risk of complete kidney failure, which may lead to renal dialysis.

We hypothesise that the raw data acquired from a non-contrast CT contains sufficient information to differentiate blood and other soft tissue components. Blood, thrombus, and artery wall are made up of different components: blood is predominantly fluid, with red/white blood cells; thrombus is predominantly fibrinous and collagenous, with red cells/platelets; artery wall predominantly contains smooth muscle cells with collagen. These individual components vary in physical density which should reflect in different (albeit subtle) HUs on a CT scan. We further hypothesise that using deep learning approaches, the subtleties between the various components of the soft tissue can be defined and amplified to enable simulation of contrast enhanced CT images without the need to inject contrast agents.

# Methods

## *Obtaining CT images from a clinical cohort*

Computerised Tomographic scans of the chest and abdomen were acquired through the Oxford Abdominal Aortic Aneurysm (OxAAA) study. The study received full regulatory and ethics approval from both Oxford University and Oxford University Hospitals (OUH) National Health Services (NHS) Foundation Trust (Ethics Ref 13/SC/0250). As part of the routine pre-operative assessment for aortic aneurysmal disease, a non-contrast CT of the chest/abdomen/pelvis and an arterial phase CT angiogram (CTA) was performed. CTA images were obtained following contrast injection in helical mode with a pre-defined slice thickness of 1.25 mm. On the other hand, non-contrast CT images were obtained with a pre-defined slice thickness of 2.5 mm. Paired contrast and non-contrast CT images were anonymised within the OUH PACS system before being downloaded onto the secure study drive. Twenty-six patients with paired non-contrast and CTA images of the abdominal region were randomly selected.

## *Manual Segmentation of CT Images (Training Dataset)*

Thirteen (of the 26) cases were randomly selected as the training dataset. Manual segmentation of the aortic inner lumen (in contrast enhanced CTAs) and aortic outer lumen (for both non-contrast and contrast enhanced CTAs) were performed using the open source ITK snap software . Segmentation of the aorta was performed from the aortic root to the iliac bifurcation.

## *Reorientation of Contrast-Enhanced Scans*

To account for voluntary and involuntary movement by the patient between scans, it was necessary to re-orientate the contrast-enhanced images obtained and corresponding segmentation masks to the non-contrast image plane. Registration of the axial slices was performed using the Image Orientation Patient (X, Y, Z coordinates) and Image Orientation Patient (patient's relative rotation) attributes found within the DICOM header. Minimal variation in the orientation of the bowel/air bubbles was observed between the contrast and non-contrast slices. Appropriate registration of 100 randomly selected axial images was confirmed by 2 blinded reviewers (NS and PL) slices prior to subsequent analysis.

## *Hounsfield Unit Sampling*

In order to investigate the regional differences, non-contrast axial slices within the aneurysmal region, 10 consecutive axial slices were taking from 10 patients, (total = 100 slices) were sampled for the underlying Hounsfield unit (HU) distribution at the lumen, intra-luminal thrombus and interface locations (**Figure 2**). These visually indistinct regions on the non-contrast CT slice were identified from their paired contrast counterpart. The reoriented binary masks were used to establish the boundary locations in the non-contrast image. To account for slight inaccuracies in the image reorientation process and minimize sampling errors, the thrombus (blue) and lumen (yellow) areas were reduced by 20% and the zone between the two regions was demarcated as the interface (red). This delineation is clearly indicated in **Figure 2**.

One-way ANOVA was used to compare the average HU intensity within each region (in individual patients) both across and within each slice for each patient. The negative control consisted of concentric sampling within the lumen region. This was used as blood on a macroscopic level is randomly distributed and relatively homogeneous and should produce similar HU intensities.

## *Image Pre-Processing*

Of the 26 patients, 13 patients were randomly allocated to the training (n~train~ = 13) cohort. Following segmentation, the original pre-aligned CT images and their corresponding image masks of patients in the training cohort were augmented using divergence transformations. These divergence transformations employ non-linear warping techniques to each axial slice, which manipulate the aorta in certain predefined locations. In this instance, both congruent and divergent local transformations were utilized to diversify the training dataset. These augmentation methods have been previously validated. Therefore, each patient's scan in the training cohort was augmented in a ratio of 10:1 to obtain a total of 23,551 2-D images (axial).

## *Cycle-GAN Architecture*

In this study, we utilized a cycle-GAN, which is a variation of the popular generative adversarial network (GAN). These networks are a class of deep learning architecture whereby two neural networks train simultaneously, with one network focused on data generation (generator) and the other network focused on data discrimination (discriminator). The two neural networks 'compete' against each other, learning the statistical distribution of the training data, which in turn allows to generate new examples from the same distribution. GANs have been applied to generate imaginary portraits, landscapes, and artworks based on real examples . Many variations of the original GAN concept have been developed, including conditional GANs (cGANs), which can learn the transformation between two paired distributions, including the transformation between images using the pixel to pixel (Pix2Pix) approach . The primary benefit of cylceGANs is that it can learn transformations from two distributions without the need for direct pairings between specific samples .

The generator and discriminator components in the NonContrast-to-Contrast (NC2C) model architecture (**Figure 3**) were explicitly defined as least-squares GAN and a 70 x 70 PatchGAN, respectively. The former incorporates an additional least-squares loss function for the discriminator, which in turn, improves the training of the generative model. On the other hand, the discriminator goes through the image pairs, in 70 x 70 patches, and is trained to classify whether the image under question is "real" or "fake".

## *Model Training and Evaluation*

The NC2C-cycleGAN (**Figure 3**) was trained with a learning rate of 2.0 \* 10^-4^ for 200 epochs on 256 x 256 images centred around the aorta. Four networks (2 generators + 2 discriminators) were trained simultaneously and various loss functions were evaluated at each iteration to document model training. In addition to the loss metrics inherent to the networks, both an identity mapping and a cycle consistency loss functions were included to ensure appropriate style transfer and regularization of the generator to allow for image translation, respectively. The accuracy of the cycleGAN output in generating the lumen and thrombus interface was assessed by comparison to the contrast image, which serves as a gold standard.

# Results

Average HU intensity in the non-contrast images was significantly different between all three regions (Lumen vs. Thrombus, Lumen vs. Interface and Interface vs. Thrombus) for all patients assessed. When assessing on a slice-by-slice basis, the average HU intensity of the thrombus was significantly different from that of the lumen 94% of the time. Histograms corresponding to the HU frequencies for each region had considerable overlap, however a shift in frequency distribution is also apparent (**Figure 4B**). **Figure 4B** further displays the differences in HU intensities for 8 axial slices obtained from one of the ten patients analyzed. The average HU intensity for the thrombus was significantly lower when compared to that of the lumen for all axial slices sampled. Furthermore, the interface was also significantly different from the other two regions, indicating a gradual change in pixel intensity from the center lumen to the peripheral thrombus. No significant differences in HU intensity were noted following concentric sampling of the aortic blood flow lumen (**Figure 4C,D**).

**Figure 5** illustrates the loss functions during the training of the NC2C-cycle GAN. The inherent generator and discriminator loss functions converge during model training. Additionally, the identity mapping and cycle consistency loss functions for both directions (1. Contrast-to-Non-Contrast, 2. Non-Contrast-to-Contrast) plateau within the first 50 epochs. Model performance was evaluated on the testing (n~test~ =13) cohort. This generative model is able to simulate the aortic lumen throughout the length of the aorta and differentiate between the lumen and intra-luminal thrombus of aneurysmal sections with strong resemblance to the ground truth. In addition to the aorta, this algorithm is able to transform other structures including the small mesenteric arteries (orange arrows), the pulmonary arteries (yellow arrow) and the kidneys (blue arow) as seen in **Figure 6**.

# Discussion

In the realm of medical image synthesis, traditional image transformation tasks have relied on patch-based regressions. These methods are able to sample a patch of an image or volume from one modality and predict its intensity in the target modality . In addition to patch-based regression models, sparse representation and atlas-based models have been used. The latter utilizes paired image atlases from the source- and target image modalities. Here, the atlas-to-image transformation is calculated for the source-modality and a similar transformation is performed for the target-modality atlas

In recent years, deep learning (DL) methods have been applied extensively in this domain and has achieved promising results. Dong et al. proposed a convolutional neural network approach to increase the resolution of medical images . Additionally, GAN-based cross-modal image synthesis methods have gained significant traction and achieved success . Here, these key methods rely on an adversarial training platform between a generator and a discriminator. The fundamental task of the generator is to produce images that are realistic and ultimately fool the discriminator, which is simultaneously trained to differentiate real and generated images . Variations of the GAN architecture include the Pix2Pix , and conditional and cycle -GAN networks . As its name suggests, the Pix2pix network uses paired data to learn the appropriate pixel-to-pixel image transformations. This network strives to maintain pixel-to-pixel similarity between the real and the generated images . On the other hand, the CycleGAN attempts to capture structural information by learning the translation mapping between unpaired images . GAN-based methods have shown great promise in synthesizing various types of medical images including CT (from MR images , low-dose to routine-dose ), retinal , MR, and ultrasound images . This study attempts, for the first time, the possibility to transform non-contrast CT images into contrast-enhanced CT Angiograms without the need the for intravenous (IV) contrast.

Currently, CT angiography relies on IV contrast injection to enhance the intrinsic contrast between the vascular tree and surrounding tissues. Therefore, this method is able to provide a unique view of the patient's vasculature, which is essential for diagnosis and surgical planning. For example, in the case of AAA disease, CTAs are the current gold-standard to visualize the intra-luminal thrombus (ILT) within the enlarging aneurysmal sac. The orientation, location and extent of the ILT as well as the complex blood-ILT interface is vital information prior to surgical intervention . Although CT angiography may provide unique insight into aneurysm morphology and the structure of the vascular tree, it is not without its disadvantages, as discussed above.

Additionally, this method to transform non-contrast CT to CTA images can be used to incorporate historic non-contrast CT scans for research purposes. A key priority for AAA research is to discovery novel indices of AAA growth prediction. Currently, this is documented by measuring the maximum antero-posterior aortic diameter of the abdominal aorta during serial duplex ultrasound scans . However, this uses a 1-dimensional measurement that is susceptible to inter-observer error to characterize a diverse and complex 3-dimensional growing process . There is emerging evidence that patient-specific geometric and volumetric measurements more readily influence AAA growth . As small AAAs enlarge, a variety of geometrical changes have been observed. Many of these changes result in a unique non-uniform distribution of wall stress and have been hypothesized to either favour AAA growth deceleration or increase rupture risk . Isolating and deciphering these changes will allow us to predict AAA growth and progression in each patient. Current work by our group has focused on generating a DL-algorithm for the automatic aortic segmentation of the aortic volume alongside feature extraction . Integration of this image synthesis model with our volume extraction methods will allow us to utilize historic non-contrast CT scans across multiple centers for this complex geometric and morphological analysis.

This image synthesis pipeline can be extended to cover other anatomical structure (veins, solid organs, etc). Our preliminary work has shown that it is capable of generating reliable CTAs of the large arteries; however, there is evidence that this model is able to account for small vessels that branch from the aorta (ex. renal arteries, vertebral arteries, etc. and can be extended to other structures (ex. venogram, solid organ contrast CTs).

# Conclusions

This study describes, for the first time, the ability to differentiate between visually incoherent soft tissue regions in non-contrast CT images using deep learning methods. Ultimately, refinement of this methodology may negate the use of intravenous contrast and prevent related complications.

# Acknowledgements

We acknowledge the support from the following: Medical Sciences Division, University of Oxford Medical Research Fund; John Fell Fund, University of Oxford; Academy of Medical Sciences Starter Grant to RL (AMS-SGL013 \1015; Oxford University Clarendon Scholarship to AC. The methods described in this manuscript is subject to a patent filing (UK priority filing, P276235GB).

[^1]: $^{1}$ Nuffield Department of Surgical Sciences, University of Oxford

[^2]: $^{2}$ Department of Engineering Science, University of Oxford

[^3]: $^{*,**}$ Authors contributed equally.


abstract:  The second-order consensus of multiple interacting non-identical agents with non-linear protocols is studied in this article. Firstly, it is shown that all agents with different non-linear dynamics can achieve consensus without a leader. Secondly, an explicit expression of the consensus value is analytically developed for the group of all agents. Thirdly, for the consensus of multiple agents with a leader, it is proved that each agent can track the position and velocity of the leader, which are different from those of the follower agents. Finally, numerical simulations are given to illustrate the theoretical results.
address:
author:
title: Second-order consensus of multiple non-identical agents with non-linear protocols

# Introduction

The problem of consensus of multiple agents has attracted great attention over the past decade. This is mainly because that multiple agent systems can be found in many applications, including cooperative and formation control of mobile vehicles, flocking of multiple agents, distributed multi-vehicles coordinated control, design of sensor network and autonomous underwater vehicles. Generally speaking, consensus means that all agents need to reach an agreement on certain quantities of interest (such as the position and velocity) under some control protocols. Vicsek *et al*. proposed a simple model for phase transition of a group of self-driven particles and numerically demonstrated complex dynamics of the model. In this model, each agent's motion is updated by using the local protocol based on the average of their own headings and neighbours. Jadbabaie *et al*. provided a theoretical explanation for the Vicsek model based on algebraic graph theory. Saber and Murry investigated a systematic framework of consensus problems with directed interconnection graphs or time-delays.

To the authors' best knowledge, most existing works concerned with the second-order consensus of multiple agents assumed that all agents have the same dynamics. As a matter of fact, the dynamics of multiple agents composed of a group can be different, for example, each agent can have different mass (inertia) and different velocity gain because of design restriction and inherent property (or the measuring instrument). In addition, non-linear factor inevitably exists as a non-linear function of a variable between the agents during the information exchange. For instance, because of the limitation of observational technique and the inaccuracy of model parameters, the velocity $v_{i}(t)$ of an agent may be unobservable in some cases. In such cases, instead, a non-linear function $f(v_{i})$ of the velocity $v_{i}(t)$ can be observed (see ). On the other hand, the linear parameters of an agent in the formation cannot be known exactly, and the agent is always subjected to external disturbances and some inevitable inaccurate data. Accordingly, in order to achieve high control performance and explore the collective dynamics of multiple agents, the consensus problem under non-linear protocols should be considered.

The main contribution of this paper is to provide an explicit expression of the consensus value, which is not only dependent on the initial positions and velocities, but also the masses (inertia) and velocity gains of all agents. For the case of agents with a leader, it will be shown that all agents can track the position and velocity of the leader, which are different from those of follower agents in the group. It will also be shown that the masses and velocity gains of all agents in the group have significant effects on the final consensus value and their influences are independent. The rest of the paper is organised as follows. Section 2 states the problem formulations and part of graph theory. Section 3 gives the main consensus results of multiple non-identical agents with and without a leader. Some numerical simulations are given in Section 4. Finally, Section 5 presents a brief conclusion to this paper.

# Problem formulations

Some mathematical notations to be used throughout this paper are given below. Let $R$ define a set of real numbers; $R^{n}$ be the $n$-dimensional real vector; $R^{n\times n}$ be the set of $n\times
n$ real matrices; and $\bar{n}=\{1,2,\ldots, n\}$ be an index set. To ensure the existence, uniqueness of the solution, and the cooperative property of the non-linear protocol, it is assumed that a set of continuously differentiable functions $f_{l}(z)$, $h_{l}(z)$ and $b_{i}(t)$ $(i\in\bar{N}, l\in\bar{n})$ satisfy the following properties (see and relevant references therein for details):

1.  $f_{l}(z)=0\Leftrightarrow z = 0;$ $z f_{l}(z)>0,\ \forall z\neq 0$; $f'_{l}(z)$ denotes the differentiation with respect to $z$;

2.  $h_{l}(-z)\,{=}\,{-}h_{l}(z),\ \forall z \,{\in}\, R$; $h_{l}(z)\,{=}\,0\,{\Leftrightarrow}\, z \,{=}\, 0$; $(z_{j}\,{-}\,z_{i}) h_{l}(z_{j}-z_{i})>0,\ \forall z_{j}\neq z_{i}\in R;$

3.  $0<b\leq b_{i}(t)\leq \bar{b}<+\infty$, $\dot{b}_{i}(t)$ are bounded, where a dot denotes the differentiation with respect to time $t$.

Consider the system consisting of $N$ agents, and the dynamics of each agent can be expressed as $$\label{eq1}
\begin{cases}
\dot{p}_{i}(t) = q_{i}(t),\\
m_{i}\dot{q}_{i}(t)=u_{i1}(t)+u_{i2}(t),
\end{cases} \quad i\in\bar{N}$$ where $p_{i}(t)=(p_{i1}(t), p_{i2}(t),\ldots, p_{in}(t))^{\top}\in
R^{n}$ and $q_{i}(t)=\break (q_{i1}(t), q_{i2}(t),\ldots,
q_{in}(t))^{\top}\in R^{n}$ are the position and velocity of agent $i$, respectively, $m_{i}$ is the mass (inertia) of agent $i$, $u_{i1}(t)\in R^{n}$ and $u_{i2}(t)\in R^{n}$ are control inputs (control protocols). Here, the control protocols are assumed to be of the form $$\begin{aligned}
\label{eq2}
u_{i1}(t)&=-b_{i}(t)\,f(q_{i}),\notag\\
u_{i2}(t)&=\sum_{j\in \mathcal{N}_{i}}c_{ij}h(p_{j}(t)-p_{i}(t)),\quad i\in\bar{N}
\end{aligned}$$ where $b_{i}(t)$ is the velocity gain of agent $i$, $f(q_{i})=(f_{1}(q_{i1}),\break  f_{2}(q_{i2}),\ldots,f_{n}(q_{in}))^{\top\!}\,{\in}\,
R^{n}$ and $h(p_{j}(t)\,{-}\,p_{i}(t))\,{=}\,(h_{1}(p_{j1}\,{-}\break p_{i1}),
h_{2}(p_{j2}-p_{i2}),\ldots,h_{n}(p_{jn}-p_{in}))^{\top}\in R^{n}$ are strictly increasing non-linear functions with $f_{l}(z)$, $h_{l}(z)$ and $b_{i}(t)$ satisfying the above assumptions (i)–(iii). The main aim of the paper is to determine $u_{i1}(t)$ and $u_{i2}(t)$ for all agents without a leader to satisfy $$\begin{aligned}
&\lim_{t\rightarrow +\infty}(p_{jl}(t)-p_{il}(t))=0,\\
&\lim_{t\rightarrow  +\infty}q_{il}(t)=0,\quad i,j\in\bar{N},\enspace l\in\bar{n}
\end{aligned}$$ and for all agents in the presence of a leader, to design the control protocols $$\begin{aligned}
\label{eq3}
u_{i1}(t)&=-b_{i}(t)f(q_{i}),\notag\\
u_{i2}(t)&=\sum_{j\in\mathcal{N}_{i}}c_{ij}h(p_{j}(t)-p_{i}(t))+c_{i{\rm L}}h (p_{\rm L}(t)-p_{i}(t)),\enspace i\in\bar{N}
\end{aligned}$$ such that $$\begin{aligned}
&\lim_{t\rightarrow  +\infty} (p_{il}(t)-p_{{\rm L}l}(t))=0,\\
&\lim_{t\rightarrow +\infty}(q_{il}(t)-q_{{\rm L}l}(t))=0,\quad
i\in\bar{N},\enspace l\in\bar{n}
\end{aligned}$$ where $c_{i{\rm L}}>0$, if there is a connection from the leader to agent $i$, and $c_{i{\rm L}}=0$, if no connection exists from the leader. $p_{\rm L}(t)\in R^{n}$ and $q_{\rm L}(t)\in R^{n}$ denote the position and velocity of the leader, respectively. The dynamics of the leader is governed by $$\label{eq4}
\begin{cases}
\dot{p}_{\rm L}(t) &= q_{\rm L}(t)\\
\dot{q}_{\rm L}(t)&=-b_{\rm L}(t)\,f(q_{\rm L})
\end{cases}$$ It should be noted that control input $u_{i2}(t)$ is a local control protocol with a neighbour-to-neighbour interaction between agents. Each agent (for example, agent $i$) can update its current velocity based on the position information of its neighbours. The protocol $u_{i2}(t)$ also includes some non-linear properties, such as uncertain linear model parameters and unpredictable external disturbances . Protocol $u_{i2}(t)$ requires neighbour's current positions only and does not need neighbour's other information.

In the analysis of the convergence conditions for multiple agents, the undirected graph $\mathcal{G}=\{\mathcal{V},\mathcal{E}, C\}$ will be used to model the interaction among $N$ agents. $\mathcal{V}=\{1,2,\ldots,N\}$ is the node (agent) set; an edge of $\mathcal{G}$ is denoted by $e_{ij} = (i, j)$ which means nodes $j$ and $i$ can receive information from each other; a set of edges $\mathcal{E}=\{(i, j)\in \mathcal{V}\times\mathcal{V}|i\sim j\}$ containing unordered pairs of nodes represents communication links; a non-negative adjacency matrix $C$ is defined as: $c_{ij}=c_{j i}>0$ if $e_{i j}\in \mathcal{E}$ (there is a connection between agents $i$ and $j$), while $c_{ij}=0$ if $e_{ij}\notin
\mathcal{E}$ (there is no connection between agents $i$ and $j$). The network neighbours of agent $i$ are assumed to belong to a set $\mathcal{N}_{i}: \mathcal{N}_{i}=\{\,j\, |\,(i,
j)\in \mathcal{E}\}\subseteq\mathcal{V}\backslash \{\,i\,\}$. The Laplacian matrix $\mathcal{L}=(l_{i j})\in R^{N\times N}$ associated with $\mathcal{G}$ is defined as $l_{i
i}=\sum_{j\in \mathcal{N}_{i}}c_{i j}$ and $l_{i j}=-c_{i
j}\ (i\neq j)$. A path of $\mathcal{G}$ is a sequence of edges of the form $(i_{1}, i_{2}),(i_{2}, i_{3}),\ldots,$ where $i_{j}\in\mathcal{V}$. An undirected (directed) graph is said to be connected (strongly connected), if there exists a path between any two distinct vertices of the graph. If there exists a path from agent $i$ to agent $j$, then agent $j$ is said to be reachable from agent $i$. For graph $\mathcal{G}$, if there is a path from every agent to agent $r$, then it can be said that agent $r$ is globally reachable in $\mathcal{G}$ . A graph $\mathcal{G}$ associated with the system consisting of $N$ agents and one leader (in the leader case) will be considered in subsequent sections.

# Consensus of non-identical agents with non-linear protocols

## Consensus of non-identical agents without a leader

**Theorem 1**.  * Consider system (1) with control protocol (2) and assume that the undirected graph $\mathcal{G}$ is connected. Then, $\lim_{t \rightarrow +\infty}
(p_{jl}(t)\,{-}\,p_{il}(t))\,{=}\,0$, $\lim_{t \rightarrow +\infty}
q_{il}(t)\,{=}\,0$ $(i, j\,{\in}\,\bar{N}, l\in\bar{n})$.*

**Proof.* Select the Lyapunov function as $$\begin{aligned}
\label{eq5}
V(t,p_{i}(t),q_{i}(t))&=\frac{1}{2}\sum_{i=1}^{N}m_{i}q_{i}^{\top}(t)q_{i}(t)+\frac{1}{2}\sum_{i=1}^{N}\sum_{j \in \mathcal{N}_{i}}\sum_{l=1}^{n}c_{ij}\notag\\
&\quad \times\int^{p_{jl}(t)-p_{il}(t)}_{0}h_{l}(s)\,{\rm d}s
\end{aligned}$$*

*Differentiating $V(t,p_{i}(t),q_{i}(t))$ with respect to time along the solution of (1) yields $$\begin{aligned}
\left.\frac{{\rm d} V}{{\rm d}t}\right|_{(1)}&=\sum_{i =
1}^{N}m_{i}q_{i}^{\top}(t)\dot{q}_{i}(t)+\frac{1}{2}\sum_{i
=1}^{N}\sum_{j \in \mathcal{N}_{i}}\sum_{l=1}^{n}c_{ij}h_{l}(p_{jl}(t)\notag\\
&\quad-p_{il}(t))(q_{jl}(t)-q_{il}(t))\nonumber\\
&=\sum_{i=1}^{N}m_{i}q_{i}^{\top}(t)\dot{q}_{i}(t)+\frac{1}{2}\sum_{i=1}^{N}\sum_{j
\in \mathcal{N}_{i}}\sum_{l=1}^{n}c_{ij}h_{l}(p_{jl}(t)\notag\\
&\quad -p_{il}(t))q_{jl}(t)\notag\\
&\quad -\frac{1}{2}\sum_{i
= 1}^{N}\sum_{j \in \mathcal{N}_{i}}\sum_{l
= 1}^{n}c_{ij}h_{l}(p_{jl}(t)-p_{il}(t))q_{il}(t)\nonumber\\
&=\sum_{i =
1}^{N}m_{i}q_{i}^{\top}(t)\dot{q}_{i}(t)+\frac{1}{2}\sum_{i=1}^{N}\sum_{j
\in \mathcal{N}_{i}}\sum_{l =
1}^{n}c_{ji}h_{l}(p_{il}(t)\notag\\
&\quad-p_{jl}(t))q_{il}(t)\notag\\
&\quad -\frac{1}{2}\sum_{i
= 1}^{N}\sum_{j \in \mathcal{N}_{i}}\sum_{l
= 1}^{n}c_{ij}h_{l}(p_{jl}(t)-p_{il}(t))q_{il}(t)\nonumber\\
&=\sum_{i =
1}^{N}m_{i}q_{i}^{\top}(t)\dot{q}_{i}(t)-\sum_{i
= 1}^{N}\sum_{j \in \mathcal{N}_{i}}\sum_{l
= 1}^{n}c_{ij}h_{l}(p_{jl}(t)\notag\\
&\quad -p_{il}(t))q_{il}(t)\nonumber\\
&=-\sum_{i =
1}^{N}b_{i}(t)q_{i}^{\top}(t)\,f(q_{i})+\sum_{i = 1}^{N}\sum_{j \in \mathcal{N}_{i}}\sum_{l =
1}^{n}c_{ij}h_{l}(p_{jl}(t)\notag\\
&\quad -p_{il}(t))q_{il}(t)\notag\\
&\quad -\sum_{i=1}^{N}\sum_{j \in \mathcal{N}_{i}}\sum_{l=1}^{n}c_{ij}h_{l}(p_{jl}(t)-p_{il}(t))q_{il}(t)\nonumber\\
&\leq-b\sum_{i=1}^{N}q^{\top}_{i}(t)\,f(q_{i})\leq0
\end{aligned}$$ then $\dot{V}(t,p_{i}(t),q_{i}(t))\leq 0$ and $\int^{t}_{t_{0}}g(s)\,ds\leq V(t_{0},p_{i}(t_{0}), q_{i}(t_{0}))-V(t,p_{i}(t),q_{i}(t))$, where $g(t)=b\sum_{i = 1}^{N}\sum_{l = 1}^{n}q_{il}(t) f_{l}(q_{il})$. Furthermore, $\lim_{t
\rightarrow +\infty}\int^{t}_{t_{0}}g(s)\,{\rm d}s\leq V(t_{0},p_{i}(t_{0}), q_{i}(t_{0}))<+\infty$`<!-- -->`{=html}exists and is finite. By considering Lyapunov function (5) and equality (6) as well as the properties of $f_{l}(z)$, $h_{l}(z)$ and $b_{i}(t)$, it can be seen that $\dot{g}(t)=b\sum_{i = 1}^{N}\sum_{l=1}^{n} 1/m_{i}[-b_{i}(t) f_{l}(q_{il})+\sum_{j \in
\mathcal{N}_{i}}c_{ij}h_{l}(p_{jl}(t)-p_{il}(t))][f_{l}(q_{il})+q_{il}(t) f'_{l}(q_{il})]$ is bounded.Therefore $g(t)$ is uniformly continuous in $t$ and $$\begin{aligned}
&m_{i}\dot{q}_{il}(t)=-b_{i}(t) f_{l}(q_{il})+\sum_{j \in
\mathcal{N}_{i}}c_{ij}h_{l}(p_{jl}(t)-p_{il}(t)),\notag\\
&\quad i,j\in\bar{N},\ l\in\bar{n}\label{eq7}
\end{aligned}$$ thus $\lim_{t \rightarrow +\infty}g(t)=0$ by applying the Barbalat lemma , which further gives $\lim_{t \rightarrow +\infty}q_{il}(t)=0$. On the other hand, $\lim_{t \rightarrow +\infty}\dot{q}_{il}(t)=0$ since $\dot{q}_{il}(t)$ is uniformly continuous in $t$. Taking the limit as $t\rightarrow +\infty$ on both sides of () gives rise to the expression $\lim_{t \rightarrow +\infty}\dot{q}_{il}(t)=\lim_{t
\rightarrow +\infty}\{\sum_{j \in \mathcal{N}_{i}}c_{ij}h_{l}(p_{jl} (t)-p_{il}(t))\}= 0$, then $\lim_{t \rightarrow +\infty} (p_{jl}(t)-p_{il}(t))=0$. ◻*

**Corollary 1**.  * Assume $f(q_{i})=q_{i}(t)$, $b_{i}(t)=b_{i}$ and suppose that the undirected graph $\mathcal{G}$ is connected. Then $$p_{i}(t)\rightarrow \frac{1}{\beta}\sum_{i=1}^{N}(b_{i}p_{i}(0)+m_{i}q_{i}(0)),\quad i\in\bar{N}$$ as $t\rightarrow +\infty$, where $\beta=\sum_{i=1}^{N}b_{i}$, $p_{i}(0)$ and $q_{i}(0)$ are the initial position and velocity of $p_{i}(t)$ and $q_{i}(t)$, respectively.*

**Proof.* Note that $\alpha(t)=\sum_{i =
1}^{N}(b_{i}p_{i}(t)+m_{i}q_{i}(t))$ is an invariant quantity and its first-order derivative $\dot{\alpha}(t)=0$ because of the symmetry of matrix $C=(c_{ij})^{N\times N}$. Applying $\alpha(0)=\alpha(+\infty)$ and the above results yields $p_{i}(t)\rightarrow1/\beta\sum_{i =
1}^{N}(b_{i}p_{i}(0)+m_{i}q_{i}(0))$ as $t \rightarrow +\infty$. ◻*

**Remark 1**.  * Lyapunov function $V(t,p_{i}(t),q_{i}(t))$ is positively definite since the properties of $h_{l}(z)$, and $V(t,p_{i}(t),q_{i}(t))=0$ if and only if $q_{il}(t)=0$ and $p_{jl}(t)-p_{il}(t)=0$ ($i, j\in\bar{N},\ l\in\bar{n}$).*

**Remark 2**.  * In proving Theorem 1, the Barbalat lemma has been used instead of Lasalle Invariance Principle since system (1) is a time-varying system. Theorem 1 indicates that the asymptotic consensus is reachable even if all agents are non-identical. Corollary 1 presents an explicit expression of the consensus value, which depends on the masses (inertial) and velocity gains, the initial positions and velocities of all agents. For given initial conditions, suitable $b_{i}$ and $m_{i}$ can be easily selected such that all agents achieve a desired consensus value. Moreover, the effects of $m_{i}$ and $b_{i}$ on the final consensus value are independent because of symmetric interaction of topology $C$.*

*By following a similar analysis to the one developed in Theorems 1, system (1) can be extended to the directed interconnected systems satisfying detailed balance conditions .*

## Consensus of non-identical agents with a leader

In some cases, all agents are required to achieve a desired consensus value, which is irrelevant to agents' initial conditions. This situation is usually referred to as the consensus in the literature. The dynamics of the leader to be considered here is non-linear and different from that of any other agents. All other agents need to reach the same dynamics. To address the consensus of non-identical agents with a leader, system (1) with a virtual leader labelled as 'L' with dynamics (4) will be investigated in this subsection ($m_{i}=1, i\in\bar{N}$).

**Theorem 2**.  * Consider system (1) with control protocol (3). Suppose that $b_{\rm L}(t)$ has the same properties with $b_{i}(t)(i\in\bar{N})$, $f_{l}(z)$ satisfying $k\,{\leq}\,{f_{l}(z)}/{z}\,{\leq}\,\bar{k}$ $(\bar{k}\,{\geq}\,
k>0, l\in\bar{n})$. Then, $\lim_{t\rightarrow +\infty}
(p_{il}(t)-p_{{\rm L}\,l}(t))=0,\,\lim_{t\rightarrow
+\infty}(q_{il}(t)-q_{{\rm L}\,l}(t))=0$ provided that the leader has a path to all agents in graph $\mathcal{G}$.*

*Proof.* Denote the position and velocity error functions between agent $i$ and the leader 'L' as $\hat{p}_{i}(t)=p_{i}(t)-p_{\rm L}(t)$, $\hat{q}_{i}(t)=q_{i}(t)-q_{\rm L}(t)\,(i\in\bar{N})$, respectively. Then the error systems obtained from (1), (3) and (4) can be written as $$\begin{aligned}
\begin{cases}
\dot{\hat{p}}_{i}(t)=\hat{q}_{i}(t),\\
\dot{\hat{q}}_{i}(t)=-b_{i}(t)f(\hat{q}_{i}(t)+q_{\rm L}(t))+b_{\rm L}(t)\,f(q_{\rm L})\\
\quad +\sum_{j\in\mathcal{N}_{i}}c_{ij}\,h(\hat{p}_{j}(t)-\hat{p}_{i}(t))-c_{i{\rm L}}\,h(\hat{p}_{i}),
\end{cases}\quad i\in\bar{N}\label{eq8}
\end{aligned}$$ Select the Lyapunov function as $$\begin{aligned}
&V(t,q_{\rm L}(t),\hat{p}_{i}(t),\hat{q}_{i}(t))\notag\\
&\quad =\frac{M}{2bk}q^{\top}_{\rm L}(t)q_{\rm L}(t)+\frac{1}{bk}\sum_{i=1}^{N}\hat{q}^{\top}_{i}(t)\hat{q}_{i}(t)\nonumber\\
&\qquad +\frac{2}{bk}\sum_{i=1}^{N}\sum_{l=1}^{n}c_{i{\rm L}}\int^{\hat{p}_{il}(t)}_{0}h_{l}(s){\rm d}s\notag\\
&\qquad +\frac{1}{bk}\sum_{i=1}^{N}\sum_{j\in\mathcal{N}_{i}}
\sum_{l=1}^{n}c_{ij}\int^{\hat{p}_{jl}(t)-\hat{p}_{il}(t)}_{0}h_{l}(s){\rm d}s\nonumber
\end{aligned}$$ Differentiation of $V(t,q_{\rm L}(t),\hat{p}_{i}(t),\hat{q}_{i}(t))$ with respect to time along the trajectory of (8) results in $$\begin{aligned}
\left.\frac{{\rm d} V}{{\rm d}t}\right|_{(8)} &=\frac{M}{bk}q^{\top}_{\rm L}(t)\dot{q}_{\rm L}(t)+\frac{2}{bk}\sum_{i=1}^{N}\hat{q}^{\top}_{i}(t)\dot{\hat{q}}_{i}(t)\\
&\quad +\frac{2}{bk}\sum_{i= 1}^{N}\sum_{l=1}^{n}c_{i{\rm L}}h_{l}(\hat{p}_{il})\hat{q}_{il}(t)\\
&\quad+\frac{1}{bk}\sum_{i=1}^{N}\sum_{j\in\mathcal{N}_{i}}\sum_{l=1}^{n}c_{ij}h_{l}(\hat{p}_{jl}(t)\\
&\quad-\hat{p}_{il}(t))(\hat{q}_{jl}(t)-\hat{q}_{il}(t))\\
&=-\frac{M}{bk}q_{\rm L}^{\top}(t)b_{\rm L}(t)f(q_{\rm L})+\frac{2}{bk}\sum_{i
=1}^{N}\hat{q}_{i}^{\top}(t)\\
&\quad\times \left\{\vphantom{\sum_{j\in\mathcal{N}_{i}}}-b_{i}(t)f(\hat{q}_{i}(t)+q_{\rm L}(t))+b_{\rm L}(t)f(q_{\rm L})\right.\\
&\quad \left.+\sum_{j\in\mathcal{N}_{i}}c_{ij}h(\hat{p}_{j}(t)-\hat{p}_{i}(t))
-c_{i{\rm L}}h(\hat{p}_{i})\right\}\\
&\quad +\frac{2}{bk}\sum_{i =1}^{N}c_{i{\rm L}}\hat{q}^{\top}_{i}(t)h(\hat{p}_{i})\\
&\quad-\frac{2}{bk}\sum_{i=1}^{N}\sum_{j\in\mathcal{N}_{i}}c_{ij}\hat{q}^{\top}_{i}(t)h(\hat{p}_{j}(t)-\hat{p}_{i}(t))\\
&=-\frac{M}{bk}q^{\top}_{\rm L}(t)b_{\rm L}(t)f(q_{\rm L})-\frac{2}{bk}\sum_{i=1}^{N}\hat{q}^{\top}_{i}(t)b_{i}(t)
f(\hat{q}_{i}(t)\\ &\quad +q_{\rm L}(t))
+\frac{2}{bk}\sum_{i=1}^{N}\hat{q}^{\top}_{i}(t)b_{\rm L}(t)f(q_{\rm L})\\\displaybreak
&\leq-M q_{\rm L}^{\top}(t)q_{\rm L}(t)-2\sum_{i=1}^{N}\sum_{l=1}^{n}(\hat{q}_{il}(t)+q_{{\rm L}l}(t))^{2}\\
&\quad +\sum_{i=1}^{N}\hat{q}^{\top}_{i}(t)\hat{q}_{i}(t)+\frac{4\bar{b}^{2}\bar{k}^{2}+2bk\bar{b}\bar{k}}{b^{2}k^{2}}\sum_{i=1}^{N}q_{\rm L}^{\top}(t)q_{\rm L}(t)\\
&\leq-\frac{1}{2}\sum_{i=1}^{N}\hat{q}^{\top}_{i}(t)\hat{q}_{i}(t)\\
&\quad -\left(M-\frac{2N(bk\bar{b}\bar{k}+3b^{2}k^{2}+2\bar{b}^{2}\bar{k}^{2})}{b^{2}k^{2}}\right)
q^{\top}_{\rm L}(t)q_{\rm L}(t)
\vspace*{-2pt}
\end{aligned}$$ so $\dot{V}(t,q_{\rm L}(t),\hat{p}_{i}(t),\hat{q}_{i}(t))\leq 0$ if choosing $M> 2N(bk\bar{b}\bar{k}+\break 3b^{2}k^{2}+2\bar{b}^{2}\bar{k}^{2})/{b^{2}k^{2}}>0$. The last term in the first inequality is obtained based on the inequality $|ab|\leq{1}/{2}(a^{2}+b^{2})$. Then, by using the same analysis procedure developed in Theorem 1, it is easy to obtain that $\lim_{t \rightarrow +\infty}
\hat{q}^{\top}_{i}(t)\hat{q}_{i}(t)\,{=}\,0$, $\lim_{t \rightarrow
+\infty}q^{\top}_{\rm L}(t)q_{\rm L}(t)=0$ and $\lim_{t \rightarrow
+\infty}\hat{p}_{il}(t)=0$, which imply that $q_{i}(t)\,{\rightarrow}\,q_{\rm L}(t)$, $p_{i}(t)\,{\rightarrow} p_{\rm L}(t)$ and $q_{\rm L}(t)\,{\rightarrow}\,0$ as $t\,{\rightarrow}\,{+}\infty$. ◻

|  |  |  |  |  |
|:---|:---|:---|:---|:---|
|  |  |  | Pioneer 2DX |  |
| Parameters | Pioneer 3DX | Pioneer 2DX | with load (4 Kg) | Units |
| $\vartheta_{1}$ | 0.24089 | 0.3037 | 0.1992 | s |
| $\vartheta_{2}$ | 0.2424 | 0.2768 | 0.13736 | s |
| $\vartheta_{3}$ | $-$`<!-- -->`{=html}9.3603e$^{-4}$ | $-$`<!-- -->`{=html}4.018e$^{-4}$ | $-$`<!-- -->`{=html}1.954e$^{-3}$ | s.m/rad$^{2}$ |
| $\vartheta_{4}$ | 0.99629 | 0.9835 | 0.9907 |  |
| $\vartheta_{5}$ | $-$`<!-- -->`{=html}3.7256e$^{-3}$ | $-$`<!-- -->`{=html}3.818e$^{-3}$ | $-$`<!-- -->`{=html}1.554e$^{-2}$ | s/m |
| $\vartheta_{6}$ | 1.0915 | 1.0725 | 0.9866 |  |
|  |  |  |  |  |

**Corollary 2**.  * Assume $b_{\rm L}(t)=b_{\rm L}$, $f(q_{\rm L})=q_{\rm L}(t)$ and suppose that the leader has a path to all agents in graph $\mathcal{G}$. Then, $p_{i}(t)\rightarrow
p_{\rm L}(0)+{q_{\rm L}(0)}/{b_{\rm L}}$ as $t\rightarrow
+\infty$, where $p_{\rm L}(0)$ and $q_{\rm L}(0)$ are the initial position and velocity of the leader, respectively.*

**Proof.* If letting $b_{\rm L}(t)=b_{\rm L}$, $f(q_{\rm L})=q_{\rm L}(t)$, then the solution of () is given by $$\begin{aligned}
p_{\rm L}(t)&=p_{\rm L}(0)+\frac{q_{\rm L}(0)}{b_{\rm L}}-\frac{q_{\rm L}(0)}{b_{\rm L}}\exp(-b_{\rm L}t)\\
q_{\rm L}(t)&=q_{\rm L}(0)\exp(-b_{\rm L}t)
\vspace*{-2pt}
\end{aligned}$$ so $p_{i}(t)\rightarrow
p_{\rm L}(0)+{q_{\rm L}(0)}/{b_{\rm L}}$ as $t\rightarrow+\infty$. ◻*

**Remark 3**.  * -1Protocol (3) contains a leader, which provides the information to other agents and does not receive any information from other agents. In protocol (3), the second term on the right hand side of $u_{i2}(t)$ suggests that the leader has relationships with some of the other agents in the group. In particular, if for any $i\in\bar{N}$, $c_{i{\rm L}}>0$, then the leader has connections with all other agents. If for a specific agent $r$ and $c_{rL}>0$, $c_{i{\rm L}}=0$ ($i\in\bar{N}, i\neq r$), then the leader has a connection with one agent in the group. Since the leader has a path to all other agents in graph $\mathcal{G}$, at least one agent can directly receive information from the leader, and the other agents receive information from the others either directly or indirectly. Theorem 2 also confirms that control of one agent in the group can ensure all agents to achieve a desired consensus value provided that the graph $\mathcal{G}$ is connected .*

# Numerical simulations

In order to illustrate the theoretical results, numerical simulations were carried out for the system consisting of six ($N=6$, $n=1$) non-identical agents with and without a leader, as shown in Fig. . In the leaderless case, the non-linear control protocols can be selected as $$\begin{aligned}
u_{i1}(t) &= -b_{i}(t) f(q_{i}),\notag\\
u_{i2}(t)&=\sum_{j \in
\mathcal{N}_{i}}c_{ij} \,(p_{j}(t)-p_{i}(t))+\sum_{j \in
\mathcal{N}_{i}}c_{ij}(p_{j}(t)-p_{i}(t))^{3}\label{eq9}
\end{aligned}$$ and in the presence of a leader, the control protocols can be chosen as $$\begin{aligned}
u_{i1}(t)&=-b_{i}(t)f(q_{i}),\notag\\
u_{i2}(t)&=\sum_{j\in\mathcal{N}_{i}}c_{ij}
(p_{j}(t)-p_{i}(t)+(p_{j}(t)-p_{i}(t))^{3})\notag\\
&\quad +c_{i{\rm L}}\, (p_{\rm L}(t)-p_{i}(t)+(p_{\rm L}(t)-p_{i}(t))^{3})\label{eq10}
\end{aligned}$$ where $f(q_{i})=q_{i}(t)+\omega_{1}\sin(q_{i})$ and $f(q_{\rm L})=q_{\rm L}(t)+\omega_{2} \sin(q_{\rm L})\break (\omega_{1}\geq 0,
\omega_{2}\geq 0)$.

Fig.  shows the consensus process of positions and velocities of all agents in system (1) with protocol (9). It indicates that all non-identical agents can achieve consensus within a short period of time. From the result of Corollary 1, the consensus value is found to be ${1}/{4.2}\sum_{i=1}^{6}(b_{i}p_{i}(0)+m_{i}q_{i}(0))\approx 1.5167$ as $t\rightarrow +\infty$ when $b_{i}(t)=0.2i$, $m_{i}=0.1i$ and $f(q_{i})=q_{i}(t)$. Fig.  illustrates that all positions and velocities of system (1) with protocol (10) can track the position and velocity of the leader. When $b_{\rm L}=0.6$, $f(q_{\rm L})=q_{\rm L}(t)$, the exact position of the leader is $p_{\rm L}(0)+{q_{\rm L}(0)}/{b_{\rm L}}=1.5$ as $t\rightarrow +\infty$ from Corollary 2. It can be concluded from Figs.  and that the theoretical results are in good agreement with numerical simulations.

# Conclusion

The consensus of multiple non-identical agents with control protocols has been studied for the agents with and without a leader. It was shown that all different agents in the group can achieve consensus, and an explicit consensus value for the group was obtained for some cases. For the agents with a leader, it was found that all agents can track the position and velocity of the leader, which are distinct from those of any follower agents. Numerical simulations were used to illustrate the theoretical results.

# Acknowledgments

Shan Cheng is indebted to the China Scholarship Council (CSC) for its financial support as a visiting scholar at the University of Technology, Sydney. This work was partially supported by the National Science Foundation of China (Grant no. 10972129), the Innovation Program of Shanghai Municipal Education Commission (Grant no. 10ZZ61), the Shanghai Leading Academic Discipline Project (Project no. S30106), and the Graduate Innovation Foundation of Shanghai University (Project no. SHUCX 101080).


abstract: Highlights in a sport video are usually referred as actions that stimulate excitement or attract attention of the audience. A big effort is spent in designing techniques which find automatically highlights, in order to automatize the otherwise manual editing process. Most of the state-of-the-art approaches try to solve the problem by training a classifier using the information extracted on the tv-like framing of players playing on the game pitch, learning to detect game actions which are labeled by human observers according to their perception of highlight. Obviously, this is a long and expensive work. In this paper, we reverse the paradigm: instead of looking at the gameplay, inferring what could be exciting for the audience, we directly analyze the audience behavior, which we assume is triggered by events happening during the game. We apply deep 3D Convolutional Neural Network (3D-CNN) to extract visual features from cropped video recordings of the supporters that are attending the event. Outputs of the crops belonging to the same frame are then accumulated to produce a value indicating the Highlight Likelihood (HL) which is then used to discriminate between positive (*i.e*.. when a highlight occurs) and negative samples (*i.e*.. standard play or time-outs). Experimental results on a public dataset of ice-hockey matches demonstrate the effectiveness of our method and promote further research in this new exciting direction.
author: Marco Godi; Paolo Rota; Francesco Setti
bibliography: refs.bib
title: Indirect Match Highlights Detection with Deep Convolutional Neural Networks

# Introduction

Sport video summarization, or highlights generation, is the process of creating a synopsis of a video of a given sport event that gives the viewer a general overview of the whole match. This process incorporates two different tasks: (1) to detect the most important moments of the event, and (2) organize the extracted content into a limited display time. While the second point is a widely-known problem in the multimedia and broadcasting community, the definition of *what is a highlight* has different interpretations in the community. According to , highlights are "those video segments that are expected to excite the users the most". In , the focus relaxes from excitement to general attention, and thus salient moments are the ones that attract audience attention the most. These two definitions would imply to explicitly design specific models for extracting excitment from the crowd in one case and attention on the other. In this paper we overcome this problem by automatically learn visual features using deep architectures that discriminate between highlights and ordinary actions.

Traditionally, extracting sport highlights has been a labor intensive activity, primarily because it requires good judgment to select and define salient moments throughout the whole game. Then, highlights are manually edited by experts, to generate a video summary that is significant, coherent and understandable by humans. State-of-the-art artificial intelligence is still far away from having solved the whole problem.

In the last years, there has been an increasing demand for automatic and semi-automatic tools for highlights generation, mainly due to the huge amount of data (*i.e*.. sport event videos) generated every day and made available through the Internet. Specialized broadcasters and websites are able to deliver sport highlights minutes after the end of the event, handling thousands of events every day. As a consequence, there has been extensive research in this area, with the development of several techniques based on image and video processing . More recently, many works started using additional sources of information to increase performances, including audio recordings , textual narratives , social networks , and audience behavior . Despite some solutions are already present on the market, performances are in general still fairly poor and we believe there is room for new research on this topic.
While previous work attempted to detect in sport videos actions that stimulate excitement  or attract attention  of the audience, in this paper we reverse the problem by analyzing the audience behavior to identify changes in emotions, that can only be triggered by highlights on the game field.

Specifically, we present a novel approach for sport highlight generation which is based on the observation of the audience behavior. This approach is based on the analysis of a set of space-time cuboids using a 3D-CNN architecture. All the samples are trained singularly, the result for each cuboid at a certain time step is then processed through an accumulator which generates a sort of highlight probability for the whole audience that will be used to perform the final ranking.

The rest of the paper is organized as follows: in Section  we briefly present the state-of-the-art in automatic highlight detection. In Section  we detail the proposed methodology, while in Section  we show some qualitative and quantitative results on a public dataset of hockey matches. Lastly, in Section  we draw some conclusions and perspectives for future works.

# Related work

Money and Angius  provide an extensive literature survey on video summarization. According to the taxonomy proposend in that paper, related work can be classified into three categories: (1) internal summarization techniques; 2) external summarization techniques; and 3) hybrid summarization techniques. By definition, *internal summarization techniques* rely only on information provided by the video (and audio) streams of the event. These techniques extract low-level image, audio, and text features to facilitate summarization and for several years have been the most common summarization techniques. *External summarization techniques* require additional sources of information, not contained in the video streams. These are usually user-based information –*i.e*.. information provided directly from users– and contextual information –such as the time and location in which the video was recorded. As for *hybrid summarization techniques*, both internal and external information are analyzed, allowing to reduce the semantic gap between the low level features and the semantic concepts.

**Social networks.** According to Hsieh *et al*.. , the quantity of comments and re-tweets can represent the most exciting moments in a sport event. A highlight can be determined by analyzing the keywords in the comments and observing if the number of comments and re-tweets passes a certain threshold. Fião *et al*..  uses emotions shared by the spectators during the match via social networks to build a system capable of generating automatic highlight videos of sports match TV broadcasts. Auxiliary sources of information are TV broadcast videos, the audio, the analysis of the movement and manual annotations (when available). The system also allows for the user to query the video to extract specific clips (*e.g*.. attacking plays of a specific team).

**Text.** In , Suksai and Ratanaworabhan propose an approach that combines on-line information retrieval with text extraction using OCR techniques. This way, they are able to limit the number of false positives.

**Audio.** Rui *et al*..  presents a method that uses audio signals to build video highlights for baseball games. It analyzes the speech of the match announcer, both audio amplitude and voice tone, to estimate whether the announcer is being excited or not. In addition, the ambient sound from the surrounding environment and the audience are also taken into considerations. Built on this work, Xiong *et al*..  handpicked the highlight events and analyzed the environment and audience sounds at each of those highlight events. They discovered that there exists a strong correlation between loud and buzzing noise and some major highlight events. This correlation exists in all the three sports being analyzed: baseball, golf, and soccer.

**Audience.** Peng *et al*..  propose the Interest Meter (IM), a system able to measure user's interest and thus use it to conduct video summarization. The IM takes account attention states (*e.g*.. eye movement, blink, and head motion) and emotion states (*e.g*.. facial expression). These features are then fused together by a fuzzy fusion scheme that outputs a quantitative interest score, determine interesting parts of videos, and finally concatenate them as video summaries. In , Conigliaro *et al*.. use motion cues (*i.e*.. optical flow intensity and direction entropy) to estimate the excitement level of audience of a team sport event and to identify groups of supporters of different teams. In , these features are used to identify highlights in team sport events using mean shift clustering.

# Method

The proposed highlights detection methodology uses a 3D Convolutional Neural Network (3D-CNN) to extract visual features from video recordings of the audience of the event, and classify them in positive samples (*i.e*.. when a highlight occurs) and negative samples (*i.e*.. standard play or timeouts).

From empirical observations, the audience reaction of a highlight (*e.g*.. a goal) lasts for at least the 10 seconds that follows the event itself. For this reason, temporal resolution is not a critical parameter and downsampling the video from 30 to 3 fps allowed us to reduce the computational burden without losing the informative part of the video. The 3D-CNN cuboid is extracted from a manually selected rectangular area that roughly contained the bulk of the audience, using a uniform grid with fixed spatial dimension of 100$\times$`<!-- -->`{=html}100 pixels, while the temporal resolution has been set to 30 frames. These parameters are the result of an a priori intuition that each block should be able to represent a portion of spectators which should not be too large, in order to reduce the computational burden, but at the same time it should not be too small since this would bring to be too much location dependent. For our model we used a sliding window with a stride of 50 pixels resulting in a maximum overlap between two crops of 50%

In order to detect and rank the most important moments in the video sequence we follow the idea of Conigliaro et al. , where information accumulators along time have been proposed to segment supporters of the two different playing teams. Our goal is however different: unlike them, we are interested in a global analysis of the excitement of the audience regardless of the supporting preference at a certain time. For this reason we are using an accumulator strategy over the whole audience location in the scene. Each spatio-temporal cuboid $C_i$, $i=1,...,N$ represents a sample that is fed into a 3D-CNN and analyzed independently; then, for each time instant the related probability score $p_i$, $i=1,...,N$ of being a positive class is accumulated over all the samples in the spatial dimension, generating a scalar value representing the *Highlight Likelihood* (HL) that is a score representing how likely a particular instant can be considered an highlight or not. A sketch of the overall system is shown in Fig. .

## Network Architecture

Inspired by earlier works on action recognition , we use a 3D Convolutional Neural Network composed by 4 convolutional and 3 fully connected layers.

The network takes as input video cuboids of 100$\times$`<!-- -->`{=html}100$\times$`<!-- -->`{=html}30, where the first two numbers refer to the spatial dimension while the third is the temporal depth (number of frames). The first two convolutional layers are composed 12 filters 3$\times$`<!-- -->`{=html}3$\times$`<!-- -->`{=html}3, to capture spatio-temporal features from the raw data. These are followed by a 2$\times$`<!-- -->`{=html}2$\times$`<!-- -->`{=html}2 max pooling layer to detect features at different scales. In the latter two convolutional layers, 8 3$\times$`<!-- -->`{=html}3$\times$`<!-- -->`{=html}3 convolutional filters have been used. In all convolutional layers the ReLU activation has been used. The network is then unfolded with a flatten layer followed by three fully connected layers of decreasing dimensionality (32, 8, and 2 neurons respectively). The final classification task is achieved by a softmax layer that outputs the probability of a test sample to belong to each of the two classes: "highlight" and "standard play".

# Experiments

In this section we provide both qualitative and quantitative results to validate our proposed methodology. For the evaluation we adopted the S-Hock dataset , a publicly available dataset composed by 6 ice-hockey games recorded during the Winter Universiade held in Trentino (Italy) in 2013. This dataset, besides a set of short videos heavily annotated on low level features (*e.g*.. people bounding boxes, head pose, and action labels), it provides also a set of synchronized multi-view full matches with high-level event annotation. In these games, the labeling consist in the time position of meaningful events such as goals, fouls, shots, saves, fights and timeouts.

In this work we considered only two matches: the final match (Canada-Kazakhstan) which is used for training the neural network, and the semi-final match (USA-Kazakhstan), used for testing.

## 3D-CNN training procedure

As mentioned briefly earlier, the positive class is named "highlights" and it represents all the spatio-temporal cuboids starting when a team scores a goal while the negative class (*i.e*.."standard play") includes other neutral situations happening during the game. In this work we excluded all the other significant annotated events (fouls, fights, etc.) to reduce the number of classes[^1]. In training phase the samples belonging to the two classes have been balanced to avoid dataset bias.

The S-Hock dataset provides a set of synchronized videos of the games including several views of the audience, at different resolution/zoom level, and of the complete game footage. The video acquisition is done from different points of view (frontal and slightly tilted to the side), in this work we used all these views to ensure a more robust model of training that is able to learn features that are more possibly scale and position invariant. Positive and negative samples are then splitted into training and validation sets with a ratio of 70%-30%. Data augmentation procedure has been performed (horizontal flips in the spatial dimension) not only to increase the amount of training data but also to augment the invariance of the network.

The final optimization is proposed as a classification problem, minimizing the categorical cross-entropy between the two classes. For this procedure we used the *RMSprop* algorithm, a generalization of the resilient backpropagation *rprop* algorithm that extends the ability to use only the sign of the gradient and to adapt the learning rate separately for each weight, to improve the work with minibatches. In our experiments we use minibatches of 64 samples each. A Dropout layer with 50% probability to disconnect the link is applied before the first two fully connected layers to reduce overfitting. The procedure iterates over the whole dataset until convergence, usually reached after about 10 epochs. The whole training procedure takes about 2 hours on a machine equipped with a NVIDIA Tesla K-80 GPU, using Keras/TensorFlow framework. The whole resulting dataset is composed of a total of 32,000 training samples.

## Quantitative Results

Here we report a quantitative performance evaluation of the 3D-CNN in detecting positive and negative highlight samples. From the second period of the testing game, we randomly selected 3000 positive samples as well as the same number of negative samples and we fed them into the trained network. In Fig.  the ROC curve is reported. The Area Under the Curve (AUC) is 0.87. Binary classification is performed by assigning the sample to the class corresponding to the higher score; under this conditions the network reaches 78% of accuracy, 69% of precision and a recall of 84%. Results themselves are quite good considering the difficulty of the task, however, our goal is different, since we are using those results in a more sophisticated framework to infer and rank interesting events during the whole game. Consequently we expect a certain amount of noise in such prediction since in many cases the sample may be partially filled with empty seats (see Fig.  ), producing a wrong or at least biased prediction toward the negative class. However, this problem is minimized with the use of the accumulator approach and due to the fact that the empty-seats location will be very little informative in the whole sequence, while the crowded locations, where most of the spectators are situated, will convey most of the information used for the final decision.

## Qualitative examples

We also provide qualitative results to validate our approach. Fig.  shows the HL score, summed over all the cuboids, at every non overlapping 10-second slice during an entire match (3 periods of 20 minutes plus timeouts). Goals are clearly identified in the first two periods, while in the third one other events also trigger the audience behavior; in particular, there are two prominent events that don't correspond to goals at 18:45 (which is caused by a player almost scoring) and at 28:15 (which is caused by a foul in front of the goaltender, and the resulting penalty). We can easily see that there is a correlation between HL score and important events in the game and that goals usually cause the biggest reaction on the spectators.

# Conclusions

In this paper we propose a method to temporally locate highlights in a sport event by analyzing solely the audience behavior. We propose to use a deep 3D convolutional neural network on cuboid video samples to discriminate between different excitement of the spectators. An spatial accumulator is used to produce a score which is proportional to the probability of having an interesting highlight in that precise time location. This enables the model to identify goals and other salient actions.

Despite being very simple, the model we present provides good preliminary result on a public dataset of hockey games, encouraging further research based on this approach. In our opinion, the main limit of this model is in the way we take into account the temporal information; indeed we extend a standard CNN to work with 3D data, where the third dimension is time. A more sophisticated model, such as recurrent neural networks (RNN) and long-short term memory (LSTM), could benefit the final inferential results. As future work we intend to replace the accumulator with such a temporal model, expanding the classification to a multiclass problem in order to detect different events. In order to do so, the dataset has to be enlarged possibly on a different location to make sure the network is learning more general discriminative features.

[^1]: These events are indeed generating different types of excitement, we could not investigate further for lack of annotated data, but this is an argument that we consider worthed of further research.


abstract: The prompt $\gamma$-ray emission of Gamma-Ray Bursts (GRBs) is currently interpreted in terms of radiation from electrons accelerated in internal shocks in a relativistic fireball. On the other hand, the origin of the prompt (and early afterglow) optical and X-ray emission is still debated, mostly because very few data exist for comparison with theoretical predictions. It is however commonly agreed that this emission hides important clues on the GRB physics and can be used to constrain the fireball parameters, the acceleration and emission processes and to probe the surroundings of the GRBs. *ECLAIRs* is a microsatellite devoted to the observation of the prompt optical and X-ray emission of GRBs. For about 100 GRBs yr$^{-1}$, *independent of their duration*, *ECLAIRs* will provide high time resolution high sensitivity spectral coverage from a few eV up to $\sim 50$ keV and localization to $\sim 5$" in near real time. This capability is achieved by combining wide field optical and X-ray cameras sharing a common field of view ($\hbox{\rlap{\raise.3ex\hbox{$>$}}\lower.8ex\hbox{$\sim$}\ }2.2$ steradians) with the coded-mask imaging telescopes providing the triggers and the coarse localizations of the bursts. Given the delays to start ground-based observations in response to a GRB trigger, *ECLAIRs* is unique in its ability to observe the early phases (the first $\sim 20$ sec) of all GRBs at optical wavelengths. Furthermore, with its mode of operation, *ECLAIRs* will enable to search for optical and X-ray precursors expected from theoretical grounds. Finally *ECLAIRs* is proposed to operate simultaneously with *GLAST* on a synchronous orbit. This combination will ensure broad band spectral coverage from eV to GeV energies for the GRBs detected by the two satellites, *ECLAIRs* further providing their accurate localization to enable follow-up studies. *ECLAIRs* relies upon an international collaboration involving theoretical and hardware groups from Europe and the United States. In particular, it builds upon the extensive knowledge and expertise that is currently being gained with the *HETE-2* mission.
author: Didier Barret
date: 2024-09-30
title: ECLAIRs: A microsatellite to observe the prompt optical and X-ray emission of Gamma-Ray Bursts

final

address=Centre d'Etude Spatiale des Rayonnements, CNRS/UPS, 31028 Toulouse Cedex 04, France, firstname.lastname@example.com,

# Introduction

GRBs occur at cosmological distances and are the most violent explosive phenomena presently observed in the Universe. For the strongest GRBs, up to $\sim 10^{54}$ erg (assuming isotropic emission) can be radiated in the $\gamma$-ray domain on a very short timescale (from milliseconds to $\sim 100$ seconds). In the currently favored models, GRBs are associated with the collapse of massive stars (collapsar, ) or mergers of two compact stars (two neutron stars or a neutron star and a black hole, e.g., ) (see e.g., for recent reviews). Since GRBs are observable across the whole Universe, if they are indeed linked to the ultimate stages of massive star evolution, their redshift distribution should reveal the formation rate of massive stars up to very high redshifts, thus making GRBs effective cosmological probes (see e.g., ).

In both the collapsar and the merger models, the final product is a stellar mass black hole and a rapidly rotating torus, from which the energy can be extracted via magnetohydrodynamic processes. The energy release in a very small volume produces a relativistic fireball with a Lorentz factor of at least a few hundred (e.g., ). When the relativistic flow decelerates in the interstellar medium (ISM), a forward and a reverse external shock are produced. The forward external shock can account for the afterglow emission observed at radio, optical and X-ray wavelengths ( for a recent review).

Whereas the physics of the afterglow is relatively well understood, the origin of the prompt emission is still debated, especially in X-rays and optical. In the relativistic fireball model, the Lorentz factor of the wind is supposed to be variable so that successive shells of plasma have large relative velocities leading to the formation of internal shocks (). In that model, the non-thermal $\gamma$-ray emission is associated with either synchrotron or inverse Compton emission of electrons accelerated in those shocks. In X-rays and optical, the picture is not as clear, mostly because there are not as many high quality observations available in that energy range compared to the $\gamma$-ray domain. There is however growing observational and theoretical evidence that this energy domain contains critical information on the GRB physics, the nature of the progenitors, the way the initial bulk energy is converted into electromagnetic radiation. A better understanding of the GRB physics is required to test models predicting that GRBs may be sources of ultra high-energy cosmic rays, neutrinos, gravitational waves (see for a recent review). This understanding is also required if one intends to make GRBs a reliable tool for cosmology and for the study of the Universe at very high redshifts.

*ECLAIRs* is a microsatellite proposed to the French Space Agency (CNES). It is specifically devoted to the observation of the prompt optical and X-ray emission of GRBs. In the next section, we briefly describe the main scientific objectives of *ECLAIRs* emphasizing on the area where it will bring an outstanding contribution. We then present the science payload and mission concept. Finally we emphasize on the complementarity of *ECLAIRs* with two other missions (*GLAST* and *SWIFT*) supposed to fly simultaneously with *ECLAIRs*.

# Scientific objectives

With *ECLAIRs* we wish to use, for the short and long duration GRBs, the prompt optical and X-ray emission to probe 1) the physics at work during the event and 2) the surrounding of the burst to get insights on their origin. In addition, thanks to its instrumental capabilities, with *ECLAIRs*, we will investigate the existence of optical and X-ray precursors expected from theoretical grounds, and whose presence would put unprecedented constraints on any GRB models.

As far as the prompt emission is concerned, as discussed above, very few data exist in optical and X-rays in contrast to the $\gamma$-ray domain. In the optical, so far only one GRB has been detected (by the ROTSE automated telescope; GRB990123, ), and for a few others, upper limits are available for the late part ($\ge 10-20$ second after the onset) of the events (e.g. ). In X-rays, the situation is slightly better, mostly thanks to the *GINGA* (e.g., ) and *Beppo-SAX* satellites .

## What can be learned from the prompt optical emission?

It has suggested that the prompt optical emission as the one observed in GRB990123 may be associated with electrons accelerated in the reverse external shock . The strength of the optical emission depends on various parameters, but can in principle yield constraints on the wind initial Lorentz factor and the interstellar medium density . Alternatively, the prompt optical emission could arise from the forward shock of the blast wave when it propagates in the pre-accelerated and pair-loaded environment . This emission can also give constraints on the radiation process itself; i.e., synchrotron versus Inverse Compton emission; a much stronger optical flash is expected in the Inverse Compton scenario (e.g., ).

The reasonable question to ask is why the prompt optical emission has so far been observed from only one GRB (GRB990123, ). The poor location accuracy of GRB detectors (e.g., BATSE), the delays in getting the finalized positions to the ground, the limited observing efficiency of automated optical telescopes, their response time, all conspire to make sensitive (below mag $\sim$ 14) and truly simultaneous observations of GRBs over the whole event almost impossible. For ROTSE, the shortest response time that has been achieved is $\sim 10$ seconds . *ECLAIRs* will not face any of these problems as the optical and X-ray cameras will operate continuously over a common field of view. Optical coverage will thus be granted for all types of bursts, independently of their duration, before, during and even after the event. This unique capability will also offer the opportunity to study the transition between the prompt and early afterglow phases at similar wavelengths.

Extinction by dust in the host galaxy may naturally prevent the detection of the prompt optical emission. This is the argument used to explain the lack of optical emission in some afterglows, otherwise detected in X-rays and in radio. Dust extinction is not unexpected if GRBs are associated with massive star formation. Djorgovski et al. (2001) have however found that the *maximum* fraction of optical afterglows hidden by dust is $\sim 50$%. This is an upper limit, as some optical afterglows may have been missed for various reasons: very high redshifts, rapid decline rate, intrinsic faintness.

*ECLAIRs* will seek optical emission down to magnitude $\sim 15$ (R band, 8 sec) for all GRBs. The properties of the prompt optical emission will be correlated with the properties of the afterglow optical emission, thus providing complementary constraints on the relativistic flow parameters and the surroundings of the event.

## What can be learned from the prompt X-ray emission?

Let us now consider the prompt X-ray emission. The good correlation between the temporal behaviour of the prompt X-ray and $\gamma$-ray emissions suggests that it is also produced in internal shocks. In X-rays however, there might be additional contributions from the reverse shock , the forward shock or from the photosphere of the fireball . With its excellent sensitivity, *ECLAIRs* will observe the prompt X-ray emission of all GRBs, allowing detailed time-resolved X-ray spectroscopy to be performed. These studies will set constraints on the Lorentz factor of the wind, its baryon loading, the emission mechanism and the relative contribution of the various shock regions in the overall emission.

The importance of observing the prompt X-ray emission has recently been reinforced by the discoveries of X-ray spectral features in *Beppo-SAX* observations: e.g., a transient absorption edge in GRB990705 and a transient emission feature in GRB990712 (spectral features are also observed in the X-ray afterglows, e.g., ). Well before these discoveries, it was predicted that effects of photo-electric absorption and Compton scattering from the circum-burst material should lead to observable changes in the intrinsic GRB spectrum, with the introduction of absorption cutoffs and features such as K-edges and emission lines . In principle, these features can be used to determine the density and composition of the ISM in the immediate vicinity of the GRB, the GRB redshift and possibly the nature of the GRB progenitor. For instance, the transient absorption edge observed from GRB990705 was satisfactorily modeled with photo-electric absorption by a medium with a large iron abundance, which could have been left there by a supernova event which occurred about 10 years before the burst . Similarly the transient emission feature seen in GRB990712 was shown to be consistent with thermal emission of a baryon-loaded expanding fireball when it becomes optically thin . The above interpretations are however made difficult by the limited statistical quality of the data. With its improved sensitivity and good time and spectral resolution, *ECLAIRs* will be able to observe the prompt emission of GRBs over the whole event and for all types of events.

## What are the short bursts ? What about the X-ray precursors ?

GRBs display a bimodal distribution in durations; the border is around 2 seconds with about 25% of the GRBs with durations less than that value. This distribution seems to correlate with spectral hardness; the shortest GRBs have on average harder spectra . It seems therefore plausible that the two distributions represent two distinct, although quite similar, physical phenomena. Extremely short GRBs may be due to primordial black hole evaporation, short GRBs to merging neutron stars, and the long ones to collapsars (see e.g., ). So far, due to observational limitations, afterglows have only been identified for the long duration GRBs and very little is known about the short GRBs. *ECLAIRs* will have the unique capability to observe both short and long duration GRBs. These observations will thus provide clues to the following questions: How does the multi-wavelength prompt emission of the short GRBs compare with those of the long GRBs? How does the prompt emission relate to the afterglow properties? What is the redshift distribution of short GRBs? Answering these questions will help in assessing whether the short duration GRBs are of different nature than the long ones.

By its mode of operation, *ECLAIRs* will also enable us to search and study X-ray and optical precursors. X-ray precursors have already been observed (), arising between 10 and 100 seconds before the main event. Several models have been put forward to explain these X-ray precursors (or soft excesses) (e.g., ), all making some specific predictions which require further data to be tested. What is the spectrum of the X-ray precursor? How does it evolves with time? How do its properties relate to the properties of the main event? These are some of the questions which will be addressed by *ECLAIRs*. In addition, Paczyński (2001) recently pointed out that there are theoretical reasons to expect strong optical flashes preceding GRBs (e.g., ), the detection of which would put stringent constraints on the range of parameters for GRB models.

# The *ECLAIRs* mission concept

The *ECLAIRs* mission concept results from the scientific goals described above and is optimized under the stringent constraints of a microsatellite: 50 kilos, 50 Watts and a total volume of $\sim 60$ cm $\times$ 60 cm $\times$ 30 cm (length, width, height) available for the science payload. A technical assessment study of *ECLAIRs* was carried out by CNES in December 2001. This study showed that *ECLAIRs* was feasible as a microsatellite, albeit with not much margins. *ECLAIRs* is now lining up for a selection at the end of 2002. This clearly leaves some time to work on the optimization of the science payload. To help us in this task, an international Science Advisory Committee (SAC) was set up for the mission. The science payload described below accounts for the results from the CNES study and for the advises received from the SAC.

|  | E-LAXT | E-SXC | E-WFOC |
|:---|:--:|:--:|:--:|
| Band pass | $\sim 3$–50/5-150 keV | 0.4–15 keV | 500-700 nm |
| Number of units | 2 | 6 (3 pairs) | 4 |
| Offset angle | $\pm 10^{\circ}$ | $-28^{\circ}, 0, +28^{\circ}$ | $\pm 25^{\circ}$ |
| Mass (kg) | 14 | 11 | 14 |
| Power (instrument + electronics) (W) | 18 | 8 | 16 |
| Field of view (one unit, FWZR) | $\sim 120^{\circ}\times120^{\circ}$ | $\sim 53^{\circ}\times 53^{\circ}$ | $\sim 50^{\circ}\times 50^{\circ}$ |
| Positionning accuracy | $\sim 0.5^{\circ}$ | 5" | 5" |
| Number of GRBs yr$^{-1}$ (total) | $\sim 100$ | $\sim 100$ | ? |
| Limiting mag. (R) (S/N=8) | … | … | 14.8, 17.4 (8, 1000 sec) |

The ECLAIRs science payload consisting of three instruments: the ECLAIRs Large Area X-ray Telescope (E-LAXT), the ECLAIRs Soft X-ray Cameras (E-SXC), the ECLAIRs Wide Field Optical Cameras (E-WFOC). Two options are considered for E-LAXT: Silicon and higher density detectors (e.g. CdZnTe). Si would cover the energy range 3-50 keV whereas CdZnTe would cover from 5 keV to 150 keV.

## The science payload

The science payload consists of three sets of instruments (see Table ). The Large Area X-ray Telescope (E-LAXT), the Soft X-ray Cameras (E-SXC), and the Wide Field Optical Cameras (E-WFOC) (see Fig ). It will be provided by a consortium of institutes which have developed a considerable expertise along the preparation and operation of missions, such as *HETE-2* and INTEGRAL. The US contribution to *ECLAIRs* will be the subject of a SMEX/MOO proposal to NASA in 2002.

### ECLAIRs - Large Area X-ray Telescope

The Large Area X-ray Telescope (E-LAXT) is made of two identical conventional 2D coded-mask imaging telescopes, with offset looking directions. The mask is located 15 cm above the detector. The detector is a pixel semiconductor detector. In the baseline, each pixel was a 2 mm thick Si PIN diodes of 1cm$^2$. The mask cells match the pixel size. Large area Si PIN detectors with their associated low-noise low-power front-end electronics are currently developed at CESR as part of an R&T program funded by CNES. At low power, the expected noise level should result in an energy resolution of $\sim 1$ keV (at 6 keV, -40C) making possible a low energy threshold of $\sim 3$ keV. The thickness of the diodes ensures an energy coverage up to $\sim 40$ keV: matrix of Si PIN diodes are therefore a possible detector solution for *ECLAIRs*.

However, as suggested by the SAC, there are alternatives to Silicon for the E-LAXT detector: CdTe, CdZnTe. Both would extend the energy range of *ECLAIRs* in the hard X-ray range, which would help for the detection of GRBs. CdZnTe, as the ones developed for *SWIFT* or AXO have excellent performances for a mW/cm$^2$ ratio similar to Silicon. In addition, using strip readout techniques they have been demonstrated to work down to 5 keV (e.g. ). Considering CdZnTe, the detector of one E-LAXT could have an effective area of $25 \times  25$ cm$^2$, covered with pixels of $5 \times  5$ mm$^2$ and 2 mm thickness. The imaging system would have an angular resolution of 2 degrees and a positioning accuracy of $\sim 0.5$ degree. Due to the stringent mass constraints on a microsatellite, the mask and shielding will be effective only below $\sim 50$ keV. Whereas the trigger will be obtained at energies above $\sim$ 50 keV, the position of the GRB will be derived from the images reconstructed below that energy. Using a simplified model for the E-LAXT and the Log(N)/Log(P) curve derived from the BATSE 4B catalog , we have estimated the rate of GRBs localized by the two E-LAXT units to be larger than $100$ GRB yr$^{-1}$.

### ECLAIRs - Soft X-ray Cameras

The Soft X-ray Cameras (E-SXC) for *ECLAIRs* are based upon the successfully-flown *HETE-2* design (see also these proceedings). The operating principle is that of a coded-mask imager, in which a 1-D coded mask is rigidly suspended above an X-ray charge-coupled device (CCDID-34). The E-SXC assembly is made of 6 camera modules, covering a field of view of 2.7 sr. The CCDID-34 (3K$\times$`<!-- -->`{=html}6K array; 10$\mu$m square pixels, 20") has an overall size of 30 mm$\times$ 60 mm and is currently in production at MIT Lincoln Laboratory. It improves over the CCID-20 used for *HETE-2* by a greater energy coverage (0.4-15 keV versus 0.8-10 keV), a better time resolution (0.25 sec versus 1 sec), and a better quantum efficiency (sensitivity of $\sim 400$ mCrab, 1 sec, $4 \sigma$). The E-SXC will provide  5" burst localizations (at S/N =8). About 100 GRBs yr$^{-1}$ should be detected in the 6 units.

### ECLAIRs - Wide-Field Optical Cameras

The Wide-Field Optical Cameras (E-WFOC) for *ECLAIRs* are derived from the star camera units successfully flown on *HETE-2* . The large field of view is achieved by four such cameras. The limiting magnitude in R is 14.8 (8 s at S/N=8) for one E-WFOC. Each of the four modules utilizes a moderately fast, well-corrected optical lens (focal length of 80 mm, f/0.9) coupled to a 2$\times$`<!-- -->`{=html}2 array of MIT CCID-34 sensors, resulting in a hybrid focal plane with 6K$\times$ 6K pixels; each pixel is 10$\mu$m x 10$\mu$m (25.8"). The integration time is 2 seconds.

To achieve the light weight and low power required for *ECLAIRs*, the drive and readout electronics, as well as the digital frame buffer memory, for the E-SXC and E-WFOC instruments will be combined to the maximum degree possible.

The operating mode for the E-WFOC relies upon digitizing and storing successive 300 MB image frames in a four stage deep buffer, requiring a total of 1.2 GB SRAM. In response to triggers from the E-LAXT or E-SXC, we will select 4$^\circ \times 4^\circ$ regions-of-interest (ie 512 x 512 subarrays) from this large buffer, centered on the suspected burst coarse localization, for transfer into an optical burst memory. In addition, neighborhoods of twenty-five stars, extending out to 64$\times$`<!-- -->`{=html}64 pixels (27'$\times$`<!-- -->`{=html}27'), will also be stored as astrometric and photometric references. The accumulation of 500 frames (=1000 sec), each with burst and reference star data, will reside in 377MB of SRAM, and require 3 minutes to downlink during an X-band contact with an *ECLAIRs* ground station. Shift-and-add summation of the digitized, two-second resolution CCD data in ground processing will permit the E-WFOC to achieve an ultimate limiting sensitivity of R=17.4 (1000s at S/N=8). Centroiding will result in bright optical transient localizations accurate to $\pm$ 2", even in the presence of spacecraft pointing drift (assumed to be 47"/s, $3\sigma$, as specified for the Myriade spacecraft). For long term optical monitoring, we will also be able to downlink 45 full image frames per day (whole field of view at full angular resolution, every $\sim 30$ minutes), each containing more than 2.5 million star images. Downlinking of the full frame data will require 13 GB/day.

We are currently investigating near-infrared cameras (NIRC, 1-2 microns) as an alternative to the E-WFOC. There are three main advantages of considering NIRC. First, in NIR, the extinction is smaller than in the optical. This means that GRBs produced in dusty regions and not visible in the optical might become detectable in NIR. Second, NIRC would have the potential to observe higher z events, due to the Lyman alpha break. Finally, the discovery space is much larger in NIR than in the optical, as the NIR sky has never been searched for variability so far.

A detailed study is now required to determine whether cooled NIRC with good sensitivity, large field of view can be accommodated on *ECLAIRs* under the stringent constraints on power, mass, etc. of a microsatellite.

## Implementation of the mission

The baseline for *ECLAIRs* is an equatorial 550 km (low inclination) orbit for a low, stable background, low radiation damage to the CCD, and for the download of the science data to be possible with a single ground station. This orbit could be achieved by various launchers, as for example a PEGASUS from the Marshall Island or from Alcantara. We plan to reuse the *HETE-2* ground segment. In particular, one of the 3 S band stations (Singapore, Kwajalein, or Cayenne) will be converted to X band. For the alert system, we plan to use the *HETE-2* network of 12 VHF stations located along the equator. As will be discussed below, *ECLAIRs* is proposed to fly simultaneously with *GLAST*; therefore an ideal launch date would be around the end of 2006. The lifetime of the mission is foreseen for 5 years for a maximum synergy with *GLAST*.

## Operational considerations

As far as the attitude control is concerned, the instruments will point in the anti-earth direction during night time, and look at the pole during day time. This way the operating temperature of the instruments will be kept low (below -50 C). The triggering system of *ECLAIRs* is relatively simple. The on-board computer monitors continuously the count rates in E-LAXT. When a transient event is detected, a signal is sent to the E-SXC and E-WFOC for the most recent data to be stored in a dedicated memory. Two images from E-LAXT are then reconstructed before and during the event. From the difference of the two images, the rough position ($\sim 0.5$ deg accuracy) of the event is obtained and sent out to the ground and to the secondary instruments which use this position to obtain the final more accurate position. After $\sim 30$ seconds, the final position (5" accuracy) is transmitted to the ground. During the next passage to the ground station a high rate X-band communication (16 Mbits/s) allows the whole data set associated with the event to be downloaded.

The mission and science operations will be performed with the help of CNES control center in Toulouse. The mission and operation center which may be combined will be responsible for receiving the data from CNES, generating the spacecraft commands on a weekly basis, monitoring the health of the spacecraft and science payload, recovering the attitude, and for the quick-look analysis of the data for rapid distribution of the GRB final positions. In addition it will be responsible for the data archives and the education and public outreach program. The center will likely be provided by a consortium of institutes including the Geneva observatory, the Strasbourg observatory and the Leicester University X-ray group.

# A mission complementary to *GLAST* and *SWIFT*

*GLAST*[^1] is scheduled to be launched in March 2006 into a low earth orbit. The satellite will carry 2 instruments: the Large Area Telescope (LAT), which will observe emission from 20 MeV to 200 GeV, and the Gamma-ray Burst Monitor (GBM), which will detect transients from 20 keV to 20 MeV. The LAT detector is  50 times more sensitive than it's predecessor, *EGRET*. While only a few GRBs were detected by *EGRET*, *GLAST* is expected to observe nearly 200 GRBs per year. These bursts will also be detected by the GBM, so that the spectrum will be measured over 7 orders of magnitude. Unfortunately, only for the brightest bursts the positions derived by the LAT will be accurate enough to be used for follow up observations. Provided that *ECLAIRs* and *GLAST* can remain on a similar orbit (adjustment and maintenance of the orbit can indeed be achieved through the chemical propulsion system of the microsatellite) *ECLAIRs* would greatly enhance the *GLAST* science, by both extending the spectra to lower energies (down to $\sim 2$ eV) and by improving the localizations in near real-time to enable follow-up observations in the afterglow regime. Given that the GBM has a FOV much larger than *ECLAIRs*, all GRBs seen by *ECLAIRs* will be also detected by the GBM, thus providing for $\sim
100$ GRB yr$^{-1}$, spectral coverage from about 7 decades in energy, and for those detected by the LAT ($\sim 80$) over 11 decades in energy! This broad band spectral coverage will enable discrimination between the various radiation processes proposed for the multi-wavelength GRB emission, including those, yet to be tested, put forward for the GeV emission (inverse Compton, Synchrotron emission, see e.g., ). This will open a completely new window on the GRB physics, setting for the first time real constraints on models predicting that GRBs are sources of ultra-high energy cosmic rays and neutrinos. Furthermore, the ability to locate the *GLAST*-GRBs precisely, making possible the identification of the host galaxies and the measure of the redshifts will enable the systematics of the GeV emission to be studied, and the *GLAST*-GRBs to be compared, as a class of events, to the GRBs detected by satellites operating at lower energies (*Beppo-SAX*, *HETE-2* and *SWIFT*). Finally, for those GRBs for which the redshift will be determined, cut-offs in the observed GeV spectrum can be used to infer the level of ultra-violet to infra-red background light which is a direct tracer of star and Galaxy formation in the early Universe . The complementarity between *GLAST* and *ECLAIRs* is best illustrated in Fig. where the observing energy range is plotted against the observing time window of the events. *ECLAIRs* was presented at the last *GLAST* GRB working group and received strong support.

*SWIFT*[^2] is a NASA mission dedicated to the study of the GRB afterglows. It should be launched in Fall 2003, with a nominal on-orbit lifetime of 3 years. It will carry three instruments: The Burst Alert Telescope (BAT) covering the 10 to 150 keV range, and two Narrow Fields Instruments (NFIs); the X-Ray Telescope (XRT, 0.2-10 keV) and the UV Optical Telescope (UVOT, 170-650 nm). The observing strategy of *SWIFT* is to point the NFIs after the detection of a GRB in the BAT. This strategy clearly means that *SWIFT* will miss the early X-ray and optical emission of all GRBs. The time to point the NFIs to the direction of the GRB should range between 20 and 70 seconds, with a mean value of 50 sec. Its ability to observe the precursors and activity during the burst will thus make *ECLAIRs* a very complementary mission to *SWIFT* (see Fig. ).

# Conclusions

Fortunately GRBs are extremely bright events which can easily be detected and studied with an instrumentation matching the stringent mass and power constraints of a microsatellite. GRBs have been proved to be highly complex phenomena whose understanding requires multi-wavelength observations of the prompt and afterglow phases and follow-up ground-based observations to determine their host galaxies and their redshifts. *ECLAIRs* will thus bring a significant contribution to a better understanding of GRBs by providing high sensitivity observations of the prompt optical/X-ray emission and accurate localization of more than 100 gamma-ray burst per year.

# Acknowledgments

It is a real pleasure for me to thank all the members of the *ECLAIRs* collaboration for their interest in the mission and their support: J.L. Atteia, M. Boër, A. Beloborodov, A. Castro-Tirado, T. Courvoisier, F. Daigne, J.P. Dezalay, B. Dingus, M. Ehanno, P. Goldoni, P. Guillout, J.M. Hameury, G. Henri, K. Hurley, P. Jean, G. Jernigan, J.P. Kneib, D. Lamb, P. Mandrou, A. Marcowith, F. Martel, L. Michel, R. Mochkovitch, C. Motch, J.P. Osborne, M. Pakull, G. Pelletier, J. Poutanen, V. Reglero, G. Ricker, J. Rodrigo, A. Short, R. Svensson and M. Ward.

A special thank also to C. Meegan and N. Gehrels for their strong support on behalf of the GLAST collaboration.

I also wish to thank the members of the *ECLAIRs* Science Advisory Committee for their very valuable inputs which will help in optimizing the mission: J.L. Atteia, A. Castro-Tirado, F. Frontera, J. Hjorth, N. Kawai, S. Kulkarni, C. Meegan, R. Mochkovitch, T. Piran, G. Ricker, B. Stern, S. Woosley.

I am very grateful to G.K. Skinner, J.L. Atteia and G. Ricker for their continuous help and support.

[^1]: http://www-glast.stanford.edu/

[^2]: http://swift.sonoma.edu/