author: Martin Monperrus et al.
bibliography: bibliography.bib

```latex
\begin{news}{3}{Human-competitive Patches in Automatic Program Repair with Repairnator}{Martin Monperrus, Simon Urli, Thomas Durieux, Matias Martinez, Benoit Baudry, Lionel Seinturier}{Software engineering research at KTH Royal Institute of Technology \& Inria}{hlabel i}

\textbf{Repairnator is a bot. It constantly monitors software bugs discovered during continuous integration of open-source software and tries to fix them automatically. If it succeeds to synthesize a valid patch, Repairnator proposes the patch to the human developers, disguised under a fake human identity. To date, Repairnator has been able to produce \numprint{5} patches that were accepted by the human developers and permanently merged in the code base. This is a milestone for human-competitiveness in software engineering research on automatic program repair.}

\vspace{.5cm}

Program repair research pursues the idea that algorithms can replace humans to fix software bugs \cite{Monperrus2015}.
A bug fix is a patch that inserts, deletes or modifies source code. For example, in the following patch, the developer has modified the condition of the if statement:

\begin{lstlisting}
- if (x < 10)
+ if (x <= 10)
  foo();
\end{lstlisting}

A program repair bot is an artificial agent that tries to synthesize source code patches. It analyzes bugs and produces patches, in the same way as human developers involved in software maintenance activities.
This idea of a program repair bot is disruptive, because today humans are responsible for fixing bugs.
In others words, we are talking about a bot meant to (partially) replace human developers for tedious tasks.

When a bot tries to achieve a task usually done by humans, it is known as a human-competitive task \cite{koza2010human}.
The empirical evaluations of program repair research \cite{defects4j-repair}  show that current program repair systems are able to synthesize patches for real bugs in real programs.
However, all those patches were synthesized for past bugs, for bugs that were fixed by human developers in the past, usually years ago. While this indicates the technical feasibility of program repair, this fails to show that program repair is human-competitive.

\subsection*{Human-competitiveness}

To demonstrate that program repair is human-competitive, a program repair bot has to find a high-quality patch before a human does so.
In this context, a patch can be considered to be human-competitive if it satisfies the two conditions of timeliness and quality.
Timeliness refers to the fact that the system must find a patch before the human developer. In other words, the prototype system must produce patches in the order of magnitude of minutes, not days.
Also, the patch generated by the bot must be correct-enough, of similar quality -- correct and readable -- compared to a patch written by a human.
Note that there are patches that look correct from the bot's point of view, yet that are incorrect (this is known as overfitting patches in the literature \cite{Smith2015,defects4j-repair}). Those patches are arguably not human-competitive, because humans would never accept them in their code base.

Consequently, for a patch to be human-competitive
1)  the bot has to synthesize the patch faster than the human developer
2) the patch has to be judged good-enough by the human developer and permanently merged in the code base.

There is one more aspect to consider. It has been shown that human engineers do not accept contributions from bots as easily as contributions from other humans, even if they are strictly identical \cite{murgia2016among}. The reason is that humans tend to have a priori biases against machines, and are more tolerant to errors if the contribution comes from a human peer. In the context of program repair, this means that developers may put the bar higher on the quality of the patch, if they know that the patch comes from a bot. This would impede our quest for a human-competitiveness proof in the context of program repair.

\begin{table*}
\begin{minipage}{1\textwidth}
\renewcommand{\arraystretch}{1.3}
\rowcolors{2}{gray!20}{white}
\begin{tabularx}{\textwidth}{p{21mm}  X  p{85mm}}\toprule
\textbf{Date} & \textbf{Contribution}  & \textbf{Developer comment}  \\ \hline

Jan 12, 2018 & \href{https://github.com/aaime/geowebcache/pull/1}{aaime/geowebcache/pull/1} & ``Thanks for the patch!'' \\

Mar 23, 2018 & \href{https://github.com/parkito/BasicDataStructuresAndAlgorithms/pull/3}{parkito/BasicDataCodeU[...]/pull/3} & ``merged commit 140a3e3 into parkito:develop'' \\

April 5, 2018 & \href{https://github.com/dkarv/jdcallgraph/pull/2}{dkarv/jdcallgraph/pull/2} & ``Thanks!''  \\

May 3, 2018 & \href{https://github.com/eclipse/ditto/pull/151}{eclipse/ditto/pull/151} & ``Cool, thanks for going through the Eclipse process and for the fix.''  \\

June 25, 2018 & \href{https://github.com/donnelldebnam/CodeU-Spring-2018-29/pull/59}{donnelldebnam/CodeU[...]/pull/151} & ``Thanks!!''\\

\bottomrule
\end{tabularx}
\end{minipage}
\caption{Human-competitive contributions on Github: patches synthesized by the Repairnator robot and accepted by the human developer.}
\label{tab:repainator-accepted-patches}
\end{table*}

To overcome this problem, we have decided early in the project that all Repairnator patches would be proposed under a fake human identity. We have created a GitHub user, called Luc Esape, who is presented as software engineer in our research lab. Luc has a profile picture and looks like a junior developer, eager to make open-source contributions on GitHub. Now imagine Repairnator, disguised as Luc Esape proposing a patch: the developer reviewing it
thinks that she is reviewing a human contribution.
This camouflage is required to test our scientific hypothesis of human-competitiveness. Now, for sake of ethics, the real identity of Luc has been disclosed on each of his pull-requests.

\subsection*{Automatic Repair and Continuous Integration}

Continuous integration, aka CI, is the idea that a server compiles the code and runs all tests for each commit made in the version control system of a software project (e.g. Git).
In CI parlance, there is a \emph{build} for each commit.
A build contains the information about the source code snapshot used (e.g. a reference to a Git commit), the result of compilation and test execution (e.g. fail or success), and an execution trace log.
A build is said to be failing if compilation fails or at least one test case fails. It has been shown that approximately one out of 4 builds fails, and that the most common cause for build failure is a test failure \cite{vassallo2017tale}.

The key idea of Repairnator is to automatically generate patches that repair build failures, then to show them to human developers, to finally see whether those human developers would accept them as valid contributions to the code base. If this happens, that would be evidence of human-competitiveness in program repair.

This setup --automatically repairing build failures happening in continuous integration -- is particularly appropriate and timely for the following reasons.
First, build failures satisfy the core problem statement of test-suite based program repair~\cite{Monperrus2015}, where bugs are specified as a failing test-cases, and those failing test cases are used to drive the automated synthesis of a patch~\cite{Monperrus2015}.
Second, it allows comparing the bots and humans on a fair basis: when a failing test is discovered on the continuous integration server, the human developer and the bot are informed about it, \emph{at the exact same time}. This test failure notification is the starting point of the human vs. bot competition.

Repairnator's focus on build failures is unique, but it fits in the big picture of intelligent bots for software \cite{Lebeuf2017}. For instance, Facebook has a tool called SapFix that repairs bugs found with automated testing. Also related,  the DARPA Cyber Grand Challenge bot attackers and defenders try to be human-competitive with respect to  security experts.

\subsection*{Repairnator in a Nutshell}

In 2017-2018, we have designed, implemented and operated
Repairnator, a bot for automated program repair.
Repairnator is specialized to repair build failures happening during continuous integration.
It constantly monitors thousands of commits being pushed to the GitHub code hosting platform, and analyzes their corresponding builds. Every minute, it launches new repair attempts in order to fix bugs before the human developer.
It is designed to go as fast as possible because it participates to a race: if Repairnator finds a patch before the human developer, it is human-competitive.

Let us now give an overview of how the Repairnator bot works.

\noindent\includegraphics[width=\columnwidth]{repairnator-overview2}

The primary input of Repairnator are continuous integration builds, triggered by commits made by developers (top part of the figure, \emph{(a)} and \emph{(b)}) based on GitHub projects \emph{(a)}.
The outputs of Repairnator are two-fold:
\emph{(1)} it automatically produces patches for repairing failing builds \emph{(g)}, if any;
\emph{(2)} it collects valuable data for future program repair research \emph{(h)} and  \emph{(k)}.
Permanently, Repairnator monitors all continuous activity from GitHub projects \emph{(c)}.
The CI builds are given as input to a three stage pipeline:
\emph{(1)} a first stage collects and analyzes CI build logs \emph{(e)};
\emph{(2)} a second stage aims at locally reproducing the build failures that have happened on CI \emph{(f)};
\emph{(3)} a third stage runs different program repair prototypes coming from the latest academic research.
When a patch is found, a Repairnator project member performs a quick sanity check, in order to avoid wasting valuable time of open-source developers.
\emph{(i)} If she finds the patch non-degenerated, she then proposes it to the original developers of the project as a pull-request on GitHub \emph{(j)}.
The developers then follow their usual process of code-review and merge.

Repairnator has to operate in a given software ecosystem. Due to our expertise with Java in past research projects, the prototype implementation of Repairnator focuses on repairing software written in the Java programming language, built with the Maven toolchain, in open-source projects hosted on GitHub, which use the Travis CI continuous integration platform.

\subsection*{Expedition Achievements}
We have been operating Repairnator since January 2017, in three different phases.
During one month, in January 2017, we performed a pilot experiment with a initial version of the prototype.
From February 1st, 2017 to December 31th, 2017, we ran Repairnator with a fixed list of \numprint{14,188} projects, we call it ``Expedition \#1''.
From January 1st 2018 to June 31th 2018, Repairnator has monitored the Travis CI build stream in real time, we call it ``Expedition \#2''

The main goal of the pilot experiment was to validate our design and initial implementation. We found that our prototype is capable of performing approximately 30 repair attempts per day, given our computing resources.
More importantly, this pilot experiment validated our core technological assumptions:
a significant proportion of popular open-source projects use Travis and the majority of them use Maven as build technology. This meant we would have a fair chance of reaching our goal of synthesizing a human-competitive patch in that context.

During Expedition \#1, whose results are presented in details in~\cite{urli:hal-01691496}, Repairnator has analyzed \numprint{11,523} builds with test failures.
For \numprint{3,551} of them (30.82\%), Repairnator was able to locally reproduce the test failure.
Out of \numprint{3,551} repair attempts, Repairnator found \numprint{15} patches that could make the CI build pass. However, our patch analysis revealed that none of those patches were human-competitive because they came too late (Repairnator produced a patch after the human developer) or were of low quality (they made the build successful coincidentally).

Expedition \#2 is the successful one. It has shown that program repair technology has crossed the frontier of human-competitiveness. Repairnator has produced 5 patches that meet the criteria of human-competitiveness defined above:
1) the patches were produced before the human ones, 2) a human developer accepted the patches as valid contributions, and the patches were merged in the main code base.
The information about those five milestone patches are shown in \autoref{tab:repainator-accepted-patches} and we now discuss the first of them.

The first patch merged by our program repair bot was accepted by a human developer on Jan 12th, 2018.
Here is the story:
on Jan 12th 2018 at 12:28pm, a build was triggered on project aaime/geowebcache\footnote{\url{https://travis-ci.org/GeoWebCache/geowebcache/builds/328076497}}. The build failed after 2 minutes of execution, because two test cases were in error.
Fourty minutes later, on Jan 12th 2018 at 1:08pm, Repairnator detected the failing build during its regular monitoring, and started to run the available program repair systems configured in Repairnator.
Ten minutes later, at 1:18pm, it found a patch.

On Jan 12th 2018, at 1:35pm, a Repairnator team member  took the patch generated by Repairnator, and validated the opening of the corresponding pull-request on GitHub. On Jan 12th 2018, at 2:10pm, the developer accepted the patch, and merged it with a comment:  "Weird, I thought I already fixed this... maybe I did in some other place. Thanks for the patch!".
That was the first patch produced by Repairnator and accepted as a valid contribution  by a human developer, definitively merged in the code base.
In other words, Repairnator was human-competitive for the first time.

After 6 more months of operation, Repairnator has had 5 patches merged by human developers, which are all listed in \autoref{tab:repainator-accepted-patches}.

Overall, the Repairnator project has fullfilled its mission. It has shown that program repair can be considered as human-competitive: Repairnator has found patches 1) before the humans, 2) that were considered of good quality by humans themselves.

\subsection*{The Future}
In addition to showing that program repair is human competitive, the Repairnator project has provided a wealth of information about bugs and continuous integration, and about the current shortcomings of program repair research, presented in \cite{urli:hal-01691496}.

Let us dwell on one point in particular, the question of intellectual property. On May 3rd, 2018, Repairnator produced a good patch for GitHub project eclipse/ditto. Shortly after having proposed the patch, one of the developers asked \emph{``We can only accept pull-requests which come from users who signed the Eclipse Foundation Contributor License Agreement.''}. We were puzzled because a bot cannot physically or morally sign a license agreement and is probably not entitled to do so. Who owns the intellectual property and responsibility of a bot contribution: the robot operator, the bot implementer or the repair algorithm designer? This is one of the interesting questions uncovered by the Repairnator project.

We believe that Repairnator prefigures a certain future of software development, where bots and humans will smoothly collaborate and even cooperate on software artifacts.

\bibliographystyle{abbrv}
\bibliography{bibliography}

\end{news}
```


abstract: Describing a video automatically with natural language is a challenging task in the area of computer vision. In most cases, the on-site situation of great events is reported in news, but the situation of the off-site spectators in the entrance and exit is neglected which also arouses people's interest. Since the deployment of reporters in the entrance and exit costs lots of manpower, how to automatically describe the behavior of a crowd of off-site spectators is significant and remains a problem. To tackle this problem, we propose a new task called crowd video captioning (CVC) which aims to describe the crowd of spectators. We also provide baseline methods for this task and evaluate them on the dataset WorldExpo'10. Our experimental results show that captioning models have a fairly deep understanding of the crowd in video and perform satisfactorily in the CVC task.
author:

title: Crowd Video Captioning
  [^1]

# Introduction

With the rapid development of the deep neural network, computers can describe the content of the video in a reasonably deep way. Video captioning has important practicality and wide potential application, and a typical one is news broadcasting.

In most great events, there are professional commentators in the stadium to broadcast the situation in real time, and some studies about automatic sports video commentary have been carried out. Outside the venue, entry and exit of spectators are also important. Reports such as *"The line of people snaked into the theater with joy in their faces."* often appear in the news, but deliberately assigning reporters to wait for spectators wastes manpower. Therefore, to report the situation of off-site spectators in real time, we use the surveillance camera to analyze the spectators' crowd and generate captions with the deep learning methods.

Recently, there has been some works on crowd counting and classification , but their output is a number of pedestrians, or the state of mobility and abnormal behaviors. None of these work can produce a descriptive sentence for the crowd as a news report.

A caption of crowd video needs to describe various attributes of a crowd, such as the number of people in the crowd, the situation of movement, direction of flow, etc. Therefore, we use a captioning framework to describe the crowd in videos. In major events, if the surveillance camera uses our system, it can be directly connected to the news broadcasting system to broadcast the real-time off-site situation to the news media.

Our framework uses convolutional neural networks to extract these crowd features, then feeds them into a classifier or a language model to produce the summary. All attributes and situations of the crowd should be included in the output descriptive words of the language model.

To validate our system, we create a crowd video captioning dataset, which is based on the crowd counting dataset: WorldExpo'10. We select some of the videos in this dataset and make captions for them. Several experiments using the proposed models have been carried out to evaluate the performances of those methods.

The main contribution of our work is the proposal of a new task called crowd video captioning **(CVC)** which aims to generate captions for the crowd video. We provide baselines and a system framework for this task, and the results of the experiments prove the feasibility of our system.

# Related Work

## Crowd Counting

In recent years, many models and datasets have been proposed for crowd counting. For example, Chan et al.  collect UCSD in the University of California, San Diego, and it is one of the earliest datasets for crowd counting. Chan et al.  have used Dynamic Textures and Gaussian methods to count the crowd in videos. After that, many new models have been proposed for this task, such as CNN and ACSVP  (a GAN-based, U-net structured model).

WorldExpo'10 proposed in is another large-scaled dataset for crowd counting. It includes more than a thousand labeled videos captured by over one hundred monitoring cameras, all from the Shanghai World Expo in 2010. We have used it in this research.

## Crowd Behaviors Analysis

In addition to counting, researches on behavior analysis of crowd are also underway. MED has been carried out as the crowd emotion dataset, but it only has 31 videos, and the people in the crowd are just walking around and making some specific movements, such as fighting, hugging.

The newest dataset, Crowd-11 proposed in , has been provided to classify the fine-grained crowd behaviors. It categorizes the flow mainly by the direction of each one in the crowd. Models including LSTM , C3D, V3G , ConvLSTM and so on, have been used to analyze the crowd abnormal behaviors in those datasets. While these systems work efficiently, they do have significant disadvantages: the accuracy of fine-grained classification for flow is generally low and they are more suitable for abnormal behavior monitoring.

## Fine-grained Video Captioning

There are also some works about fine-grained video captioning, including broadcasting for tennis videos , and Fine-grained Sports Narrative dataset . Models like LSTM-YT and S2VT have been used to complete those tasks. But the aims of those works are all for professional sports broadcasting.

# Methods

A conventional video captioning pipeline can be divided into two stages: feature extraction and caption generation.

In the first stage, we can use the model for image classification as a frame feature extractor, or the model for video classification as a video feature extractor. The features extracted by these two methods are different, so suitable models should be selected for different tasks..

In the second stage, a sequence to sequence model is fed with the extracted features and generate sentences. Therefore, a language model such as a recurrent neural network (RNN) can be used to construct this decoder.

## Frame Feature Extraction

The high-level 2D features of the frame can be extracted by a convolutional neural network (CNN) for image classification, feeding it with frames of the video one after another, and getting the feature of each frame from the last layer of the model.

Inception V3 is one of those networks, evolved from GoogLeNet . It decomposes a 2D convolution mask which is $n\times n$ into two 1D masks which are $1\times n$ and $n\times 1$ respectively. It can not only accelerate the calculation but also increase the depth of the network.

In addition to Inception V3, ResNet and Inception V4 are also available for the frame feature extraction. Their inputs have a direct influence to the output through skip connections.

## Video Feature Extraction

C3D is a network for extracting features from videos. Unlike the 2D convolution in 3D space which can not slide in the temporal domain, 3D convolution can extract the features of the same region in different periods.

C3D network has eight convolution and five pooling layers. Using Principal Component Analysis (PCA), the output of the last layer (fc7) in C3D is shown in Fig. , where every point represents the feature of its corresponding video, and the 8 different colors represent 8 different classes. Before training (step=0), all points from different categories are mingled together. But after training (step=89), points of the same category are collected together. There are 70 points in the figure, the length of their tensor is 4096.

## Caption Generation

S2VT is a typical sequence-to-sequence network for generating captions for videos. It's made up of two-layer recurrent neural network (RNN), and long short term memory (LSTM) is used as the cells of this RNN.

As Fig.  shows, S2VT takes the features extracted from the video frames as the input sequences. The "words" of those embeddings are fed into the LSTM cells of the first layer one by one, then, after several iterations, the words for captioning are continuously generated by the second layer.

S2VT generates words by taking the words which are already generated. It uses $\textless BOS\textgreater$ to indicate the begin-of-sentence and $\textless EOS\textgreater$ for the end-of-sentence tag. And $\textless pad\textgreater$ is used when there is no input at the time step. Those labels are all utilized as the references to produce the next word.

# Details of Models

In this section, we first introduce the definition of our crowd video captioning task and the overview of our frameworks. Then, we describe two alternative models to caption the crowd video.

## Task Definition and Overview

Crowd video captioning aims to describe $q$ attributes of a crowd in natural language, such as the number of people in the crowd, the situation of movement, the direction of flow, etc. Our model can be divided into two parts: encoder and decoder. First, we need to identify the crowd from videos. It is easy to detect the changing areas between frames because the pixels representing pedestrians tend to move together as a whole. Then, $n$ frames are randomly selected from a video. the attributes and situations of the crowd can be extracted from the following features: crowd extent size, density, individual movements, pedestrian situations and so on. Our framework employs convolutional neural networks as the feature extractor to get these features embeddings.

The features extracted from the videos include the attributes of the crowd. Therefore, crowd's feature can be directly extracted from the extractor frame by frame, and then joined into one sequence $F_{frame}=(f_1,f_2\cdot \cdot \cdot f_n )$. Crowd's features can also be encoded into a vector sequence $F_{video}=(x_1, x_2\cdot \cdot \cdot x_t)$ through the video feature extractor. Finally, this sequence is fed to a classifier or a language model as the caption generator to produce the description $Y=(y_1, y_2, \cdot\cdot\cdot y_m)$ of the crowd.

## Classification Model

If there are $t$ words in the dataset, the number of $m$-word sentences can be formed theoretically is $t^m$. But very few of those randomly generated sentences are grammatical. So we use $p$ grammatical sentences as the tags that classifier needs to recognize. In this model, after the video features $\boldsymbol{F}_{video}=(x_1, x_2\cdot \cdot \cdot x_t)$ are extracted from the encode, a classifier is used as the decoder.

Let's take p-category classifier based on C3D as example, a linear layer is used as the classifier followed in the rear of C3D, as shown in Fig. . The linear layer converts $t=4096$ dimensional vectors $\boldsymbol{F}_{video}=(x_1,x_2,\cdot\cdot\cdot ,x_{4096})$ to p-dimensional outputs, then the probabilities $\boldsymbol{Z}=(z_1,z_2,\cdot\cdot\cdot ,z_p)$ for which label to be output can be calculated by *softmax* as follow: $$z_j=softmax(x_j)=\frac{e^{x_j}}{\sum_{j=1}^pe^{x_j}}$$ where $j\in{\{x|1\leq x\leq p, x\in\mathbb{N}\}}$, $p$ is the total number of categories. After that, the corresponding value of the category with the maximal probability is set to 1.

## Captioning Model

Since classification model can only generate captions within the label set, we adopt the captioning model in our **CVC** system framework. To caption the video, our captioning model needs to generate natural language sequences. Given an frame feature sequence $\boldsymbol{F}_{frame}=(f_1, f_2\cdot \cdot \cdot f_n)$, the goal of captioning model is to generate proper words: $\boldsymbol{Y}=(y_1,y_2\cdot\cdot\cdot y_m)$. The model estimates the probability: $$P(Y|\boldsymbol{F}_{frame})=\prod_{i=1}^{m}P(y_i|y_{<i},\boldsymbol{F}_{frame})$$ where $y_{<i}$ represents all the generated words before the $i^{th}$ word. When captioning model outputs each word, it chooses the word with the highest probability of this position on the basis of all the words that have been output before.

The overall design of our **CVC** system framework of captioning model is shown in Fig. .

Firstly, the video frames feature $\boldsymbol{F}_{frame}$ or the whole video feature $\boldsymbol{F}_{video}$ about the crowd is obtained from the encoder, such as C3D, ResNet or Inception.

After that, we take the language model, like S2VT shown in Fig. , as the decoder for captioning. In addition to the LSTM, we apply a gated recurrent neural network (GRU) as the cell of S2VT, which has fewer parameters than LSTM and can avoid over-fit. Researches and practical experiences show that these two cells each have their own advantages and disadvantages.

The following steps show how the model produces the captions.

First of all, vocabulary which contains all the words in the dataset is built, and every word is encoded into a vector.

Next, the features $\boldsymbol{F}_{frame}$ are input to the model, and then which word $y_i$ to be selected from the vocabulary depends on the hidden state $h_{n+i-1}$ and the network parameters $\theta$:

$$y_i^*=\mathop{\arg\max}_{y_i} \sigma(y_i,h_{n+i-1},\theta)$$

Finally, supposing that all the hidden states of the network are $\boldsymbol{H}=(h_1, h_2\cdot\cdot\cdot h_{n+m})$, from the beginning of the input sentence to the end of the output sentence, the model optimizes the parameters by maximizing the sum of a log-likelihood probability of the generated words as follow: $$\theta^*=\mathop{\arg\max}_{\theta} \sum_{i=1}^{m}\log P(y_i|h_{n+i-1},y_{i-1};\theta)$$

## Loss Function

During training, we use the Cross-Entropy Loss as the loss function, which is defined as follow: $$\boldsymbol{L}=-[\boldsymbol{Z}\log \boldsymbol{\hat{Z}}+(1-\boldsymbol{Z})\log(1-\boldsymbol{\hat{Z}})]$$ where $\boldsymbol{Z}$ is the ideal embedding of the words created by humans and the $\boldsymbol{\hat{Z}}$ is the computer-generated words embedding. The greater the difference between the predicted results and the ground truth is, the higher the gradient of the loss function is, and the faster the convergence rate is.

# Experiments

To caption the crowd videos, we select WorldExpo'10 from a series of datasets for crowd analysis. And in order to evaluate our baseline and captioning methods in our dataset, C3D and other feature extractors have been chosen as the encoder, and experiments on S2VT with LSTM and GRU have been token.

## Dataset

Crowds in most videos of the WorldExpo'10 dataset are messy because this dataset is mainly used for crowd counting. In order to simplify the task, we select 98 videos of it and caption them based on the crowd. These videos are captured by 7 surveillance cameras.

Keywords of captions in our dataset are shown in Fig. , they describe the number of people in the crowd, the situation of movement and the direction of flow respectively. Because "running" accounts for a small percentage in the WorldExpo'10 dataset, so it only accounts for 21% in our dataset. We define the direction to be close to the camera as "in", and the direction to be far away from the camera as "out".

The size of the vocabulary is 6, the number $q$ of attribute pairs that make up the descriptive sentences is 3. So $p=2^q=8$ captions are formed in our dataset.

## Baseline Based on C3D

The first baseline is using C3D directly as $p$-category classifier. Since the words in our dataset can make up $p=2^q=8$ sentences, we divide all the videos into eight categories. The label of each category is one of these sentences, such as "Many people walk in".

The dimension of the last layer in C3D is $t=4096$, we add a linear full connected layer after it as the classifier. The dimension of the output of this linear layer the total number of categories, which is 8 in our experiment.

The model is first pre-trained with UCF101 . We then fine-tune the network on our dataset. The split for training, validating and testing is 70:19:9. In order to fit the C3D model, frames are resized into $128\times171$, and the randomly cropped to $112\times112$.

The accuracy and loss curves for training, validating and testing epochs are shown in Fig. , where the number of frames inputted to the model is 16, the learning rate is set to $10^{-4}$, and the schedule is set to divide the learning rate by 2 every 10 epochs. And the curves in Fig.  are smoothed by 0.8, while the original values are reported in faint polylines.

The curves show that in the training epochs, the loss almost converges to zero, and the accuracy can achieve fast convergence as well. It reaches 0.9714 and 0.6842 on training and validating corpus, but it's only 0.4444 on testing one. Their loss can be reduced to approximately zero on the training set, but not on the validation set or the test set.

## Evaluation Metrics for Captioning

In this section, we introduce several frequently used evaluation metrics for video captioning.

-   **BLEU.** It is based on modified n-gram precision. To begin with, the modified precision $p_n$ is defined as the candidate counts clipped by their corresponding reference maximum value, summed, and divided by the total number of candidate n-grams. In the second place, supposing $c$ is the length of the candidate translation and $r$ is the effective reference corpus length, we compute the brevity penalty: $$BP=\left\{
                \begin{array}{lcl}
                    1, & c > r. \\
                    e^{(1-\frac{r}{c})}, &  c\leq r.
                \end{array}
            \right.$$ Ultimately, we use n-grams from 1-gram up to length N to calculate the BLEU score with the weighted precision: $$BLEU=BP\cdot \exp(\sum_{n=1}^{N}w_n\log(p_n))$$

-   **CIDEr.** It measures the similarity of a sentence to the majority, or consensus of how most people describe the image. For instance, sentences such as "Mike has a baseball and Jenny has basketball" is more representative of the consensus descriptions than the sentence "Jenny brought a bigger ball than Mike". CIDEr is proposed to capture those sentences with more broad consensus.

-   **METEOR.** It is based on the weighted precision $P$ and recall $R$ of the matched content-function words in hypothesis and reference. Fragmentation penalty $Pen=\gamma\cdot(\frac{ch}{m})^\beta$ is defined to account for differences in word order, where the chunks $ch$ defined as a series of matches that is contiguous and identically ordered in both sentences, and $m$ is the average number of matched words over hypothesis and reference. After the parameterized harmonic mean $E_{mean}$ of $P$ and $R$ is calculated, the METEOR score is computed as follow: $$METEOR=(1-Pen)\cdot E_{mean}$$ It's always small than 1 even if the sentences predicted are the same as the references, because the $Pen$ never equals zero.

-   **ROUGE.** It counts the number of overlapping units such as n-gram, word sequences, and word pairs between the predicted captions and the referential summaries.

## Details and Results of Model based on S2VT

We use C3D, ResNet-152 and Inception V3 (V4) pre-trained on UCF101 or ImageNet as the feature extractor of every frame from our dataset, then train the S2VT model directly with those features. LSTM and GRU are used as the RNN cell of the S2VT. For the best performance, the split for training and testing is adjusted to 45:4. We follow the experimental setting from the default. We set the dimension of features of video frames to 2048 and set the number of hidden layers to 512.

```latex
\begin{table*}[htbp]\caption{Performances of S2VT model with different RNN cell and feature}
    \begin{center}
        \begin{tabular}{|c|c|c|c|c|c|c|c|c|}
            \hline
            \textbf{Metric}&\multicolumn{4}{|c|}{\textbf{BLEU(\%)}} &
            \multirow{2}{*}{\textbf{CIDEr(\%)}} & \multirow{2}{*}{\textbf{METEOR(\%)}} &
            \multirow{2}{*}{\textbf{ROUGE\_L(\%)}} & \multirow{2}{*}{\textbf{Accuracy}} \\
            \cline{2-5}
            \textbf{(n-gram)} & \textbf{\textit{1-gram}} & \textbf{\textit{2-gram}} & \textbf{\textit{3-gram}} & \textbf{\textit{4-gram}} & & & & \\
            \hline
            C3D and LSTM & 82.76 & 76.89 & 74.24 & 75.64 & 63.46 & 50.30 & 82.83 & 0.625 \\
            C3D and GRU & \underline{92.86} & \underline{88.84} & \underline{86.97} & \underline{90.06} & \underline{72.53} & 56.97 & 91.67 & \underline{0.75} \\
            ResNet152 and LSTM & 78.52 & 69.13 & 59.37 & 50.92 & 47.02 & 43.24 & 80.57 & 0.625 \\
            ResNet152 and GRU  & \underline{92.86} & \underline{88.84} & 83.97 & 81.63 & 70.44 & 56.97 & \underline{92.71} & \underline{0.75} \\
            % LSTM with Incept.v3 & 82.76 & 76.89 & 71.39 & 68.35 & 61.36 & 50.30 & 83.87 & 0.625 \\
            Incept.v3 and LSTM & 86.21 & 83.54 & 81.27 & 80.95 & 72.07 & \underline{58.38} & 86.99 & \underline{0.75} \\
            Incept.v3 and GRU  & \textbf{96.43} & \textbf{95.71} & \textbf{94.34} & \textbf{95.73} & \textbf{81.15} & \textbf{67.81} & \textbf{95.83} & \textbf{0.875} \\
            Incept.v4 and LSTM & 82.09 & 81.62 & 80.68 & 84.35 & 68.65 & 57.90 & 83.33 & \underline{0.75} \\
            Incept.v4 and GRU  & \underline{92.86} & \underline{88.84} & 83.97 & 81.63 & 70.44 & 56.97 & \underline{92.71} & \underline{0.75} \\
            \hline
        \end{tabular}
        \label{tab1}
    \end{center}
\end{table*}
```

We report BLEU, CIEDEr, METEOR, and ROUGE_L captioning scores for this method, the results on the testing set are provided in TABLE. . The learning rate is set to $4\times 10^{-5}$, the scheduler is set to decay the learning rate by 0.8 every 200 epochs.

Results obtained from the S2VT with different RNN cell and feature are shown in TABLE. , best performances are presented in bold, and second best performances are underlined. For our task, the most appropriate feature extractor is inception V3, followed by C3D. And GRU has fewer parameters than LSTM, so it converges more easily than LSTM when the dataset size is small. It is the reason why GRU works better than LSTM on our dataset.

The accuracy is calculated in the purpose of being compared with the results of the C3D method, defined as the proportion of complete correct sentences to all of them. It is higher than the counterpart of the C3D classifier. And the two incorrect hypotheses generated by LSTM from Inception V3 features are just predicted with the mistake on verb and direction respectively, as shown in Fig. . The second failed case is due to several people who interfere with the judgment. Although the sentence structure is not set manually, the output is completely consistent with the grammar, such as the singular-plural rule.

## Experimental Conclusion

Compared with the baseline, this method based on S2VT outperforms the classifier with C3D due to the comprehension of words in the sentence separately. This verifies that it is not necessary to know what the specific features represent when extracting features, and the analysis of crowd features can be directly handed over to the language model.

Moreover, it proves that image classification models such as Inception and ResNet can extract the feature of the crowd in each frame, and the sequence composed of these features can be used for the crowd captioning.

This reflects the power of the video captioning model, especially if more information needs to be described in the summary later. Video captioning models can interpret the temporal information from the features extracted by the convolution neural network. It can also understand which information each word represents, and make them into a sentence without grammatical mistakes.

# Conclusion and Future Work

In this paper, we propose a new video captioning task of describing the off-site audiences or visitors crowd, called crowd video captioning (**CVC**). In our encoder-decoder system for this task, we use a deep convolutional neural network to extract the features of crowd video and feed them to a language model for crowd description generating. We create a dataset based on WorldExpo'10. On this dataset, our experimental trials prove that our **CVC** system works well to accomplish this task, and they have achieved high accuracy. It shows that the language model can deeply comprehend the information about the crowd which feature extractors don't understand.

In our approach, S2VT with features extracted from Inception V3 works better than other methods in the **CVC** task because our dataset is small and the captions are simple. For future work, these models should be adjusted to fit this fine-grained captioning task for the crowd. The number of videos and the complexity of captions needs to be increased as well in the dataset.

[^1]: This work is supported in part by the National Science Foundation China (NSFC) 61761136005, and ARC through DP190100887 and DP160104500.


author: Stefan Hild, Simon Chelkowski and Andreas Freise

email:email@example.com
date: 2024-09-30
title: Pushing towards the ET sensitivity using 'conventional' technology

------------------------------------------------------------------------

height 1pt

# The Scope of this Document

Recently, the design study 'Einstein gravitational wave Telescope' (ET) has been funded within the European FP7 framework . The design study represents the first coordinated effort towards the design of a third-generation gravitational wave detector. The ambitious goal of this project is to provide a conceptual design of a detector with a hundred times better sensitivity than currently operating detectors, which corresponds to a capability of scanning a one million times larger volume of the universe! It is expected that this challenging goal requires the development and implementation of new technologies, which go beyond the concepts employed for the first and second detector generations, especially for the reduction of quantum noise. One task within the design study is to review such new technologies and study their feasibility for gravitational wave detection. In this context, also the application of completely different detector geometries and topologies is being discussed .

However, it is a very interesting and educational exercise to imagine a Michelson interferometer in which conventional technologies have been pushed to - or maybe beyond - their limits to reach the envisaged sensitivity for the Einstein Telescope. In this document we present a first sketchy analysis of what modifications and improvements are necessary to go, step-by-step, from second generation gravitational wave detectors to the Einstein Telescope. We restrict our analysis to a configuration similar to Advanced LIGO (Dual-Recycled Michelson interferometer featuring arm cavities) but with an increased arm length of 10 km and we explicitly put a focus on rather conventional techniques.

# Assumptions and Constraints

As described above we want to restrict ourselves to more or less *conventional* technologies which fulfill at least one of the following two criteria:

-   The technology was already successfully demonstrated on prototypes, such as the injection of squeezed light into a Michelson interferometer or interferometry with cryogenic mirrors.

-   The technology is an up-scaling of currently used technology without any change of the fundamental physics involved, for instance using suspensions of 50 m length.

Thus, we consider 'non-conventional', any technique or technology which so far only exist on paper, or has been demonstrated only in proof-of-principle experiments which are very different from the target interferometers. Examples for such technologies include displacement-noise-free interferometry, optical bars and optical levers or the use of higher-order Laguerre-Gauss-modes. These 'non-conventional' techniques have been omitted in our analysis.

A further constraint we imposed on our analysis is that we did not consider the possibility to split ET into separate smaller, optimised interferometers, for example a high-power high-frequency device and a low-power low-frequency device. Although such an approach might be promising for reducing the quantum noise, it would not show the limits of conventional techniques as clearly as a single broadband detector.

Restricting our analysis to a single interferometer also implicates, that we do not evaluate any properties which are closely related to a potential network of detectors, such as for instance sky-coverage, null-stream availability or redundancy.

# A GWINC Model for ET

The Advanced Virgo design group makes use of a well documented noise model , which is based on the GWINC code developed within the LIGO scientific collaboration. We use this Advanced Virgo noise model as starting point for our investigations.

Figure shows the fundamental noise contributions of an detector with basic parameters similar to a possible Advanced Virgo design, using an arbitrary detector configuration. Please note that the resulting sensitivity (solid blue line) is for an imaginary advanced detector only and is not connected to the ongoing design process for Advanced Virgo. The solid red line is an approximation of the ET design target (blue line from Figure ). Please note that every single noise source reaches above the ET target, at least for some signal frequencies. Therefore, each of the displayed noise curves has to be lowered by changing the interferometer performance. In the next section we will change step by step the relevant parameters of our noise model, starting with a noise model for an advanced detector, in order to suppress each individual noise contribution until we achieve compatibility with the ET design target.

# From Advanced Detectors to the ET Design Target

The following list indicates briefly which parameter changes were performed to go from an advanced detector to the Einstein Telescope. Please note: The items listed below represent only **one** possible approach to achieve the ET target sensitivity. There might be other, perhaps more elegant or more feasible scenarios even with conventional technologies. A realistic detector design will very likely contain some of the techniques labeled as non-conventional in this document.

-   In a first step we **increased the arm length from 3 to 10 km** (`ET_sthild_2.m`), which reduces all displacement noises (seismic, gravity gradients, suspension thermal, coating Brownian, coating thermo-optic and substrate Brownian noise) by a factor 3.3 and the residual gas pressure noise by about a factor $\sqrt{3.3}$. Please note that all other detector parameter, such as the beam size at the test masses are kept constant.

-   Then we adjusted the Signal Recycling from detuned (SR-phase = 0.15) to **tuned Signal Recycling** (SR-phase = 0) in order to maximize the detector bandwidth (by resonant enhancement of both signal sidebands). Additionally we have modified the Signal Recycling mirror transmittance slightly from 11% to 10% as a compromise of peak sensitivity and detector bandwidth (`ET_sthild_3.m`).

-   In a further step we **increased the laser input power** from 125 Watt to 500 Watts which yields an intracavity power of about 3 MW (`ET_sthild_4.m`). The value of 500 Watts has been chosen by eye as a trade off between improved shot noise at high frequencies and higher radiation pressure noise at low frequencies.

-   Then we introduced a **quantum noise suppression factor** of 10 dB (`ET_sthild_5.m`), which yields a broadband reduction of photon shot noise and radiation pressure noise of about a factor 3.

-   Next we chose to decrease coating Brownian noise by **increasing the beam size** at the main test masses. We enlarged the beam radius (1/$e^2$ in power) from 6 to 12 cm (`ET_sthild_6.m`). This requires test mass radii of curvature of 5070 m for input and end mirrors. This increase of beam size reduces coating and substrate thermal noise by about a factor of 2. In addition, also the residual gas pressure is slightly improved, due to the larger volume of the beam. Please note that assuming such large beams has the consequence that the mirror coatings have to have a diameter of 60 to 70 cm in order to keep the clipping losses within an acceptable range.

-   Since the coating noise was still limiting we also had to consider cooling of the optics. Consequently we introduced a **reduction of the temperature** from 290 to 20 K (`ET_sthild_7.m`). Please note that going to cryogenic temperatures could involve a change of the test mass material from fused silica to sapphire.

-   The next step was to reduce the seismic noise. We replaced the Virgo super attenuator by a **suspension consisting of 5 stages each 10 m long**.[^1] We used a very simplistic model for the suspension transfer function comprising of just 10 simple poles with a corner frequency of 0.158 Hz. For the seismic we used a simplified spectrum ($1\cdot 10^{-7}$ m$/f^2$ for $f>1$ Hz) of the horizontal seismic measured at Virgo site (`ET_sthild_9.m`). Please note that for our new suspensions the suspension thermal noise is considered to be not limiting the ET sensitivity and has therefore been omitted in the following.

-   In order to further improve the seismic noise we had to go **underground**. We assumed the underground seismic to be about $5\cdot 10^{-9}$ m$/f^2$ for $f>1$ Hz which corresponds roughly to the seismic level measured in the Japanese Kamioka mine (Slide 26 of ). This is assumed to reduce both the seismic noise as well as the gravity gradient noise by a factor of 20 (see `ET_sthild_10.m`).

-   Even when going underground the gravity gradient noise seems still to be above the ET target sensitivity for frequencies below 6 Hz (see Figure ET_sthild_10 in the Appendix section). Therefore it will be necessary to find some way to further reduce this noise by another factor 50 (maybe by a clever shaping of the caves) to finally get it below the ET sensitivity target (`ET_sthild_11.m`). Please note that this somehow *magic* reduction of gravity gradient noise is the only aspect of our analysis which is not compatible with our initial definition of *conventional*.

-   Finally, in order to bring the radiation pressure below the ET design target we had to **increase the weight of the mirrors** from 42 kg to 120 kg (`ET_sthild_12.m`).

|  | advanced detector | potential ET design |
|:---|:--:|:--:|
| Arm length | 3 km | 10 km |
| SR-phase | detuned (0.15) | tuned (0.0) |
| SR transmittance | 11 % | 10 % |
| Input power (after IMC) | 125 W | 500 W |
| Arm power |  0.75 MW |  3 MW |
| Quantum noise suppression | none | 10 dB |
| Beam radius |  6 cm | 12 cm |
| Temperature | 290 K | 20 K |
| Suspension | Superattenuator | 5 stages of each 10 m length |
| Seismic | $1\cdot 10^{-7}\,{\rm m}/f^2$ for $f>1$ Hz (Cascina) | $5\cdot 10^{-9}\,{\rm m}/f^2$ for $f>1$ Hz (Kamioka) |
| Gravity gradient reduction | none | factor 50 required (cave shaping) |
| Mirror masses | 42 kg | 120 kg |
| BNS range |  150 Mpc |   2650 Mpc |
| BBH range |   800 Mpc |   17700 Mpc |

Summary of the parameter changes necessary to go from the advanced detector sensitivity to the ET design target.

The final result of our analysis is shown in Figure . With the changes described above we can achieve a sensitivity (black solid line) close to the ET design target (red solid line). Table  shows a summary of the parameter changes to transform an advanced detector into a new detector which achieves a sensitivity compatible with the ET design target. Our detector model gives an inspiral range for binary neutron stars of about 2530 Mpc and of about 17500 Mpc for binary black hole systems.

# Summary and Outlook

The analysis presented in this document indicates that it is, in principle, possible to achieve the ET target sensitivity with *conventional* technology. Only the rather unexplored gravity gradient noise needs to be suppressed by non-conventional techniques in order to be compatible with the ET sensitivity at the very low end of the detection band (below 6 Hz). However, we already know that some of the required up-scalings of existing technologies will introduce new noise couplings. The best example for that is of course the high circulating light power which can introduce stability problems to the arm cavities. Therefore we do not believe that up-scaling existing technologies is always the best or even a viable option for the design of a third generation detector. It is important to explore new detector topologies and new technologies (such as broadband QND-techniques, Sagnac-topologies, displacement noise free interferometry, to list only a few examples) and compare their costs and feasibility to conventional technologies. The purpose of this document is simply to provide the parameters shown in Table  as a reference for that work.

# Acknowledgment

The research leading to these results has received funding from the European Community's Seventh Framework Programme (FP7), under the Grant Agreement 211743, ET project.

# Appendix: Intermediate Steps of our Analysis

This section contains all noise budgets from each intermediate step of our analysis presented in Section . The file names in the title of each plot corresponds to the item containing the same file name as reference.

![image](ET_sthild_2.pdf) ![image](ET_sthild_3.pdf)

![image](ET_sthild_4.pdf) ![image](ET_sthild_5.pdf)

![image](ET_sthild_6.pdf) ![image](ET_sthild_7.pdf)

![image](ET_sthild_9.pdf) ![image](ET_sthild_10.pdf)

![image](ET_sthild_11.pdf)

[^1]: A completely different but also somewhat conventional approach to tackle the seismic noise would be to use suspension point interferometry .