abstract: While deep learning has been widely used for video analytics, such as video classification and action detection, dense action detection with fast-moving subjects from sports videos is still challenging. In this work, we release yet another sports video dataset **P$^2$A** for *P*ing *P*ong-*A*ction detection, which consists of 2,721 video clips collected from the broadcasting videos of professional table tennis matches in World Table Tennis Championships and Olympiads. We work with a crew of table tennis professionals and referees to obtain fine-grained action labels (in 14 classes) for every ping-pong action appeared in the dataset, and formulate two sets of action detection problems—*action localization* and *action recognition*. We evaluate a number of commonly-seen action recognition (e.g., TSM, TSN, Video SwinTransformer, and Slowfast) and action localization models (e.g., BSN, BSN++, BMN, TCANet), using **P$^2$A** for both problems, under various settings. These models can only achieve 48% area under the AR-AN curve for localization and 82% top-one accuracy for recognition, since the ping-pong actions are dense with fast-moving subjects but broadcasting videos are with only 25 FPS. The results confirm that **P$^2$A** is still a challenging task and can be used as a benchmark for action detection from videos.
author: Jiang Bian, Qingzhong Wang, Haoyi Xiong, Jun Huang, Chen Liu,
Xuhong Li, Jun Cheng, Jun Zhao, Feixiang Lu, Dejing Dou
bibliography: sample-base.bib
title: **P$^2$A**: A Dataset and Benchmark for Dense Action Detection from Table Tennis Match Broadcasting Videos

# Introduction

Videos have become one of the most popular media in our every day life and video understanding draws much attention of researchers in recent years, including video tagging , retrieval , action recognition and localization . There are many applications of video understanding, such as recommendation systems . One of the most attractive applications is understanding sports videos, which is able to benefit coaching , player training and sports broadcasting . To better understanding sports videos, localizing and recognizing the actions in untrimmed videos play crucial roles, however, they challenging tasks with unsolved problems.

First, there are many types of sports, such as team sports like football, basketball, volleyball, and individual sports like tennis, table tennis, golf, and gymnastics, and each of them has specific actions. Therefore, it is difficult to build a dataset covering all sports and their specific actions. Moreover, the duration of actions in different sports are diverse. For example, the strokes in tennis and table tennis are extremely fast, usually less than one second, while actions in soccer could endure for several seconds, such as long passes and corner ball. Hence, it is difficult to model the actions with various lengths using one model.

Second, data annotation of sports actions is challenging. Compared with common actions in our daily life like walking, riding bicycles that can be easily recognized by annotators, it could be difficult to identify the actions in sports videos, for example, Axel jump, Flip jump and Lutz jump in figure skating, hence, professional players should be involved in data annotation. In addition, players normally perform deceptive actions, for example, table tennis players can perform similar actions, but serve balls with different kinds of spin (top spin, back spin and side spin), which is difficult for annotators to distinguish one action from the others.

Recently, researchers pay much attention to sports videos to address the challenges, including building datasets and proposing new models . In this paper, we focus on a specific sport – table tennis and propose a dataset – *Ping Pang Action* ($\mathbf{P^2A}$) for action recognition and localization to facilitate researches on fine-grained action understanding. The properties of $\mathbf{P^2A}$ are as follows:

-   We annotate each stroke in videos, including the category of the stroke and the indices of the starting and end frames. Plus, the stroke labels are confirmed by professional players, including Olympic table tennis players.

-   To the best of our knowledge, $\mathbf{P^2A}$ is the largest dataset for table tennis analysis, composed of 2,721 untrimmed broadcasting videos, and the total length is 272 hours. Though propose table tennis datasets, the number of videos is smaller and they use self-recorded videos where there is only one player, which is much easier than using broadcasting videos. Moreover, the datasets proposed in are only for action recognition, whereas $\mathbf{P^2A}$ can also be used for action localization.

-   The actions are fast and dense. The action length ranges from 0.3 seconds to 3 seconds, but more than 90% actions are less than 1 second. And typically, there are around 15 actions in 10 seconds, which is challenging for localization.

With the properties, $\mathbf{P^2A}$ is a rich dataset for research on action recognition and localization, and a benchmark to assess models (see section for more details). We further benchmark several existing widely used action recognition and localization models using $\mathbf{P^2A}$, finding that $\mathbf{P^2A}$ is relatively challenging for both recognition and localization.

To sum up, the main contributions of this work are twofold. First, we develop a new challenging dataset – $\mathbf{P^2A}$ for fast action recognition and dense action localization, which provides high-quality annotations confirmed by professional table tennis players. Compared with existing table tennis datasets, $\mathbf{P^2A}$ uses broadcasting videos instead of self-recorded ones, making it more challenging and flexible. In addition, $\mathbf{P^2A}$ is the largest one for table tennis analysis. Second, we benchmark a number of existing recognition and localization models using $\mathbf{P^2A}$, finding that it is difficult to localize dense actions and recognize actions with imbalance categories, which could inspire researchers to come up with novel models for dense action detection in the future.

# Related Work

#### **Action Localization**

The task of action localization is to find the beginning and the end frames of actions in untrimmed videos. Normally, there are three steps in an action localization model. The first step is video feature extraction using deep neural networks. Second, classification and finally, suppressing the noisy predictions. Z. Shou *et al.* propose a temporal action localization model based on multi-stage CNNs, where a deep proposal network is employed to identify the candidate segments of a long untrimmed video that contains actions, then a classification network is applied to predict the action labels of the candidate segments and finally the localization network is fine-tuned. Alternatively, J. Yuan *et al.* employ *Improved Dense Trajectories* (IDT) to localize actions, however, IDT is time-consuming and requires more memory. In contrast, S. Yeung *et al.* treat action localization as decision making process, where an agent observes video frames and decides where to look next and when to make a prediction. Similar to Faster-RCNN , X. Dai *et al.* propose a temporal context network (TCN) for active localization, where proposal anchors are generated and the proposals with high scores are fed into the classification network to obtain the action categories. To generate better temporal proposals, T. Lin *et al.* propose a boundary sensitive network (BSN), which adopts a local-to-global fashion, however, BSN is not a unified framework for action localization and the proposal feature is too simple to capture temporal context. To address the issues of BSN, T. Lin *et al.* propose a boundary matching network (BMN). Recently, M. Xu *et al.* introduce the pre-training-fine-tuning paradigm into action localization, where the model is first pre-trained on a boundary-sensitive synthetic dataset and then fine-tuned using the human annotated untrimmed videos. Instead of using CNNs to extract video features, M. Nawhal *et al.* employ a graph transformer, which significantly improves the performance of temporal action localization.

#### **Action Recognition**

Action recognition lies at the heart of video understanding, an elementary module that draws much attention of researchers. A simple deep learning based model is proposed by A. Karpathy , where a 2D CNN is independently applied to each video frame. To capture temporal information, the fusion of frame features is used, however, this simple approach cannot sufficiently capture the motion of objects. A straightforward approach to mitigate this problem is to directly introduce motion information into action recognition models. Hence, K. Simonyan *et al.* proposed a two-stream framework, where the spatial stream CNN takes a single frame as input and the temporal steam CNN takes a multi-frame optical flow as input. However, the model proposed by only employs shallow CNNs, while L. Wang *et al.* investigate different architectures and the prediction of a video is the fusion of the segments' predictions. To capture temporal information without introducing extra computational cost, J. Lin *et al.* propose a temporal shift module, which can fuse the information from the neighboring frames via using 2D CNNs. Another family of action recognition models is 3D based. D. Tran *et al.* propose a deep 3D CNN, termed C3D, which employs 3D convolution kernels. However, it is difficult to optimize a 3D CNN, since there are much more parameters than a 2D CNN. While J. Carreira *et al.* employs mature architecture design and better-initialized model weights, leading to better performance. All above mentioned models adopt the same frame rate, while C. Feichtenhofer *et al.* propose a SlowFast framework, where the slow path uses a low frame rate and the fast path employs a higher temporal resolution and then the features are fused in the successive layers.

Recently, researchers pay more attention to video transformer – a larger model with multi-head self-attention layers. G Bertasius *et al.* propose a video transformer model, termed TimeSformer, where video frames are divided into patches and then space-time attention is imposed on patches. TimeSformer has a larger capacity than CNNs, in particular using a large training dataset. Similarly, A. Arnab *et al.* propose a transformer-based model, namely ViViT, which divides video frames into non-overlapping tubes instead of frame patches. Another advantage of the transformer-based model is that it is easy to use self-supervised learning approaches to pre-train the model. C. Wei *et al.* propose a pre-training method, where the transformer-based model is to predict the *Histograms of Oriented Gradients* (HoG) of the masked patches and then the model is fine-tuned on downstream tasks. Using the pre-training-fine-tuning paradigm, we can further improve the performance of action recognition.

In this paper, we adopt multiple widespread action recognition and localization models to conduct experiments on our proposed dataset – $\mathbf{P^2A}$, showing that $\mathbf{P^2A}$ is relatively challenging for both action recognition and localization since the action is fast and the categories are imbalance (see sections and for more details).

# **P$^2$A** Dataset

Our goal for establishing the PingPangAction dataset is to introduce a challenging benchmark with professional and accurate annotations to the intensive and dense human action understanding regime.

## Dataset Construction

**Preparation.** The procedure of data collection takes the following steps. First, we collect the raw broadcasting video clips of International/Chinese top-tier championships and tournaments from the ITTF (International Table Tennis Federation) Museum & China Table Tennis Museum. We are authorized to download video clips from 2017 to 2021 (five years), including the Tokyo 2020 Summer Olympic table tennis competition, World Cups, World Championships, etc.

As a convenience for further processing (e.g., storing and annotating), we split the whole game videos into 6-minute chunks, while ensuring the records are complete, distinctive and of standard high-resolution, e.g., 720P ($1280\times 720$) and 1080p ($1920\times 1080$). Then, the short video chunks are ready for annotation in the next step.

**Annotation.** Since the actions (i.e., the ping pong strokes) in video chunks are extremely dense and fast-moving, it is challenging to recognize and localize all of them accurately. Thus, we cooperate with a team of table tennis professionals and referees to regroup the actions into broad types according to the similarity of their characteristics, which are the so-called 14-classes (14c) and 8-classes (8c). As the name implies, 14c categorizes all the strokes in our collected video chunks into 14 classes, which are shown in Fig.  (right side). Compared to 14c, we further refine and combine the classes into eight higher-level ones, where the mapping relationships are revealed accordingly. Specifically, since there are seven kinds of service classes at the beginning of each game round (i.e., the first stroke by the serving player), we reasonably combine these service classes aside from those non-serving classes. Note that, all the actions/strokes we mentioned here are presented in the formal turns/rounds either conducted by the front-view player or back-view player from the broadcasting cameras (the blue-shirt and red-shirt player in Fig. ), and others presented in highlights (e.g., could be in a zoom-in scale with slow motions), Hawk-Eye replays (e.g., with the focus on the ball tracking), and game summaries (e.g., possibly in a zoom-out scale with a scoreboard) are ignored as redundant actions/strokes. For each of the action types in 14c/8c, we define a relatively clear indicator for the start frame and the end frame to localize the specific action/stroke. For example, we set the label of the non-serving stroke in a +/- 0.5 seconds window based on the time instance when the racket touches the ball, where the serving action/stroke may take a long time before the touching moment so that we label it in a +0.5/-1.5 seconds window as the adjustment. The window size is used as the reference for the annotator and can be varied according to the actual duration of the specific action. We further show the detailed distribution of the action/stroke classes in Section 3.2.

Once we have established the well-structured classes of actions, the annotators then engage to label all the collected broadcasting video clips. Unfortunately, the first edition of annotated dataset achieves around $85\%$ labeling precision under sampling inspection by experts (i.e., international table tennis referees). It is reasonable that even with the trained annotators, labeling the actions/strokes of table tennis players is still challenging, which lies in the existence of 1) the fast-moving and dense actions; 2) the deceptive actions; 3) the entire/half-sheltered actions. To improve the quality of the labeled dataset, we work with a crew of table tennis professionals to further correct the wrong or missing samples in the first edition. The revised edition can achieve about $95\%$ labeling precision on average, which is confirmed by the invited international table tennis referee .

Note that, beyond labeling the segment of each action/stroke, we also label some additional information along with it (after finishing each action), which includes the placement of the ball on the table, the occasion of winning or losing a point, the player who committed the action (the name with the status of front view or back view in that stroke), whether it is forehand or backhand, and the serving condition (serve or not). The full list of labeling items is attached in the open-sourced dataset repository on Github[^1].

**Calibration.** The last step for annotation is to clean the dataset for further research purposes. Since the whole 6-minute video dataset has some chunks without valid game rounds/turns (e.g., the opening of the game, the intermission and break, and the award ceremony), we filter out those unrelated chunks and focus on only action-related videos. Then, to accomplish the main goals of the research, which are tasks of *recognition* and *localization*, we further screen the video chunks to reserve the qualified ones meeting the following criteria:

-   ***Camera Angle*** – The current most-common camera angle for broadcasting is the top view, where the cameras are hosted on a relatively high position upon the game court and overlook from the back of two sides of players. As shown in Fig. , we only select the videos recorded in the standard top view and remove those with other views (e.g., side view, "bird's eye" view, etc.) to keep the whole dataset consistent. Note that, the broadcasting recordings with non-standard broadcasting angles rarely (less than $5\%$) appear in our target pool of competition videos, where only a few WTT (World Table Tennis Championships) broadcasting videos experimentally use the other types of camera angles.

-   ***Game Type*** – Considering the possible mutual interference in double or mixed double games, we only include the single games in **P$^2$A**. In this case, there at most two players appear simultaneously in each frame, where usually one player is located near the top of the frame and another is at the bottom of the frame.

-   ***Audio Track*** – Although we do not leverage the audio information for the localization and recognition tasks, the soundtracks are along with the videos and awaiting to be explored or incorporated in the future research. We also remove those with broken or missing soundtracks.

-   ***Video Resolution*** – We select the broadcasting videos with a resolution equal to or higher than 720P ($1280\times 720$) in MP4 format and drop those with low frame quality.

-   ***Action Integrity*** – Since we trim the original broadcasting video into 6-minute chunks, some actions/strokes may be cut off when delivering the splitting. In this case, we make efforts to label those broken actions (about $0.2\%$) and it is optional to incorporate the broken actions or not in a specific task.

In the next subsection, we introduce the basic statistics of calibrated **P$^2$A** dataset and provide some guidance on pre-processing that might be helpful in specific tasks.

```latex
\begin{table*}[h]\begin{minipage}{\textwidth}
\label{tab:comparison}
\caption[xxx]{Comparison of existing sports video datasets for action detection and related purposes.}
\vspace{-3mm}
\begin{center}
\scalebox{1}{
\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|}
 \hline
 Dataset & Duration & Samples & Segments & Classes & Tasks\footnote[2]{We denote the task of recognition, localization, and segmentation as ``Rec'', ``Loc'', and ``Seg''.} & Density\footnote[3]{We calculate the Density here as the value of Segments/Samples.} & Action Length & Source & Year \\ [0.5ex]
 \hline\hline
 Olympic~\cite{niebles2010modeling} & 20 h & 800 & 800 & 16 & Rec + Loc & 1.00 & 5s $\sim$ 20s & YouTube & 2010\\
 \hline
 UCF-Sports~\cite{soomro2014action} & 15 h & 150 & 2,100 & 10 & Rec + Loc & 14.00 & 2s $\sim$ 12s & Broadcast TV & 2014\\
 \hline
 ActivityNet~\cite{caba2015activitynet}\footnote[1]{We also include some popular video-based action detection datasets, which are not only limited to sports-related actions, but with a considerable amount of them in sub-genre.} & 648 h & 14,950 & 23,064 & 200 & Rec + Loc & 1.54 & 10s $\sim$ 100s & YouTube & 2015\\
 \hline
 TenniSet~\cite{faulkner2017tenniset} & 9 h & 380 & 3,568 & 10 & Rec + Loc & 9.40 & 1.08s $\sim$ 2.84s & YouTube & 2017 \\
 \hline
 OpenTTGames~\cite{voeikov2020ttnet} & 5 h & 12 & 4,271 & 3 & Rec + Seg & 355.92 & 0.12s $\sim$ 0.13s & Self Recorded & 2020  \\
 \hline
 FineGym~\cite{shao2020finegym} & 708 h & 303 & 4,885 & 530 & Rec + Loc & 16.12 & 8s $\sim$ 55s & Broadcast TV & 2020 \\
 \hline \hline
 \textbf{\textbf{P$^2$A}{}} & 272 h & 2,721 & 139,075 & 14 &  Rec + Loc & 52.70 & 0.32s $\sim$ 3s & Broadcast TV & 2022 \\
 \hline
\end{tabular}}
\end{center}
\end{minipage}
\end{table*}
```

## Dataset Analysis

In this section, we conduct a comprehensive analysis of the statistics of **P$^2$A**, which is the foundation of the follow-up research and also serves as the motivation for establishing **P$^2$A** in the video analysis domain. The analysis lies in four parts, which are (1) general statistics, (2) category distribution, (3) densities, and (4) discussion. In the rest of this subsection, we go through each part in detail.

**General Statistics.** The **P$^2$A** dataset consists of 2,721 annotated 6-minutes-long video clips, containing 139,075 labeled action segments and last 272 hours in total. They are extracted from over 200 table tennis competitions, involving almost all the top-tier ones during 2017-2021. These events include the World Table Tennis Championships, the Table Tennis World Cup, the Olympics, the ITTF World Tour, the Asia Table Tennis Championships, the National Games of China, as well as the China Table Tennis Super League. Since we intend to establish an action-focused table tennis dataset, the quality and standard of actions are expected to achieve a relatively high level, where the official HD broadcasting videos selected from the above-mentioned top-tier competitions lay a good foundation as the source of the target actions.

For the general statistics of actions, Table 1 presents the basic statistics of **P$^2$A** compared with the popular streams of sports-related datasets. The length of action for **P$^2$A** varies from 0.32 to 3 seconds, which is significantly shorter than most of the other datasets except the OpenTTGames and the TenniSet. Since the OpenTTGames targets only the in-game events (i.e. ball bounces, net hits, or empty event targets) without any stroking annotations, the action length is even shorter and with a considerably higher Density. Compared to these two table tennis related datasets, our **P$^2$A** dataset has a longer Duration (over 30 times, and with more samples and segments), where it means a lot for accurately reflecting the correlation and distribution of actions/strokes in different classes. Furthermore, compared to the sources of the OpenTTGames and the TenniSet, **P$^2$A** leverages the Broadcast TV games with the proper calibration by the international table tennis referee, which are more official and professional. Compared to the commonly used datasets such as ActivityNet and FineGym, **P$^2$A** focuses on the actions/strokes with the dense and fast-moving characteristics. Specifically, the action length of **P$^2$A** is around 30 times shorter and density is 5 to 50 higher than these two datasets, which means **P$^2$A** introduces a series action/strokes in a totally different granularity. We later show that the SOTA models confront a huge challenge when applying on **P$^2$A**. Although the number of classes of **P$^2$A** is relatively small due to the nature of the table tennis sports itself, the action recognition and localization tasks are barely easy on **P$^2$A** for most of the mainstream baselines. In the next subsection, we show the distribution of categories in **P$^2$A**, which is one of the vital components causing the aforementioned difficulties.

**Category Distribution.** Figure  right side shows the targeting categories of actions/stokes in **P$^2$A**. The **P$^2$A** dataset has two main branches of strokes – **Serve** and **Non-serve**, where each of them owns 7 types of sub-strokes. Since the strokes are possibly taken by different players in the same video frames, we annotate strokes from both the front-view and the back-view players without time overlaps. For example, Figure  represents consecutive frames of annotations on a front-view player, where a player is producing a "Side Spin Serve" at the start of a turn. Notice that this player stands at the far-end of the game table and turns around from the side-view to the front-view, then keeps the front-view in the rest of this turn. A similar annotating is taken to record the player's strokes at the near-end of the game table.

As aforementioned, the **P$^2$A** dataset consists of two versions, which are 14c (14 classes) and 8c (8 classes). The 14c flattens all the predefined classes in Figure  and treats them equally in the future tasks. We measure the number of strokes for each of the classes in Figure . Overall, the drive category contains more than half of the samples (we sort the categories in descending order), which means there exists unbalancing category phenomenon in **P$^2$A**. We also observe an interesting point that the non-serve categories generally last a shorter duration than the serve categories (as shown from the blue bars). This is because of the long preparation of the serve strokes and could be an important feature to distinguish the serve and the non-serve strokes. Another fact is that the non-serve categories dominate the whole dataset, where there are seven non-serve categories out of the left-most eight categories (in the descending order). This unbalancing phenomenon leads to the creation of the second version – 8C to combine all the serve categories into one unified "Serve" category.

Figure  measures the number of actions in the serve, non-serve , and combined 8c categories separately. As a result, the sub-categories are unbalanced, where "Side Spin" and "Drive" dominates the serving and non-serving actions in (a)&(b). In the combined 8c dataset, "Drive" unsurprisingly takes a large proportion among all eight categories due to its frequent appearances in modern table tennis competitions. Thus, it is necessary to adopt additional sampling techniques to mitigate the side effects, and the implementation details are introduced in Section 4.

**Densities.** As one of the unique characteristics compared to other datasets, the high density of actions/strokes plays a vital role in **P$^2$A**. We analyze the density of actions/strokes in two aspects, where the first one is the duration distribution of each actions/strokes category. Figure (a) shows the duration distribution of all classes of strokes in **P$^2$A**. We can observe that most of the action types have relatively stable distributions (with short tails) except the "Control" and the "Neutral" types. The "Neutral" is one of the serving strokes representing those irregular/unknown actions in the serving process, where the duration of it widely ranges from 0.4 to 3 seconds in Figure (a). It is similar that the "Control" category stands for the non-serving strokes which cannot be categorized to any other classes, and has a long tail of 2 seconds at most. On the whole, the average duration of strokes sticks to around 0.5 seconds, which demonstrates the actions in **P$^2$A** is fast-moving compared to other datasets in Table 1. We also compare the duration distribution between the serve and non-serve classes in Figure (b), where the stacked plot of both classes presents that the serve class has a slightly longer tail than the non-serve class.

For the second respect, we measure the action frequency and the appearance density in each turn of the game. One turn in a game starts from a serving action by one player and ends with one point acquired by either of the players. Thus, the action frequency can reflect the nature of the table tennis competition (e.g., the strategy and the pace of the current table tennis game) that how many strokes could be delivered in a single turn. As shown in Figure (a), most of the turns have around 3 $\sim$ 5 actions/strokes and the distribution is also long-tailed to a maximum 28 actions/strokes per turn. It is reasonable that the pace of the game today becomes faster than ever, and a point could be outcome within three strokes for both players (i.e., summed up within 6 strokes). To further differentiate from other sports datasets, we additionally measure the appearance density in 10 seconds, which is usually the time boundary for long-duration and short-duration sports actions. Figure (b) depicts the histogram of the action counts in 10-second consecutive video frames. The results follow a normal-like distribution with average counts as 15, which means the actions appear densely and frequently in a short time in the most of the time in **P$^2$A**. The density can be achieved as about 1.5 actions per second on average and it demonstrates the characteristics of dense actions for the **P$^2$A** dataset.

**Summary.** In this section, we comprehensively report the statistics of **P$^2$A**. Specifically, we leverage rich measurements to have a thorough view of the volume of the dataset, the category distribution, the duration of samples, and the densities. In conclusion, the **P$^2$A** dataset is unique comparing to the other mainstream sports datasets in the following sides,

-   Comparing to the table tennis datasets , **P$^2$A** has an adequate sample size (10 times $\sim$ 200 times) and more subtle set of categories.

-   Comparing to the large datasets in other sports domain , **P$^2$A** focuses on the dense (5 times $\sim$ 50 times) and fast-moving (around 0.1 times) actions.

-   Comparing to most of the aforementioned sports datasets, **P$^2$A** includes two players' actions back-to-back (the action appears in turns with a fast pace), which is more complicated and mutually interfering when analyzing.

In the next section, we investigate the performance of the state-of-the-art models and solutions for the predefined action recognition and action localization tasks on our **P$^2$A** dataset. Then, we observe the challenges those algorithms confront and demonstrate the research potential of the **P$^2$A** dataset.

# Benchmark Evaluation

In this section, we present the evaluation of various methods on our **P$^2$A** dataset and show the difficulties when applying the original design of those methods. Then, we provide several suggestions to mitigate the ineffectiveness and tips to improve the current models.

## Baselines

We adopt two groups of baseline algorithms to separately tackle the action recognition and action localization tasks. For **Action Recognition**, we includes the following trending algorithms,

-   **Temporal Segment Network (TSN)**  is a classic 2D-CNN-based solution in the field of video action recognition. This method mainly solves the problem of long-term behavior recognition of video, and replaces dense sampling with sparsely sampling video frames, which can not only capture the global information of the video, but also remove redundancy and reduce the amount of calculation.

-   **Temporal Shift Module (TSM)**  is a popular video action recognition model with shift operation, which can achieve the performance of 3D CNN but maintain 2D CNN's complexity. The method of moving through channels greatly improves the utilization ability of temporal information without increasing any additional number of parameters and calculation costs. TSM is accurate and efficient: it ever ranked first place on the Something-Something  leaderboard. Here, we adopt the industrial level variation from the deep learning platform PaddlePaddle . We improve the original design with additional knowledge distillation, where the model is pretrained on Kinetics400  dataset.

-   **SlowFast**  involves a Slow pathway operating at a low frame rate to capture spatial semantics, and another Fast pathway operating at a high frame rate to capture motions in the temporal domain. SlowFast is a powerful action recognition model, which ranks in second place on AVA v2.1 datasets .

-   **Video-Swin-Transformer (Vid)**  is a video classification model based on Swin Transformer . It utilizes Swin Transformer's multi-scale modeling and the efficient local attention module. Vid shows competitive performances in video action recognition on various datasets, including the Something-Something and Kinetic-series datasets.

For **Action Localization**, we investigate the following classical or trending algorithms,

-   **Boundary Sensitive Network (BSN)**  is an effective proposal generation method, which adopts "local to global" fashion. BSN has already achieved high recalls and temporal precision on several challenging datasets, such as ActivityNet-1.3  and THUMOS14' .

-   **BSN++**  is a new framework which exploits complementary boundary regressor and relation modeling for temporal proposal generation. It ranked first in the CVPR19 - ActivityNet challenge leaderboard on the temporal action localization task.

-   **Boundary-Matching Network (BMN)**  introduces the Boundary-Matching (BM) mechanism to evaluate confidence scores of densely distributed proposals, which leads to generating proposals with precise temporal boundaries as well as reliable confidence scores simultaneously. Combining with existing feature extractors (e.g., TSM), BMN can achieve state-of-the-art temporal action detection performance.

-   **Self-Supervised Learning for Semi-Supervised Temporal Action Proposal (SSTAP)**  is one of the self-supervised methods to improve semi-supervised action proposal generation. SSTAP leverages two crucial branches, i.e., the temporal-aware semi-supervised branch and relation-aware self-supervised branch to further refine the proposal generating model.

-   **Temporal Context Aggregation Network (TCANet)**  is the championship model in the CVPR 2020 - HACS challenge leaderboard, which can generate high-quality action proposals through "local and global" temporal context aggregation and complementary as well as progressive boundary refinement. Since it is a newly released model designed for several specific datasets (the original design shows unstable performance on **P$^2$A**), we slightly modify the architecture to incorporate a BMN block ahead of TCANet. The revised model is denoted as TCANet+ in our experiments.

## Experimental Setup

**Datasets.** As aforementioned, we establish two versions – 14-classes (14c) and 8-classes (8c) – in **P$^2$A**. The difference is that we combine all the serving strokes/actions in a single "Serve" category in 8c version. We evaluate the baseline action recognition algorithms on both versions and compare the performances side by side. Since the performances of the current action localization algorithms on 14c are far from satisfactory, we only report the results on 8c for reference.

**Metrics.** We use the Top-1 and Top-5 accuracy as the metric for evaluating the performance of baselines in the action recognition tasks. For the action localization task, We adopt the area under the Average Recall vs. Average Number of Proposals per Video (AR-AN) curve as the evaluation metric. A proposal is a true positive if it has a temporal intersection over union (tIoU) with a ground-truth segment that is greater than that or equal to a given threshold (e.g, tIoU\>0.5). AR is defined as the mean of all recall values using tIoU between 0.5 and 0.9 (inclusive) with a step size of 0.05. AN is defined as the total number of proposals divided by the number of videos in the testing subset. We consider 100 bins for AN, centered at values between 1 and 100 (inclusive) with a step size of 1, when computing the values on the AR-AN curve.

**Data Augmentation.** The category distribution analysis in Section 3.2 reveals that **P$^2$A** is an imbalanced dataset, where it might cause trouble to the recognition task. To fully utilize the proposed **P$^2$A** dataset, we design a simple yet effective data augmentation method to train the baseline models. Specifically, we introduce an up/down-sampling procedure to balance the sample size of each category. First, we calculate the mean size of all the samples (denoted $N$ samples) by the defined number of categories (e.g., 8c or 14c) as $M$, where $M = N/8$ or $N/14$. Then, we sample the actual number of action segments $S_A$ within a range $M \leq S_A \leq 2M$. For those categories with less number of segments than $M$, we up-sample by the random duplication. For those with much more samples over $2M$, we random down-sample from the original sample set and try to make the sampled actions cover all the video clips (i.e., uniform sampling from each 6-minute chunk). We apply this data augmentation strategy to all the recognition tasks and the significant performance gains. As shown in Figure , we report the ablation results of action recognition baseline TSM on **P$^2$A** with/without the data augmentation. In Figure (a), the confusion matrix illustrates that TSM falsely classifies most of the strokes/actions into the "Drive" category without the class balancing, while this kind of misclassification is significantly alleviated after we apply the data augmentation. Note that, since the localization tasks normally require feature extractor networks as the backbones (e.g., TSM is one of the backbones), localization performance could also be benefited from the designed data augmentation. The baseline algorithms in the following results section also involve data augmentation.

## Results

```latex
\scalebox{0.83}{
    \begin{tabular}{c|c|cc|cc}
        \toprule
        Actions & Method & Acc1@14c & Acc5@14c & Acc1@8c & Acc5@8c \\
         \hline
        \multirow{4}*{Face} &
         TSN & 59.77 & 90.13 & 73.84 & 94.85\\
        & TSM & \textbf{69.97} & 98.08 & \textbf{82.35} & \textbf{99.06}\\
        & SlowFast & 63.19 & 92.79 & 77.41 & 96.07\\
        & Vid & 68.85 & \textbf{98.16} & 81.03 & 98.96\\
        \midrule
        Face\&Back & TSM (Best) & 59.46 & 87.11 & 68.39 & 91.73 \\
        \bottomrule
    \end{tabular}}
```

This section first presents the performance results of action recognition tasks. Table  provides a summary of these results in terms of top1/top5 accuracy on both 8c and 14c datasets. As we can observe, the top5 accuracy is unsurprisingly higher than the top1 accuracy for all the methods. For comparing the 8c and 14c datasets in the same setting, the performances of baselines on 8c are relatively better than on 14c, which confirms that the combination of serving classes is reasonable and actually eases the recognition task. In general, our refined TSM (PaddlePaddle version) method achieves the highest accuracy in all four settings except the top5 accuracy on the 14c datasets. The popular transformer-based method Vid also shows competitive performances in the second place as a whole. Note that, since the same strokes/actions done by the player facing the camera and with back towards the camera differs drastically (e.g., the blocked views and the opposite actions), we mainly focus on the facing players' actions/strokes as the recognition targets. As for the back view player's actions/strokes, the performances are poor for all methods. Alternatively, we report the performance of the mixed (i.e., face and back) one, the results are far below the expectation, where even the best performer, TSM, only achieves 59.46 top 1 accuracy on the 14c and 68.39 top5 accuracy on the 8c dataset. Although the top5 accuracy of all the methods appears to be promising, considering the total number of categories in **P$^2$A**, which is 14 or 8 accordingly, the results are trivial to some degree. In conclusion, for the action recognition task with the mainstream baseline algorithms, **P$^2$A** is considerably a challenging dataset and there is a vast room and potential for improvement and research involvement.

For the action localization task, we summarize the performances in Figure . We selected several representative state-of-the-art action localization methods that are publicly available and retrained them on our **P$^2$A** dataset. We report the area under the AR-AN curve for the baselines as the measurements of the localization performance. For example, in Figure (a), we draw the AR-AN curves with a varying value of tIoU from 0.5 to 0.95 separately and one mean curve (solid blue curve) to represent the overall performance of BSN on **P$^2$A**. Among all the methods, TCANEt+ shows the highest overall AUC score which is 47.79 in Figure (f) and also outperforms other baselines in each tIoU level. To straightforwardly compare the localization results, we visualize the located action segments from each method in Figure . The top figure is on a large scale to show the predicted segments compared with the ground-truth segments in a complete 6-minute video chunk, where the bottom one zooms in a single turn for a series of consecutive actions/strokes by the player facing the camera. From such visual observations, we can find that TCANet+ can almost reproduce the short action segments from the ground-truth labeling. However, the first period of serving action is barely located and TCANet+ mis-predicts with a much shorter one instead. Compared to the action recognition task, the action localization seems more difficult on **P$^2$A** since the best AUC score is even in a massive gap from the average achievement of these baselines on ActivityNet  (47.79 $\ll$ 68.25 on average from top35 solutions in the 2018 leader-board).

**Summary.** From the above experimental results, we can conclude that the **P$^2$A** is still a very challenging dataset for no matter the action recognition and the action localization tasks. Compared to the current release sports datasets, the representative baselines hardly adapt to the proposed **P$^2$A** dataset and the performance is far lower than our expectation. Thus, the **P$^2$A** dataset provides a good opportunity for researchers to further explore the solution in a domain of fast-moving and dense action detection. Note that, the **P$^2$A** dataset is not limited to the action recognition and localization tasks only, where it is also potential to be used for video segmentation and summarizing. Actually, we are on the way to explore more possibilities of **P$^2$A** and will update the released version to include more use cases. Due to the limited space, we will release the implementation details for all the baselines in our Github repository with the dataset link once the paper gets published.

# Conclusion

In this paper, we present a new large-scale dataset **P$^2$A**, which is currently the largest publicly available table tennis action detection dataset to the best of our knowledge. We have evaluated several state-of-the-art algorithms on the introduced dataset. Experimental results show that existing approaches are heavily challenged by fast-moving and dense actions and uneven action distribution in video chunks. In the future, we hope that **P$^2$A** could become a new benchmark dataset for fast-moving and dense action detection especially in the sports area. Through a comprehensive introduction and analysis, we intend to help the follow-up users and researchers to better acknowledge the characteristics and the statistics of **P$^2$A**, so as to fully utilize the dataset in action recognition and action localization tasks. We believe the **P$^2$A** dataset will make an early attempt in driving the sports video analysis industry to be more intelligent and effective.

# Supplementary

Due to the page limit in the main body of the manuscript, we supplement the complete implementation details and evaluation results with brief discussions in this supplementary section.

```latex
\begin{table*}[htp!]\centering
    \caption{Key parameters of implemented algorithms.}
    \vspace{-2mm}
    \begin{tabular}{c|c|c|c|c|c}
    \toprule
        Recognition Method & Backbone & Sampling & Num\_Seg & Target Size & Others \\
        \midrule
        TSN & ResNet50-D & TenCrop & 8 & 224 & Distillation with ResNet152-CSN\\
        TSM & ResNet50-D & Dense & 8 & 224 & Distillation with ResNet152-CSN\\
        SlowFast & ResNet50-D & Time-stride & 8 & 224 & Mutigrid speedup~\cite{wu2020multigrid}\\
        TimeSformer & ViT-base-patch16-224 & UniformCrop & 8 & 224 & Linspace when testing\\
        ViD & SwinT-base-1k-224 & UniformCrop & 32 & 224 & -\\
        Attention-LSTM & Inception-ResNet-v2 & RandomNoise & 8 & 224 & -\\
        MoViNet & MoViNet-A0 & UniformCrop & 50 & 172 & AutoAugment~\cite{cubuk2018autoaugment}\\
         \midrule
         Localization Method & Backbone & Sampling & NMS & Window Size & Others \\
         \midrule
         BSN & TSM (ResNet50-D) & - & Soft &100  & - \\
         BSN++ & TSM (ResNet50-D) & - & Soft &100 & Proposal extension~\cite{su2020bsn++}, Self-attention \\
         SSTAP & TSM (ResNet50-D) & - & Soft &100 & Self-Supervised Learning\\
         BMN  & TSM (ResNet50-D) & - & Soft &100& Proposal extension~\cite{su2020bsn++} \\
         TCANet+  & TSM (ResNet50-D) & RandomNoise & Soft & 300 & LGTE, SEBlock~\cite{qing2021temporal}
          \\
         \bottomrule
    \end{tabular}
    \label{tab:parameters}
\end{table*}
```

**Remarks.** In this paper, we mainly target the 2D video action recognition algorithms considering the following sakes,

-   Traditional 3D/4D video action recognition algorithms such as I3D , R3D , S3D , and Non-local  are widely acknowledged to be inefficient in computation and optimization compared to the 2D algorithms (i.e., it is barely affordable and scalable in some real world applications), even though the 3D/4D algorithms generally have a more robust performance  on video recognition tasks.

-   Analyzing table tennis broadcasting video is not only limited to academic purposes, which includes the research on dense, fast-moving, and noisy (e.g., multiple action sources from players) action recognition and localization, it also benefits table tennis sports affairs, for example, real-time competition analysis, and actions/events summarizing on stream broadcasting videos. Thus, the effectiveness and efficiency are of equal importance for the solutions in industrial practices, where the 2D and the lightweight video action recognition algorithms have received a great deal of industrial and research attention in recent years.

-   We also include one 3D action recognition algorithm – SlowFast . Compared to the traditional 3D networks, SlowFast does not heavily rely on the concatenation of 3D CNNs, where its fast pathway could be very lightweight by reducing its channel capacity, so as to largely improve the overall efficiency. Furthermore, SlowFast is proved to be competitive with other full-size 3D recognition algorithms on several benchmark datasets .

Notice that the proposed **P$^2$A** is not only available for the action recognition and the action localization tasks. For the general action detection task, which incorporates the recognition and localization in a unified objective, the **P$^2$A** dataset is still challenging and remains explored in future study. As shown in Figure , we conduct the experiments for the action detection task on **P$^2$A** using a pipeline with the components of (1) TSM (PaddlePaddle variation), (2) BMN, and (3) Attention-LSTM. Specifically, we first extract the frame features using a TSM module (TSM module is pre-trained on a recognition task on the **P$^2$A** dataset). Once we have the well-represented features, we then feed the features into a BMN module to generate the action proposals. The last step is to train an Attention-LSTM module to obtain the action class within each proposal. We further provide ablation tests in Table . As we can observe, several optimization strategies improve the average mAP (mean average precision) of the proposed pipeline, which includes the proposal extension and data augmentation. Since the proposal generated by the BMN module sometimes cannot cover the entire action, we intend to extend the proposal duration before and after for 1 second. However, we find that not all the actions benefit from this kind of extension, where we additionally design a partial proposal extension strategy for the target actions. Although applying the above mentioned strategies, we hardly obtain a satisfied average mAP (IoU from 0.5 to 0.9 with a step size of 0.05), where the value is only 49.74 for the best model after the fine-tuning. It makes sense with the unsatisfactory result that the performance of action detection on **P$^2$A** depends on the performance of both the action recognition and the action localization tasks, which have been shown as two challenging tasks in this paper.

```latex
\scalebox{0.85}{
    \begin{tabular}{c|c|c|c|c}
        \toprule
         \diagbox[width=15em, innerrightsep=5em]{Pipelines}{mAP}{IoU} & 0.5 & 0.75 & 0.95 & Average \\
         \hline
        TSN + BMN + LSTM & 37.71 & 16.33 & 4.10 & 18.99 \\
        TSM + BMN + LSTM & 40.39 & 22.01 & 4.94 & 20.34  \\
        TSM + BMN + LSTM + EPE & 45.14 & 22.94 & 5.89 & 22.06 \\
        TSM + BMN + LSTM + PPE & 46.03 & 25.79 & 7.17 & 24.35 \\
        TSM + DA + BMN + LSTM + PPE & 49.75 & 30.68 & 8.03 & 33.35 \\
        \bottomrule
    \end{tabular}}
```

**Additional recognition algorithms.** Beyond the most representative action recognition algorithms introduced in the main body, we also conduct the experiments using the following algorithms,

-   Vanilla-TSN/TSM: the original version of TSN  and TSM  models. Note that, the revised TSM (denoted as TSM) includes (1) better backbones: ResNet50-D series; (2) Two-stage knowledge distillation: TSM adopt a semi-supervised distillation (SSD) strategy for the first stage and leverages the CSN  as the teacher network with a ResNet152 backbone in the second stage.

-   Attention-LSTM : recurrent neural networks (RNN) are often used in the processing of sequence data, which can model the sequence information of consecutive frames, and are commonly used methods in the field of video classification. As one of the RNNs, Attention-LSTM uses a two-way long and short-term memory network with attention layers to encode all the frame features of the video in sequence.

-   MoViNet : Movinet is a mobile video network developed by Google research. It uses causal convolution operators with stream buffer and temporal ensembles to improve the accuracy of video classification. It is a lightweight and efficient video model designed for online video stream reasoning.

-   TimeSformer : TimeSformer is a video classification model based on a vision transformer, which equips with the global receptive field, and strong time series modeling without convolutions. At present, it has achieved SOTA accuracy on the Kinetics-400 data set, approximating the performance of classic 3D-CNN based video classification models, while it has a considerably short training time.

As shown in Table , we present the full performance results of the implemented video action recognition algorithms in different settings. The TSM (revised) algorithm outperforms the other logarithms in most cases. We also observe that the recognition task of Face&Back is even more challenging than the recognition task of Face only, where it is reasonable since the actions/strokes from the back-view player are commonly blocked partially/entirely by the body of the player and the mixed actions/strokes from both players in the overlapped frames might affect the learning process of the baseline models. The overall performance of the baseline algorithms demonstrates the action recognition task on **P$^2$A** has a lot of room to be explored and improved in terms of the performance.

**Implementations.** We list the key hyper-parameters of all the implemented action recognition and action localization algorithms in Table . Note that, the TSN and TSM stands for the revised versions compared to the released version from the original paper. For other baselines, we follow the original design of the critical components from each algorithm and adjust specific hyper-parameters to fit them on the **P$^2$A** dataset.

```latex
\scalebox{0.75}{
    \begin{tabular}{c|c|cc|cc}
        \toprule
        Actions & Method & Acc1@14c & Acc5@14c & Acc1@8c & Acc5@8c \\
         \hline
        \multirow{9}*{Face} &
         TSN & 59.77 & 90.13 & 73.84 & 94.85\\
        & Vanilla-TSN & 57.28 & 89.36 & 72.07 & 93.89 \\
        & TSM & \textbf{69.97} & 98.08 & \textbf{82.35} & \textbf{99.06}\\
        & Vanilla-TSM & 63.33 & 93.21 & 76.79 & 95.63 \\
        & SlowFast & 63.19 & 92.79 & 77.41 & 96.07\\
        & Vid & 68.85 & \textbf{98.16} & 81.03 & 98.96\\
        & Attention-LSTM & 53.19 & 81.98 & 73.34 & 89.31 \\
        & MoViNet & 54.79 & 85.31 & 74.09 & 88.93 \\
        & TimeSformer & 64.01 & 94.87 & 77.39 & 95.56 \\
        % \midrule
        % \multirow{9}*{Back} &
        %  TSN & & & & \\
        % & Vanilla-TSN & & & & \\
        % & TSM & & & & \\
        % & Vanilla-TSM & & & & \\
        % & SlowFast & & & & \\
        % & Vid & & & & \\
        % & Attention-LSTM & & & & \\
        % & MoViNet & & & & \\
        % & TimeSformer & & & & \\
        \midrule
        %Face\&Back & TSM (Best) & 59.46 & 87.11 & 68.39 & 91.73 \\
        \multirow{9}*{Face\&Back} &
         TSN & 53.31 & 78.03 & 54.13 & 82.46 \\
        & Vanilla-TSN & 50.47 & 76.69 & 53.94 & 80.31\\
        & TSM & \textbf{59.46} & 87.11 & \textbf{68.39} & \textbf{91.73} \\
        & Vanilla-TSM & 55.72 & 84.05 & 63.39 & 86.32 \\
        & SlowFast & 56.39 & \textbf{87.98} & 65.10 & 89.93 \\
        & Vid & 58.54 & 87.20 & 67.28 & 91.05\\
        & Attention-LSTM & 51.06 & 75.39 & 52.79 & 80.67 \\
        & MoViNet & 52.45 & 77.19 & 54.12 & 82.07 \\
        & TimeSformer & 56.49 & 86.31 & 64.93 & 90.30 \\
        \bottomrule
    \end{tabular}}
```

[^1]: The link will be released later.


abstract: A real-time capable core turbulence tokamak transport model is developed. This model is constructed from the regularized nonlinear regression of quasilinear gyrokinetic transport code output. The regression is performed with a multilayer perceptron neural network. The transport code input for the neural network training set consists of five dimensions, and is limited to adiabatic electrons. The neural network model successfully reproduces transport fluxes predicted by the original quasilinear model, while gaining five orders of magnitude in computation time. The model is implemented in a real-time capable tokamak simulator, and simulates a 300 s ITER discharge in 10 s. This proof-of-principle for regression based transport models anticipates a significant widening of input space dimensionality and physics realism for future training sets. This aims to provide unprecedented computational speed coupled with first-principle based physics for real-time control and integrated modelling applications.
address: $^1$FOM Institute DIFFER – Dutch Institute for Fundamental Energy Research, P.O. box 6336, 5600 HH Eindhoven, The Netherlands; $^2$CEA, IRFM, F-13108 Saint Paul Lez Durance, France; $^3$Eindhoven University of Technology, Department of Mechanical Engineering, Control Systems Technology Group, P.O. Box 513, 5600MB, Eindhoven, The Netherlands; $^4$Istituto di Fisica del Plasma "P. Caldirola", Associazione Euratom-ENEA-CNR, Milano, Italy; $^5$Aix-Marseille Universit´e, CNRS, PIIM UMR 7345, 13397 Marseille, France
author: J. Citrin$^{1,2}$, S. Breton$^2$, F. Felici$^3$, F. Imbeaux$^2$, T. Aniel$^2$, J.F. Artaud$^2$, B. Baiocchi$^4$, C. Bourdelle$^2$, Y. Camenen$^5$, J. Garcia$^2$
bibliography: neuralbib.bib
title: Real-time capable first principle based modelling of tokamak turbulent transport

*Introduction*.– Particle, momentum, and heat transport in the tokamak core is dominated by turbulence driven by plasma microinstabilities . An accurate predictive model for turbulent transport fluxes is thus vital for the interpretation and optimization of present-day experiments, and extrapolation to and control of future machines.

Direct numerical simulation with massively parallel nonlinear gyrokinetic codes has provided tremendous insight to the underlying transport physics and success in reproducing experimental fluxes in many regimes. However, the computational cost – typically $10^5$ CPU hours for a local flux calculation at a single radial point – precludes the routine use of such codes for integrated tokamak transport simulations which demand $\sim10^3$ flux computations per 1 s of plasma evolution on JET scale devices.

Reduced turbulent transport models have been constructed to increase tractability. They are based on the quasilinear approximation, which is proven to be largely valid in the core of tokamak plasmas . These rely on nonlinear simulations for validating their ansatzes and normalizing factors. These models have proven successful in reproducing experimental profiles in many cases. Examples are TGLF  and QuaLiKiz . A $\sim$`<!-- -->`{=html}6 orders of magnitude speedup is gained in quasilinear calculations compared to nonlinear simulations. However, while extremely useful, the tractability of such models is still marginal for convenient large-scale scenario development over discharge timescales. For example, 1 s of JET plasma evolution can take up to 10 hours with 10 processors, depending on the integrated modelling platform used. This speed is also insufficient for applications such as trajectory optimization, and simulations for developing real-time controllers. Furthermore, any increase in physics fidelity in the models often results in a trade off with further decrease in tractability.

This Letter illustrates an approach to overcome these challenges. The central point is to relegate the expensive flux calculations to a stage precedent to its use in a transport simulation. Instead, analytical formulae are to be used in the simulation, based on a neural network (NN) nonlinear regression of quasilinear fluxes previously compiled in a database. The advantage is twofold: 1) the numerical resolution of analytical formulae is orders of magnitude faster than original flux calculation; 2) the computation time required for compiling the database is independent from the computation time spent during the tokamak simulation itself, hence the training set for NN regression can include results from more complete codes than used in contemporary integrated transport modelling.

Neural networks have found multiple applications in tokamak research, including: nonlinear regression for energy confinement scaling ; neoclassical transport ; rapid determination of equilibria , electron temperature profiles , and charge exchange spectra ; classification of disruption  and L-H transition onsets . Most related to this work is a regression of DIII-D heat fluxes from experimental power balance databases .

*Quasilinear transport model and training set*.– The QuaLiKiz quasilinear gyrokinetic transport model  was employed in this work. QuaLiKiz solves a linear gyrokinetic dispersion relation for calculating wavenumber spectra of instability growth rates and frequencies. Then, integrating over the spectra, the transport fluxes are calculated via quasilinear flux integrals and nonlinear saturation rules. The bulk of the computational time is spent in the first stage, the dispersion relation solver. QuaLiKiz has been coupled to the CRONOS  integrated modelling suite, and has successfully reproduced temperature and density profiles of JET and Tore-Supra discharges . Following recent upgrades , the computational time for the QuaLiKiz eigenvalue solver at a single wavenumber in QuaLiKiz is on the order of $\sim$`<!-- -->`{=html}1 s.

A database of QuaLiKiz solutions was constructed, in the ion temperature gradient (ITG) instability regime. This instability is often the primary driver of tokamak microturbulence. The code was run with adiabatic electrons for simplicity, which also decreases the computational time to $\sim$`<!-- -->`{=html}300 ms. The database covers four input parameters known to have significant impact on ITG transport fluxes in this regime: the driving normalized logarithmic ion temperature gradient $R/L_{Ti}$, the ion to electron temperature ratio $T_i/T_e$, the safety-factor $q$, and the magnetic shear $\hat{s}\equiv\frac{r}{q}\frac{dq}{dr}$. In addition, the input normalized wavenumber $k_\theta\rho_s$was scanned, constricted to above ion-Larmor-radius scales, where $\rho_s\equiv\sqrt{T_em_i}/(Z_iq_eB)$. The following parameters were maintained fixed: the normalized logarithmic density gradient $R/L_n=3$, normalized radial location $r/a=0.5$. No Shafranov shift was assumed in the geometry. The database consists of a dense grid of points summarized in table , from which the training sets for the neural network were sifted. The QuaLiKiz outputs we investigate are: growth rates and frequencies, which correspond to 5D input space; ion heat flux, which corresponds to 4D input space due to integration over wavenumbers. The database includes cases corresponding to unstable modes, and cases where no instabilities were found by QuaLiKiz, and the outputs are set to 0.

A regression of the ion heat flux has immediate application for transport modelling. However, a regression of the more primitive linear output has its own specific applications. For example, since the dispersion relation solver is the slowest part of the code, a fast reproduction of growth rates, frequencies and eigenfunctions would allow rapid tests of various saturation rule formulations throughout parameter space. These saturation rules typically evolve following continuous comparisons with nonlinear simulations and experiments. In this sense, a database consisting of the complete outputs of linear codes does not become obsolete, while a quasilinear flux database can.

```latex
\scalebox{0.84}{\begin{tabular}{c c c c}
\label{tab:summary1}
Parameter  & Min value & Max value & No. of points \\
\hline
$R/L_{Ti}$& 2 & 12 & 30 \\
$T_i/T_e$& 0.3 & 3 & 20 \\
$q$& 1 & 5 & 20 \\
$\hat{s}$& 0.1 & 3 & 20 \\
$k_\theta\rho_s$& 0.05 & 0.8 & 16 \\
\hline
\multicolumn{3}{c}{Total no. of points} & 3 840 000\\
\hline
\end{tabular}}
```

*Neural networks*.– The goal is to find analytical formulae which robustly reproduce the various QuaLiKiz outputs. To this end, a multilayer perceptron neural network is used, which is a nonlinear function with tunable variables (weights and biases), with the property of universal approximation . For an overview, with an emphasis on applications for fusion, see Ref.. Linear combinations of the inputs and biases are propagated through a series of nonlinear transfer function vectors (named 'hidden layers'), until eventually linearly combined to an output layer. With two hidden layers and a single output value (as used in this work), this is represented as: $$\label{eq:NN}
y = b_3  + \sum_i^N{w^2_i}g\left(b^2_i + \sum_j^Mw^1_{ij}g\left(b^1_j+\sum_k^{I}w^{in}_{jk}x_k\right)\right)$$ Where $y$ is the output 'neuron' containing the output value (i.e. growth rate, frequency, or ion heat flux), $x_k$ the vector of input values, $b^n$ the bias vectors, $w^{in}$ the M$\times$I weight matrix connecting the input vector to the 1st hidden layer, $w^{1}$ the N$\times$M weight matrix connecting the two hidden layers, and $w^{2}$ the weight vector connecting the 2nd hidden layer to the output neuron. $g$ is the nonlinear transfer function, defined as a sigmoid in this work: $$\label{eq:sig}
g(x)=\frac{2}{1+e^{-2x}}-1$$ Following a series of optimization tests, two hidden layers, as shown in equation , were employed here. The hidden layer sizes M and N were set to 40. The input layer size, I, is 4 for ion heat fluxes, and 5 for growth rates and frequencies.

The key stage is the determination of the optimized values of the weights and biases. This is done by minimizing a cost function consisting of the average squared error between the network output and known target output. This set of target output, known as the 'training set', is a subset of the QuaLiKiz output values from the database. The BFGS algorithm , implementing a quasi-Newton method, was used for the weight and bias optimization. All NN training in this work was carried out with the MATLAB neural network toolbox . Following training, the network output then emulates the original model within the database input parameter envelope. This is validated by comparison to validation sets sifted from the database, which are different from the training set.

To avoid overfitting the data, regularization techniques were used in the regression. This corresponds to adding a penalty term in the cost function related to the sum of squares of the network weights and biases, leading to smoother output. The use of regularization ensures that the NN response is smooth (e.g. without strong oscillations) in sparse regions of training set parameter space or when extrapolating beyond the training set envelope.

The analytic form of the nonlinear regression function allows for the calculation of analytical gradients of the outputs with respect to the inputs. This is vital for the efficient solution of fast implicit schemes in real-time capable core transport simulators such as RAPTOR . The regularization also ensures smooth gradients throughout parameter space, important for the stability of such implicit schemes.

*Regression results*.– We focus on the ion heat flux NN regression, due to its direct relevance for transport modelling applications. Successful regressions of the growth rates and frequencies were also obtained but for brevity not discussed here.

To capture the instability thresholds with high fidelity, the regression was only carried out for a training set corresponding to unstable modes. The NN output for the stable regions in the validation set was then negative, since the regularized network tends to smoothly extrapolate the trends observed towards the training set envelope. For the final heat flux output, these negative values were then set to zero to represent stability. This scheme avoids having the regularized regression network attempt to directly fit the discontinuous gradients at the instability thresholds, which would be performed poorly due to to the regularization constraint. This is an important point since tokamak transport often tends to be maintained near the critical temperature gradient thresholds, especially in high temperature regimes.

The network was trained with a training set of 35000 points chosen randomly from the set of unstable modes in the database. A comparison between the regression NN and QuaLiKiz outputs for a validation set of 10000 unstable cases (different from the training set) is shown in figure . The regression network has an RMS error of 0.77 in gyroBohm units ($\chi_{iGB}=\frac{T_i^{3/2}m_i^{1/2}}{(Z_iq_eB)^2r}$) when compared to the validation set. This RMS error is similar for the training set itself and is primarily due to the regularization constraint. The impact of this error on the simulated profiles is minor. This is due to stiffness, defined here as the local gradient of $\chi_i$ with respect to the driving $R/L_{Ti}$. To quantify this, a comparison was made between the $R/L_{Ti}$ values predicted by the NN and QuaLiKiz to balance a representative $\chi_{iGB}=1$. This was done for all values of $q$, $\hat{s}$, and $T_i/T_e$ in the database. The RMS error was ${\Delta}\left(R/L_{Ti}\right)=0.29$, which corresponded to an average relative error of only 4.2%.

The typical quality of the fits can be seen in figure , displaying scans of the 4 separate input parameters while the others remained fixed. Negative outputs of the NN network are set to zero. Note the resulting excellent fit of the instability thresholds. In addition, extrapolating the NN scans beyond the range of the training set maintains the trend observed in the data, due to the regularization. This is very encouraging with regard to extension of this approach to more sparse datasets in higher dimensions. However, we do not intend to routinely use NN models in poorly represented regions of parameter space, as the quality of extrapolation cannot be determined a priori. Rather, the training sets should be continuously expanded to cover such encountered sparse or empty regions, and the NN then periodically retrained. Nevertheless, the smoothness of the regularized NN response when extrapolating ensures its robustness and stability during practical use as a transport model, including during phases when such sparse regions are encountered."

Each NN output is calculated on a sub 10 ${\mu}s$ timescale in MATLAB on a Intel(R) Xeon(R) E5450 CPU @ 3.00GHz. This is a 5 order of magnitude speedup in comparison to the original QuaLiKiz calculations.

*Application in transport codes*.–A transport model based on the trained neural network was constructed, and implemented both in the CRONOS  and RAPTOR  integrated modelling codes.

In CRONOS, the validity of the NN transport model was assured by a successful comparison with a JET baseline H-mode shot 73342 with ion and electron heat transport previously simulated  with the full QuaLiKiz model. For brevity we will not focus further on this case. Rather, we focus on the real-time simulation capabilities offered by coupling the NN model to RAPTOR.

Presently, RAPTOR only models electron heat transport. The NN model output was thus modified to roughly approximate ITG regime electron heat transport with kinetic electrons. This was done by assuming that heat fluxes in ITG kinetic electron cases are higher by factor 3 compared with adiabatic electron cases, and furthermore assuming an ion to electron heat flux ratio of $q_i/q_e=3$. These approximations are based on typical nonlinear and quasilinear observations in the ITG regime .

In figure , we compare a RAPTOR simulation of an ITER hybrid scenario, using the QuaLiKiz NN model for electron heat transport, with a simulation of the same case originally carried out  using CRONOS and the GLF23  transport model. Using GLF23 allows to compare over ITER-scale discharge times of $>$`<!-- -->`{=html}100s, which is less tractable using the original QuaLiKiz model. For heat transport in a pure ITG regime, GLF23 and QuaLiKiz predictions are expected to be similar, as illustrated in specific single-time-slice comparisons .

The RAPTOR simulation uses all the same actuator (source) inputs and density evolution as the CRONOS simulation. Ion temperatures were held fixed at $T_i/T_e\sim0.8$ in L-mode and $T_i/T_e\sim0.9$ in H-mode. The NN model was operational within a normalized toroidal flux coordinate ($\rho$) range of 0.25 to 0.95. For $\rho>0.95$, $\chi_e$ was feedback controlled to maintain a prescribed edge pedestal temperature of 4 keV. For $\rho<0.25$, a constant $\chi_e$ was assumed to maintain a reasonable level of transport, since GLF23 and QuaLiKiz both predicted stability within that region. A RAPTOR simulation of an entire 300 s ITER discharge took 10 s on a single CPU, corresponding to 30x faster than real-time. This combination of simulation speed and first-principle modelling is unprecedented. With CRONOS/GLF23, the same simulation took 24 hours.

*Conclusions and outlook*.– A neural network fit to a restricted subspace of quasilinear gyrokinetic transport model calculations, relevant in the ITG regime, was carried out and applied as a transport model for integrated modelling. While the quasilinear model, QuaLiKiz, is 6 orders of magnitude faster than nonlinear simulations, the NN regression leads to a further 5 order of magnitude speedup. This model is thus real-time capable while still being based on first-principles, which is unprecedented. This model has been coupled to the CRONOS integrating modelling suite, and validated against a full QuaLiKiz simulation in the ITG regime. The model is also coupled to the real-time capable RAPTOR tokamak simulation code, and can model a 300s ITER discharge within 10s, with good agreement with previous modelling using CRONOS and the GLF23 transport model.

This opens up many new possibilities for real-time controller design and validation, scenario preparation and optimization, and real-time discharge supervision. Such models can be used to design controllers for the plasma profiles using model-based controller design methods (e.g. or ). The transport model can be used in closed-loop simulations to validate the designed controllers. Recent work on plasma ramp-up trajectory optimization was carried out with an ad-hoc transport model, and can now be improved using this first-principle-based transport model. Also, this transport model can be used in real-time simulations to verify the measured plasma evolution and warn a supervisory control system of any unexpected deviations during the discharge . Specifically for ITER, the faster-than-real-time opens up the possibility of (on-line) real-time optimization of the discharge evolution in response to such unexpected events.

While applications in the ITG transport regime are already feasible with this model, there remains much scope for expanding the number of input dimensions in the databases used for the fits, as well as employing slower yet more complete linear gyrokinetic codes for populating the database. Neural network topology complexity favourably scales linearly with input dimensionality. However, we estimate that uniform density population of the input dimensions, as carried out in this work, is feasible up to N$\sim$`<!-- -->`{=html}10. This is due to constraints on the NN training and quasilinear database calculation times. For higher dimensionalities, a training set which captures the natural correlations of the input parameters is then vital. This can be done by basing the training sets on experimental parameters and reasonable extrapolations thereof. This is a feasible goal, as also evidenced in Ref. , and this work is ongoing.

*Acknowledgments*.– This work is part of the research programme 'Fellowships for Young Energy Scientists' (YES!) of the Foundation for Fundamental Research on Matter (FOM), which is financially supported by the Netherlands Organisation for Scientific Research (NWO). The authors greatly thank O. Meneghini, S. Smith, U. von Toussaint and P. Xanthopolous for inspiring discussions.

# References


author: Carlos Gershenson and Francis Heylighen
bibliography: carlos.bib; sos.bib
title: Protocol Requirements for Self-organizing Artifacts: Towards an Ambient Intelligence

# A Scenario

The diversity and capabilities of devices we use at home, school, or work, are increasing constantly. The functions of different devices often overlap (e.g. a portable computer and a mobile phone have agendas; a radio-clock and a PDA have alarms), but most often we cannot combine their capabilities automatically (e.g. the PDA cannot tell the radio to set its alarm for the early Tuesday's appointment), and users need to repeat the same tasks for different devices (e.g. setting up an address book in different devices). Moreover, using the functionality of some devices in combination with others would be convenient (e.g. if my computer has an Intelligent User Interface, I would like to use it to ask for coffee, without the need of having speech recognition in the coffee machine: The computer should be able to ask the coffee machine for cappuccino).

Could we build devices so that they would *automatically coordinate*, combining their functions, and possibly producing new, "emergent" ones? The technology to achieve this is already at hand. What we lack is a proper design methodology, able to tackle the problems posed by autonomously communicating artifacts in a constantly changing technosphere. In this paper we try to delineate the requirements that such a design paradigm should fulfill. The scenario we imagine considers a nearby future where technological artifacts *self-organize*, in the sense that they are able to communicate and perform desirable tasks with minimal human intervention.

This vision is closely related to the concept of "Ambient Intelligence" (AmI), which envisages a future where people are surrounded by "smart" and "sensitive" devices. AmI would be the result of the integration of three technologies: Ubiquitous Computing , Ubiquitous Communication, and Intelligent User Friendly Interfaces. The first one conceives of a seamless integration of computation processes taking place in the variety of artifacts that surround us, being part of "The Grid", the network that would allow anyone anywhere to access the required computing power. The present paper focuses on the aspect of Ubiquitous Communication that attempts to obtain seamless information exchange between devices. Intelligent User Friendly Interfaces should enable an intuitive, effortless interaction between users and devices.

With current approaches, this scenario would be possible, since we have the technology, but extremely expensive, since people would need to buy from the same producer all of their devices. We can see a similar casein the area of Home Automation: the technology is available on the market, but it is not possible to buy today ventilation for a house, and in five years integrate the system with a new fire detector. An engineer needs to integrate them manually, so that the ventilation system could be activated if smoke is detected, simply because the ventilation system was not designed to receive such signals. These limitations increase the price and restrict the market of devices for Home Automation, since complete solutions should be bought in order to have full coordination and functionality between devices. People would be more willing to invest in Home Automation if they could have the possibility of acquiring it progressively.

# Requirements for self-organizing artifacts

We see self-organization as a paradigm for designing, controlling, and understanding systems . A key characteristic of a self-organizing system is that structure and function of the system "emerge" from interactions between the elements. The purpose should not be explicitly designed, programmed, or controlled. The components should *interact* freely with each other and with the environment, mutually adapting to reach an intrinsically "preferable" or "fit" configuration (attractor), thus defining an emergent purpose for the system . By "self-organizing artifacts" we mean a setup where different devices, with different fabrications and functionalities, and moving in and out of different configurations, can communicate and integrate information to produce novel functionalities that the devices by themselves could not achieve.

A first requirement for such communication is cross-platform compatibility. This is already achieved for programming with Java, and for documents with XML. Another requirement is wireless communication, which is offered by technologies such as IR, Bluetooth and WiFi. Near Field Communications (NFC) is a newly envisioned standard, proposed by a consortium headed by Sony, Nokia, and Philips, which would allow information to be transmitted between devices that come in close spatial proximity ("touching").

Even with such a standard, the problem remains that the user generally would need to specifically request such communication between devices (e.g. "transfer this file from here to there"). Ideally, the devices would *know* what we want them to do and how to do it. User Interfaces already help us to tell them our wishes. Still, one device cannot tell *another* device what we want, especially if they are produced by different manufacturers. This is a general problem of communication between artifacts: they can recognize standard messages, but they do not "know" what the messages *mean*. To avoid endless debates, we can say that the meaning of a message is determined by its *use* : if a device has received a message, and does "the right thing" (for the user), then it has "understood" the meaning of the message. Thus, the user's satisfaction is the ultimate measure of the effectiveness of the artifacts' performance.

Another issue is how to deal with changes in technology. We do not want to reconfigure every artifact each time a new device arrives. Moreover, we want the old devices to be able at least to cope with the functionality of new ones. New devices should configure themselves as automatically as possible. Older ones may require user intervention at first (as they cannot know beforehand which functions will be required), but they should be able to cope with new technology being added to the network. The overall system must be *adaptive*, *extensible*, and *open*.

An adaptive system can cope with unexpected changes in its environment, as exemplified by the constantly changing technology. Having flexibility built into our systems is desirable: they should at least be able to tolerate events they were not designed for without breaking down, but preferably try to find adapted solutions, or at least ask assistance from the user. For example, home appliances have a limited set of functions. To have them self-organize (e.g. the alarm clock coordinating with the microwave oven, and the oven with the kettle), their functions could be easily programmed to respond to unknown messages. If a new device arrives, and an old one does not know what to do when it receives a message, it can check what the user wants, thus learning how to respond appropriately. The possibility to add more devices to an existing configuration may be called *extensibility*.

Suppose that a company develops adaptable and extensible devices that interact seamlessly with each other. This would still leave the problem that customers cannot add devices from other companies, as these would follow their own standards, thus creating compatibility problems. We believe that the solution is to have *open* technologies, in the spirit of GNU. Open means that everyone has free access to their specifications. The advantage is that they can develop much faster, meeting the requirements of more people, because they are developed by a global community that can try out many more approaches than any single company. Still, a company can benefit in promoting an open technology, since this would provide them with free publicity while everyone is using their protocol (e.g. Sun's Java).

Open technology can respond to the needs of the largest variety of people, while allowing problems and errors to be detected and corrected more easily. Another advantage is that it allows people to get updates developed by other users for free. For example, if I program my "old" toaster to integrate with my new mobile phone, it costs me nothing to make the program available on the Internet to anyone else who might need it. Thus, updates, extensions, and specialized applications can flow much more quickly from a global community than from a private company. Still, companies would benefit from this approach, since people would be more willing to buy new devices as integrating them into their existing, open setup, will be easier.

# Achieving self-organization

We can divide the problem of self-organizing integration into three subproblems: 1) devices should learn to *communicate* with each other, even when they have no a priori shared understanding of what a particular message or function means; 2) devices should learn which other devices they can trust to *cooperate*, avoiding the others; 3) devices should develop an efficient *division of labour* and workflow, so that each performs that part of the overall task that it is most competent at, at the right moment, while delegating the remaining functions to the others.

These issues are all part of *collective intelligence* or *distributed cognition* : a complex problem cannot be tackled by a single device or agent, but must be solved by them working together, in an efficiently coordinated, yet spatially distributed, system, where information flows from the one agent to the other according to well-adapted rules. Until now, distributed cognition has been studied mostly in existing systems, such as human organizations or animal "swarms" , that have evolved over many generations to develop workable rules. Having the rules self-organize from scratch is a much bigger challenge, which has been addressed to some degree in distributed AI and multi-agent simulations of social systems. Inspired by these first explorations, we will propose a number of general mechanisms that could probably tackle the three subproblems. However, extensive simulation will clearly be needed to test and elaborate these mechanisms.

# Learning to communicate

To communicate effectively, different agents must use the same concepts or categories. To achieve effective coordination, agents must reach a shared understanding of a concept, so that they agree about which situations and actions belong to that category, and which do not. A group of agents negotiating such a consensus may self-organize, so that a globally shared categorisation emerges out of local interactions between agents.

Such self-organization has been shown in different simulations of the evolution of language . Here, interacting software agents or robots try to develop a shared lexicon, so that they interpret the same expressions, symbols, or "words" in the same way. In these simulations agents interact according to a protocol called a "language *game*". There are many varieties of such games, but the general principle is that two agents "meet" in virtual space, which means that through their sensors they experience the same situation at the same time. Then they try to achieve a consensus on how to designate one of the components of their shared experience by each in turn performing elementary *moves*.

In a typical move, the first agent produces an "utterance" referring to a phenomenon that belongs to one of its inbuilt or previously learned categories, and the second one finds the best fitting category for that phenomenon in its knowledge base. The second agent then indicates a phenomenon belonging to that same category. If this phenomenon also belongs to the same category for the first agent, both categorisations are reinforced, otherwise they are reduced in strength. In the next move of the "game", another phenomenon is indicated, which may or may not belong to the category. The corresponding categorisation is strengthened or weakened depending on the degree of agreement. After a number of moves the game is stopped, each agent maintaining the mutually adjusted categories. Each agent in turn is coupled to another agent in the system, to play a new game using different phenomena. After some games a stable and coherent system of categories shared by all agents is likely to emerge through self-organization. A good example of such a set-up can be found in Belpaeme's simulation of the origin of shared colour categories.

If for some reason devices are not able to communicate, they should be able to notify the user, and ask for the correct interpretation of the message. This is easy, since devices have a limited functionality. It would be possible to "teach" a device what to do if it receives a particular message, and the device should "learn" the meaning of the message.

Research has been done in multi-agent systems where agents negotiate their protocols , which could be extended for a setup of self-organizing artifacts. However, agent communication standards, such as FIPA, still do not contemplate adaptation to new meanings. Nevertheless, there is promising research going on in this direction.

# Learning to cooperate

Integrated devices should not only communicate, but cooperate. Cooperation may seem self-evident in preprogramed systems, where the components are explicitly designed to respond appropriately to requests made by other components. However, this is no longer the case in open, extensible configurations.

For example, a person at the airport would like her PDA to collaborate with the devices present at the airport, so that it can automatically warn her when and where she has to go, or tell her which facilities are available in the airport lounge. Yet not all devices at the airport may be ready to help a PDA, e.g. because of security restrictions, because they are proprietary and reserved for paying customers, or because they simply do not care about personal wishes. Moreover, devices may be ready to share certain types of services but not others, e.g. telling users when the flight is scheduled to depart, but not how many passengers will be on it. As another example, devices may not only be uncooperative, but malevolent, in the sense that they try to manipulate other devices in a way detrimental to their user. Such devices may be programmed, e.g. by fraudsters, spies, or terrorists.

There exists an extensive literature on the evolution of cooperation between initially "selfish" agents, inspired by the seminal work of Axelrod that compared different strategies for playing a repeated "Prisoners' Dilemma" game. However, this game does not seem directly applicable to information exchanging devices. Moreover, the chief result, while sensible, may seem trivial: the most effective strategy to achieve robust cooperation appears to be *tit for tat*, i.e. cooperate with agents that reciprocate the cooperation, stop cooperating with those that do not. More recent, tag-based models (e.g. start from a simpler situation than the Prisoners' Dilemma, in which one agent "donates" a service to another one, at a small cost to the donor but a larger benefit to the recipient. The main idea is that agents are identified by "tags", and that they cooperate with those agents whose tags are similar to their own. The rationale is that agents with the same type of tag belong to the same group, "family" or "culture", following the same rules, so that they can be trusted to reciprocate.

For artifacts, a tag may include such markers as brand, model, and protocols understood. This would show that a device is capable and willing to lend particular services to another one, thus obviating the need for a repeated, "tit-for-tat-like" interaction probing the willingness to reciprocate. Yet extensible environments should allow the addition of very dissimilar devices, made by different companies using different standards and functionalities. Therefore, we propose a different approach, combining some advantages of tags and tit-for-tat strategies.

Consider a game with the following moves: an agent makes a request and the other agent either cooperates (donates) or "defects". Agents learn from these interactions in the following manner: if the result is positive (cooperation), the agent will get more "trust" in the other agent's cooperativeness. Thus, the probability increases that it will make further requests to that agent in the future, or react positively to the other's requests. Vice-versa, a negative result will lead to more "distrust" and a reduced probability to make or accept requests to/from this agent. Still, to recognise this agent, it has to take its clue from the tag, which is usually not unique to that agent. This means that a later interaction may be initiated with a different agent that carries a similar tag, but that is not necessarily willing to cooperate to the same extent. We may assume that if the first few interactions with agents having similar tags all generate positive (negative) results, the agent will develop a default propensity to react positively (negatively) always to agents characterised by that type of markers.

We expect that in this way the initially undirected interactions will produce a differentiation in clusters of similarly marked agents that cooperate with each other (e.g. all devices belonging to the same user or organization), but that are reluctant to interact with members of other groups (e.g. devices belonging to rival organizations). The tags and their association thus develop the function of a mediator that increases the probability of positive interactions by creating a division between "friends" (in-group) and "strangers" or "foes" (out-group). Note, however, that there is no assumption that an agent only cooperates with agents bearing the same tag as itself: by default it cooperates with anyone having a tag similar to the one of agents that were cooperative in the past. This means that there can be groups with which everyone cooperates (e.g. "public" devices), but also that specific types of "symbiosis" can develop in which one group systematically seeks out members of a different group to cooperate with because of their complementary capabilities. This brings us to the more complex issue of the division of labour.

# Learning to coordinate

After having ascertained that our devices can communicate and cooperate, we still need to make sure that the functions they perform satisfy the user. This desired functionality can be viewed as a complex of tasks that need to be executed. The tasks are mutually dependent in the sense that a certain task (e.g. locating a file) has to be accomplished before subsequent tasks (e.g. downloading and playing the file) can be initiated. Each agent can either execute a task itself, or delegate it to another agent. Initially, we may assume that all agents that have a certain functionality built in (e.g. playing a sound file) are equally competent at performing that type of task. However, in practice the satisfaction of the user can vary. For example, a recording is likely to be played with a higher sound quality by a music installation than by a PDA or television. By default, devices can use certain preprogramed rules-of-thumb to decide who takes precedence (e.g. newer or more specialized devices are preferred to older, less specialized ones). Yet in an open environment there is no guarantee that such simple heuristics will produce the best result. Again, we may tackle this problem through individual learning coupled with collective self-organization.

Assume that the user regularly expresses his/her overall satisfaction with the ambient intelligence environment (e.g. explicitly by clicking on a scale from one to ten, or implicitly by facial or physiological cues that express happiness/unhappiness). This score can be used as a feedback signal to the network of devices, allowing it to reinforce the more successful rules, while weakening the less effective ones. We will assume that the agent who delegated a task will increase its trust in the competence of the agent that performed that task, and thus increase its probability to delegate a similar task to the same agent in the future. Otherwise, it will reduce its trust. As demonstrated by the simulation of Gaines , this assumption is sufficient to evolve a self-reinforcing division of labour where tasks are delegated to the most "expert" agents.

However, when the tasks are mutually dependent, selecting the right specialist to carry out a task is not sufficient: First the preparatory tasks have to be done by the right agents, in the right order. When the agents do not know a priori what the right order is, they can randomly attempt to execute or delegate a task, and, if this fails, pick out another task. Eventually they will find a task they can execute, either because it requires no preparation, or because a preparatory task has already been accomplished by another agent. Each completed task enables the accomplishment of a series of directly dependent tasks. In this way the overall problem will eventually be solved. In each problem cycle, agents will learn better when to take on which task by themselves, or when to delegate it to a specific other agent.

We expect that this learned organisation will eventually stabilise into a system of efficient, coordinated actions, adapted to the task structure. When new devices are added to the system, system and device should mutually adapt, producing a new organization. While no single agent knows how to tackle the entire problem, the knowledge has been "distributed" across the system. The "tags" that identify agents, and the learned associations between a tag and the competence for a particular task, play the role of a mediator , delegating tasks to the right agents and coordinating their interactions so that the problem is tackled as efficiently as possible.

# Conclusions

We cannot keep on adding functions to personal computers. They serve as text editors, game consoles, televisions, home cinemas, radios, agendas, music players, gateway to the Internet, etc. Such general devices will never produce the same quality as specialized appliances. Our PCs are like ducks: they can swim, but not as well as fish; fly, but not as well as hawks; and walk, but not as well as cats. Rather than integrate so many functions in a single device, it seems preferable to entrust them to an ever expanding network of specialized devices that is kept coordinated through an ongoing process of self-organization. We have described a number of general requirements and approaches that may enable our artifacts to learn the most effective way of cooperation.

In our overall scenario, we have assumed that standard functions and interaction rules are preprogrammed by a global community to handle the most common, default situations, but that the system is moreover ready to extend its own capabilities, adapting to newly encountered tasks, situations, or devices. This ability to adapt should be already present in the interaction rules. The adaptation may be achieved through the self-organization of the system of agents, using recurrent, "game-like" interactions, in which the agents learn what messages mean and who they can trust to perform which task. Most of this can happen outside of, or in parallel with, their normal "work", using idle processing power to explore many different communication and collaboration configurations. Thus, we can imagine that our future, intelligent devices, like young animals or children, will learn to become more skilful by exploring, "playing games" with each other, and practising uncommon routines, so as to be prepared whenever the need for this kind of coordinated action appears.

# Acknowledgements

We thank Peter McBurney for useful comments. C. G. was supported in part by CONACyT of Mexico.