abstract: In between transportation services, trains are parked and maintained at shunting yards. The conflict-free routing of trains to and on these yards and the scheduling of service and maintenance tasks is known as the train unit shunting and service problem. Efficient use of the capacity of these yards is becoming increasingly important, because of increasing numbers of trains without proportional extensions of the yards. Efficiently scheduling maintenance activities is extremely challenging: currently only heuristics succeed in finding solutions to the integrated problem at all. Bounds are needed to determine the quality of these heuristics, and also to support investment decisions on increasing the yard capacity. For this, a complete algorithm for a possibly relaxed problem model is required. We analyze the potential of extending the model for multi-agent path finding to be used for such a relaxation.
author: Jesse Mulderij[^1]; Bob Huisman[^2]; Denise Tönissen[^3]; Koos van der Linden; Mathijs de Weerdt
bibliography: TUSP-Mendeley.bib; TUSSP.bib
date: email: `email@example.com`
title: Train Unit Shunting and Servicing: a Real-Life Application of Multi-Agent Path Finding

# Introduction

The multi-agent path finding (MAPF) problem is an abstraction of routing problems such as that of robots or automated guided vehicles through an environment whilst preventing collisions. The state-of-the-art of complete solvers such as Conflict-Based Search  and Branch-and-Cut-and-Price  has recently progressed to facilitate routing as many as 50 agents on various benchmark instances . Therefore, this seems to be the opportune moment to make this abstract model more realistic, and start solving real problems.

The application domain discussed in this paper is that of shunting and servicing trains, which considers all operations on trains outside the timetable. When at the end of the day trains have completed their service, they are parked on a nearby shunting yard, cleaned (sometimes also on the outside on a special washing track), repaired if necessary, and they receive regular checks and maintenance. After this, trains of the right type need to be ready at the right time to start executing their service in the timetable of the following day. For a small number of trains and a large shunting yard, finding feasible paths and schedules for the trains and their operations can be done by hand, but this becomes very difficult when a shunting yard is operating close to its capacity.

has coined the name Train Unit Shunting and Service (TUSS) problem to indicate this application domain. It is an extension to the Train Unit Shunting Problem (TUSP), that originally focused on parking and matching arriving train units to departures only. TUSP has been investigated by several researchers in railway engineering and operations research, who provided mixed-integer programming (MIP) formulations for parts of the problem , where typically the routing is considered separately. Later several authors added one or more maintenance related service tasks to be scheduled simultaneously. The full scope of the problem has been brought in by the ROADEF 2014 competition . This formulation is based on a real-life situation, and the results of that competition made clear that for practical application this problem could only be solved by heuristics . Since then these heuristics have been further developed by , , and . However, we cannot estimate the quality of the solutions found by these heuristics.

Only a complete algorithm can conclude that a given instance is infeasible and can provide a bound on the quality of heuristic methods. Moreover, such a complete algorithm is especially important for supporting decisions on investing in logistic capacity, since it can provide proof that capacity is inherently insufficient, and possibly even indicate the main bottlenecks. If such an algorithm is fast enough, it can also be used to support finding feasible solutions, for example by providing an optimistic heuristic in heuristic search, or by indicating combinations of choices as infeasible. Bounds on heuristics and advice on investment decisions can even be provided when abstract models are used, as long as such an abstraction is a relaxation and the algorithm is complete.

The main objective of this work is therefore to identify whether MAPF models (and algorithms) can be used or extended to provide a relaxation of the TUSS problem. The contributions of this paper include an introduction of TUSS from the viewpoint of MAPF, and an analysis of similarities and differences to arrive at a relaxation of TUSS based on MAPF.

The next section contains a detailed description of the TUSS problem including the context and argumentation for the importance of providing bounds. Then, in Section  a model of MAPF is given and the similarities and differences from TUSS are analyzed, with in particular attention for aspects of the MAPF model that invalidate its use as a relaxation of TUSS. Section  contains the consequential formal extensions of MAPF, including an analysis whether this addresses such a relaxation issue, or makes the model more precise, hopefully leading to better bounds. In a concluding section we reflect on these extensions and provide an outlook for next steps to be taken to let MAPF algorithms contribute to the practice of railway scheduling.

# Background

First we provide more context and details on TUSS as it occurs in practice. Then we highlight some relevant work from the literature.

## Conceptual model

This subsection describes the real-life problem, and those aspects that need to be incorporated in the formal models.

In a railway network the nodes represent major cities. For passenger transportation these hubs incorporate a station (or a few closely connected ones) and one or more yards. Many train services start and terminate at these hubs. One of the functions of a hub is to provide physical trains for these services, while terminating services cause a return flow. Then, at a yard trains are checked, cleaned, repaired, and parked. Functionally it resembles a repair shop with trains being the components to be delivered to the transportation process.

Figure shows the interaction between the transport and yard processes. The timetable and related train-type assignment specifies the required output flow of the shunting and servicing site (i.e., the 'repair shop') and the return flow. It prescribes which types of trains, and in which composition, must be delivered at a specific station platform at what time. Vice versa it specifies the train units that become available when train services terminate. Although in some countries the timetable specifies the physical-train identifier, more often it specifies only the train type. Consequently the return flow contains different physical trains every day, even when a time table is executed repeatedly.

The maintenance program for a train type specifies the tasks to be executed with respect to the condition. For example, the need for a check depends on mileage. Because different trains arrive at a hub every day, and a train does not require daily the same maintenance services, the tasks to be scheduled vary over time.

At a yard certain tracks allow specific tasks to be executed. This resembles a flexible flow shop , where a physical train is represented by a job and the associated maintenance tasks to be executed by the operations. (Note that in the literature on flexible flow shops often an operation is what is called a task in this paper.) A set of tracks on which an operation can be performed is the equivalent of a set of similar machines. However, due to the rail infrastructure the routing is much more restricted than in a general flexible flow shop, and the intermediate stocks are limited due to track lengths.

Informally speaking, the objective of the *Train Unit Shunting and Service Problem* is to find a feasible schedule for all activities required to take train units from a station platform to a yard, find conflict-free routes to tracks that are needed for the trains to get serviced and parked, and deliver the trains to the transportation process as specified, while respecting all kinds of resource constraints. In the following we briefly describe the sub-problems involved.

## Sub-problems to be addressed

First, each train to be delivered needs to be mapped to one of the trains arriving and present on the yard. This is called the *matching problem*. In general any train can be chosen for delivery if its type is compliant. However, sometimes a unique train is specified in order to direct it to a workshop. Second, a *resource-constrained scheduling problem* can be identified which represents the allocation of service resources. In addition to tasks that are known a priori, a schedule should include conditional tasks such as shunting between service tracks and staff that has to walk over the yard from one task to the other. Assigning a resource flow, that is the job sequence a resource unit follows, yields transportation tasks for trains and staff, which depend on spacial distances and site characteristics. Third, since staff needs to meet specific shift restrictions such as maximum number of subsequent hours driving a train, assigning a resource flow involves a special kind of *staff scheduling and rostering problem*. Fourth, a *parking problem* involves assigning a part of a track to each train for a certain time interval. Because a track can only host a relatively small number of trains, and each train type has its own length, this resembles a bin packing problem with additional constraints on the parking time (e.g., at platforms). Moreover, trains usually arrive and depart in some composition of two or more. To find feasible or optimal plans, *decoupling and coupling* becomes relevant. Finally, a *routing problem* is to be solved for trains to drive to and from the yard, while avoiding conflicts with regular passenger and cargo trains, and to visit the appropriate service and parking tracks as scheduled.

A typical approach is to solve these problems sequentially: for example, first do the matching for decoupled train units, then schedule the tasks and resources, allocate the parking places, find out where to couple train units, define the routing and finally assign personnel. This sequential approach has reached it limits, and is unable to solve today's hub problems due to the high traffic intensities. In particular, such sequential approaches cannot provide bounds.

## Bounds for the logistic hub capacity

Traditionally, railways produce timetables that are used for several months or sometimes even longer. Repeating the same solution for TUSS for every day (or week) is possible only by including sufficient slack in time and space, because, as described above, servicing requirements depend on the condition of individual trains. However, public transportation volumes keep on growing and unfortunately the rail infrastructure cannot be extended accordingly. Not only are typically the most important hubs situated in urban areas with often little space left to extend shunting yards, such extensions of the infrastructure determine to a high degree the total cost of railways, which should be reduced to let the industry be competitive in the future. If the planning of servicing and shunting can be done dynamically, such that only those activities are scheduled that are required based on up-to-date information about arriving trains and their condition, then the infrastructure can be used more efficiently. Furthermore, it is expected that such dynamic planning will also contribute to make the overall railway network more robust, because it increases its ability to compensate for disruptions. However, such an approach raises a new problem: How can the logistic capacity of a hub under dynamic planning be assessed to support strategic and tactical planning?

NS, the national railway operating company in the Netherlands, has developed a method to simulate dynamic hub logistics with on-line scheduling using local search. Although this has improved logistic analysis, the gap between the hub capacity of solutions found this way, and the upper bound found by optimization of a relaxed problem formulation remains significant. The currently used relaxed problem formulation covers all sub-problems mentioned before, except for the routing. It is expected that with routing included the relaxation gap will be reduced. The relaxed problem formulation should be complete, meaning that no feasible solution to the original problem is discarded. This also means that when the relaxed problem is infeasible, the original problem is infeasible as well. This is the main motivation for applying MAPF to TUSS.

## Related Work

To the best of our knowledge, provide the first attempt to capture a train unit shunting problem (TUSP) in a mathematical model. They split the problem in two sub-problems: the track assignment problem and the matching problem. They are able to solve both models, and use real-life instances of the NS. extend this formulation by not only looking at the assignments of train units to tracks and departures, but by also including the routing of the trains through the shunting yard. and provide an integrated approach for the matching and parking problem. They aimed to include all possible configurations of train units and shunting yards in their models. provides a comparison benchmark of multiple solution approaches for the TUSP, based on instances of the Danish State Railway and the NS. Besides using solution approaches from the literature they also introduce three new methods: a constraint programming formulation, a column generation approach, and a randomized greedy heuristic.

TUSS extends the TUSP by including service tasks, such as cleaning, regular checks, and maintenance. show that for practical applications the TUSS problem can currently only be solved by heuristics. provides a simulated annealing algorithm that finds solutions for the TUSS problem. Furthermore, combine machine learning with local search with the goal to provide bounds of the logistic hub capacity. However, solving a (large) number of instances does not provide a guarantee that always solutions for instances with similar properties can be found, nor can be this be used to indicate instances as infeasible. For this, a complete solver is required.

Methods that can solve TUSS problems at scale do not exist, but for multi-agent path finding (MAPF), complete solvers such as Conflict-Based Search (CBS)  and Branch-and-Cut-and-Price (BCP)  can deal with instances of relevant size (e.g., 50–100 agents). In both solvers, paths for individual agents are computed independently. When such paths imply a collision, CBS considers these collisions pair-wise and creates two branches, one for each agent receiving priority over the other. BCP uses a master LP that can select among an increasing number of shortest paths for the agents, and uses the dual of these collisions to define the cost of using that specific node in a next iteration of path finding.

MAPF with task allocation is an extension of MAPF where tasks have to be allocated to the agents. consider the assignment of (single) target task locations to agents before finding the routes. In the more general multi-agent pickup and delivery problem (MAPD) multiple tasks are assigned to agents and collision-free paths are planned for them to execute their tasks. In this recent work, first a task sequence for each agent is found by solving a special traveling salesman problem, and then paths are planned accordingly. This approach is complete if the problem is well-formed, i.e., if (a) the number of tasks is finite, (b) the parking location of each agent is different from all task/end points, and (c) there exists a path between any two task/end points which traverses no other task/end points. MAPD does not include the matching problem and the related individual constraints on arrival and departure times, nor the staff scheduling and (de)coupling sub-problems, but includes most of the other elements. In TUSS the tasks are maintenance and cleaning; although they come with the train, there is some form of allocation, because sometimes there are different locations where the tasks can be performed.

Although the summary of related work above is far from complete, in our literature search we have not encountered any method that can solve the TUSS problem optimally at scale, and relaxations of TUSS appear not to have been a topic of study on itself thus far.

# The Multi-Agent Path Finding Problem

It is important to be clear about the variant of MAPF that we assume to be given before we can propose (new) extensions. In this section, we provide a formal definition of the Multi-Agent Path Finding problem that is analogue to the one for example used by .

Let $G=(L,E)$ be a connected, undirected graph that consists of a set of nodes $L$ called "locations" and a set of edges $E\subset L\times L$. Let $V= L \times \mathbb{Z}_{+}$ be the *time-extended vertex set*, where a vertex $v \in V$ corresponds to a combination $v = (l,t)$ of a location $l\in L$ and a time $t\in \mathbb{Z}_{+}$. We denote $\hat{E} \subset V\times V$ to be the *time-extended edge set*, where an edge $e = (v_{1},v_{2}) = ((l_{1},t_{1}),(l_{2},t_{2})) \in \hat{E}$ if and only if $t_{2} = t_{1} + 1$ and either $(l_{1},l_{2})\in E$ or $l_{1}=l_{2}$. We define the *reverse edge* $e'$ of $e$ to be $e' = ((l_{2},t_1),(l_{1},t_2))$. We say that the directed graph $\hat{G} = (V,\hat{E})$ is the *space-time graph* corresponding to $G$.

Let us be given a set $A$ of which the elements $a\in A$ are called "agents". For each agent we are given a start- and goal location $s_{a}, g_{a}\in L$. A *path* $p$ of length $k \in \mathbb{Z}_{+}$ for agent $a$ is a vector of locations $(l_{0},l_{1},\cdots,l_{k-1})$ where $l_{0} = s_{a}$ and $l_{k-1} = g_{a}$ and $((l_t,t),(l_{t+1},t+1)) \in E_t$ for all $t\in\{0,1,\cdots,k-2\}$. For $t>k$, agent $a$ remains at its goal location. Let $c:\hat{E}\to \mathbb{R}_{+}$ be a cost function on the edges of $\hat{G}$. For now we let $c(e) = 1, \forall e\in \hat{E}$. The cost $c(p)$ of a path $p$ is the sum of the traversed edges' cost. A feasible solution to the MAPF problem is a set of paths, one for each agent such that each vertex $v \in V$ is visited at most once and each edge $e$ and its reverse $e'$ are visited at most once in total.

Common objectives are to minimize the sum of individual costs (SIC) of the paths that agents take, or to minimize the latest time an agent arrives at its goal location, i.e., the makespan. Here we opt for SIC minimization. Formally, the PSPACE-complete decision problem corresponding to the first of these objectives is described below.

The minimization variant of this problem is finding the minimum $k$ for which the question is still answered affirmatively. Unlike most MAPF definitions that assume a grid-world, we specifically use a graph representation as it allows for a more precise modeling of a shunting yard.

The MAPF problem has similarities with the routing sub-problem of the TUSS problem, but it can also solve part of the job scheduling sub-problem. Consider the following translation of (part of) an instance of TUSS to an instance of MAPF. The trains in the shunting yard can be seen as agents in the MAPF problem. The shunting yard can be modeled as a graph where each track is represented by a number of nodes. A natural choice for this number is to take the length of the track and divide it by the length of the shortest train, and rounding up the result. This allows for at least as much space for trains to move around on the graph is would be the case on the real shunting yard, making sure the formulation is a relaxation of the application. For service tasks that need to be performed at certain tracks, the goal is to find paths to these tracks for each of the trains that are not colliding (either on tracks or in resource allocation) during which the service tasks are completed.

In order to use MAPF as a relaxation, we plan to formulate a series of MAPF modifications that are all relaxations of the real-life version of the TUSS problem, but become progressively more restrictive, closing the gap between solutions to TUSS and bounds produced by the MAPF modifications. It is important to observe that the original MAFP formulation as given above is not a relaxation of TUSS. Therefore, it is important to first provide, as an essential starting point, a variant of MAPF that is a relaxation of TUSS, and then further refine this variant.

# Extensions

To guarantee this being a relaxation, it is essential that the first variant includes some basic model of the matching problem. In the MAPF problem each agent has its own unique goal location $\mathbf{g} \subset L$ in the graph, while in TUSS any physical train of the appropriate type can be assigned to a departure. A relaxed version of this problem is to assume that all train units are interchangeable, and that the problem thus defines that each goal needs obtained by (at least) one of the agents. Second, the direction of movement of trains is taken into account. Third, we propose to include a set of service tasks for each agent, resembling cleaning, repair and maintenance check-ups. Finally we mention several further possible extensions to reduce the gap between an optimal solution for the MAPF extension and feasible solutions to TUSS.

## Matching as a sub-problem

In practice there are multiple types of train units. At the end of the timetable trains, which are compositions of train units, enter the shunting yard while at the end of the maintenance period trains depart, possibly in different compositions, to enter service the following day. It does not matter which specific train units fill the slot in the departing train, as long as they are of the prescribed type. This property reminds us of robots that can perform specific tasks, which is modeled by the multi-commodity flow problem . For the moment, we presume all trains are decoupled, but address the extension of matching train units of certain types. Another difference we address with this extension is that of differing arrival and departure times of trains. In the classical MAPF problem, all agents start their routes at the same moment while trains leave and re-enter the transporting service at different moments throughout the day. The start- and goal nodes of an agent are therefore not only location dependent, but also time specific. We assume that for each train type, as many train units arrive as depart.

Every agent $a$ is assigned a type $o \in O$. The expected time $t \in \mathbb{Z}_+$ and location $l\in L$ of an agent's arrival is accurately represented by the vertex $(l,t)\in \hat{G}$. The start locations of agents are thus replaced with starting vertices in the time-extended graph. The goal locations are also replaced by vertices such that they carry a time specification. Goal vertices are also no longer attached to a specific agent. Instead, a set of goal vertices $g=\{g_{1}=(l_1,t_1,o_i), g_{2},\hdots,g_{n}\}$ is introduced, where $o_i$ is the specified train unit type that departs at that vertex. An agent of type $o$ can only depart from a goal vertex of the corresponding type. In the end, all departures have to be satisfied. Thus, a matching problem between the agents and the departures has to be solved implicitly or explicitly.

This extension is important, because without it agents have fixed goal locations. This would effectively limit the options and might exclude practically feasible solutions. The model would then no longer represent a relaxation of the TUSS problem. Ignoring the coupling and decoupling, and thus treating train units as agents does not reduce the number of options. The proposed extension is thus a relaxation of TUSS. Even the case where all agents are considered to be of the same type, all arrivals are at time 0, and all goals at the latest time in the input is still a relaxation, and could be a good start point.

## Direction of movement

A possible next step in refining the relaxation is keeping track of the direction of movement of an agent. This extension is added to make the model more precise, resulting in a better capacity bound. When a train moves in a given direction, it means that the train driver must be located on the front-end of the train. Therefore, a train cannot simply reverse direction. The train driver has to walk towards the other side of the train, this process takes some time. Ignoring this may lead to infeasible solutions because of missed deadlines, but does not remove potentially feasible solutions from the modeled search space.

We propose to model this restriction of a train's movement by adapting the graph. Every node where a train is allowed to reverse its direction is split in two nodes as illustrated in Figure . Nodes $v$ on which a train is not allowed to reverse direction are split in the same manner, except there is no edge between $v_l$ and $v_r$. For every pair of nodes $v_l$ and $v_r$ it is required that only one agent can occupy either node.

## Including service tasks

A further important extension of MAPF to model TUSS more accurately is the inclusion of service tasks. This is the most prevalent reason that trains are required to not only park, but also move over the yard throughout the maintenance period. This extension is thus aimed at improving the capacity bound.

During the maintenance period trains are checked, cleaned, repaired, and parked at the shunting yard. These tasks will be referred to as service tasks. Each train has a set of service tasks that it has to complete during the maintenance period. Each service task can be performed at several locations.

We propose to model the service tasks in the form of waypoints that agents should pass at some moment during their path. Each agent $a\in A$ is assigned a set of service tasks $S_a=\{s_1,\hdots,s_k\}$, where each service task $s_i$, for $i\in \{1,\hdots,k\}$, corresponds to a set of locations $L_{i}\subset L$ at which the service task can be completed. An agent that is being serviced has to remain at the service location for the duration $\tau_i$ of the service task. Therefore, a service task $s_i\in S_a, i\in \{1,\hdots,k\}$ is said to be completed when the path $p\in P_a$ of agent $a$ includes the same location $l\in L_i$ for $\tau_i$ consecutive time steps. A path $p\in P_a$ is only considered feasible when all maintenance tasks of agent $a$ are completed. By defining the service tasks in this manner we ensure two properties: 1) the order of the service tasks is not of pre-defined, and 2) agents can move over unoccupied nodes that are also service locations just like any other nodes.

## Other extensions and open questions

Next we discuss several more practical extensions we believe to be of less importance.

The first of such extensions is that some locations in the shunting yard are not eligible for parking, such as track changes and platforms. Trains can only remain on such locations for several time steps at most. A strict way in which this can be modeled is to remove all wait edges for the corresponding vertices. This might however be too strict, such that the model is no longer a relaxation of the real-life TUSS application.

For the moment we also ignore the composition of train units into trains and the related coupling and decoupling activities. Train units (just as agents) are atomic. Maintenance tasks are assigned to train units. Trains are not atomic. Arriving events and departure goals are defined for trains. It is still an open question what an effective extension of MAPF is to represent this. Without this, however, the current model is a relaxation.

A basic way in which tracks in the shunting yard can be represented by a graph is to divide every track into multiple nodes. A natural choice would be to divide the track length by the length of the shortest train type and round up to obtain the number of nodes that together represent that track. This method ensures there is at least as much space for trains to move around in the model as there is on the represented shunting yard, resulting in a relaxation. To obtain a more accurate representation, which is more strict, the train length can be paired with train types. This assigns a length to every train unit. A track can be modelled by one or more nodes, where a constraint needs to ensure that the sum of the lengths of the trains on a track is not more than the length of the nodes. Additionally, such a representation should then accurately reflect the train order on the track to guarantee that only trains at the beginning or the end of the track can leave the track, and also only in that direction.

An extension that is very much part of the real-life problem is the planning of personnel. Personnel is required in the form of train drivers, maintenance crews and cleaners. Employees of each category are limited in their number, ability and working hours. Operational costs are very high if all trains have their own driver for the duration of the entire maintenance period and the maintenance period extends for longer than one shift. Also, employees can only perform certain tasks. In all likelihood, the train drivers do not have the know-how required for a technical maintenance check of the train, the maintenance engineers are unaware of how to properly clean the train, and the cleaners cannot drive the train around the shunting yard. Moreover, when crew has to move from one service track to another, or from one train to another, walking times have to be taken into account. If indeed there are known limitations on the available personnel, then scheduling personnel is a very important extension. However, when the main objective of the model is to provide a bound on the capacity of the (very expensive) yard infrastructure, this extension is not necessarily a relaxation.

Lastly, more detail could be added in terms of physical correctness. The speed of trains can vary during movement. Because trains have such a high mass, changing the momentum by accelerating or breaking can take a significant time. There are also safety measures that limit the speed of trains and install a minimum time between two trains crossing the same track change. The way in which these extensions are implemented should conserve the relaxation property of the model.

# Conclusion and Future work

The real-life application of the Train Unit Shunting and Servicing problem contains many practical details that are hard to account for in an abstract model. We aim to find an upper bound on the servicing capacity of a shunting yard and it is therefore important to extend the model in a manner that such a bound can be guaranteed.

The main conclusion from the presented analysis is that it is essential to first extend MAPF with some representation of the matching problem. This, in its simplest form has strong similarities to the task allocation and planning MAPF models discussed in related work . If done properly, this extension provides a relaxation of TUSS.

This paper described further refinements to obtain better bounds: time-specific arrival and departure times can be modeled such that agents no longer start moving towards their goals at the same time, but instead enter the shunting yard at the arrival time and leave shortly before their departure from one of the platforms. Keeping track of the direction of movement of agents can be achieved by splitting each node into two nodes. Each new node then resembles one direction of movement. The capacity of the new nodes is in total still only enough to harbour one agent at a time. The addition of service tasks in the form of waypoints resembles the need for trains to be checked, cleaned and repaired at pre-defined locations on the shunting yard. Each agent has their own set of service tasks to complete, and each service task can be completed at a given set of locations. The service tasks also take a specified amount of time to complete, in which the agent has to remain on the service location. We provide arguments why each of the resulting variants is a relaxation of TUSS. By formulating these extensions precisely, we aim to interest researchers in this real-life problem and leverage existing work on MAPF to solve TUSS.

Keeping the goal of relaxation in mind, the extensions proposed in Subsection  should be further formalized. Apart from these extensions, we propose several other interesting directions for future research.

A similar exercise as reported in this paper could be done for the use of finding feasible solutions to the TUSS problem. This would require the model variants to be restrictions rather than relaxations. The big advantage is that any feasible solution to a restriction can be translated into a feasible solution of the original problem, so a complete or optimal algorithm is not necessary.

Another future research direction is to introduce disturbances into the problem. For the TUSS the exploration of so-called robust schedules seeks to construct the maintenance schedule such that disturbances in arrival times, train or track defects and personnel running late, can be absorbed by making no or only a very small change to the schedule. Although, there are no papers that provide robust schedules for the TUSS problem, several robustness measures to measure the robustness of TUSS schedules are proposed by .

Finally, we have considered MAPF and its extensions mainly as a feasibility problem. In practice, costs emerge mostly from the personnel that is required to fulfill all the scheduled tasks. It would therefore be beneficial to minimize the number of simultaneous actions, for example with regards to train movements.

[^1]: Delft University of Technology

[^2]: NS, Utrecht, Netherlands

[^3]: Vrije Universiteit Amsterdam


abstract: The Murchison Widefield Array (MWA) is a Square Kilometre Array (SKA) Precursor. The telescope is located at the Murchison Radio–astronomy Observatory (MRO) in Western Australia (WA). The MWA consists of 4096 dipoles arranged into 128 dual polarisation aperture arrays forming a connected element interferometer that cross-correlates signals from all 256 inputs. A hybrid approach to the correlation task is employed, with some processing stages being performed by bespoke hardware, based on Field Programmable Gate Arrays (FPGAs), and others by Graphics Processing Units (GPUs) housed in general purpose rack mounted servers. The correlation capability required is approximately 8 TFLOPS (Tera FLoating point Operations Per Second). The MWA has commenced operations and the correlator is generating 8.3 TB/day of correlation products, that are subsequently transferred 700 km from the MRO to Perth (WA) in real-time for storage and offline processing. In this paper we outline the correlator design, signal path, and processing elements and present the data format for the internal and external interfaces.
author: S. M. Ord$^{4}$$^,$$^{20}$, B. Crosse$^{4}$, D. Emrich$^{4}$, D. Pallot$^{4}$, R. B. Wayth$^{4}$$^,$$^{20}$, M. A. Clark$^{23}$$^,$$^{5}$$^,$$^{22}$, S. E. Tremblay$^{4}$$^,$$^{20}$, W. Arcus$^{4}$, D. Barnes$^{25}$, M. Bell$^{14}$, G. Bernardi$^{21}$$^,$$^{24}$$^,$$^{5}$, N. D. R. Bhat,$^{4}$, J. D. Bowman$^{1}$, F. Briggs$^{2}$, J. D. Bunton$^{3}$, R. J. Cappallo$^{6}$, B. E. Corey$^{6}$, A. A. Deshpande$^{9}$, L. deSouza$^{3}$$^,$$^{14}$, A. Ewell-Wice$^{7}$, L. Feng$^{7}$, R. Goeke$^{7}$, L. J. Greenhill$^{5}$, B. J. Hazelton$^{16}$, D. Herne$^{4}$, J. N. Hewitt$^{7}$, L. Hindson$^{19}$, H. Hurley-Walker$^{4}$, D. Jacobs$^{1}$, M. Johnston-Hollitt$^{19}$, D. L. Kaplan$^{18}$, J. C. Kasper$^{13}$$^,$$^{5}$, B. B. Kincaid$^{6}$, R. Koenig$^{3}$, E. Kratzenberg$^{6}$, N. Kudryavtseva$^{4}$, E. Lenc,$^{14}$$^,$$^{20}$ C. J. Lonsdale$^{6}$, M. J. Lynch$^{4}$, B. McKinley$^{2}$, S. R. McWhirter$^{6}$, D. A. Mitchell$^{3}$$^,$$^{20}$, M. F. Morales$^{16}$, E. Morgan$^{7}$, D. Oberoi$^{11}$, A. Offringa,$^{2}$$^,$$^{20}$, J. Pathikulangara$^{3}$, B. Pindor$^{12}$, T. Prabu$^{9}$, P. Procopio$^{12}$, R. A. Remillard$^{7}$, J. Riding$^{12}$, A. E. E. Rogers$^{6}$, A. Roshi$^{8}$, J. E. Salah$^{6}$, R. J. Sault$^{12}$, N. Udaya Shankar$^{9}$, K. S. Srivani$^{9}$, J. Stevens$^{3}$, R. Subrahmanyan$^{9}$$^,$$^{20}$, S. J. Tingay$^{4}$$^,$$^{20}$, M. Waterson$^{4}$$^,$$^{2}$, R. L. Webster$^{12}$$^,$$^{20}$, A. R. Whitney$^{6}$, A. Williams$^{4}$, C. L. Williams$^{7}$, J. S. B. Wyithe$^{12}$$^,$$^{20}$

$^{1}$Arizona State University, Tempe, USA
$^{2}$The Australian National University, Canberra, Australia
$^{3}$CSIRO Astronomy and Space Science, Australia
$^{4}$International Centre for Radio Astronomy Research (ICRAR), Curtin University, Perth, Australia
$^{5}$Harvard-Smithsonian Center for Astrophysics, Cambridge, USA
$^{6}$MIT Haystack Observatory, Westford, MA, USA
$^{7}$MIT Kavli Institute for Astrophysics and Space Research, Cambridge, USA
$^{8}$National Radio Astronomy Observatory, Charlottesville, USA
$^{9}$Raman Research Institute, Bangalore, India
$^{10}$Swinburne University of Technology, Melbourne, Australia
$^{11}$National Center for Radio Astrophysics, Pune, India
$^{12}$The University of Melbourne, Melbourne, Australia
$^{13}$University of Michigan, Ann Arbor, USA
$^{14}$University of Sydney, Sydney, Australia
$^{15}$University of Tasmania, Hobart, Australia
$^{16}$University of Washington, Seattle, USA
$^{17}$University of Western Australia, Perth, Australia
$^{18}$University of Wisconsin–Milwaukee, Milwaukee, USA
$^{19}$Victoria University of Wellington, New Zealand
$^{20}$ARC Centre of Excellence for All-sky Astrophysics (CAASTRO)
$^{21}$Square Kilometre Array South Africa (SKA SA), Cape Town, South Africa
$^{22}$California Institute of Technology, California, USA
$^{23}$NVIDIA, Santa Clara, California, USA
$^{24}$Department of Physics and Electronics, Rhodes University, Grahamstown, South Africa
$^{25}$Monash University, Melbourne, Australia
bibliography: journals.bib
nocite: ;
title: The Murchison Widefield Array Correlator

# Introduction

The MWA is a 128 element dual polarisation interferometer, each element is a 4x4 array of analog beam formed dipole antennas. The antennas of each array are arranged in a regular grid approximately 1m apart, and these small aperture arrays are known as tiles. The science goals that have driven the MWA design and development process are discussed in the instrument description papers, and the MWA science paper . These are (1) the detection of redshifted 21cm neutral hydrogen from the Epoch of Re-ionization (EoR); (2) Galactic and extra-Galactic surveys; (3) time-domain astrophysics; (4) solar, heliospheric and ionospheric science and space weather.

## Specific MWA Correlator Requirements

The requirements and science goals have driven the MWA into a compact configuration of 128 dual-polarisation tiles. 50 tiles are concentrated in the 100-m diameter core, with 62 tiles distributed within 750-m and the remaining 16 distributed up to 1.5-km from the core.

The combination of the low operating frequency of the MWA and its compact configuration allow the correlator to be greatly reduced in complexity, however this trade-off does drive the correlator specifications. Traditional correlators are required to compensate for the changing geometry of the array with respect to the source in order to permit coherent integration of the correlated products. In the case of the MWA, no such corrections are performed. This drives the temporal resolution specifications of the correlator and forces the products to be rapidly generated in order to maintain coherence. list the system parameters and these include a temporal resolution of 0.5 s.

The temporal decoherence introduced by integrating for 0.5 s without correcting for the changing array geometry; and the requirement to image a large fraction of the primary beam, drive the system channel resolution specification to 10 kHz. It should be noted that this does not drive correlator complexity as it is total processed bandwidth that is the performance driver, not the number of channels. Albeit the number of output channels does drive the data storage and archiving specifications. The MWA correlator is required to process the full bandwidth as presented by the current MWA digital receivers, which is 30.72 MHz per polarisation.

In summary, in order to meet the MWA science requirements, the correlator is required to correlate 128 dual polarisation inputs, each input is 3072x10kHz in bandwidth. The correlator must present products, integrated for no more than 0.5 seconds at this native channel resolution, to the data archive for storage.

## Benefits of Software Correlator Implementations

The correlation task has previously been addressed by Application Specific Integrated Circuits (ASICs) and Field Programmable Gate Arrays (FPGAs). However the current generation of low frequency arrays including the MWA , LOFAR , PAPER and LEDA have chosen to utilise, to varying degrees, general purpose computing assets to perform this operation. The MWA leverages two technologies to perform the correlation task, a Fourier transformation performed by a purpose built FPGA-based solution, and a cross-multiply and accumulate task (XMAC) utilising the xGPU library[^1] described by , also used by the PAPER telescope and the LEDA project. The MWA system was deployed over 2012/13 as described by , and is now operational.

The MWA is unique amongst the recent low frequency imaging dipole arrays, in that it was not designed to utilise a software correlator at the outset. The flexibility provided by the application of general purpose, GPU based, software solution has allowed a correlator to be rapidly developed and fielded. This was required in-order to respond to a changed funding environment, which resulted in a significant design evolution from that initially proposed by to that described by . Expeditious correlation development and deployment was possible because General Purpose GPU (GPGPU) computing provides a compute capability, comparable to that available from the largest FPGAs, in a form that is much more accessible to software developers. It is true that GPU solutions are more power intensive than FPGA or ASIC solutions, but their compute capability is already packaged in a form factor that is readily available and the development cycle is identical to other software projects. Furthermore the GPU processor lifecycle is very fast and it is also generally very simple to benefit from architecture improvements. For example, the GPU kernel used as the cross-multiply engine in the MWA correlator will run on any GPU released after 2010. We could directly swap out the GPU in the current MWA cluster, replace them with cards from the Kepler series (K20X), and realise a factor of 2.5 increase in performance (and a threefold improvement in power efficiency).

The organisation of this paper begins with a short introduction to the correlation problem, followed by an outline of the MWA correlator design and then a description of the sub-elements of the correlator following the signal path; we finally discuss the relevance of this correlator design to the Square Kilometre Array and there are several Appendices describing the various internal and external interfaces.

# The Correlation Problem

A traditional telescope has a filled aperture, where a surface or a lens is used to focus incoming radiation to a focal point. In contrast, in an interferometric array like the MWA the purpose of a correlator is to measure the level of signal correlation between all antenna pairs at different frequencies across the observing band. These products can then be added coherently, and phased, or focussed, to obtain a measurement of the sky brightness distribution in any direction

The result of the correlation operation is commonly called a *visibility* and is a representation of the measured signal power (brightness) from the sky on angular scales commensurate with the distance between the constituent pair of antennas. Visibilities generated between antennas that are relatively far apart measure power on smaller angular scales and vice versa. When all visibilities are calculated from all pairs of antennas in the array many spatial scales are sampled (see for an extensive review of interferometry and synthesis imaging). When formed as a function of observing frequency ($\nu$), this visibility set forms the *cross-power spectrum*. For any two voltage time-series from any two antennas $V_{1}$ and $V_{2}$ this product can be formed in two ways. First the cross correlation as a function of lag, $\tau$, can be found, typically by using a delay line and multipliers to form the lag correlation between the time-series. $$(V_{1} \star V_{2})(\tau) = \int_{-\infty}^{\infty} V_{1}(t)V_{2}(t-\tau) dt.
\label{eqn:xcorr}$$ The cross power spectrum, $S(\nu)$, is then obtained by application of a Fourier transform: $$S_{12}(\nu) = \int_{-\infty}^{\infty} (V_{1} \star V_{2})(\tau)e^{-2\pi i \nu\tau} d\tau.
\label{eqn:XF}$$ When the tasks required to form the cross power spectrum are performed in this order (lag cross-correlation, followed by Fourier transform) the combined operation is considered an *XF* correlator. However the cross correlation analogue of the convolution theorem allows Equation to be written as the product of the Fourier transform of the voltage time series from each antenna: $$S_{12}(\nu) = \int_{-\infty}^{\infty} V_{1}(t) e^{-2\pi i \nu t} dt  \times \int_{-\infty}^{\infty} V_{2}(t) e^{-2\pi i \nu t} dt.
\label{eqn:FX}$$ Implemented as described by Equation the operation is an *FX* correlator. For large-N telescopes the FX correlator has a large computational advantage. In an XF correlator for an array of $N$ inputs the cross correlation for all baselines requires $\mathcal{O}(N^{2})$ operations for every lag, and there is a one to one correspondence between lags and output channels, F, resulting in $\mathcal{O}(FN^{2})$ operations to generate the full set of lags. The Fourier transform requires a further $\mathcal{O}(F\log_{2}F)$ operations, but this can be performed after averaging the lag spectrum and is therefore inconsequential. For the FX correlator we require $\mathcal{O}(NF\log_{2}F)$ operations for the Fourier transform of all input data streams, but only $\mathcal{O}(N^{2})$ operations per sample for the cross multiply (although we have F channels the sample rate is now lower by the same factor). Therefore as long as $N$ is greater than $\log_{2}F$ there is a computational advantage in implementing an FX correlator. XF correlators have been historically favoured by the astronomy community, at least in real-time applications, as until very recently $N$ has been small, and there are disadvantages to the FX implementation. The predominant disadvantage is data growth: the precision of the output from the Fourier transform is generally larger than the input, resulting in a data rate increase. There is also the complexity of implementing the Fourier transform in real-time.

# The MWA Correlator System

The tasks detailed in this section cover the full signal path after digitisation, including fine channelisation, data distribution, correlation and output. In the MWA correlator the F stage is performed by a dedicated channeliser, subsets of frequency channels from all antennas are then distributed to a cluster of processing nodes via an ethernet switch. The correlation products are then assembled and distributed to an archiving system. An outline of the system as a whole is shown in Figures and . As shown in Figure the correlator is conceptually composed of 4 sub-packages, the Polyphase Filterbank (PFB), that performs the fine channelisation, the Voltage Capture System (VCS), responsible for converting the data transport protocol into ethernet and distributing the data, the correlator itself (XMAC), and the output buffering that provides the final interface to the archive. Figure presents the physical layout of the system.

## The System Level View

The computer system consists of 40 servers, grouped into different tasks and connected by a 10GbE data network and several 1GbE monitoring, command and control networks. The correlator computer system has been designed for ease of maintenance and reliability. All of the servers are configured as *thin-clients* and have no physical hard-disk storage that is utilised for any critical functions - any hard-disk is used for non-critical storage and could be removed (or fail) without limiting correlator operation. All of the servers receive their Operating System through a process known as the PXE (Pre-eXecution Environment) boot process, and import all of their software over NFS (Network File System) and mount it locally in memory. The servers are grouped by task into the VCS (voltage capture system) machines that house the FPGA capture card and the servers that actually perform the cross-multiply-accumulate (XMAC) and output the visibility sets for archiving. Any machine can be replaced without any more effort than is required to physically connect the server box and update an entry in the IP address mapping tables.

## The MWA Data Receivers: Input to the Correlator

As described in there are 16 digital receivers deployed across the MWA site. Power and fibre is reticulated to them as described by . The individual tiles are connected via coaxial cable to the receivers, the receivers are connected via fibre runs of varying lengths to the central processing building of the MRO, where the rest of the signal processing is located.

### The Polyphase Filterbank

Polyphase Filter Banks (PFBs) are commonly used in digital signal communications as a realisation of a Discrete Fourier Transform (DFT; see for a detailed explanation). It specifically refers to the formation of a DFT via a method of *decimating* input samples into a polyphase filter structure and forming the DFT via the application of a Fast Fourier Transform to the output of the polyphase filter .

As described by the first or *coarse* PFB is as an 8 tap, 512 point, critically sampled, polyphase filter bank. This is implemented as; an input filtering stage realised by 8x512 point Kaiser windowing function, the output of which is subsampled into a 512 point FFT. This process transforms the 327.68 MHz input bandwidth into 256, 1.28  MHz wide, subbands.

The second stage of the F is performed by four dedicated FPGA based polyphase filter bank boards housed in two ATCA (Advanced Telecommunications Computing Architecture) racks, which implement a 12 tap, 128 point filter response weighted DFT. Because the first stage is critically sampled the fine channels that are formed at the boundary of the coarse channels are corrupted by aliasing. These channels are processed by the pipeline, but removed during post-processing, the fraction of the band excised due to aliasing is approximately 12%. The second stage is also critically sampled so there is a similar degree of aliasing present in each 10kHz wide channel. The aliased signal correlates, and the phase of the correlation does not change significantly across the narrow channel, any further loss in sensitivity is negligible.

Development of the PFB boards was initially funded through the University of Sydney and the firmware initially developed by CSIRO. The boards were designed to form part of the original MWA correlator which would share technology with the Square Kilometre Array Molongolo Prototype (SKAMP) . Subsequent modifications to the firmware were undertaken at MIT-Haystack and the final boards, with firmware specific to the MWA, were first deployed as part of the 32-tile MWA prototype.

### Input Format, Skew and Packet Alignment

The PFB board input data format is the Xilinx serial protocol RocketIO, although there have been some customisations at a low level, made within the RocketIO CUSTOM scheme. The data can be read by any multi-gigabit-transceiver (MGT) that can use this protocol, which in practice is restricted to the Xilinx FPGA (Vertex 5 and newer). The packet format as presented to the PFB board for the second stage channeliser is detailed in .

A single PFB board processes 12 fibres that have come from 4 different receivers. The receivers are distributed over the 3 km diameter site and there exists the possibility of there being considerable difference in packet arrival time between inputs that are connected to different receivers. The PFB boards have an input buffer that is used to align all receiver inputs on packet number 0 (which is the 1 second tick marker see Appendix ). The buffer is of limited length ($\pm 8$ time-samples, corresponding to several hundred metres of fibre) and if the relative delay between input lines from nearby and distant receivers is greater than this buffer length, the time delay between receivers will be undetermined. We have consolidated the receivers into 4 groups with comparable distances to the central processing building and constrained these groups to have fibre runs of 1380, 925, 515, and 270 metres. Each of the 4 groups is allocated to a single PFB (see Figure ). This has resulted in more fibre being deployed than was strictly necessary, but has guaranteed that all inputs to a PFB board will arrive within a packet-time. The limited buffer space available can therefore be utilised to deal with variable delays induced by environmental factors and will guarantee that all PFB inputs will be aligned when presented to the correlator.

This packet alignment aligns all inputs onto the same 1 second mark but does not compensate for the *outward* clock signal delay, which traverses the same cable length, and produces a large cable delay between the receiver groups, commensurate with the differing fibre lengths. The largest cable length difference is 1110 metres, being the difference in length between the shortest and longest fibres and is removed during calibration.

### PFB Output Format

As the MWA PFB boards were originally purposed as the hardware F-stage of an FPGA based FX correlator, the output of the PFB was never intended to be processed independently. The output format is also the Xilinx serial protocol RocketIO, and the PFB presents eight output data lanes on two CX4 connectors. These connectors generally house four-lane XAUI as used by Inifiniband and 10Gb Ethernet, but these are simply used as a physical layer. In actuality all of the pins on the CX4 connectors are driven as transmitters, as opposed to receive and transmit pairs, therefore only half of the available pins in the CX4 connectors were being used. We were subsequently able to change the PFB firmware to force all output data onto connector pins that are traditionally transmit pins, and leave what would normally be considered the receive pins unused. We then custom designed breakout cables that split the CX4 into 4 single lane SFP connectors (see Appendix ).

## The Voltage Capture System

In order to capture data generated by the PFBs with off-the-shelf hardware, the custom built SFP connectors were then plugged into an off-the-shelf Xilinx based RocketIO capture cards that are housed within linux servers. Each card can capture 2 lanes from the PFB cards, therefore 4 cards were required per PFB, and 16 cards required in total. A set of 16 machines are dedicated to this voltage capture system (VCS), these are 16 CISCO UCS C240 M3 servers. They each house two Intel Xeon E5-2650 processors, 128 GB of RAM, and 2x1TB RAID5 disk arrays. These machines also mount the Xilinx-based FPGA board, supplied by Engineering Design Team Incorporated (EDT) that is used to capture the output from the PFB. All of the initial buffering and packet synchronisation is performed within these machines.

### Raw Packet Capture

The data capture and distribution is enabled by a succession of software packages outlined in Figure . First, the EDT supplied capture card transfers a raw packet stream from device memory to CPU host memory. The transfer is mediated by the device driver and immediately copied to a larger shared memory buffer. At this stage the data are checked for integrity and aligned on packet boundaries to ensure efficient routing without extensive re-examination of the packet contents. Each of the 16 MWA receivers in the field obtain time synchronisation via a distributed 1 pulse per second (PPS) that is placed into the data stream in the "second tick" field as described in Appendix . We maintain synchronisation with this "tick" and ensure all subsequent processing within the correlator labels the UTC second correctly. As each machine operates independently the success or failure of this synchronisation method can only be detected when packets from all the servers arrive for correlation, at which point the system is automatically resynchronised if an error is detected. No unsynchronised data are correlated.

### The Demultiplex

Data distribution in a connected element interferometer is governed by two considerations; all the inputs for the same frequency channels for each time step must be presented to the correlator; and the correlator must be able to keep up with the data rate.

As there are four PFB boards (see Figure ), each processing one quarter of the array, each output packet contains the antenna inputs to that PFB, for a subset of the channels for a single time step. Individual data streams from each antenna in time order need to be *cross-connected* to satisfy the requirement that a single time step for *all antennas* is presented simultaneously for cross correlation. To achieve this we utilise the fact that each PFB packet is uniquely identified by the contents of its header (see Appendix ). We route each packet (actually each block of 2000 packets) to an endpoint based upon the contents of this header, each VCS server sends 128 contiguous, 10kHz channels to the same physical endpoint, from all of the antennas in its allocation. This results in the same 128 frequency channels, from all antennas in the array, arriving at the same physical endpoint. This is repeated for 24 different endpoints – each receiving a different 128 channels. These endpoints are the next group of servers, the cross multiply and accumulate (XMAC) machines.

This cross-connect is facilitated by a 10 Gb ethernet switch. In actuality this permits the channels to be aggregated on any number of XMAC servers. We use 24 as it suits the frequency topology of the MWA. Regardless of topology some level of distribution is required to reduce the load on an individual XMAC server in a flexible manner, as compute limitations govern how many frequency channels can be simultaneously processed in the XMAC stage. demonstrates that the XMAC engine performance is limited to processing approximately  4096 channels per NVIDIA Fermi GPU at the 32 antenna level and  1024 channels at the 128 antenna level. This is well within the MWA computational limitations as we are required to process 3072 channels, and have up to 48 GPU available. However, the benefit of packet switching correlators in general is clear: if performance were an issue we could simply add more GPU onto the switch to reduce the load per GPU without adding complexity. Conversely, as GPU performance improves we can reduce the number of servers and still perform the same task.

Each of the VCS servers is required to maintain 48 concurrent TCP/IP connections, two to each endpoint, because each VCS server captures two lanes from a PFB board . This amounts to 48 open sockets that are being written to sequentially. As there are many more connections than available CPU cores individual threads, tasked to manage each connection, the operation is subject to the linux thread scheduler, which attempts to distribute CPU resources in a *round robin* manner. Each thread on each transmit line utilises a small ring buffer to allow continuous operation despite the inevitable device contention on the single 10 Gb ethernet line. Any such contention causes the threads without access to the interface to block. This time spent in this *wait condition* allows the scheduler to redistribute resources. We also employ some context-based thread waiting on the receiving side of the sockets to help the scheduler in its decision making and to even the load across the data capture threads. We have found that this scheme results in a thread scheduling pattern that is remarkably fair and equitable in the allocation of CPU resources and provided that a reasonably-sized ring buffer is maintained for each data line, all the data are transmitted without loss.

### VCS Data Output

To summarise, we are switching packets as they are received and they are routed based upon the contents of their header. Therefore any VCS server can process any PFB lane output. Each server holds a single PCI-based capture card, and captures two PFB data lanes. The data from each lane is grouped in time contiguous blocks of 2000 packets (50 milliseconds worth of samples for 16 adjacent 10 kHz frequency channels), from all 32 of the antennas connected to that PFB. The packet header provides sufficient information to uniquely identify the PFB, channel group, and time block it is, and the packet block is routed on that basis. Further information is given in Appendix . A static routing table is used to ensure that each XMAC server receives a contiguous block of frequency channels, the precise number of which is flexible and determined by the routing table, but it is not alterable at runtime. [^2]

### Monitoring, Command, Control and the Watchdog

Monitor and control functionality within the correlator is mediated by a watchdog process that runs independently on each server. The watchdog launches each process, monitors activity, and checks error conditions. It can restart the system if synchronisation is lost and mediates all start/stop/idle functionality as dictated by the observing schedule. All interprocess command and control, for example propagation of HALT instructions to child processes and time synchronisation information, is maintained in a small block of shared memory that holds a set of key-value pairs in plain text that can be interrogated or set by third-party tools to control the data capture and check status.

### Voltage Recording

The VCS system also has the capability to record the complete output of the PFB to local disk. The properties and capabilities of the VCS system will be detailed in a companion paper .

## The Cross-Multiply and Accumulate (XMAC)

The GPU application for performing the cross-multiply and accumulate is running on 24 NVIDIA M2070 Fermi GPU housed in 24 IBM iDataplex servers. These machines are housed in racks adjacent to the VCS servers and connected via a 10Gb ethernet network. The package diagram detailing this aspect of the correlator is presented in Figure and the packet interface and data format is described in Appendix . The cross-multiply servers receive data from the 32 open sockets and internally align and unpack the data into a form suitable for the GPU. The GPU kernel is launched to process the data allocation and internally integrates over a user defined length of time. There are other operations possible, such as incoherent beam forming and the raw dump of data products (or input voltages). These modes will be expanded upon during the development of other processing pipelines and will be detailed in companion papers when the modes are available to the community.

### Input Data Reception, and Management

A hierarchical ring buffer arrangement is employed to manage the data flow. Each TCP connection is managed by a thread that receives a 2000 packet block and places it in a shared memory buffer. Exactly where in a shared memory block each of the 2000 packets is placed is a function of the time/frequency/antenna block that it represents. A small *corner-turn*, or transpose, is required as the PFB mixes the order of time and frequency. The final step in this process is the promotion of the sample from 4 to 8 bits, which is required to enable a machine instruction that performs rapid promotion from 8 to 32 bits within the GPU. This promotion is not performed if the data are just being output to local disk after reordering.

Each of the 32 threads is filling a shared memory block that represents 1 second of GPU input data, and a management thread is periodically checking this block to determine whether it is full. Safeguards are in place that prevent a thread getting ahead of its colleagues or overwriting a block. The threads wait on a condition variable when they have finished their allocation, so if a data line is not getting sufficient CPU resources it will soon be the only thread *not* waiting and is guaranteed to complete. This mechanism helps the Linux thread scheduler to divide the resources equitably. The use of TCP allows the receiver to actually prioritise which threads on the send side of the connection are blocked to facilitate an even distribution of CPU resources.

Once the management thread judges the GPU input block to be full, it releases it, launches the GPU kernel and frees all the waiting threads to fill the next buffer while the GPU is running.

### xGPU - Correlation on a GPU

It is possible to address the resources of a GPU in a number of ways, utilising different application programming interfaces (API), such as OpenCL, CUDA, and openGL. The MWA correlator uses the xGPU library as is described in detail in . The xGPU library is a CUDA application which is specific to GPUs built by the NVIDIA corporation. There are many references in the literature to the CUDA programming model and examples of its use .

The performance improvement seen when porting an application to a GPU is in general due to the large aggregate FLOPS and memory bandwidth rates, but to realise this performance requires effective use of the large number of concurrent threads of execution that can be supported by the architecture. This massively parallel architecture is permitted by the large number of processing cores on modern GPUs. The TESLA M2070 are examples of NVIDIA Fermi architecture and have 448 cores, grouped into 14 *streaming multiprocessors* or SM, each with 32 cores. The allocation of resources follows the following model: threads are grouped into a *thread block* and a thread block is assigned to an SM; there can be more thread blocks than SM, but only one block will be executing at a time on any SM; the threads within each block are then divided into groups of 32, (one for each core of the associated SM), and this subdivision called a *warp*; execution is serialised within a warp with the same instruction being performed by all 32 threads, but on different data elements in the *single instruction multiple data* paradigm.

GPGPU application performance is often limited by memory access bandwidth. A good predictor of algorithm performance is therefore *arithmetic intensity*, or the number of floating point operations per byte transferred. A complex multiply and add for two dual polarisation antennas at 32 bit precision requires 32 bytes of input, 32 FLOPS (16 multiply-adds), and 64 bytes of output. The arithmetic intensity of this is 32/96=0.33. Compare this to the Fermi C2070 architecture which has a peak single-precision performance of 1030 GFLOPS and memory bandwidth of 144 GB/s; the ratio of which (7.2) tells us that the performance of the algorithm will be completely dominated by memory traffic. The high performance kernel developed by Clark et al. (2011) increases the arithmetic intensity in two ways. Firstly the output memory traffic can be reduced by integrating the products for a time, I, at the register level. Secondly, instead of considering a single baseline, groups of ($m \times n$) baselines denoted tiles (see Figure 6), are constructed cooperatively by a block of threads. In order to fill an $m \times n$ region of the final correlation matrix, only two vectors of antennas need be transferred from host memory. A thread block loads two vectors of antenna samples (of length $n$ and $m$) and forms $nm$ baselines. These two steps then alter the arithmetic intensity calculation to

$$\begin{aligned}
\mathrm{Arithmetic\,\,Intensity} & = & \frac{32mnI}{16(m+n)I + 64mn} \\
\end{aligned}$$

which implies that the arithmetic intensity can be made arbitrarily large by increasing the tile size until the number of available registers is exhausted. In practice a balance must be struck between the available resources and the arithmetic intensity and this balance is achieved in the xGPU implementation by *multi-level tiling* and we direct the reader to for a complete discussion.

The MWA correlator is not a delay tracking correlator. Therefore, we do not have to adjust the correlator inputs for whole, or partial sample delays. The correlator always provides correlation output products phased to zenith. The system parameters (integration time, baseline length, channel width) have all been chosen with this in mind and the long operating wavelengths result in the system showing only minimal ( 1%) decorrelation even far from zenith for typical baseline lengths, integration times and channel widths (see Figure ). Typical observing resolutions have been between 0.5 and 2 seconds in time and 40 kHz and 10 kHz in frequency.

### Output Data Format

The correlation products are accumulated on the GPU device and then copied asynchronously back to host memory. The length of accumulation is chosen by the operator. Each accumulation block is topped with a FITS header and transferred to an archiving system where the products are concatenated into larger FITS files with multiple extensions, wherein each integration is a FITS extension (see Appendix ). The archiving system is an instantiation of the Next Generation Archiving System (NGAS) and was originally developed to store data from the European Southern Observatory (ESO) telescopes. The heart of the system is a database that links the output products with the meta-data describing the observation and that manages the safe transportation and archiving of data. In the MWA operating paradigm all visibilities are stored, which at the native resolution is 33024 visibilities and 3072 channels or approximately 800 MBytes per time resolution unit (more than 6 Gb/s for 1 second integrations). Due to the short baselines and long wavelengths used by the array it is reasonable to integrate by a factor of two in each dimension (of time and frequency, see Figure for the parameter space), but even this reduced rate amounts to 17 TBytes/day at 100% duty cycle. The large data volume precludes handling by humans and the archiving scheme is entirely automated. NGAS provides tools to access the data remotely and provides the link between the data products to the observatory SQL database. As discussed in , the data are taken at the Murchison Radio Observatory in remote Western Australia and transferred to the Pawsey HPC Centre for SKA Science in Perth, where 15 PB of storage is allocated to the MWA over its five-year lifetime. Subsequently the archive database is mirrored by the NGAS system to other locations around the world: MIT in the USA; The Victoria University of Wellington, NZ; and the Raman Research Institute in India. These remote users are then able to access their data locally.

# Instrument Verification and Commissioning

After initial instrument commissioning with simple test vectors and noise inputs the telescope entered a commissioning phase in September 2012. The commissioning team initially consisted of 19 scientists from 10 institutions across 3 countries and was led by Curtin University. This commissioning phase was successfully concluded in July 2013 giving way to "Early Operations" and the MWA is now fully operational.

## Verification

The development of the MWA correlator proceeded separately to the rest of the MWA systems and the correlator GPU based elements were verified against other software correlator tools. The MWA correlator is comparatively simple, it does not track delays, performs the correlation at 32-bit floating point precision, and does not have to perform any Fourier transformations. The only verification that was required was to ensure that the cross multiply and accumulate was performed to sufficient precision and matched results generated by a simple CPU implementation.

Subsequent to ensuring that the actual cross multiply and accumulate operation was being performed correctly the subsequent testing and integration was concerned with ensuring that the signal and path was maintained. This was a relatively complex operation due to the requirement that the correlator interface with legacy hardware systems. The details of these interfaces are presented in the Appendices. In Figure a correlation matrix is shown, the colours representing the length of the baseline associated with that antenna pair. The following Figure shows the response of the baselines when the telescope is pointed at the radio galaxy Centaurus A, demonstrating the response of the different baseline lengths of the interferometer to structure in the source. Images of this object obtained with the MWA can be found in McKinley et al. 2013. The arrows on Figure indicate those baselines associated with the densely-packed core .

## Verification Experiments

The MWA Array and correlator is also the verification platform for the Aperture Array Verification System (AAVS) within the SKA reconstruction activities pursued by the Low Frequency Aperture Array (LFAA) consortium. This group have recently performed a successful verification experiment, using both the MWA and the AAVS telescopes. Using an interferometric observation of the radio galaxy Hydra A taken with the MWA system, they have obtained a measurement of antenna sensitivity (A/T) and compared this with a full-wave electromagnetic simulation (FEKO[^3]). The measurements and simulation show good agreement at all frequencies within the MWA observing band and are presented in detail in .

## Commissioning Science

In order to verify the instrumental performance of the array, correlator and archive as a whole, the MWA Science Commissioning Team have performed a 4300 deg$^2$ survey, and have published a catalogue of flux densities and spectral indices from 14,121 compact sources. The survey covered approximately 21 h $<$ Right Ascension $<$ 8 h, -55$^{\circ}$ $<$ Declination $<$ -10$^{\circ}$ over three frequency bands centred on 119, 150 and 180 MHz. This survey will be detailed in Hurley-Walker et al. (2014). Data taken during and shortly after the commisioning phase of the instrument has also been used to demonstrate the tracking of space debris with the MWA , novel imaging and deconvolution schemes , and to present multi-frequency observations of the Fornax radio galaxy and the galaxy cluster A3667 .

# The Future of the Correlator

## Upgrade Path

We do not fully utilise the GPU in the MWA correlator and we estimate that the X-Stage of the correlator could currently support a factor of 2 increase in array size (to 256 dual polarisation elements). As discussed in the operational life-span of the MWA is intended to be approximately five years. The GPUs (NVIDIA M2070s) the MWA correlator are from the Fermi family of NVIDIA GPU and are already almost obsolete. One huge advantage of off-the-shelf signal processing solutions is that we can easily benefit from improvements in technology. We could swap out the GPU in the current MWA cluster, replace them with cards from the Kepler series (K20X) with no code alteration, and would benefit from a factor of 2.5 increase in performance (and a threefold improvement in power efficiency) as not only are the number of FLOPS provided by the GPUs increasing, but also their efficiency (in FLOPS/Watt) is improving rapidly. This would permit the MWA to be scaled to 512 elements. However, not all of our problem is with the correlation step and increasing the array to 512 elements would probably require an upgrade to the networking capability of the correlator.

## SKA Activities

The MRO is the proposed site of the low-frequency portion of the Square Kilometre Array. Curtin University is a recipient of grants from the Australian government to support the design and pre-construction effort associated with the construction and verification of the Low-Frequency Aperture Array (LFAA) and software correlation systems for the SKA project. The MWA correlator will therefore be used as a verification platform within the prototype for the LFAA, known as the Aperture Array Verification System (AAVS). One of the benefits of a software based signal processing system is flexibility. We have been able to easily incorporate AAVS antennas into the MWA processing chain to facilitate these verification activities, which have been of benefit to both LFAA and the MWA . Furthermore we will be developing and deploying SKA software correlator technologies throughout the pre-construction phase of the SKA using the MWA correlator for both testing and verification.

# Summary

This paper outlines the structure, interfaces, operations and data formats of the MWA Hybrid FPGA/GPU correlator. This system combines off-the-shelf computer hardware with bespoke digital electronics to provide a flexible and extensible correlator solution for a next generation radio telescope. We have outlined the various stages in the correlator signal path and detailed the form of all internal and external interfaces.

# Acknowledgements

We acknowledge the Wajarri Yamatji people as the traditional owners of the Observatory site. Initial correlator development was enabled by equipment supplied by an IBM Shared University Research Grant (VUW & Curtin University), and by an Internal Research and Development Grant from the Smithsonian Astrophysical Observatory.

Support for the MWA comes from the U.S. National Science Foundation (grants AST CAREER-0847753, AST-0457585, AST-0908884 and PHY-0835713), the Australian Research Council (LIEF grants LE0775621 and LE0882938), the U.S. Air Force Office of Scientific Research (grant FA9550-0510247), the Centre for All-sky Astrophysics (an Australian Research Council Centre of Excellence funded by grant CE110001020), New Zealand Ministry of Economic Development (grant MED-E1799), an IBM Shared University Research Grant (via VUW & Curtin), the Smithsonian Astrophysical Observatory, the MIT School of Science, the Raman Research Institute, the Australian National University, the Victoria University of Wellington, the Australian Federal government via the National Collaborative Research Infrastructure Strategy, Education Investment Fund and the Australia India Strategic Research Fund and Astronomy Australia Limited, under contract to Curtin University, the iVEC Petabyte Data Store, the Initiative in Innovative Computing and NVIDIA sponsored CUDA Center for Excellence at Harvard, and the International Centre for Radio Astronomy Research, a Joint Venture of Curtin University and The University of Western Australia, funded by the Western Australian State government.

# MWA F-Stage (PFB) to Voltage Capture System (VCS)

The contents of this Appendix are extended from the original MWA ICD for the polyphase filterbank written by R. Cappallo. We have incorporated the changes made to the physical mappings, electrical connections and header contents to enable capture of the PFB output by general purpose computers.

## Physical

-   PFB presents data on two CX-4 connectors. ALL pins are wired with transmit drivers, there are no receivers on any pins

## Interface Details

-   Two custom made CX4 to SFP breakout cables. This cable breaks a CX4 connector into 4 1X transmit and receive pairs on individual SFP terminated cables.

-   Data flows only in a single direction (simplex) from PFB to CB. Each (normally Tx/Rx) pair of differential pairs is wired as two one-way data paths (Tx /Tx). We have distributed these one-way data paths to occupy every other CX-4 pin. Therefore all cables are wired correctly as Tx/Rx but the Rx channels are not utilised.

-   The serial protocol to be used a custom protocol as described below. The interface to the Xilinx Rocket I/O MGTÕs is GT11_CUSTOM, which is a lean protocol. The word width will be 16 bits.

-   The mean data rate per 1X cable will be 2.01216 Gb/s, carried on a high speed serial channel of burst rate of 2.08 Gb/s formatted, or 2.6 Gb/s including the 10B/8B encoding overhead.

## Data Format

-   Each data stream carries all of the antenna information for 384 fine frequency channels. In the entire 32T correlation system there are 8 such data streams going from the PFB to the CB, each carrying the channels making up a 3.84 MHz segment of the processed spectrum.

-    Each sample in the stream is an 8-bit (4R+4I) complex number, 2Õs complement with 8 denoting invalid data, representing the voltage in a 10 kHz fine frequency channel.

-   Data are packetised and put in a strictly defined sequence, in f1t\[f2a\] (frequency-time-frequency-antenna) order. The most rapidly varying index, a, is the antenna number (0, 16, 32, 48, 1, 17, 33, 49, 2,18, É, 15, 31, 47, 63), followed by f2, the fine frequency channel (n..n+3), then time (0..499), and most slowly the fine frequency channel group index (0, 4, 8, 12, 128, 132, 136, 140, ,É, 2944, 2948, 2952, 2956 for fibre 0; increment by 16n for fibre n). The brackets indicate the portion of the stream that is contained within a single packet. The antenna ordering is non intuitive as for historical reasons each PFB combines its inputs into an order originally considered conducive to its interface to a full mesh backplane and a hardware FPGA based correlator.

-   A single packet consists of all antenna data for a group of four fine channels for a single time point. Thus there are 500 sequential time packets for each of the 96 frequency channel groups. Over the interval of 50 ms there are 48,000 (=96channel groups x 500 time samples) packets sent per data stream, so there are 960,000 packets/second.

-    Using a packet size of 264 bytes allows a 16-bit packet header (0x0800), two 16-bit words carrying the second of time tick and packet counter, 256 bytes of antenna data, finally followed by a 16-bit checksum (see Table 1).

-   The header word is a 16-bit field with value 0x0800, used to denote the first word in a packet. Note that it cannot be confused with the data words, since the value 8 is not a legal voltage sample code.

-   The second tick in word two is in bit 0, and has the value 0 at all times, except for the first packet of a new second, when it is 1.

-   The leading bits of word two, and all of word three contains the following counter values that can be decoded to determine the precise input and channel that a given packet belongs to.

    -   *sec_tick* (1 bit): Is this the first packet for this data lane, for this one second of data? 0=No, 1=Yes. Only found on the very first packet of that second. If set, then the mgt_bank, mgt_channel, mgt_group and mgt_frame will always be 0. The preceding 3 bits are unused.

    -   *pfb_id* (2 bits): Which physical PFB board generated this stream \[0-3\]. This defines the set of receivers and tiles the data refers to. A lane's pfb_id will remain constant unless physically shifted to a different PFB via a cable swap.

    -   *mgt_id* (3 bits): Which 1/8 of coarse channels this Rocket–IO lane contains. Should be masked with 0x7, not 0xf. Contains \[0-7\] inclusive. A lane's mgt_id will remain constant unless physically shifted to a different port on the PFB via a cable swap.

    Within an individual data lane, the data packets cycle in the following order, listed from slowest to fastest.

    -   *mgt_bank* (5 bits): Which one of the twenty 50ms time banks in the current second is this one? \[0-19\] \[0-0x13\].

    -   *mgt_channel* (5 bits): Which coarse channel \[0-23\] \[0-0x17\] does the packet relate to?

    -   *mgt_group* (2 bits): Which 40KHz wide packet of the contiguous 160KHz does this packet contain the 4 x 10KHz samples for? \[0-3\]

    -   *mgt_frame* (5 bits): Which time stamp within a 50ms block this packet is. \[0-499\] or \[0-0x1F3\]. Cycles fastest. Loops back to 0 after 0x1F3. There are 20 complete cycles in a second.

-   The last word of the packet is a 16-bit checksum, formed by taking the bitwise XOR of all 128 (16-bit) words of antenna samples in the packet.

# VCS to XMAC Server

## Physical

-   Data presented on either fibre optic or direct attach copper cables terminated with SFP pluggable transceivers. Note that all optical transceivers used in the CISCO UCS servers must be supplied by CISCO, the firmware within the 10Gb ethernet cards requires it.

-   A single 10GbE interface is sufficient to handle the data rate

## Interface Details

-   Ethernet IEEE 802.3 frames, with TCP/IPv4

-   The packet format is precisely as described in the PFB to VCS interface

-   The communication is from the VCS to the XMAC is uni-directional and mediated by a switch.

-   Each XMAC requires a data packet from all VCS machines. This requires 32 open connections mediated by TCP

-   Data from all sources is stripped of header information and assembled into a common 1 second buffer which requires 20 blocks of 2000 packets from each of the 32 connections to be assembled.

-   As the packets for four PFB are being combined the antenna ordering is also being concatenated. The order within each packet is maintained with the 64 inputs concatenated into a final block of 256 inputs.

# XMAC to GPU Interface

## Physical

The interface is internal to the CPU host, being a section of memory shared between the GPU device driver and the host.

## Interface details

### GPU Input

The data ordering in the assembly buffer for input buffers, running from slowest to fastest index is time(t), channel(f), station(s), polarisation(p), complexity(i). The data is an 8 bit integer, promoted from 4-bit twos-complement sample via a lookup table. The correlator is a dual polarisation correlator as implemented, but the input station/polarisation ordering is not in a convenient order as we have concatenated 4 PFB outputs during the assembly of this buffer. Two options presented themselves. One that we re-order the input stations and polarisation to undo the partial corner turn. The disadvantage begin that each input packet would need to be broken open and the stations reordered - which would have been extremely compute intensive. Or to correlate *without* changing the order and remapping the output products to the desired order. This is preferable as although there are now N$^2$ products instead of N stations, the products are integrated in time and can be reordered in post-processing.

### GPU Output

The data are formated as 32-bit complex floating point numbers. An individual visibility set has the order (from slowest index to fastest index): channel(f), station(s1), station(s2), polarisation(p1), polarisation(p2), complexity(i). The full cross-correaltion matrix is Hermitian so only the lower, or upper, triangular elements need to be calculated, along with the on-diagonal elements which are the auto-correlation products. Therefore only a packed triangular matrix is actually transferred from the GPU to the host. The GPU processing kernel is agnostic to the ordering of the antenna inputs and the PFB output order is maintained. Therefore the correlation products cannot be simply processed without a remapping.

The remapping is performed by a utility that has been supplied by the builders to the data analysis teams that simply reorders the data products into one that would be expected if no PFB reordering had taken place. It is important to keep track of which products have been generated but the XMAC as conjugates of the desired products, and which need conjugating. It should further be noted that this correction is performed to the order as presented *to* the PFB and may not the order as expected by the ordering of the receiver inputs. Care should be taken in mapping antennas on the ground as the order of the receiver processing within a PFB is not only function of the physical wiring but the firmware mapping. We have carefully determined this mapping and the antenna mapping tables are held within the same database as the observing metadata.

For historical reasons the files are transferred into an intermediate file format that was originally used during instrument commissioning before subsequent transformation to UVFITS files (or measurement sets) by the research teams.

# XMAC Server to NGAS (Next Generation Archiving System)

## Physical

The interface is internal to the CPU host, being a section of memory shared between the XMAC application and an NGAS application running on the same host.

## Interface details

The data are handed over as a packed triangular matrix with the same ordering as was produced by the GPU. It has a FITS header added and is padded to the correct size as expected by a FITS extension. The FITS header includes a time-tag.

Once the buffer is presented to the NGAS system it is buffered locally and presented to a central archive server, subsequently it is added to a database and mirrored to multiple sites across the world.

[^1]: xGPU available at: https://github.com/GPU-correlators/xGPU

[^2]: In order to achieve optimal loop unrolling and compile-time evaluation of conditional statements, xGPU requires that the number of channels, number of stations, and minimum integration time are compile-time parameters.

[^3]: www.feko.info


bibliography: auto_generated.bib
date: 2024-10-01
title: Alignment of the CMS tracker with LHC and cosmic ray data

$Revision: 230857$ $HeadURL: svn+ssh://firstname.lastname@example.com/reps/tdr2/papers/TRK-11-002/trunk/TRK-11-002.tex$ $Id: TRK-11-002.tex 230857 2014-03-10 15:51:29Z alverson$

# Acknowledgements

We congratulate our colleagues in the CERN accelerator departments for the excellent performance of the LHC and thank the technical and administrative staffs at CERN and at other CMS institutes for their contributions to the success of the CMS effort. In addition, we gratefully acknowledge the computing centres and personnel of the Worldwide LHC Computing Grid for delivering so effectively the computing infrastructure essential to our analyses. Finally, we acknowledge the enduring support for the construction and operation of the LHC and the CMS detector provided by the following funding agencies: the Austrian Federal Ministry of Science and Research and the Austrian Science Fund; the Belgian Fonds de la Recherche Scientifique, and Fonds voor Wetenschappelijk Onderzoek; the Brazilian Funding Agencies (CNPq, CAPES, FAPERJ, and FAPESP); the Bulgarian Ministry of Education and Science; CERN; the Chinese Academy of Sciences, Ministry of Science and Technology, and National Natural Science Foundation of China; the Colombian Funding Agency (COLCIENCIAS); the Croatian Ministry of Science, Education and Sport, and the Croatian Science Foundation; the Research Promotion Foundation, Cyprus; the Ministry of Education and Research, Recurrent financing contract SF0690030s09 and European Regional Development Fund, Estonia; the Academy of Finland, Finnish Ministry of Education and Culture, and Helsinki Institute of Physics; the Institut National de Physique Nucléaire et de Physique des Particules / CNRS, and Commissariat à l'Énergie Atomique et aux Énergies Alternatives / CEA, France; the Bundesministerium für Bildung und Forschung, Deutsche Forschungsgemeinschaft, and Helmholtz-Gemeinschaft Deutscher Forschungszentren, Germany; the General Secretariat for Research and Technology, Greece; the National Scientific Research Foundation, and National Innovation Office, Hungary; the Department of Atomic Energy and the Department of Science and Technology, India; the Institute for Studies in Theoretical Physics and Mathematics, Iran; the Science Foundation, Ireland; the Istituto Nazionale di Fisica Nucleare, Italy; the Korean Ministry of Education, Science and Technology and the World Class University program of NRF, Republic of Korea; the Lithuanian Academy of Sciences; the Ministry of Education, and University of Malaya (Malaysia); the Mexican Funding Agencies (CINVESTAV, CONACYT, SEP, and UASLP-FAI); the Ministry of Business, Innovation and Employment, New Zealand; the Pakistan Atomic Energy Commission; the Ministry of Science and Higher Education and the National Science Centre, Poland; the Fundação para a Ciência e a Tecnologia, Portugal; JINR, Dubna; the Ministry of Education and Science of the Russian Federation, the Federal Agency of Atomic Energy of the Russian Federation, Russian Academy of Sciences, and the Russian Foundation for Basic Research; the Ministry of Education, Science and Technological Development of Serbia; the Secretarı́a de Estado de Investigación, Desarrollo e Innovación and Programa Consolider-Ingenio 2010, Spain; the Swiss Funding Agencies (ETH Board, ETH Zurich, PSI, SNF, UniZH, Canton Zurich, and SER); the National Science Council, Taipei; the Thailand Center of Excellence in Physics, the Institute for the Promotion of Teaching Science and Technology of Thailand, Special Task Force for Activating Research and the National Science and Technology Development Agency of Thailand; the Scientific and Technical Research Council of Turkey, and Turkish Atomic Energy Authority; the National Academy of Sciences of Ukraine, and State Fund for Fundamental Researches, Ukraine; the Science and Technology Facilities Council, UK; the US Department of Energy, and the US National Science Foundation.

Individuals have received support from the Marie-Curie programme and the European Research Council and EPLANET (European Union); the Leventis Foundation; the A. P. Sloan Foundation; the Alexander von Humboldt Foundation; the Belgian Federal Science Policy Office; the Fonds pour la Formation à la Recherche dans l'Industrie et dans l'Agriculture (FRIA-Belgium); the Agentschap voor Innovatie door Wetenschap en Technologie (IWT-Belgium); the Ministry of Education, Youth and Sports (MEYS) of Czech Republic; the Council of Science and Industrial Research, India; the Compagnia di San Paolo (Torino); the HOMING PLUS programme of Foundation for Polish Science, cofinanced by EU, Regional Development Fund; and the Thalis and Aristeia programmes cofinanced by EU-ESF and the Greek NSRF.

# The CMS Collaboration