abstract: The VEDLIoT project targets the development of energy-efficient Deep Learning for distributed AIoT applications. A holistic approach is used to optimize algorithms while also dealing with safety and security challenges. The approach is based on a modular and scalable cognitive IoT hardware platform. Using modular microserver technology enables the user to configure the hardware to satisfy a wide range of applications. VEDLIoT offers a complete design flow for Next-Generation IoT devices required for collaboratively solving complex Deep Learning applications across distributed systems. The methods are tested on various use-cases ranging from Smart Home to Automotive and Industrial IoT appliances. VEDLIoT is an H2020 EU project which started in November 2020. It is currently in an intermediate stage with the first results available.
author:
title: VEDLIoT: Very Efficient Deep Learning in IoT [^1]

# The VEDLIoT Approach

Deep Learning has become a strong driver in IoT applications. Typically, those applications have very challenging computational demands coupled with a low energy budget. The goal of VEDLIoT is to integrate IoT with Deep Learning, accelerate applications, and optimize them towards energy efficiency. shows the architecture of VEDLIoT. The project is presented following a bottom-up approach, starting with customizable hardware platforms using modular microservers and specialized hardware accelerators for heterogeneous computing. The optimization of use cases is systematically accompanied from an early stage of development by using requirement engineering and verification techniques for AIoT, also developed within VEDLIoT. Expert-level knowledge of different domains is combined to create a powerful middleware for optimizing the underlying neural networks of deep learning algorithms and ease the development with frameworks for testing, benchmarking and deployment. A lot of development work goes into guaranteeing a high level of safety and security, which is essential for the VEDLIoT use cases.

# Accelerated AIoT hardware platform

The hardware development within VEDLIoT focuses on extending and refining the already available platforms RECSBox and t.RECS, which primary targets cloud and near edge computing. uRECS is developed from scratch within VEDLIoT and focuses on compact dimensions, low cost, and high energy efficiency to better suit low-cost / low-power devices for AI and ML applications (). Using the RECS hardware platform, VEDLIoT covers the complete range from embedded via edge to cloud computing.

## Heterogeneous hardware platform

All RECS hardware platforms share a modular approach, which leads to a heterogeneous, adaptable hardware architecture supporting a wide range of applications and allowing for a future-proof design by an exchangeable/upgradable hardware basis  and . Another common feature is the scalable communication-driven infrastructure, realizing efficient communication between heterogeneous microservers via 1 G/ 10 G Ethernet and high-speed low-latency connections, reconfigurable during run-time .

Most supported microservers are based on mid- or high-performance Computer-on-Module (COM) form factors, e.g., RECSBox supports COM Express microservers and t.RECS the recently released COM-HPC Server and Client standards. As shown in , several other, well-established form factors focus on low-power embedded computing. SMARC modules, for example, provide a smaller footprint and support with x86, ARM and FPGA-SoC more target architectures.

uRECS closes the gap in hardware platforms towards embedded/ far edge computing with a power consumption of less than 15 W as required by some use cases. Next to SMARC microservers, it also natively supports Jetson Xavier NX modules. By using adaptor-PCBs, uRECS also integrates Xilinx Kria, and Raspberry Pi compute modules. Extension slots based on USB and M.2 can be used to use additional hardware accelerators or peripherals.

VEDLIoT extends the classically static hardware architecture towards a dynamically configurable infrastructure for increased resource-efficiency and robustness. The RECS ecosystem enables easy exchange of computing resources and seamless switching between the different heterogeneous components on the system level. On the communication level, e.g., the networking topology or protocol parameters can be adapted to cope with changing real-time or bandwidth requirements. Finally, reconfigurable devices (FPGAs) are utilized to enable the integration of new architectural concepts developed in VEDLIoT. On this level, partial reconfiguration is used to adapt to changing application requirements at run-time, e.g., using implementations with different power/performance footprints.

## Accelerators and Microservers

One of the key components to delivering the required performance for the Deep Learning (DL) applications are the hardware accelerators. VEDLIoT focuses on developing new dedicated hardware accelerators tailored explicitly towards specific applications requirements. However, the software abstraction layers that have helped the independent development of both software and hardware in the past cannot be used any longer to achieve the best performance and efficiency for the most demanding workloads. The solution is to focus on hardware-software co-design . In VEDLIoT, four different types of DL accelerators are explored: (1) existing off-the-shelf; (2) statically configured; (3) dynamically reconfigurable; and (4) fully simultaneous co-design accelerator. Evaluation and exploration are done using existing accelerators for fast deployment of the required performance. FPGAs are used to develop accelerator prototypes that can achieve higher performance and efficiency for use-case applications. In addition, different modes of operation are identified that offer dedicated accelerators to those modes of operation. These accelerators follow the partial co-design principles by mapping the DL models into the hardware components. Nevertheless, preliminary results have shown that no single accelerator can provide a better match to different models. Consequently, the fully simultaneous co-design is explored where the hardware is developed together with the software. In addition to mapping the models to hardware, feedback is given to the models so that optimizations can be tuned for better hardware utilization. In addition to the accelerator design, an in-depth study of how the memory is utilized in current accelerators and exploring new approaches for the memory hierarchy for future DL accelerators is performed.

With the many moving parts in the space of DL processing acceleration, VEDLIoT uses Renode, an open-source simulation framework , to test the FPGA accelerator prototypes. Renode, a functional simulator for complex heterogeneous systems, provides an ability to simulate complete SoCs and run the same software that would be used on hardware. VEDLIoT benefits from Renode's testing and introspection capabilities, using it both for interactive development of accelerator prototypes and within a Continuous Integration environment. This does not only ease the development process but also makes the final result more reliable. During the course of the project, Renode is enhanced with capabilities of simulating Custom Function Units, or CFUs. A CFU is an accelerator tightly coupled with the CPU, providing functionality explicitly designed for the planned ML workflow. Programmed in a Hardware Description Language, CFUs are used as an input for Renode to extend simulated cores.

## Performance Evaluation

A wide variety of hardware accelerators for deep learning is emerging on the market, targeting a wide range of applications from small embedded systems with power budgets in the order of milliwatt to cloud platforms with a power consumption exceeding 400 W. Figure  summarizes the accelerators, that have been analyzed in detail in . It has to be noted that the diagram shows a very high-level view. The data is based on the peak performance values (in Giga Operations per Second), provided by the vendors. No normalization to a specific technology node is performed, and the architectures vary in the used precision, ranging from FP32 to INT8 and even binary weights are included. Nevertheless, an interesting fact is that most architectures cluster around an energy efficiency of about 1 Tera Operation per Ws (1 TOPS/W), independent of their individual performance (or power demand).

For performance evaluation, the DL models ResNet50, MobileNetV3 and YoloV4 were chosen to determine comparable performance values of available DL accelerators. Depending on the supported quantization of the hardware, the tests were executed using INT8, FP16 or FP32 datatypes. The tools used for best utilization are chosen based on the manufacturer's recommendations, e.g., TensorRT for NVIDIA. In addition, performance and hardware utilization were optimized by varying the batch size from 1 to 8, which is represented in Figure  by B1, B4 and B8. In this figure, the performance (in GOPS) and the measured power consumption (in Watt) are shown exemplarily for YoloV4. The investigated platforms include x86 CPUs (Epic3451 and D1577), GPUs (GTX1660), eGPUs (Xavier AGX (in high performance and low power mode), Xavier NX and Jetson TX2), FPGAs (Zynq ZU15 and ZU3) and ASICs (Myriad). In VEDLIoT, performance and energy efficiency evaluations are an important basis for selecting DL accelerators to be integrated into the RECS platform, tailoring it towards the use cases.

# Optimizing tool chain for heterogeneous hardware

Trained deep learning models have redundancy in their computational graph that can be exploited for optimizations. In some cases, models have been compressed down to 49x of their original size, with negligible accuracy loss. This can be achieved by combining methods that remove connections and/or neurons, quantize parameters and activations and encode the parameters in a more compact form . Although there has been much recent research in the area, most of the results are theoretical speed-ups based on metrics, e.g. number of operations and reduction of parameters. The theoretical speed-ups do not always translate to more efficient execution in hardware . In the VEDLIoT project, novel methods for hardware-aware optimization are developed. Furthermore, the industry-standard ONNX, which is an open format to represent machine learning models , is used as input to ensure compatibility with the current open ecosystem. The model's computational graph undergoes significant surgery in the optimization phase to optimize its execution latency, power consumption and/or memory footprint. Utilizing the knowledge of the target hardware leads to optimizations that translate to improved execution metrics when deployed. Deploying deep learning models on edge devices usually involves the following steps: (1) Preparation and analysis of the dataset, preparation of data pre-processing and output post-processing routines. (2) Model training (usually transfer learning), if necessary. (3) Evaluation and improvement of the model until its quality is satisfactory. (4) Model optimization, usually hardware-specific optimizations (e.g., operator fusion, quantization, neuron-wise or connection-wise pruning). (5) Model compilation to a given target. (6) Model deployment and execution on a given target. There are many different frameworks for most of the above steps (training, optimization, compilation and runtime). The cooperation between those frameworks differs and may provide different results.

Kenning , an open-source framework developed by Antmicro, addresses the problem of enabling the tools to cooperate with each other. The interoperability is achieved by converting the models into a common representation using the Open Neural Network Exchange (ONNX) format. All intermediate conversions and optimizations are performed on ONNX models. At the final stage, Kenning converts the model to a selected neural network runtime and deploys it on the target hardware. Based on the implemented interfaces, the Kenning framework can measure the inference duration, resource usage, and processing quality on a given target. Depending on a target, Kenning can monitor inference time, mean CPU usage, and CPU and GPU memory usage. Kenning can automatically benchmark the processing quality of a given neural network mode and generate a confusion matrix for classification models and recall/precision graphs for detection algorithms. In addition to Kenning, VEDLIoT uses Renode, an open-source simulation framework , which has been introduced in Section .

# Safety, Security and Requirements for distributed AIoT systems

When combining deep learning with the properties of IoT, new concerns might arise that are not yet foreseen by standards and literature. The new concerns include aspects such as data quality, heuristic deep learning modelling, learning of the models, or even new ethical considerations. Additional stakeholders such as data engineers enter the stage, and common languages or interfaces need to be found between the different stakeholders. Typical architectural frameworks, such as the ISO 42010 or the IEEE 2413 standard, cannot cope with concerns for systems that include some form of machine learning. One major challenge identified in VEDLIoT is the difficulty to keep track of dependencies, e.g. through correspondence rules, between the different architectural views. Another problem of current architectural frameworks is the lack of a clear system development hierarchy, which would support the early identification and mapping of dependencies between different architectural views .

## Requirements concepts for AIoT

Designing a large, distributed system is a hierarchical process . The architectural framework for VEDLIoT not only supports the seamless design and integration of traditional software components and deep learning components but also allow for all necessary quality concerns to be taken into account as early as possible in the design process. The VEDLIoT architectural framework is organized by two aspects: Clusters of concerns, and level of abstraction. These aspects form a 2-dimensional grid of architectural views that guide the concept and design of a VEDLIoT system. Typical clusters of concerns for a system with deep learning components are *logical behavior, process behavior, context and constraints, learning setting, deep learning model, hardware, information, communication, ethical concerns, safety, security, privacy,* and *energy*. Levels of abstraction are *knowledge level, conceptual level, design level*, and *run-time level*. Each architectural view is categorized by the two aspects cluster of concern and level of abstraction. In VEDLIoT, it is shown that dependencies between the architectural views only exist vertically between the views of the same cluster of concern or horizontally between architectural views on the same level of abstraction. This reduces the complexity of the system design challenge and allows for better traceability. Knowledge can become available on all levels of the architectural framework at any time. Traditionally, requirement engineering would be organized in a top-down fashion. However, the architectural framework supports middle-out systems engineering, which is a widely common practice, combining traditional top-down systems design with integration of designated lower-level hardware, software, AI models, or other components .

## Safety aspects

Safety requirements must guide any model construction for systems using deep neural networks. Safety standards emphasize processes for software development that help with avoiding systematic mistakes during the design of systems. However, safety standards that base on the EN-IEC 61508 standard[^2], such as ISO 26262 for the automotive industry, assume that for software, only systematic faults exist. However, due to the probabilistic nature of deep learning, the assumption that only systematic faults exist in software does not hold anymore. There is no absolute guarantee that a deep neural network performs as intended under the desired circumstances. The desired behavior of the deep neural network depends on the data used for training and validation. Therefore, safety standards for deep neural networks must encompass the deep neural networks and especially the data used for training and validation.

From the perspective of defining an architecture and implementing system solutions to increase the robustness of deep learning processes, VEDLIoT focuses on monitoring approaches to detect faulty situations and trigger appropriate reactive measures. The work is being developed in two directions. Firstly, the problem of characterizing the quality of the input data is considered, detecting situations in which these data may have been accidentally or even maliciously compromised. A large set of data errors may be easily identified, may be corrected, or the affected data may be removed to avoid the propagation of these errors through the DL models. Different monitoring and error detection mechanisms are developed, depending on the kinds of input data (e.g., time series, image) and on the error types (e.g., outliers, image noise). Secondly, the problem of detecting errors on the output data is analyzed when these errors derive from systematic faults affecting the execution of DL models on devices or edge nodes. It is considered that these faults may have been triggered or injected during run-time (e.g., hardware faults, attacks). In brief, the approach consists in periodically submitting both the input and the output data to a robustness service, which holds a copy of the DL model and can verify the correctness of the output data. To support all these monitors and monitoring mechanisms, an architectural pattern comprising two separate parts is considered, based on the concept of architectural hybridization .

## Security considerations

VEDLIoT implements several hardware- and system-level tools to improve the dependability and security of edge applications. Hardware features are leveraged for trusted execution environments combined with well-established dependability techniques to support both middleware and applications layers of the project. So far, the project has focused on developing end-to-end trust through a distributed attestation mechanism, secure execution and communication of critical code (e.g. for monitors, see Section ) on edge devices. The hardware protection offered by Intel SGX enclaves is leveraged, and an open-source WebAssembly runtime implementation to build a trusted runtime environment without dealing with language-specific APIs. An evaluation shows that SQLite can be fully executed inside an SGX enclave via WebAssembly and existing system interface, with small performance overheads .

As part of developing a hardware-trusted execution environment, a novel Trusted Execution Environment (TEE) support for VexRISC-V, an open-source RISC-V soft processor, has been developed. The implementation takes the form of a highly optimized RISC-V Physical Memory Protection (PMP) unit that enables secure processing by limiting the physical addresses accessible by software running on a processor. The PMP unit is configurable in the highest privilege level (the machine mode) and can be used to specify read, write and execute access privileges for a specific memory region. In small devices that only support machine mode (M-mode) and user mode (U-mode), the PMP configurations can efficiently ensure the secure execution of software in M-mode and U-mode. The PMP implementation is part of the official VexRISC-V implementation, and the source code and documentation are openly available for the research community.

Apart from x86 and RISC-V, also ARM SoCs are considered, using TrustZone as a TEE, combined with the open-source and trusted operating system OP-TEE. TrustZone splits the operating system into two parts: the normal and secure worlds. Trusted applications can only run in the secure world, and the operation necessary to change context between worlds is rather complex and cannot be done at user-level. To implement remote attestation for WebAssembly code running in ARM processors, a TEE specification defining how the trusted environment behaves and how the normal world can interact with the secure world is realized. The implementation is based on a root-of-trust provided by the hardware and a secure boot mechanism, preventing an attacker from substituting the trusted software.

# VEDLIoT applications

VEDLIoT applications focus on both very high energy efficiency and high-security and safety requirements. These requirements are in line with the general need for "trusted IoT and edge computing platforms" and "development and deployment of next-generation computing components" as identified in the EU research agenda .

## Automotive

Amongst the numerous potential problems in the automotive sector, the Pedestrian Automatic Emergency Breaking (PAEB) was chosen as a well-specified example, which can be benchmarked and compared against state-of-the-art systems. The major development goals are the distribution of the deep learning models and the decision making between different on-car systems and edge devices at varying speeds and reliability of mobile networks. Dynamic distributing of sensor data to edge stations is a quite new research topic. It requires quick monitoring of available mobile networks, their speed and latency, available computing resources of the edge devices and a management system that can quickly react to the current situation. The overall goal is to optimize the energy efficiency in total and minimize the on-car energy consumption. Sending raw sensor data via a mobile network to an edge station always implies a high-security risk. Therefore, an integration of VEDLIoT's remote attestation approach is of importance.

## Industrial IoT

VEDLIoT supports two Industrial IoT use cases: Motor Condition Classification and Arc Detection in DC power distribution cabinets. The Motor Condition Classification use case aims to design and build a prototype of a battery-powered ultra-low energy deep learning-driven small box that can be attached to large electric asynchronous motors and continuously monitors the motor. The states to monitor are the operational, thermal and mechanical conditions of the motor, and upon specified events, e.g. a ball bearing failure, a message is sent to an operator.

The Arc Detection use case aims to design and build a prototype system that can detect unwanted arcs in DC power distribution cabinets using deep learning technology. A challenge is to guarantee a very low latency from the first spark till inference, including sensing and pre-processing, and an ultra-low false-negative error rate for a smooth operation. In general, arc localization helps for faster fault detection and repair of broken units.

## Smart Home

This use case targets the development and acceleration of AI-based methods for a demand-oriented interaction between the user and a smart home. To achieve this goal, an intuitive and natural operating interface is crucial, which is realized by a smart-mirror device. As seen in Figure , a camera and a microphone are providing input data, and four different neural networks are used to detect gestures, faces, objects and speech to interact with people. The distribution of data to the cloud is not desirable because of privacy concerns of the residents. Therefore, all sensing and interaction is performed on-site in real-time, making low power and energy efficiency computations a prime concern for this use-case.

# Summary

VEDLIoT addresses the challenge of bringing Deep Learning to IoT devices with limited computing performance and low-power budgets. The VEDLIoT AIoT hardware platform provides optimized hardware components and additional accelerators for IoT applications covering the full spectrum from embedded via edge to the cloud. A powerful middleware to ease the programming, test and deployment of neural networks to this heterogeneous hardware. New methodologies for requirement engineering, coupled with safety and security concepts, incorporate the new challenges arising from the use of Deep Learning techniques are designed and applied throughout the complete framework. The concepts are tested driven by challenging use cases in key industry sectors like automotive, automation, and smart home.

In addition, an open call for cascaded funding is foreseen to explore new opportunities by extending the application of the VEDLIoT platform to a more extensive set of new and relevant use cases. Typical open call projects leverage VEDLIoT technologies for their own AI-related IoT use case, thereby broadening the VEDLIoT use-case basis and making the overall concept more robust. The envisaged run-time of the satellite projects is in the range of 9 – 12 months, with an average funding of up to 120,000 € (including 25 % indirect costs), at a funding/reimbursement rate of 70 %. More detailed information, including available VEDLIoT technology and the application procedure, is scheduled to be published in early 2022.

[^1]: This publication incorporates results from the VEDLIoT project, which received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 957197.

[^2]: Functional Safety of Electrical/Electronic/Programmable Electronic Safety-related Systems


abstract: When elementary quantum systems, such as polarized photons, are used to transmit digital information, the uncertainty principle gives rise to novel cryptographic phenomena unachievable with traditional transmission media, e.g. a communications channel on which it is impossible in principle to eavesdrop without a high probability of being . With such a channel, a one-time pad can safely be reused many times as long as no eavesdrop is detected, and, planning ahead, part of the capacity of these uncompromised transmissions can be used to send fresh random bits with which to replace the one-time pad when an eavesdrop finally is detected. Unlike other schemes for stretching a one-time pad, this scheme does not depend on complexity-theoretic assumptions such as the difficulty of factoring.
author: Charles H. Bennett
(IBM Yorktown) [^1]; Gilles Brassard
(Univ. de Montreal) [^2]
; Seth Breidbart
(Box 1526, NY 10268)
date: November 1982
title: Quantum Cryptography II:

**Note written on 29 June 2014:** This paper was written in November 1982 and originally submitted to the *Fifteenth Annual ACM Symposium on Theory of Computing*, but it was rejected. Shortly thereafter, two of the authors (Charles H. Bennett and Gilles Brassard) discovered what became known as BB84, which seemed like such a better idea that they gave up on resubmitting this earlier work. On the occasion of the 30th anniversary of the BB84 paper, *Natural Computing* has invited us to get this paper finally published. This freshly typeset version is scrupulously faithful to the original 1982 submission, except for the correction of about one dozen typographical mistakes and a few footnotes written in retrospect by the authors.

# Introduction

In conventional information theory and cryptography, it is taken for granted that a digital message can always be copied easily, even by someone ignorant of its meaning. Analog messages (e.g. handwritten signatures) are somewhat harder to copy, but not really infeasible, and digital data can be protected to a considerable extent by interposing a restrictive hardware interface between the data and the outside world (e.g. smart credit cards); but in both these cases, the difficulty of copying is only technological, not fundamental. However, when elementary quantum systems such as polarized photons are used as the transmission medium, routine copying of messages is no longer possible even in principle. In particular, there are ways of encoding messages so that they can be copied reliably only with the help of certain key information used in forming the message.

Quantum coding was first described in , along with two applications: making money that is in principle impossible to counterfeit, and multiplexing two or three messages in such a way that only one can be read. More recently , quantum coding has been used in conjunction with public key cryptographic techniques to yield several schemes for unforgeable subway tokens. Here we show that quantum coding considerably enhances the usefulness of another standard cryptographic device, the one-time pad.

Mathematically, a polarized photon acts like a two-bit read-once memory one of whose bits ($k$) serves as a read key for the other ($m$). Querying the memory with the correct $k$ yields the correct value of $m$. Querying with the wrong $k$ yields a random bit instead of $m$, and in either case querying resets the memory so that subsequent queries yield no new information. Even after a query, it is generally impossible to infer the initial state of bit, because the memory gives no indication of whether its response was the correct response to the correct key or a random response to the wrong key. Because it represents the behavior of an elementary quantum system, this kind of restricted-access memory should be thought of as a natural information-processing primitive, not as a complex technological device that could probably be circumvented in principle.

Ordinarily, when one thinks of a technological restricted-access memory, one has in mind an information-storage device. Photons can also be stored (e.g. between mirrors, or in a closed optical fiber), but they cannot in practice be stored for very long, and their natural application is in the transmission of information. We thus have a situation in which restricted-access memory, as a storage device, is possible in practice but not in principle via conventional technology, and in principle but not in practice via storage of polarized photons. On the other hand, restricted-access transmissions, which can be read or copied only with the help of a key, are possible both in principle and in practice using polarized photons.

# Essential Properties of Polarized Photons

Polarized light can be produced by sending ordinary light through a polarizing apparatus such as a Polaroid filter or Nicol prism. A beam of polarized light is characterized by its polarization axis, which is determined by the orientation of the polarizing apparatus in which the beam originates. polarization is a continuous variable, and in principle can be measured as accurately as desired by passing the polarized beam through a second polarizing apparatus, the uncertainty principle forbids measurements on any photon from revealing more than one bit about the beam's polarization. In particular, if a beam with polarization axis $\alpha$ is sent into a polarizer oriented at angle $\beta$, the individual photons behave dichotomously and probabilistically, being transmitted with probability $\cos^2(\alpha - \beta)$ and with the complementary probability $\sin^2 (\alpha - \beta)$. The photons behave deterministically only when the two axes are parallel (certain transmission) or perpendicular (certain absorption).

If the two axes are not perpendicular, so that some photons are transmitted, one might hope to learn additional information about $\alpha$ by measuring the transmitted photons again with a polarizer oriented at some third angle; but this is to no avail, because the transmitted photons, in passing through the $\beta$ polarizer, emerge with exactly $\beta$ polarization, having lost all memory of their previous polarization $\alpha$. Any other elementary two-state quantum system, such as a spin-$1/2$ atom, behaves similarly dichotomously and probabilistically.

Another way one might hope to learn more than one bit from a single photon would be not to measure it directly, but rather somehow amplify it into a clone of identically polarized photons, then perform measurements on these; but this hope is also vain, because such cloning can be shown to be inconsistent with the foundations of quantum mechanics .

# Quantum Coding

In order to encode a message bit $m$ into a photon that can be read or copied reliably only with the help of a key bit $k$, we generate a photon with a selected one of the four polarization directions $0$, $45$, $90$ and $135$ degrees. \[Generating a single photon of known polarization is possible by variation of the Einstein-Podolsky-Rosen setup , in which a decaying atom emits two oppositely polarized photons. By polarizing and counting one photon, the other's presence is assured and its polarization fixed without measuring it directly.\] If the key bit is a $0$, then the photon is polarized rectilinearly, i.e. $0$ or $90$ degrees according to whether the message bit is $0$ or $1$. If the key bit is a $1$, then the photon is polarized diagonally, i.e. $45$ or $135$ degrees according to the message bit.

*Def.* The *quantum encoding* $Q_K (M)$ of a message $M$ by a key $K$ of equal length is the train of photons obtained by applying the above procedure bitwise to $M$ and $K$.

To read a quantum-encoded message with the help of its key, one simply reads each photon with a polarizer oriented so as to cause it to behave deterministically, for example, reading the rectilinear photons with a $0$-degree polarizer and the diagonal photons with a $45$-degree polarizer. An attempt to read a photon with the wrong key causes it to behave randomly, losing its stored information. For example, if a $45$-or $135$-degree photon is read with a $0$-degree polarizer, it will be transmitted with $50$ per cent probability in either case, and all evidence of its original polarization will be lost.

Suppose an eavesdropper intercepts and attempts to read a quantum transmission $Q_K(M)$ without being detected. Consider first the case in which the message $M$ and key $K$ are both random. Not knowing $K$, the eavesdropper makes the wrong measurement on half the photons, and thus obtains a message $M'$ differing from $M$ in $1/4$ of its bit positions (of course the eavesdropper does not know which ones). Having destroyed the original transmission by reading it, the eavesdropper must now, in order to remain undetected, inject a forged transmission designed to approximate the intercepted one as well as possible. Not knowing which measurements are wrong, the eavesdropper's best strategy is to produce a new train of photons in agreement with the results of the measurements, as if they had all been right. Half of the photons in such a forged transmission will be correct; the other half have wrong key values (i.e. will be diagonal when they should be rectilinear, or vice versa), and when subsequently measured with the correct key by the intended receiver, these will give wrong answers half the time. Thus the error probability is $1/4$ per bit, both for reading the quantum transmission without knowing its key, and for having a forged replacement agree with what the original message would have said when decoded by the intended receiver. Of course, if the intended receiver knew only $K$ but had no prior knowledge of $M$, the eavesdropping would still [^3] go undetected, since a random message with random errors still looks random. Quantum money  corresponds to the case where the intended receiver (the bank) has perfect knowledge of both $M$ and $K$, while the counterfeiter knows neither. The usual message $M$ sent over communication channels is intermediate between these extremes: the receiver has partial prior knowledge of it (e.g. expecting it to be in English).

Simply encoding an arbitrary message $M$ with a random quantum key $K$ has two disadvantages: 1) if the message is too random the receiver won't be able to detect eavesdropping, for the reason mentioned above; 2) if the message is too redundant (e.g. English), eavesdropping will be detected, but by then the eavesdropper will have gained significant information about the message, perhaps even enough to decrypt it uniquely, because eavesdropping errors in only $1/4$ of the bits. (In this respect quantum coding differs from ordinary one-time pad encryption, where ignorance of the key prevents the eavesdropper from learning anything about the encrypted message,[^4] though of course it can be freely copied.)

We now define a stronger kind of coding that overcomes both these disadvantages. The trick is to make the message redundant with an error-detecting code $M \rightarrow E(M)$, then hide the redundancy from the eavesdropper by an ordinary one-time pad $J$, before applying quantum coding.

*Def.* For any error-detecting code $E$ (assumed known to the eavesdropper) let the *strong* *quantum code* $S^E$ be defined as follows: let $J$ and $K$ be two random key strings of length $|E(M)|$ not known to the eavesdropper.[^5] Then the *strong quantum encoding* $S^E_{J,K}(M)$ of message $M$ is the train of photons $Q_k(J \text{ xor } E(M))$.

It is obvious (because of the one-time pad $J$ ) that the eavesdropper can learn about $M$ from $S^E_{J,K}(M)$. Moreover, for suitable error-correcting codes,[^6] eavesdropping a high risk of being detected. Even the rudimentary code of repeating the message twice $E(M)=MM$ suffices to detect eavesdropping with probability at least $1-0.79^k$ when $k$ photons have been intercepted, quite close to the optimum $1-0.75^k$ implied by the independent, probabilistic nature of eavesdropping-induced errors.

Although the simple code $E(M)=MM$ is nearly optimal for eavesdropping-detection, a more complex code would be preferable for another reason: the detection of deliberate message alteration. Although randomly quantum-coded photons cannot be read reliably, they can be altered reliably. For example, the polarization axis of a photon can be rotated by $90$ degrees, without measuring or otherwise disturbing it, by passing the photon through an appropriate sequence of mirrors (or, more mysteriously, through a sugar solution). If this manipulation were applied to the first and st photons of a $2n$-photon transmission coded as above, both would be altered with certainty in such a way as to induce an undetected alteration in the message. A more complex error-detecting code, e.g. concatenating $MM$ with a check sum of the addresses of the ones in $M$, would make such alterations unlikely to escape detection. In the next section, where quantum transmissions are used to carry key information for future transmissions, it will be necessary to use an error-correcting code [^7] that provides some 'diffusion', in the sense of making each bit of $E(M)$ depend on many bits of $M$. This prevents the eavesdropper who has luckily guessed a few bits of the present key from thereby efficiently inferring any bits of future keys. Finally, in section , we will need a code $E$ that corrects errors as well as detecting them, to make up for photons that arrive at the receiver but fail to be detected.

#

We consider a situation in which two users of an insecure communications channel, who initially share a finite secret key, wish to communicate as long as they can. In a classical channel, where eavesdropping is undetectable in principle, they must assume that all their communications are being listened to, and the volume of safe communication is only in the size of the key, unless they resort to pseudorandom key-expansion schemes , which are at best (assuming $\text{P} \neq \text{NP}$) only computationally secure.

We show that by strongly quantum-coding their messages with suitable error-detecting codes, the sender and receiver can safely reuse the same $J$ and $K$ keys indefinitely until an eavesdrop is detected. (The safety is not absolute. There is an exponentially small chance ($O(2^{-|K|})$) that the eavesdropper, having guessed the entire quantum key $K$ correctly, will be able to eavesdrop on all the transmissions without detection and go on to break the reused $J$ key in the usual manner, as well as a moderate chance for the eavesdropper to learn a few bits of the $K$ key correctly and go on to intercept and decrypt a few bits of each message; but these risks do not increase with the number of times an apparently secure key is reused.) An eavesdropper who tampers with or suppresses messages will also be detected with high probability, as will one who injects false messages.

When an eavesdrop is detected, the sender and receiver can go on communicating with only slightly diminished safety by replacing their compromised keys by fresh random information sent over the channel in previous uncompromised transmissions. With high probability they will be able to continue communicating in this fashion for an exponential ($2^{O(|K|)}$) number of key changes,[^8] unless the eavesdrops become so frequent before then that they are forced to use up key information faster than they can replace it, in which case they will (with high probability) be able to cease communication before any of their transmissions have become uniquely decodable by the eavesdropper.

Because the replacement keys are truly random, rather than being pseudorandomly computed from an original seed, the security of the scheme would not be reduced by allowing the eavesdropper unlimited computing power. Neither would it be compromised by technological improvements in the art of eavesdropping. The scheme does incorporate one technologically unrealistic assumption, viz. that photons can be detected with perfect efficiency (cf. section , where this assumption is not made).

We sketch how these advantages can be achieved. The ability to send many messages without loss of security (when no eavesdropping is detected) follows from the exponential decline of the probability of escaping detection with the number of distinct bit positions ever subjected to eavesdropping, whether these bit positions are listened to all at once, or a few at a time over the course of many transmissions. For this reason, a quantum channel could even be used to safely send arbitrarily many copies of the *same* strongly coded transmission, without the eavesdropper being able to forge it accurately, provided the copies were sent one at a time, each only on confirmation that the preceding one had apparently not been listened to. By contrast, if many identical transmissions were sent all at once, the eavesdropper could intercept them all, reliably determine each polarization by multiple measurements, and then escape detection by forging many trains of photons with the now known sequence of polarizations.

In order to be sure that no key is reused after a detected eavesdropping, the two communicating parties must alternate strictly in their use of each key. Otherwise the eavesdropper could, for example, intercept and absorb a message from A without forwarding it to B and then wait for B to use the same key on a subsequent message. The effect of absorbing a message is thus the same as that of spoiling it through eavesdropping: neither party reuses the key with which it was sent. If the initial body of shared key information included several keys reserved for first use by A and several for first use by B, the parties could alternate in the use of each key without strictly alternating transmissions. Of course if multiple keys were in use, and particularly if some transmissions were being absorbed by the eavesdropper, the communicating parties would have to prefix each quantum transmission by a (cleartext) indication of which key it was encoded with, to avoid reading a message with the wrong quantum key and spoiling it.

The ability to change keys without serious loss of security depends on using a somewhat diffusive error-detecting code when new key information is transmitted. With the simple non-diffusive code $E(M)=MM$, an eavesdropper who by good luck has correctly guessed the first and st bits of the current $J$ and $K$ keys will know what measurements to make to reliably read and forge the corresponding bits of a fresh pair of random keys $J'$, $K'$ when these are sent through the channel in four transmissions strongly encoded by $J$ and $K$; as well as confirming, by the consistency of decoding of the error-detecting code, that the guessed bits of $J$ and $K$ are indeed correct. Subsequent lucky guessing on further generations of keys (along with unlucky guessing causing some keys to be rejected due to detected eavesdropping) could be used to discover additional bits until, in linear time, some pair of keys $J''$ and $K''$ became entirely known.

To delay this collapse for an expected exponential number [^9] of key changes $2^{O(n)}$, it suffices to use an error-detecting code that diffuses information about each bit of its argument $M$ among many bits of its value $E(M)$; so that knowledge, say, of any $n/4$ bits of $E(M)$ reveals little or nothing about any bit of $M$. Many error-detecting codes have this property, e.g. a random mapping from $n$-bit strings to $2n$-bit strings, or the linear code obtained by mod-$2$ multiplying $MM$ by an appropriate nonsingular matrix. With a diffusive code, knowledge of a few bits of $J$ and $K$ would not enable the eavesdropper to make reliable measurements of any bits of the replacement keys $J'$ and $K'$.

# A Practical Implementation

Although visible light photons can be polarized with nearly perfect efficiency (e.g. a Nicol prism can split a beam into two beams, very nearly perfectly polarized at right angles to each other, whose total intensity is scarcely less than that of the incoming beam), and transmitted with nearly perfect efficiency (in a vacuum the only significant losses are due to diffraction, and these can be made negligible by using a beam diameter considerably greater than the square root of the product of the transmission distance and the wavelength of light), current technology allows them to be detected with only about thirty per cent efficiency.[^10]

Fortunately, the scheme of the preceding section can be modified to accommodate finite detector efficiency, at the cost of using a more complicated error-correcting code $M \rightarrow E(M)$ in place of the error-detecting code, and a more complicated criterion for key rejection than the detection of a single error on decoding $E(M)$. Somewhat surprisingly, the modified scheme remains secure against an eavesdropper with a more efficient, or even perfectly efficient, photon detector. The volume of safe communication for this scheme is more than linear, but may be less than exponential, in the initial key size.

The main modification is to use standard faint pulses of polarized light instead of single photons, each pulse being of such a size that when it is sent into a detector of the given efficiency (e.g. $30$ per cent), or split into several fainter pulses (e.g. by a half-silvered mirror, or a Nicol prism) and sent into several such detectors, the total number of photons detected obeys a Poisson distribution of mean $1$. Such a standard faint pulse can easily be produced by filtering a standard bright pulse of polarized light to reduce its intensity by the requisite constant factor.

A standard faint pulse of a given polarization resists copying nearly as well as a single photon would. The best strategy for an eavesdropper to copy a faint pulse is to use a half silvered mirror and two Nicol prisms to split the incoming pulse into four beams, one of each canonical polarization, and monitor each beam by a photon detector. Most of the time, only one of the detectors will register a photon, and the eavesdropper will be no better off than in the single photon case. Occasionally two or three detectors will register, yielding more information. Only when three detectors register will the pulse's polarization be known unambiguously (e.g. if both diagonal detectors and the vertical detector register, then the pulse must have been vertically polarized). The faint pulse works well because the chance of three detectors responding to the same pulse is only about $2$ per cent (for a Poisson distribution of mean $1$). The other $98$ per cent of the time, the eavesdropper does not learn the pulse's polarization unambiguously, and, as with single photons, cannot reliably copy it. Even a technologically advanced eavesdropper, with perfectly efficient photon detectors, could not copy faint pulses reliably. For example, if the advanced eavesdropper uses $100$ per cent detectors to analyze a pulse intended for $30$ per cent detectors, an average of $3.3$ photons will be detected per pulse, but the chance that these will appear in three different beams, and thus reveal the pulse's polarization unambiguously, would still be only about $25$ per cent.

The converse phenomenon, namely statistical failure to detect even one photon when a pulse arrives, requires that the rejection test be made more complicated. Even if a transmission is not subjected to eavesdropping, about $1/e$ of its light pulses go undetected, due to normal bad luck at the detectors. The rejection test must begin by deciding whether the number of missing light pulses is so great as to raise the suspicion of eavesdropping (a wise eavesdropper now might not bother to forge replacements for the intercepted pulses, but instead let them remain missing, hoping to pass them off as pulses that arrived but were not detected). If the number of missing pulses is not too great, the error-correcting code must reliably restore the data they would have carried, as well as checking for polarization , which as before would indicate interception and forgery of some of the pulses. A convolutional code  appears most suitable for achieving the desired high efficiency of error-correction in a channel with a large number of erasures ($1/e$). on the purity of polarization available from the Nicol prisms, the code could be made to tolerate and correct a small number of polarization errors, but reject a larger number as evidence of forgery. Since the capacity of a binary channel with $1/e$ erasure probability is $0.632$, a four-fold expansion in $E(M)$ offers ample room for efficient error detection and correction. This in turn means that eight transmissions, each containing $n$ fresh key bits, would have to be accepted to replace the $8n$ bits sacrificed in a rejection.

The most problematical aspect of the modified scheme is the decision of when to reject a transmission and change keys. By contrast with the scheme of the previous section, it is now necessary to change keys periodically (at least every $n^{1/2}$ transmissions) even in the absence of any evidence of eavesdropping, in order to prevent an eavesdropper from intercepting all of the bit positions, a few at a time, over the course of many apparently safe transmissions. The expected number of safe key changes has not been worked out, but it is not implausible that it is still exponential in the key size.[^11]

# Acknowledgements

We wish to thank Stephen Wiesner for numerous helpful discussions of quantum theory, John Denker for drawing our attention to the analogy between choice of basis (e.g. rectilinear vs. diagonal) and a cryptographic key, Greenberg for pointing out that photons in flight could be used to test a channel for eavesdropping, and Lalit Bahl for advice on error-correcting codes.[^12]

[^1]:  2014 update on email: `firstname.lastname@example.com`.

[^2]:  2014 update on email: `email@example.com`;

[^3]:  This word ("still") appears to be superfluous. The authors do not understand in 2014 what they could have meant by it when they wrote the original 1982 manuscript.

[^4]:  In 2014, the authors realize that the phrase "the encrypted message" was ambiguous and confusing. They intended it to mean the "the message whose meaning had been concealed by encryption"—i.e. the plaintext—rather than what would nowadays be seen as its more likely meaning in a cryptologic context, "the message in encrypted form"—i.e. the ciphertext. Eavesdropping on a classical one-time pad transmission of course yields complete information on the ciphertext but none on the plaintext.

[^5]:  In 2014, the authors noticed a possible ambiguity in this sentence. It is the random key strings $J$ and $K$ that are unknown to the eavesdropper, not their length $|E(M)|$.

[^6]:  In 2014, the authors noticed that they had meant "error-detecting codes" here.

[^7]:  In 2014, the authors noticed that they had meant "error-detecting code" here as well.

[^8]:  Note added in 2014: this 1982 use of the asymptotic notation "$O$" was an example of the common physics usage, where it may mean either an upper or lower bound depending on context. Here, we intended a lower bound, which in modern computer science usage would be denoted $2^{\Omega(|K|)}$.

[^9]:  Note added in 2014: as in the previous footnote, modern computer science usage would have us write an expected $2^{\Omega(n)}$ number of key changes.

[^10]:  Note added in 2014: this was the approximate quantum efficiency of photomultiplier tubes available in 1982.

[^11]:  Decades after these words were written, the basic idea behind this paper was reinvented *independently* by Ivan Damgård, Thomas Pedersen and Louis Salvail, but they worked out the complete analysis of its security, which is missing here. Fittingly, these two papers will appear together in a special issue of *Natural Computing* celebrating 30 years of BB84.

[^12]:  We also thank Ilana Frank Mor for typesetting this paper in 2014 from the original 1982 manuscript and for detecting most of the typographical mistakes that have been corrected here.


abstract: High Energy Nuclear Physics (HENP) collaborations' experience show that the computing resources available at a single site are often neither sufficient nor satisfy the need of remote collaborators eager to carry their analysis in the fastest and most convenient way. From latencies in the network connectivity to the lack of interactivity, work at distant computing centers is often inefficient. Having fully functional software stack on local resources is a strong enabler of science opportunities for any local group who can afford the time investment. The situation becomes more complex as vast amount of data are often needed to perform meaningful analysis.
  .
  Prague's heavy-ions group participating in STAR experiment at RHIC has been a strong advocate of local computing as the most efficient means of data processing and physics analyses. To create an environment where science can freely thrive, a Tier 2 computing center was set up at a Regional Computing Center for Particle Physics called "Golias". It is the biggest farm in the Czech Republic fully dedicated for particle physics experiments.
  .
  We report on our experience in setting up a fully functional Tier 2 center leveraging the minimal locally available human and financial resources. We discuss the solutions chosen to address storage space and analysis issues and the impact on the farms overall functionality. This includes a locally built STAR analysis framework, integration with a local DPM system (a cost effective storage solution), the influence of the availability and quality of the network connection to Tier 0 via a dedicated CESNET/ESnet link and the development of light-weight yet fully automated data transfer tools allowing the movement of entire datasets from BNL (Tier 0) to Golias (Tier 2). We will summarize the impact of the gained computing performance on the efficiency of the local physics group at offline physics analysis and show the feasibility of such a solution that can used by other groups as well.
address: Nuclear Physics Institute
Academy of Sciences of the Czech Republic
Prague, Czech Republic; Brookhaven National Laboratory
Upton, USA
author: Petr Chaloupka, Pavel Jakl, Jan Kapitán and Michal Zerola; Jérôme Lauret for the STAR collaboration
title: Setting up a STAR Tier 2 Site at Golias/Prague Farm

# Overview of STAR experiment computing

The STAR (Solenoidal Tracker At RHIC) experiment at Brookhaven National Laboratory (BNL) is a collaboration of over $500$ scientists from 55 institutions in 12 countries. It is the largest high energy nuclear experiment in the world currently taking data. One of RHIC's goals is to study nuclear matter at extreme conditions (pressure and temperature) similar to those that existed in the very early universe by colliding heavy ion nuclei. In parallel with this program, with increasing importance, there is also a unique proton-proton program to study proton spin structure. To achieve these ambitious tasks, STAR collected large volumes amount of data since its first run in year 2000 and the rate at which data are accumulated is increasing. Over the 9 years of running, STAR has collected $~7$PB of physics data in total.

In order to remain on the forefront of nuclear research, the STAR and RHIC are undergoing upgrades heading toward high-luminosity beams and detectors focused to harvest the fruits of the most challenging physics topics. The STAR upgrades will include new detector instrumentation targeted to enhance STAR's acceptance, particle identification capability, and effective sampling of luminosity. In particular, new front end electronics for the STAR Time Projection Chamber (TPC) and data acquisition readout electronics will lead to an order of magnitude increase in the bandwidth for acquisition of minimum bias data and operation for rare triggers with nearly zero dead time, increasing the effective luminosity. These improvements will lead to a dramatic increase in the volume of data accessible for physics research with these probes by 60-70% in heavy ion and proton-proton interactions.

To capitalize on these investments, it is essential that the computing capability of the STAR experiment, now and into the future, be strategically positioned to receive and analyze the flood of data which the upgraded STAR detector will collect. Part of the plan to accomplish smooth delivery of physics results is to leverage the use of Tier 2 computing centers for user's analyses. This approach utilizes locally available storage and computing power and increases the productivity of scientific work at a favorable hardware and human resource cost affordable by the local group.

# STAR computing model

Located at BNL, the RHIC Computing Facility (RCF) at BNL serves as a Tier 0 center for the STAR experiment. Its primary mission is to provide storage resources for mission critical data as well as at least one pass data reconstruction and some analysis bandwidth at a lower priority. The collected data are stored in the primary mass storage system and reconstructed later at the RCF. The final physics-ready data is then made available to users. At the present time the RCF has 350 computing nodes with 1500 CPUs and 500TB of disk space dedicated to STAR only. The STAR's needs for CPU and storage space (Figure ) are expected to steadily grow hand in hand with the planned increase of the volume of collected data.

Until recently STAR had a single Tier 1 center: the Parallel Distributed Systems Facility (PDSF) located at the National Energy Research Supercomputing Center (NERSC) at LBNL. PDSF's main purpose is to provide STAR with supplemental user analysis cycles and simulation (aka embedding) cycles. In addition STAR will soon open a second Tier 1 center at the Korean Institute of Science and Technology (KISTI) in Daejon, South Korea, a center which has the capacity to serve as a regional data redistribution hub for STAR collaborators in Asia and provide resource boost for data production and simulations.

Although initially STAR mainly relied on centralized user analysis facilities (BNL and PDSF) to provide the bulk of the analysis power for STAR collaborators and scientists, the steadily growing amount of experimental data taken and increased demand for storage and processing power required to reconstruct data caused the computing model to naturally evolve toward multi-Tier structure, as shown in the Figure . As analyses become more complex, individual users compete for available resources along with simulation and reconstruction processes at the main Tier 0 and Tier 1 centers. Since there can be no analysis without reconstructed data, the requirement to allocate resources for reconstructing data in a timely fashion often takes precedence over user analyses. A squeeze-out at Tier 0 and Tier 1 centers motivates local groups with their own resources to set up Tier 2 centers. The STAR software and computing plan partly relies on this migration of analysis toward Tier 2 centers serving as a buffer to resource constraints at primary facilities. Furthermore, it has been empirically observed in STAR that productivity seem to increase as local resources are utilized making such a computationally distributed model even more attractive.

# Paradigm of local computing: bringing data to users

In world wide collaborations such as STAR, users connect to the main computing Tiers from distant locations, often over unreliable and slow networks. This combined with having to share computing resources and disk storage with reconstruction and simulation jobs renders their work inefficient. Especially, interactivity necessary to carry out analysis is hindered as well as the research objective. These problems are even more apparent before major conferences when more computing resources are required by users. However, at many places computing and storage resources are available at smaller local computing farms that can be utilized as Tier 2 centers for data processing and physics analyses of the local physics groups after some basic prerequisites are fulfilled and the STAR software stack is deployed. Although the resources available at Tier 2 sites are modest, these sites strongly leverage the capabilities at the STAR Tier 0 and Tier 1 sites with respect to scientific productivity due to the involvement of scientist end-users in data manipulation and handling. Such an approach allows running analyses and simulation on locally stored copies of data, either full data sets or specifically pre-filtered subsets. The short distance between the user and the computing center makes fast interactive work viable and quick retrieval of computed results becomes possible thus shortening overall turn-around time for scientific papers.

The advantage of a Tier 2 center is not only in increase of the efficiency of scientific work, but also in cost effectiveness. The setup of Tier 2 centers relies on building on already existing infrastructure, such as a network or cooling system, and sharing maintenance costs with other users of the computing farm. As the users are expected to work predominantly with copies of data and performance (number of users) is much lower than what is required for a Tier 0 facility, it is possible to use cheap retail class storage components such as SATA disk arrays to store the data, and further decrease the total computing costs. Example of above mentioned approach is a setup of a STAR Tier 2 site at the already existing infrastructure of Golias farm in Prague.

# The Regional Computing Center for Particle Physics in Prague - the Golias farm

The Regional Computing Center for Particle Physics (Figure ), named Golias, is the biggest site in the Czech Republic. It is mainly dedicated to simulations and data processing for particle physics experiments. The farm resides in the Institute of Physics of the Academy of Sciences of the Czech Republic and provides computing and storage services for particle physics experiments as well as for solid state physics and astrophysics communities. The computing and storage resources consisted of of about 450 CPUs and 50 TB of disc space in the end of 2008 and will be gradually expanded to about 1550 CPUs/3,3 MSI2k and 216 TB of disc space in 2009.

The network connectivity of the farm is maintained by CESNET (Czech Education and Scientific NETwork), a non-profit association founded by universities and the Academy of Sciences of the Czech Republic. Its aim is to conduct research in advanced network technologies and provide network connection to educational and research institutions. Excellent network connectivity with other institutions, both national and worldwide (1 or 10 Gb/s) (Figure ) allows Golias to be integrated to other projects (LCG, EGEE, SAMGrid) through their experiments' specific middle-ware components and offer its capacity to all qualified members of supported projects (ATLAS, ALICE and D0). The installed gLite grid middle-ware components include CE, SE, UI, MON box, site BDII and have been configured with YAIM, integrated into Cfengine used for the management of the local site changes. Since 2005, the site is a certified Tier 2 center of the LCG project and in 2008 signed the Memorandum of Understanding of the Worldwide LHC Computing Grid Collaboration (WLCG). Part of the farm's capacity is available for local users from other projects such as AUGER and STAR and spare cycles can be used by anyone, making the facility a good example of efficient resource usage.

Job management at the Golias farm is performed using the workload management system PBSPro (current version 9.2). Golias is in the process of evaluating alternatives such as Torque and the cluster scheduler MAUI. The hardware, software and network status are extensively monitored using many standard packages including Nagios, Munin, MRTG, IPAC as well as a set of locally developed custom tools tailored to the facility's need.

# Setup of the local STAR computing environment

The local ultra-relativistic heavy ion collisions group from the Nuclear Physics Institute of the Academy of Sciences of the Czech Republic (NPI ASCR) has been participating in the STAR experiment since its very first collisions in 2000. It's main activities have been data analyses in the area of correlations and high momentum physics as well as detector research and development activities.

The road towards productive work was however paved with difficulties. In the beginnings the group had struggled with running analyses on the major Tier centers from its remote location given the slow connectivity over the Atlantic. This was extremely inefficient, as then the network capabilities were an order of magnitude slower than the ones available today. The idea of developing local STAR computing environment has been tested since 2002 and a decisive advance was achieved by joining the Golias farm at 2005, leveraging the knowledge acquired over the years to set up the necessary components for a Tier 2 center. Since then, the STAR physics group has used the Golias farm for short-time and mid-scale analyses. Over this period of time we have tested multiple approaches and identified key components for the successful setup of local Tier 2 computing center. A schematic layout of the STAR Tier 2 at Golias in the configuration described by this article is shown in the Figure .

## Deployment of STAR software framework

First the STAR software framework must be deployed and adapted to the specifics the of local farm. In-situ compilation of the STAR software stack and import of user environment to emulate the RCF is a key aspect making the work at any facility interchangeable hence transparent (learn once, use many times). The deployment of the STAR software stack includes a version of ROOT and a set of add-ons customized by the STAR experiment called "root4star". For the needs of the Golias farm, the ROOT framework was patched to support the RFIO protocol which is used to access locally stored data, as will be explained in the next section. To simplify the selection of data for analysis and subsequent job submission STAR has developed the "STAR Unified Meta Scheduler" (SUMS) tool and middle-ware. Customizing SUMS for the Golias environment requires an adaptation of policies to make use of PBSpro and Torque for scheduling, but from a user stand point, the use of SUMS and its job syntax is identical regardless of the facility, allowing the user to simply migrate their work from the Tier 0 to the Tier 2 center using the same job description files.

## Data storage,transfer and aggregation strategy

Data management was found to be a key component of our local computing strategy. The overall aim is to allow physicists to easily download data sets from the RCF to local storage and also to provide them with a simple and fast way to connect their jobs running on computing nodes of the farm with the storage. As trivial as it may sound, at the same time one requirement was that the access to the data not strain internal network of the farm, as would mounted NFS. This combined with an additional requirement of a low purchase and maintenance cost resulted in the use of the SRM-based Disk Pool Manager (DPM) grid-aware solution developed at CERN. In order to keep the farm maintenance cost low DPM was actually located outside of the farm premises and is connected to Golias via a dedicated 1Gbit/s optical fiber link. The DPM offers various interfaces for data access (RFIO, XROOTD) used in the STAR computing framework. The DPM solution is very convenient for end-users since it affects their jobs in a minimal way and allows for easy scalability and maintenance by the administrators. The DPM manages about 20 TB of disk space and with sophisticated data-mover tools developed by the group at Prague and STAR software staff, physicists can download almost 1 TB of data per 24 hours from BNL.

## Network connectivity

Fast networking available to the users of the Golias farm makes the main difference between working remotely at major Tier centers and working locally at smaller farm for the end-users. As already stated above, the Golias has an excellent connection to other major scientific institutions via CESNET. In order to efficiently transfer data to and from the RCF, a dedicated 1Gbit/s intercontinental link between BNL and Golias was established through cooperation between CESNET, ESNet and BNL. More than a year of operations show that about 1TB of data per day can be easily transferred from BNL and written to DPM storage using SRM transfer tools. Since DPM storage lies outside of the farm the transfer of data to running jobs at working nodes is of a concern. At this point a 1Gbit optic fiber link is setup between the storage facility and computing the farm. Stress tests show that under conditions of typical physics analyses jobs run by the group the DPM is able to supply data to $~200$ running jobs without significant slow down of the jobs. Above this threshold, the IO is a limiting factor which may require the consolidation of the Local Area Network.

## Local database mirror

Database access to the main calibration, geometry and status information database at the local farm is a small, but very helpful addition to the local setup. Even though we have high throughput connection at our disposal, the large distance of jobs from main database located at the RCF caused major prolongations of running jobs due to latencies inherent to the communication protocol. Access to a local database mirror enables running detector simulations and embedding, if needed, at the same speed as at the Tier 0 facility. Since the database mirroring is a fully automatic process done through a MySQL master/slave replication, the local database mirror is a maintenance-free addition to the Tier 2 center.

# Performance

The setup of a STAR Tier 2 center at the Golias farm, in the above described configuration, has been tested by the Prague physics group for more then a year in conditions typical for small to mid-size physics working group. While such a group does not always need high processing power it needs intermittent on-demand access to computing resources. The most common tasks performed at the Golias site are physics analyses on locally stored, usually preprocessed, data sets on the order of $10-1000$GB. With the available connection to the RCF facility, where primary copies of data are stored, users are able to transfer data for their analyses with minimal time delays. Hence, switching between analyses and data sets is possible and a wide range of studies have been carried out locally.

Since the setup of a Tier 2 center must to be cost-effective and simple, it has limitations and future bottlenecks are predicted. We expect to increase the number of running jobs in future increasing the data transfer from the DPM storage. Our disk-array setup of DPM is currently based on a single bus/NIC which may cause a bottle neck for data-hungry jobs in the future. For this reason another separate storage server is being purchased to distribute the disk-space among several loosely coupled disk-servers and balance the overall load. Another limitation follows from the chosen low-cost solution. Our experience shows that this solution works well at a smaller site, like ours, however if Golias Tier 2 were to grow to the size of a regular Tier 1 center, such a solution would be rather human resource hungry and likely inefficient.

# Conclusions and perspectives

The experience with the Tier 2 center in Prague shows that a light-weight distributed computing approach, concentrating on the redistribution of data, can improve analysis viability, sustainability, and reliability and increase the local scientific opportunity by carrying out local analyses and leveraging local human and hardware resources. This approach not only relieves the load from main computing centers (BNL/RCF and NERSC/PDSF) but also takes advantage of faster data access and shorter waiting times for the local users.

In the near future STAR experiment has to move to grid-oriented computing. Such a need is driven by steadily increasing data volumes expected in the next 5-10 years. It is thus very likely that the Golias farm as well as the NPI ASCR local data storage will be used to process grid-submitted jobs.

# Acknowledgements

The authors wish to thank the team of Golias farm for their support of the STAR Tier 2 project. We also thank the BNL computing team for continuous support and development of the off-site computing facilities. We would like to extend our gratitude to people from ESNet, and CESNET.

This work was supported in part by the Grant Agency of the Czech Republic, Grant 202/07/0079 and Ministry of Education of the Czech Republic, Grant LC 07048.

# References