abstract: Urban intersections are prone to delays and inefficiencies due to static precedence rules and occlusions limiting the view on prioritized traffic. Existing approaches to improve traffic flow, widely known as automatic intersection management systems, are mostly based on non-learning reservation schemes or optimization algorithms. Machine learning-based techniques show promising results in planning for a single ego vehicle. This work proposes to leverage machine learning algorithms to optimize traffic flow at urban intersections by jointly planning for multiple vehicles. Learning-based behavior planning poses several challenges, demanding for a suited input and output representation as well as large amounts of ground-truth data. We address the former issue by using a flexible graph-based input representation accompanied by a graph neural network. This allows to efficiently encode the scene and inherently provide individual outputs for all involved vehicles. To learn a sensible policy, without relying on the imitation of expert demonstrations, the cooperative planning task is considered as a reinforcement learning problem. We train and evaluate the proposed method in an open-source simulation environment for decision making in automated driving. Compared to a first-in-first-out scheme and traffic governed by static priority rules, the learned planner shows a significant gain in flow rate, while reducing the number of induced stops. In addition to synthetic simulations, the approach is also evaluated based on real-world traffic data taken from the publicly available inD dataset.
author: Marvin Klimke$^{1,2}$, Benjamin Völz$^{1}$, and Michael Buchholz$^{2}$ [^1] [^2] [^3]
bibliography: references.bib
title:  **Cooperative Behavior Planning for
  Automated Driving Using Graph Neural Networks**

# Introduction

Urban traffic regularly exhibits disturbances and inefficiencies caused by simple traffic management schemes faced with large volumes of vehicles. Especially smaller intersections are typically handled by static priority rules, resulting in vehicles approaching from a minor road having to yield. Moreover, occlusions through buildings or other objects are highly prevalent in urban areas, limiting the view for both human drivers and vehicle-bound sensory systems.

The increasing use of connected vehicles (CVs) and connected automated vehicles (CAVs) opens up new opportunities to increase the traffic efficiency. Those vehicles can announce their presence and possibly share perception data with surrounding road users via a communication link. Moreover, with edge computing resources becoming available in urban areas, it is viable to build and maintain a local environment model of, e.g., an intersection and its surroundings. Such an edge server can distribute the environment model to connected vehicles in the operational area. CAVs, for instance, can make use of the information by incorporating it into their planning algorithms.

In the publicly funded project MEC-View, research on connected automated driving was conducted using a testing site at a suburban three-way intersection in the city of Ulm in Germany . Due to buildings occluding the view onto the priority road, an automated vehicle merging from the side road has to decelerate strongly, before being able to safely enter the intersection based solely on its own perception system. With the support of the environment model provided by the edge server, the automated vehicle can transition smoothly onto the main road, given that appropriate space is available. We build upon this approach and discuss the potential of multi-agent planning schemes that are executed on the server. Based on the fused environment model, a cooperative plan for handling intersection traffic is derived that can be proposed to the connected vehicles as behavioral instructions. Explicit deviations from static priority rules become possible. For instance, vehicles on the main road can be requested to slow down and thus letting a vehicle from the side road merge into the emerging gap, as depicted in Fig. .

Prior research on automatic intersection management (AIM) primarily focuses on non-learning algorithms, like reservation-based or optimization-based programs. At the same time, machine learning-based approaches show remarkable results on prediction tasks in automated driving as well as planning for a single ego vehicle. The lack of fitting ground-truth data prevents the application of supervised learning for cooperative behavior planning. To bridge this gap, the present work proposes to train a reinforcement learning (RL) policy for multi-agent planning in a simulated environment, resulting in the following contributions:

-   Leveraging machine learning to perform cooperative multi-agent planning for urban automated driving,

-   To the best of our knowledge, we propose the first AIM system exploiting graph neural networks,

-   Evaluation based on real-world traffic data taken from a publicly available urban driving dataset.

The remainder of the paper is structured as follows: Section  discusses related work in the field of AIM and machine learning-based planning for automated driving. The proposed behavioral planning scheme is comprehensively introduced in Section . Afterwards, evaluation results obtained in synthetic simulations and based on real-world traffic data are given in Section . Section  concludes the paper and gives an outlook on future work.

# Related Work

The analysis on the state of the art first considers existing approaches to AIM. Because machine learning is seldom used for AIM, we subsequently investigate learning-based behavioral planning methods. Due to the large body of existing literature, we present a selection of commonly used approaches and refer the reader to surveys for a more extensive overview.

## Automatic Intersection Management

Past research brought forth a variety of AIM schemes, surveyed for instance by . The authors identify centralization as a crucial feature for distinguishing AIM schemes. Thereby, a fully centralized scheme exhibits a single coordination unit that is in charge of planning the intersection traversal and acts as the communication partner for all vehicles. In a fully distributed AIM, a cooperative plan is negotiated by the vehicles on their own.

A centralized, reservation-based AIM system is proposed in , which employs a first-in-first-out policy for assigning clearance to cross the intersection. A driver agent places a request when its vehicle is about to enter the monitored intersection area, covering possible conflict points with other paths. The intersection manager maintains tile-based reservations and confirms the request if the affected tiles are free. This approach can be combined with a traffic light to enable the co-usage of the intersection by human drivers and automated vehicles.

Optimization-based intersection management systems are published, for instance, in and . Those works assume full penetration of CAVs that laterally follow predefined lanes on urban intersections. The distributed energy-optimizing approach further disallows turning maneuvers and the utilization of two conflicting paths at the same time. In , the longitudinal control of vehicles is performed by a centralized intersection coordination unit employing a model predictive control (MPC) scheme. Both works demonstrate efficiency gains in time and fuel consumption by comparison to a traditional signalized intersection. A prevalent issue with optimization-based approaches is the unfavorable scaling of computational demand with increasing traffic density. In , a novel AIM scheme is presented, that is also capable of handling mixed traffic, i.e., simultaneous usage by automated and human-driven vehicles.

The concept of platooning can also be used for AIM. Based on a so-called virtual inter-vehicle distance, a pair of vehicles can adapt their velocities to cross their conflict point with sufficient clearance. The authors acknowledge that significant adaptions would have to be made for managing an intersection under mixed traffic.

## Machine Learning-Based Planning

Machine learning-based approaches to automated driving experience rising interest of researchers during the last years. A survey of recent deep reinforcement and imitation learning planning methods for a single ego vehicle can be found in . The authors categorize published works by the type of input data (e.g. sensor measurements or object detections) and output representation (e.g. behavioral planning or direct control outputs). Because individual sensor measurements are not suited for cooperative planning over multiple vehicles, we limit our analysis to methods that require a prior perception system to be in place. Readers interested in machine learning-based prediction for automated driving are referred to comprehensive surveys on the topic, like . On the output side, multi-agent planning requires an intermediate representation that can be passed to various vehicles in the scene, laying the focus on high-level behavioral planning approaches.

Imitation learning describes the application of supervised learning techniques to automated driving by training on expert drivers' demonstrations, which can be obtained from datasets or accordingly equipped testing vehicles. Based on object detections from a dedicated perception system and high-definition map information, a typical approach is to render the surroundings of the ego vehicle in a raster image that is subsequently processed by a convolutional neural network (CNN) . To address the problem of distributional shift between training data and closed-loop test conditions, various improvements have been proposed, like perturbing a random subset of training trajectories to teach the model to recover from atypical states . Being based on supervised learning, those techniques share the large needs for high-quality training data. This limits their prospective transfer to cooperative multi-agent planning because ground-truth data showing cooperative maneuvers is virtually not available. Urban traffic datasets (e.g. the inD dataset ) show road users obeying to static priority rules or traffic lights; both entities that shall become obsolete with cooperative planning.

In contrast to supervised learning, RL approaches evade the requirement for large datasets by instead exploring possible actions in a simulated environment and exploiting a reward signal to learn the desired behavior. In , a driving policy for controlling the acceleration and steering angle is trained through RL that is applied to multiple vehicles in a common simulated environment. As there is no explicit communication between the different vehicles' policies, no cooperation is shown in traffic. Based on a raster image representation, this approach shares the unfavorable scaling of computational load with the number of participants in the scene, because each vehicle requires an individual image, centered on its pose for sensible inference. An alternative RL-based approach to coordinated driving on an urban intersection was published in . By maintaining a tile-based reservation of the intersection, the decentralized policies can choose from the set of actions that do not cause a collision. Apart from this limitation of the action space, there is no further inter-agent communication that could enable cooperative maneuvers.

When encoding the semantic environment of a vehicle in urban traffic, the number of potentially relevant entities (e.g. other vehicles) is highly dynamic. This makes fixed-size network architectures and input representations often used in RL unsuitable for the task at hand. In , it is proposed to encode input features per vehicle using a multilayer perceptron followed by a permutation invariant operation for pooling the resulting features. The aggregated feature vector is then propagated through another fully connected network to finally infer actions for a single ego vehicle. The authors extend their work to encode whole traffic scenes including lanes and traffic signs and compare it to using a graph convolutional network for the same task in . Similarly, proposes to encode the vehicles being present in the scene as graph vertices. However, none of the described works can handle multi-agent planning.

# Proposed Approach

In this section, our proposed approach is presented, beginning with a discussion on learning paradigms for multi-agent usage. Afterwards, the graph-based input representation is introduced, followed by details on the network architecture and reward engineering.

## Learning Paradigm

In RL algorithms, an agent typically interacts with an environment in discrete time steps. The agent observes the current state of the environment and subsequently chooses an action, whose effect is evaluated by a reward signal. With multiple entities to be controlled, one can pursue different learning paradigms depending on the degree of centralization in multi-agent reinforcement learning . Instead of having the various agents interact individually with the environment and learn independent policies, cooperative planning is modeled best by the centralized training centralized execution paradigm. Because different agents, which shall take part in a cooperative maneuver, have to be able to communicate explicitly. This holds not only during training, but also at inference time. In paradigms relying on decentralized execution, the agents would have to learn an implicit communication scheme through their behavior. With the fused environment model being available to the server-side planner, considering the planning problem over all vehicles in the scene using a joint RL agent allows for explicit communication and hence better cooperation.

Like many RL problems, the cooperative planning problem can be denoted as a Markov decision process (MDP), defined as the tuple $(S,\,A,\,T,\,R)$. It consists of a set of states $S$ that fully describe the traffic scene at a given time. $A$ denotes the set of actions the RL agent can choose from while interacting with the environment. The transition function $T(s\,,a\,,s')$ describes the probability of changing from state $s \in S$ to $s' \in S$ when applying action $a \in A$, whereas the reward signal is determined by the function $R(s,\,a)$. Since the multi-agent planning problem contains different vehicles in the scene, the dimensionalities of the state space and the action space depend on the number of vehicles currently present and may vary over time.

## Input Representation

A well-suited input representation is crucial for applying artificial neural networks successfully. We identify three major requirements for an input scene representation to be used in cooperative multi-agent planning:

-   Invariance on the number of vehicles in the scene,

-   Permutation invariance of the input nodes,

-   Permutation equivariance regarding the output nodes.

Simple tabular representations already lack the invariance properties. The limitations of fixed-sized inputs for behavior planning are elaborated more extensively in . A rendered raster image of the scene, as often used for CNNs, fulfills the invariance requirements, but typically requires a target agent to be centered around . This process must be repeated to produce individual outputs for each agent, making the application to a large number of vehicles computationally infeasible. Permutation equivariance means that the inferred outputs of given agents in the scene are independent of their ordering in the input vector. Hence, our work proposes to use a lean and flexible graph-based scene representation, shown in Fig. , which fulfills all above requirements. The current state of the environment is thus defined as $S=(V,\,E,\,U)$, with $V$ being the set of vertices corresponding to the vehicles in the scene and $E$ denoting edges depending on the pairwise relation between vehicles. For each vehicle, one vertex in $V$ stores the corresponding input features. Each of the directed edges is assigned one of two edge types in $U$, either *same lane*, or *crossing*: $$(v_1,\,r,\,v_2) \in E = V \times U \times V.$$ Two vehicles in front of the intersection whose paths cross or merge are being connected bidirectionally with crossing edges, like $v_1$, $v_2$, and $v_3$ in the figure. The same lane edge, in contrast, is used to connect two vertices of vehicles on the same path, pointing from the predecessor to the following one (e.g. $v_6$ and $v_5$). This is motivated by the observation that vehicles should adapt their behavior to the preceding vehicle and not vice-versa. Note that the graph does not need to be connected. Some vehicles may form a disjoint sub-graph, if they share no conflicts with the remaining vehicles, as it is the case for $v_5$ and $v_6$ or $v_7$ in Fig. .

The input feature vector for each vehicle consists of three values and is denoted as $h^{(0)} = [s,\,v,\,d]^T$, where the upper index denotes the layer number. The longitudinal position of the vehicle along its path is denoted by $s$, with $s=0$ defined as the point where the path leaves the intersection area. This corresponds to the longitudinal coordinate of a Frenet coordinate pair. Because different maneuvers (e.g. straight driving and right turns) cause a difference in the path length on the intersection, the effective path length is scaled to a common reference length $s_{\mathrm{ref}}$, as depicted in Fig. . Thereby, the entry point to the intersection area is located at $s=-s_{\mathrm{ref}}$ consistently, ensuring that the localization on the incoming lanes is independent of the maneuver to be driven. Moreover, this normalization makes the scene representation robust to slight changes in intersection geometry.

The second input feature $v$ denotes the scalar velocity of the corresponding vehicle normalized over the speed limit of the lane it is currently driving on. To allow the network to sense immediate proximity of other vehicles, the input features are complemented by a distance measure $d$ based on the Mahalanobis distance . The distance measured from vehicle $i$ to vehicle $j$ is calculated as $$\label{eq:dist}
d_{ij} = \sqrt{(\boldsymbol{p}_j - \boldsymbol{p}_i)^T \boldsymbol{\varSigma}_i^{-1} (\boldsymbol{p}_j - \boldsymbol{p}_i)},$$ where $\boldsymbol{p}_i$ denotes the position of vehicle $i$ in cartesian coordinates. The covariance matrix is given as $$\boldsymbol{\varSigma}_i =
    \boldsymbol{R}_{\psi_i}
    \begin{bmatrix}
        l^2/4 & 0 \\
        0 & w^2/4
    \end{bmatrix}
    \boldsymbol{R}_{\psi_i}^T,$$ with $l=\SI{5}{\meter}$ and $w=\SI{2}{\meter}$ describing the standardized length and width of a vehicle. $\boldsymbol{R}_{\psi_i}$ denotes the 2D rotation matrix using the heading angle $\psi_i$. To determine the input feature for a particular vehicle, the distance to each other vehicle is computed according to , and the inverse of the minimum distance value is passed to the network. Using the inverse instead of the plain distance value proved to yield better model convergence.

## Network Architecture

In the present work, the behavioral control of vehicles is performed by applying a commanded longitudinal acceleration within the range $[a_{\mathrm{min}},\,a_{\mathrm{max}}]$, requiring an RL algorithm suited for continuous control. We propose to use the twin delayed deep deterministic policy gradient (TD3) algorithm , an extension of the deep deterministic policy gradient (DDPG) . Both methods are actor-critic RL algorithms for actions in continuous space. TD3 consists of two function approximators, namely the *actor* and the *critic*. Based on a given state and action input, the critic network is trained to predict the discounted reward as $Q$ value estimates. The actor gets the current environment state as an input and outputs an action to be performed in the particular time step, optimized by using the critic output as the loss.

The proposed graph neural network (GNN) architecture is depicted in Fig. . For each vertex, the low-dimensional input features are first processed by a dense layer `enc`. Note that this operation is performed individually for each vertex using shared parameters, disregarding the graph structure defined by the edges. The encoded vertex features are then propagated through two relational graph convolution layers, `conv_1` and `conv_2`. In contrast to simple graph convolution layers, multiple weight matrices corresponding to the different edge types are used for message passing . During a forward pass, the hidden features of the vertices are propagated along the outgoing edges, while being multiplied by the respective weight matrix. Hence, each node receives a variable amount of such messages that have to be integrated into its own feature vector. This is done using a permutation invariant aggregation function like the element-wise maximum, mean, or sum. In the present work, the maximum operation delivered the best results. The update for the hidden feature vector of node $i$ is thus given as $$\boldsymbol{h}_i^{(l+1)} = \sigma \left( \Sigma_{r \in U} \max_{j \in \mathcal{N}_i^r} \boldsymbol{W}_r^{(l)} \boldsymbol{h}_j^{(l)} + \boldsymbol{W}_0^{(l)} \boldsymbol{h}_i^{(l)} \right),$$ where the set of neighbor nodes connected to the target node $i$ by incoming edges is denoted by $\mathcal{N}_i^r$. The weight matrices for each edge type $r \in U$ are called $\boldsymbol{W}_r$, while the previous target node vector is multiplied by $\boldsymbol{W}_0$. In case a vertex has no incoming edges (like $v_4$ or $v_7$ in Fig. ), the node update coincides with a single dense layer on the node's own state. Finally, the resulting feature vector is passed through a non-linear activation function, empirically chosen as rectified linear unit (ReLU).

While the actor network and the critic network are constructed analogously up to the point described above, they differ in the output layer, depicted on the right side of Fig. . The actor network is responsible for deriving an action for each entity in the scene to be executed for a given time horizon, while the critic is in charge of estimating the Q value for the entire graph. With each vehicle being represented by a vertex in the graph, there is one regression target per vertex that describes the commanded acceleration for the corresponding vehicle in the actor network. The latent feature output by the GNN is reduced to a single unit using a final dense layer `a_dec`, whose weights are shared across nodes. To limit the action output to a defined range, a tangens hyperbolicus activation function is used on the output layer. The normalized value range is subsequently mapped to an acceleration between −5 m s^−2^ and 5 m s^−2^. With the critic network's output being a performance measure in form of a single Q value estimate, an aggregation function for the latent feature vector of all nodes is required. The Q values' range is not limited, hence a final dense layer `a_dec` with linear activation is used as the output layer. Because the critic network requires the chosen action in addition to the state representation, the action values are concatenated with all vertex input features.

## Reward Engineering

Apart from the network architecture described above, the RL algorithm requires a reward scheme to learn a reasonable behavior within the simulation environment. The reward signal is composed of a weighted sum of reward components $$R = \sum\limits_{k \in \mathcal{R}} w_k R_k,$$ where the set of reward components is given as $\mathcal{R} = \{\mathrm{velocity,\,action,\,idle,\,proximity,\,collision}\}$. The velocity reward is the main driver for learning a non-trivial solution through rewarding large velocities and is defined as $$R_\mathrm{velocity} = \begin{cases}
    1.25 \frac{v}{v_\mathrm{lim}}      &       \frac{v}{v_\mathrm{lim}} \le 0.8 \\
    1.0                                & 0.8 < \frac{v}{v_\mathrm{lim}} \le 1.0 \\
    6.0 - 5.0 \frac{v}{v_\mathrm{lim}} & 1.0 < \frac{v}{v_\mathrm{lim}},
    \end{cases}$$ where $v_\mathrm{lim}$ describes the vehicle's lane speed limit. Regularizing the model against applying large acceleration magnitudes is done by the action penalty that is defined as the negative absolute commanded acceleration. When striving to avoid collisions, the simplest solution is to stop the whole traffic, which is not desirable. Therefore, the idle penalty is set to $R_\mathrm{idle}=1$ in case all vehicles are standing still. To teach the model to keep suitable safety distances to nearby vehicles, the proximity component is used to penalize actions that cause two vehicles to get dangerously close. This penalty is calculated based on the aforementioned modified Mahalanobis distance measure (cf. ), which takes the relative direction of the obstacle into account. In the case that two vehicles collide, the collision penalty $R_\mathrm{collision}=1$ is used to let the model implicitly learn collision avoidance, while aborting the episode on the spot preventing further positive rewards from being accumulated.

# Experiments

Training and evaluation of RL algorithms require a suited simulation environment. For the task of behavioral planning in automated driving, the simulator should at least provide a kinematic vehicle model and reasonable interaction between vehicles. In the present study, the open-source environment Highway-env is used and slightly adapted to be employed for centralized multi-agent planning. The simulation of vehicle kinematics is done according to the kinematic bicycle model , which suffices for behavioral planning. The graph-based scene representation and graph neural network layers are based on the PyTorch Geometric API .

| **Reward** | velocity | action | idle | proximity | collision |
|:-----------|---------:|-------:|-----:|----------:|----------:|
| **Weight** |     0.03 |   0.01 | 0.01 |       0.2 |       1.0 |

Reward weights

The choice of reward weights is based on a grid search that was conducted on a reduced variant of the simulation environment resembling the key behavior, while being much less computationally demanding. The reward weights used throughout this study are given in Table . During training, the latest model is evaluated on a separate validation environment for ten episodes every 5000 time steps. The validation environment is constructed the same way as the training environment but initialized with a different seed. Each time the validation shows a new best validation reward, the current model parameters are saved to disk.

We benchmark our approach against two baselines in synthetic simulation: static priority rules (PR) and a first-in-first-out scheme (FIFO), resembling the currently prevalent approach in real-world and a seminal AIM scheme. Traffic obeying to static priority rules is simulated using the driver models provided by Highway-env, which rely on the intelligent driver model (IDM) for longitudinal control and additional logic for handling intersections. We applied minor tweaks to the driver models to obtain reasonable results for more dense traffic:

-   The derivation of the commanded acceleration is modified to correctly handle a target speed of zero.

-   Scheduling at an intersection is based on vehicle priorities that are inferred from their intended maneuver instead of the current lane priority.

-   The yielding logic is extended to respect a specific stop point in front of the intersection.

The FIFO scheme, on the other hand, prioritizes the incoming vehicles based on their distance to the intersection. Thereby, non-conflicting paths can be used at the same time, possibly allowing multiple vehicles on the intersection at a given time. Note that this policy does not enforce a strict FIFO ordering on the whole intersection, but rather groups of conflicting paths, which leads to a considerable performance increase.

The way of generating simulated traffic differs between training and evaluation runs. During training, vehicles are being spawned on all incoming lanes featuring enough space at a certain probability with their destination also being chosen randomly. In the course of training, the spawn probability is continuously increased until it saturates at 5 % per time step. Thereby, the intersection is kept busy and allows the RL algorithm to obtain meaningful data samples to learn from. The evaluation runs are based on scenario definitions that are generated using a slightly different scheme. In that case, the time between vehicles appearing on a particular lane is governed by a shifted exponential distribution. This resembles a Poisson process, where the shift on the distribution of the spawn period ensures a minimum distance between vehicles. If a traffic jam has formed so that there is no space for a vehicle to be spawned, the generation is suspended to prevent immediate collisions. For each scenario to be generated, the desired vehicle rate is chosen randomly from a uniform distribution over the interval $[0.2,\,0.4]$ vehicles per second and major road lane. The vehicle rate on the minor road lanes is set to half of that value. In both training and evaluation, the initial vehicle velocity is chosen uniformly between 60 % and 100 % of the corresponding lane speed limit.

## Synthetic Simulation

All experiments within this subsection were performed on a four-way intersection, whose layout is analogue to the one depicted in Fig. . The *flow rate* describes the number of vehicles that cross an intersection (or other road infrastructure) during a given time frame. Figure  shows the flow rate distribution over 100 evaluation runs (each of 100 s length) of varying traffic density. It can be observed that already the rather simple FIFO scheme achieves a benefit over static priority rules, while the learned RL planner outperforms both baselines regarding the median values. To further investigate the performance gain, we analyze the ratio of vehicles that had to stop during the maneuver. A vehicle trajectory is considered to contain a stop, if the velocity falls below 0.3 m s^−1^ for at least one time step. This threshold was chosen due to numeric reasons. By categorizing the vehicles by their incoming road priority, the effect on traffic approaching from a minor road becomes apparent, as depicted in Fig. . Clearly, the static priority rules induce a significant traffic buildup on the minor road that forces nearly all vehicles to stop. The FIFO policy manages to let more vehicles from the minor road pass the intersection, but in turn causes a large proportion of stops also on the major road. In contrast, the RL planner succeeds to get a large amount of vehicles across the intersection, while keeping the traffic flow mostly intact. This behavior might be explained by the planner's learned ability to adapt the vehicles' velocities early to fit into an emergent gap. It should be mentioned that the RL planner cannot completely eliminate collisions, as denoted in the first row of Table . However, the occurrence is extremely rare, making it very challenging to further reduce them, given the RL algorithm has to implicitly learn it via the reward signal. In practice, the remaining failure cases are not an issue, because the cooperative maneuver will only be advertised to the connected vehicles, if it fulfills sanity checks like being collision-free. In the case no viable cooperative plan was found, the vehicles simply resort to local planning.

| **Intersection** | **Priority rules** | **FIFO scheme** | **RL planner** |
|:-----------------|-------------------:|----------------:|---------------:|
| Synthetic 4-way  |              0.0 % |           0.0 % |        0.028 % |
| inD              |              0.0 % |         1.918 % |        0.584 % |

Collision rates

## Simulation Based on Real-World Traffic Data

Apart from the simulation based on synthetic data, we also evaluated the cooperative planning scheme on real-world urban traffic data taken from the inD dataset . The dataset contains tracks of vehicles and vulnerable road users that were recorded at four urban intersections in Germany. We selected a four-way intersection connecting a priority road with a minor road that is managed by static priority rules. The major road also features isolated lanes for turning left, as depicted in Fig. . Simulating the traffic according to the cooperative planning approach on this intersection makes the following assumptions inevitable. Firstly, the intersection geometry is only approximated in simulation. However, this is not an issue for behavioral planning, which is mostly independent of road geometry. Considering the road curvature, vehicles may not traverse it with arbitrary speed, which is ensured by defining lane-dependent velocity limits. As the dynamic properties of the vehicles shown in the dataset are unknown, a default parameter set is used in simulation. Moreover, the real-world tracks deviate from the lane center lines that are used for guiding the simulated vehicles. The recorded intersection is also used by vulnerable road users like pedestrians and bicyclists that cannot be modeled in the simulation as of now.

Compared to the synthetic simulation, not all metrics are viable for evaluation when using the dataset as the baseline. The flow rate, for instance, cannot be improved by any intersection management system, because the number of vehicles in the scene is specified by the dataset. In total, the used dataset excerpt provides 2446 vehicle tracks over a recording time of 3.08 hours, resulting in an average traffic density of 794  h^−1^ or 0.221  s^−1^. Compared to the flow rates obtained in the synthetic simulations, those numbers are rather small, which might indicate that there are not many interesting situations during most of the recording. Evaluating the RL planner and FIFO policy on real-world data is performed by spawning vehicles in simulation according to the appearance time in the dataset and subsequently simulating their motion based on the vehicle models.

Figure  compares the number of vehicles that had to come to a complete stop categorized into the maneuvers left turn, straight driving, and right turn. The total number of vehicles being managed by the FIFO policy and the RL planner has to be identical to the amount given by the dataset recording. It is clearly visible from the real-world data that many left turning vehicles have to come to a stop before being able to safely pass the intersection. Note that although one minor road access is governed by a stop sign, the real-world recordings show that by far not all road users obey to it. The FIFO scheme naturally distributes the stops to all maneuvers including vehicles driving straight on. Meanwhile, the RL planner is able to avoid the vast majority of stops and maintains a smooth flow of traffic.

As it can be seen in the last row of Table , both the FIFO and the RL planner suffer from an increased collision rate. This can, at least in parts, be attributed to the way the real-world tracks are mapped to simulation. Especially the minor road is only depicted for a very short distance in front of the intersection (cf. Fig. ), which makes it difficult for the planner to influence the incoming vehicles before they enter the intersection area. In case one vehicle waits at the stop point while a second vehicle enters the scene at high speed, a collision might be inevitable. This is merely an issue of the evaluation and does not diminish the remarkable improvement in traffic efficiency that is raised by the RL planner.

# Conclusion

In this work, a novel multi-agent behavioral planning scheme for connected automated vehicles at urban intersections was presented. We chose a reinforcement learning algorithm to leverage recent advances in machine learning while evading the need for ground truth data that is virtually unavailable for cooperative maneuvers. The developed graph-based input representation effectively encodes the semantic environment at the operational area. By employing graph neural networks, our approach confidently handles the varying number of vehicles in the scene. The proposed approach was evaluated in synthetic simulation and additionally based on real-world traffic data. Compared to static priority rules and a FIFO scheme as baselines, the learned planner increases the vehicle throughput significantly. In addition, the number of induced stops is reduced which indicates better traffic flow.

The proposed behavioral planning framework can serve as a sound foundation for solving more sophisticated planning problems. In the future, we plan to extend this work to be applicable to intersection layouts that were not seen during training. Moreover, cooperative planning in mixed traffic, i.e. human drivers and automated vehicles sharing the road, shall be addressed.

[^1]: $^{1}$The authors are with the Robert Bosch GmbH, Corporate Research, D-71272 Renningen, Germany. E-Mail: `{marvin.klimke, email@example.com`

[^2]: $^{2}$The authors are with the Institute of Measurement, Control and Microtechnology, Ulm University, D-89081 Ulm, Germany. E-Mail: `email@example.com`

[^3]: Part of this work was financially supported by the Federal Ministry for Economic Affairs and Climate Action of Germany within the program "Highly and Fully Automated Driving in Demanding Driving Situations" (project LUKAS, grant numbers 19A20004A and 19A20004F).


abstract: The process of decomposing target images into their internal properties is a difficult task due to the inherent ill-posed nature of the problem. The lack of data required to train a network is a one of the reasons why the decomposing appearance task is difficult. In this paper, we propose a deep learning-based reflectance map prediction system for material estimation of target objects in the image, so as to alleviate the ill-posed problem that occurs in this image decomposition operation. We also propose a network architecture for Bidirectional Reflectance Distribution Function (BRDF) parameter estimation, environment map estimation. We also use synthetic data to solve the lack of data problems. We get out of the previously proposed Deep Learning-based network architecture for reflectance map, and we newly propose to use conditional Generative Adversarial Network (cGAN) structures for estimating the reflectance map, which enables better results in many applications. To improve the efficiency of learning in this structure, we newly utilized the loss function using the normal map of the target object.
author: Mingi Lim$^{1}$ and Sung-eui Yoon$^{2}$ [^1] [^2]
title:  **A Method for Estimating Reflectance map and Material using Deep Learning with Synthetic Dataset**

# INTRODUCTION

In a traditional computer vision tasks, there are several works to do decomposition of a target image into internal properties. These kinds of works are usually called as inverse rendering.
The way people can see things in real life or how cameras can take pictures of their surroundings is widely understood by basic science such as physics. In short, when light hits an object in a particular direction, the material properties of the object determine the direction and proportion of the light's reflection, so that it can be recognized or photographed by the contact of such a reflected light with the eye or camera. However, in the opposite case, such as finding the properties of the real object through the result image is a very difficult problem because the same appearance results can be made by a combination of multiple internal properties.
To estimate those properties, it is a common practice to assume that one or more properties are generally known or simplified to estimate other properties. For example, traditional approaches to estimate internal properties assume lambertian materials , or point lights .In this way, in order to simplify the problem, shapes are often provided as 3D models that is used for synthesize dataset or restrict the class of shapes.
In this work, we extract high quality reflectance maps from images of a target object with complex shapes and specular materials under complex natural illumination. Futhermore, we use this network for estimating BRDF parameters, environment map, and classifying materials.
Based on a previous work, we change several things to make improvements. At first, we introduce cGAN for estimating better reflectance maps. Next, we use our novel loss function using object normal for estimating more exact reflectance map. Finally, we do several applications using estimated reflectance maps such as estimating BRDF parameters, environment maps and classifying materials.
A reflectance map maintains the orientation-dependent appearance of a fixed material under a certain surrounding illumination. In other words, a reflectance map is an image of an object's material itself, mixed with the surrounding environment map. By separating this reflectance map, we can obtain an information that we want, material and illumination. Under assumptions such as single material, shadow, distant light source and distant viewer, the relationship between surface orientation and appearance is fully explained by the reflectance map. It can represent all surrounding illuminations and material properties. In particular, it can represent specular materials under high-frequency natural illumination precisely. Therefore, without a better understanding and an analysis of a 2D image system, the ability to estimate reflectance maps comes to a wide spectrum of applications, including estimating BRDF parameters of input objects, environment maps, and classifying materials with additionally synthesized images. The input of our reflectance map estimator is a segmented 2D image where an object from an input image with various shape and the output is a reflectance map.
In conclusion, at first, we propose an improved approach that estimates a reflectance map from the input image with deep-learning based cGAN with novel loss using an observable point map. Next, we suggest a model structure that use estimated reflectance map to estimate Cook-Torrance BRDF parameters, that is used to render materials of our synthetic dataset and also it is used to estimate environment maps that represent the surrounding illuminations. We make following key contributions:

-   We propose a deep learning-based method to estimate reflectance maps from 2D images with a cGAN structure and decompose the estimated reflectance map into the material parameters and the environment map.

-   We suggest a new loss for extracting reflectance maps from input images, using object normal maps.

-   We introduce a new dataset that includes synthetic images to encourage our training process of our deep-learning sub-models that estimate reflectance maps, environment maps and BRDF parameters.

More commonly, destabilizing perturbations are reduced by selecting a low order loss function and stable learning rate. Low order loss functions; such as absolute and squared distances, are effective because they are less prone to destabilizingly high errors than higher-order loss functions. Indeed, loss function modifications used to stabilize learning often lower loss function order. For instance, Huberization reduces perturbations by losses, $L$, larger than $h$ by applying the mapping $L \rightarrow \min(L,(hL)^{1/2})$.

# Related Works

The appearance of 3D object relates to its surface geometry, material, and illumination. Estimating these factors is a fundamental problem in a computer vision. After a release of depth sensors, geometry reconstruction have made rapid progress recently. However, the estimation of material and illumination remains relatively more challenging these days. Previous approaches for material and illumination estimation need strong assumptions. For such assumptions, for example, necessity of a depth sensor, multiple images of the same object under varying illumination, limited lighting conditions, a fixed rotation under static illumination, an object of a given class, or requiring user's intervention.
**Reflectance Maps** Reflectance maps assign a surface appearance to its orientation, therefore it is a combination of material reflectance and surrounding illumination that can be used in many important applications. For example, if we have 3D models, we can render virtual 3D objects with the same material and the illumination of a real existing object or change the appearance of two objects. In computer graphics, there are many ways to represent object materials, but a reflectance map is a one of the popular way to represent targets' material with their surrounding environments easily. In this work, we want to solve challenges of the related previous work, Georgoulis et al. with estimating reflectance map more exactly using conditional GAN . These estimated reflectance maps can be used in many applications and make the results better, such as BRDF parameter estimation, environment map estimation, material classification.
**Material Estimation** Depending on the assumption of a geometry, we can divide the material estimation area into two main categories. At first, there are approaches that assume geometry are known. Methods that require the surface geometry of objects to be known can work on any type of surface geometry when object geometry is provided. Dong et al. estimate spatially-varying reflectance from the video of a rotating object of known geometry. Wu and Zhou and Knecht et al. propose a method for appearence estimation at on-the-fly frame rates using a Kinect sensor. Li et al. targets for estimating planer object's surface appearance from single images using self-augmented CNNs. And next, there are approaches for specific object classes of unknown geometry. There are some recent works, Rematas et al. , Georgoulis et al. and Liu et al. Unlike works described earlier, these three works can estimate materials without the need for additional sensors to obtain geometry. However, for these works, the actual available target's shape is limited to the shape class of the object used for training. Therefore, their work only works on certain types of objects, such as cars and chairs. By contrast to these methods, our approach uses general surface geometry so do not restricted to specific object classes.

# Backgrounds

Before going into explanation of our methods, we introduce some basic definitions that can be used in our paper. We can represent reflectance map $L \left( \omega \right) \in S^{+} \rightarrow  R^{3}$ which is a map from orientations $\omega$ in the positive half-sphere $S^{+}$ to the RGB radiance value L leaving that surface to a distant viewer. It combines the effect of reflectance and illumination.
Horn and Sjoberg suggest to use positional gradients to parametrize orientation $\omega$. Instead, Georgoulis et al. parametrize the orientation simply by s and t those are normalized surface normal's x and y components. We follow their approach to parameterize a reflectance map. They also drop the z coordinate which is equivalent to drawing a sphere under orthographic projection and it can be represented as a single image. Note that this only covers the upper half-sphere $S^+$, so they need to parametrize a half-sphere.
We explain the rendering equation first to approach the concept of the reflectance map. Let's start with the rendering equation below.. $$\begin{gathered}
    L_{o}(x, \omega_{o}) = L_{e}(x, \omega_{o}) + \\ \int_{\omega^{+}} f_{r}(x, \omega_{i}, \omega_{o})L_{i}(x, \omega_{i}) \left< \omega _{i}, n(x) \right>^{+} d\omega _{i}
\end{gathered}$$ where $L_o$ is the outgoing radiance, $L_e$ the emitted radiance, $L_i$ the incoming radiance, $f_r$ the BRDF, and $n(x)$ the surface orientation. Radiances can be represented as functions of position x and direction $\omega$. The reflection part is the integral over the upper hemisphere $S^{+}$ of the product of incoming light $L_{i}$, BRDF $f_r$, and the dot product of surface normal n(x) and integration direction $\omega_{i}$. Refer to a recent book on the rendering  for more detailed information.
In this work, there are several assumptions. We only consider 1) general materials without emissive, translucent materials and 2) a single material which means one surface reflectance model. We also consider that 3) the object is seen under orthographic projection from an infinitely far-away observer and 4) illuminated under image-based lighting technique with no shadows. These assumptions can make the rendering equation the following function: $$\begin{gathered}
    L_{o}(\omega_{o}) = \int_{\Omega^{+}} f_{r}(\omega_{i}, \omega_{o})L_{i}(\omega_{i}) \left< \omega _{i}, n \right>^{+} d\omega _{i}
\end{gathered}$$ where $L_{o}$ is the reflectance map, $L_{i}$ is the illumination, and $f_{r}$ is the surface reflectance. In addition, we refer to the surface reflectance $f_{r}$ as the material for simplification. We suggest to use Cook-Torrance Model for rendering realistic, physical based material used widely. : $$\begin{aligned}
    f_{r}=k_{d}f_{lambert}+k_{s}f_{cook-torrance}, \\
    f_{lambert}=\pi / c,
\end{aligned}$$ where $k_{d}$ is called the diffuse color, $k_{s}$ the specular color. We use the lambert reflection model to model a diffuse reflection. $$\begin{gathered}
    f_{cook-torrance}\left( \omega_i, \omega_o \right) = \frac{ DFG }{ \left(n \cdot \omega_{i} \right) \left( n \cdot \omega_{o} \right)}.
\end{gathered}$$ Cook-Torrance specular model consists of 3 different functions that models the laws of physics in real world, D stands for normal distribution function, G stands for geometric shadowing function, and F stands for Fresnel function. The formula for each parameters is as follows: $$\begin{aligned}
    D_{GGX} \left( m \right) = \frac{ \alpha ^{2} }{ \pi \left( \left( n \cdot m \right)^{2} \left(\alpha^{2}-1 \right) +1 \right) ^{2} } \\
    G_{GGX}  \left( v \right) = \frac{ \alpha ^{2} }{ \left( n \cdot v \right)   + \sqrt{ \alpha^{2} +  \left( 1 - \alpha^{2} \right) \left( n \cdot v \right)^{2} } }
\end{aligned}$$ $$\begin{aligned}
    F = F_{0} + \left( 1 - F_{0} \right) \left( 1 - cos\left( \theta \right) \right)^{5} \\
    F_{0} = \left( \frac{ n_{1} - n_{2} }{ n_{1} + n_{2} } \right).
\end{aligned}$$ As both the illumination $L_{i}$ and the reflectance map $L_{o}$ are two-dimensional functions of direction $\omega$, we represent them as images using s, t that we explained above.

# Network Architectures

We devise network architectures to accomplish our goals as shown in Fig. . There are 4 separate networks. First, there is a network that estimates the reflectance map from an input image. We opt for cGAN  to estimate the reflectance map more precisely and design a new loss to help the network train well.

## Network for Reflectance Map

We want to estimate the reflectance map of a target object in an input image. A target object is consist of a single material and an input image is 2D RGB image. This is equivalent to estimating an appearance of sphere with the same material as the target object would look like from the same camera position and under the same illumination. Previous work uses an encoder-decoder network with L2 loss . We want to improve the quality of the reflectance map itself with expecting performance improvement of other applications. Therefore we used cGAN with a loss using object normal for estimating reflectance map. There are several works using cGAN for transforming images. We first deploy to extract reflectance map from input images and then we applied some changes appropriate for estimating reflectance map. First, we do not use patches for estimating a reflectance map (known as PatchGAN ). Because there are no patch-to-patch relations between input and output images, we adopt our novel loss using objects' normal for estimating a more realistic and exact reflectance map.
Also, our deep-learning networks are trained and evaluated using our new synthesized dataset. In our approach, our network for estimating reflectance map learn a mapping between an input image and its reflectance map, following a cGAN architecture.
Fig. shows the proposed architecture. We introduce half-connected U-Net architecture. In the beginning, there are several convolutional layers with batch normalization, ReLU, and pooling layers, the size of the input feature maps is reduced to 1 x 1. Next on, there are several up-convolutional layers until the output size becomes 256 x 256 pixels. In 3 low-level layers, each incomming input features is concatenated with outcomming CNN features. Every convolutional layer has a one stride and zero paddings where the output can be the same size as the input. L1 loss between the ground-truth reflectance map and prediction is used in the final layer. Furthermore, this generator is trained with adversarial loss calculated using real/fake classifier and loss with observable point map.
In short, the intrinsic goal of this network is find a mapping function from the input image to a reflectance map. This model is going to learn not only how to map the image pixels to locations on the reflectance map, but also to interpolate appearance from observed normals. Our goal is a challenging problem as a kind of domain transfer problem that encodes image to a directional domain. In the previous approach, they used encoder-decoder structure, which actively utilizes CNN structure, is used for training. Furthermore, they use a sparse reflectance map as one of inputs, an image that records material appearances for observable normal orientation, and a general object image is used as an another input. As such, they try several ways, but do not show much difference in results between different two inputs. Inspired by this, we want to use cGAN to improve the quality of the result image more.
And then, we will explain our loss for the training network for reflectance estimation. At first, the objective of conditional GAN can be expressed as: $$\begin{gathered}
    {L_{cGAN}}(G,D) ={E_{x,y}}[logD(x, y)]+ \\ {E_{x,z}}[log(1 - D(x,G(x, z))],
\end{gathered}$$ where G tries to minimize this objective against an adversarial D that tries to maximize it. Various previous studies have often shown good results using a combination of a GAN loss function and a traditional loss function. The role of discriminator is decision which the target is real or fake, and the generator should not be only trained in the direction of deceiving the discriminator, but also in a way that is substantially less different with pixel values. In this case, the L1 loss function results in a clearer image than the L2 loss function according to the previous work. In this work, based on good results from previous works, bringing better results by using an additional loss function.

$$L_{L1}(G) = E_{x,y,z}[||y - G(x, z)||_{1}].$$ We then use the ideal object normal to define an observable normal map of an input object. The estimated reflectance map is derived from observable surfaces of the target object therefore unobservable areas of the reflectance map are just estimated from those observable surfaces. Therefore, we propose a new loss to weight on the observable area of the target object as below: $$L_{N}(G) = E_{x,y,z}[||(y - G(x, z)) \circ N ||_{1}].
    \label{eq:normal_loss}$$ where $N$ is an observable point map of the input object, and $\circ$ means a pixel-wise product between images. You can see details in Fig. . We call this loss as normal loss because it uses observable normal of input object for calculating loss. An observable normal map is a sphere map of observable object normal orientation. It is a mapped image from an object normal to an orientation similar to the mapping method of a reflectance map. An observable point map is a sphere map of observable points' pixel value with their orientation. we obtain an observable point map by a pixel-wise product with binarized observable normal map and the estimated reflectance map. The observable point map is a map that has pixel values on each orientations for observable points on the reflectance map and an expectation for the observable point map is used to train.

Therefore, our final objective is: $$\begin{gathered}
 \label{eqn_fin}
    G* = arg min_{G} max_{D}L_{cGAN}(G,D) + \\ \lambda _{1}L_{L1}(G) + \lambda_{2}L_{N}(G).
\end{gathered}$$

## Network for Environment Map and BRDF Parameters

In the previous section, we showed our model and training method for estimating a reflectance map of a target object from a RGB input image. In this section, we are going to explain how to decompose the estimated reflectance map into its internal properties: materials and illumination.
The input of our material and illumination decomposition is a reflectance map estimated from the reflectance map estimator. Commonly, a reflectance map can be obtained in several ways. For example, when a spherical sample of the real material is available, it can be obtainable directly with surrounding illumination. In addition, in the case of different shape, if the shape is known, more specifically, its normals are known, its reflectance map can be retrieved, at least for all observed surface orientations. By the way, If the shape is unknown, there are several alternatives that have been tested to acquire it such as 3D scanning, structure-from-motion, depth sensors, CNN-based depth extraction or directly estimating the normals using deep learning. In this paper we going to use estimated reflectance map, that is given from our first network, reflectance map estimator. Our outputs explained in this section are Cook-Torrance BRDF parameters, and an environment map. The environment map is rendered using HDR spherical image, expressing surrounding illumination. We show details of the networks in Fig. and Fig. . All networks take a reflectance map as input. BRDF estimator make a 4-dimensional parameter vector, which are corresponding to each parameter of the Cook-Torrance reflectance model: three color values for the specular, and a roughness value, which defines how rough the material is. The environment map estimator estimates the environment map.
For BRDF estimator network, we use Huber loss  for regression and for the environment map estimator network, we use L2 loss. We also decide the resolution of the output of the environment map estimator as 256 x 256, which is more improved quality comparing to the previous work. The input type of this network is a 2D image of the reflectance map and the output type is Cook-Torrance parameter vector. The design of the network is shown in Fig. .
Overall, the environment map estimator consists of multiple convolutional layers, and deconvolution layers. Those convolutional layers have batch normalization and ReLU. Next, there are several fully-connected layers for parameter regression. As we already mentioned, an input of these networks is the same reflectance map as in the BRDF estimator. We opt for U-Net architecture to estimate an environment map with preserving what input features have. As other model previously mentioned, these models apply ReLU and batch normalization in each CNN layer units.

# Synthetic Dataset

A lot of images are required to train our network. But it is very difficult to find a large dataset. We need a dataset contains many real images with 3D shape, material, and HDR illumination that can be used as ground-truth. Therefore, we consider to synthetize imaginary rendered images for the training process. And also, there is another problem which is a lack of large databases of real material samples and HDR illumination maps. Accordingly, we generated a dataset for training each networks. It includes a set of input images, reflectance maps, environment maps, object normal images, observable normal maps, observable point maps. We also synthesized some results to train different methods for baseline comparison.
Synthetic images were produced with random 1) views, 2) 3D shapes, 3) materials and 4) illumination. An example can be seen in Fig. . This example was sampled from a random position around the object, looking at the center of the object. We could get 60 3D shapes from the free repository. For each sample, the object orientation and camera view around the y-axis were selected randomly. Illumination was provided by 61 free HDR illumination maps collected from the internet. For materials, we decided to randomize parameters. Overall 107 k sample images were generated. We defined a training-test split where no shape, material or illumination is shared between train and test dataset.

# Experimental Setup

We propose a network to estimate target images from input images with an end-to-end architecture using conditional GAN. Because we want to improve a performance of the previous work and the performance of indirect translation from an input to a reflectance map of the existing work is not significantly improved from the direct method, we suggest the method using the input image directly without estimating normal. This result is validated under our synthesized dataset. Let me explain a training process. We train our reflectance map estimator using the synthetic dataset and next, we train the rest of networks using the same dataset. We train BRDF estimator and environment map estimator with results obtained by reflection map estimator as inputs. In the following sub-sections, we share quantitative results obtained using our synthetic dataset, and then figure out how much performance there has been over the traditional method. For rendering our test and train dataset, 3D models, camera pose, rotation of objects, BRDF parameters, and environment map are randomly set and used, as described in the previous section, and there is no sharing of 3D models and environment maps between test dataset and train dataset.

# Results

```latex
\begin{table*}
    \begin{center}
        \begin{tabular} {cccccc}
            \hline\hline
            & Methods & DSSIM & MSE & MAE &\\
            \hline
            & \textbf{Our method} & \textbf{0.3248} & \textbf{0.1223} & \textbf{0.2514}&\\
            \hline
            & Ours w/o normal loss & 0.3516 & 0.1455 & 0.2808&\\
            \hline
            & ReflectanceNet\cite{RSD1} & 0.4719 & 0.3377 & 0.4706&\\
            \hline
            & Half-connected U-Net & 0.4087 & 0.2296 & 0.3697&\\
            \hline\hline
        \end{tabular}
    \end{center}
    \captionof{table}{Results of different methods}
    \label{tbl:Result}
\end{table*}
```
```latex
\begin{table*}
    \begin{center}

        \begin{tabular} {cccc}
            \hline \hline
            Task & Huber & MSE & MAE\\
            \hline
            Material estimation & 0.0223 & 0.0446 & 0.1650\\
            \hline \hline
        \end{tabular}

        \bigskip

        \begin{tabular} {cccc}
            \hline \hline
            Task & DSSIM & MSE & MAE\\
            \hline
            Environment map estimation & 0.3092 & 0.0855 & 0.2052\\
            \hline \hline
        \end{tabular}

    \end{center}
    \captionof{table}{Results of material estimation, and environment map estimation}
    \label{tbl:Result2}
\end{table*}
```

**Quantitative Results** Our quantitative results are summarized in Table . We quantitatively analyze our method's performance to validate our model designs. We compare each networks with our test dataset that contains 15,300 synthetic test images. We compare our approach to three different methods using the same training set. We use three different metric for comparing each different methods such as DSSIM, MSE, and MAE. Structural dissimilarity (DSSIM) is an image distance metric, that corresponds better to the human perception than MAE or RMSE. Mean Squared Error(MSE) measures the average of the squares of the errors that is, the average squared difference between the estimated values and the actual value. Mean Absolute Error (MAE) is the average distance between each pixel point.
We also test different methods. 1. Our network as-is, but without the novel normal loss (Eq. ). Exclusion of the normal loss leads to reduced accuracy in the reflectance map estimates. 2. This method uses encoder–decoder structure of Georgoulis et al. using our synthetic training dataset, and estimates a reflectance map of size 32 x 32. This approach performs poorly comparing to other methods. We suspect this might be due to an encoder-decoder model which results in the loss of high-frequency information rather than our cGAN based model with additional losses. 3. We bring the previously mentioned generator structure of reflectance map estimator, half-connected U-Net, which consist of encoder-decoder structure. We tried to construct a training method similar to previous works using L2 loss rather than using our conditional GAN based structure or normal loss suggested in this paper. We train those all networks until convergence. We can find that it records higher loss than our method, that is, we can infer that our methods produce better results.
We also report the accuracy of our sub-network for estimating an environment map and BRDF parameters in Table . We estimate BRDF estimator using Huber loss function. Huber loss function describes the penalty incurred by an estimation procedure. We want to show that this application sub-modules are well trained using an estimated reflectance map.

**Qualitative Results** Figure shows material estimation results for various methods with same materials and general objects. As you can see, our approach estimates a reflectance map at high quality. To take a closer look, ReflectanceNet is a network proposed by the existing work, while the Half-connected U-Net is an encoder-decoder model that is trained by L2 Loss. To train that ReflectanceNet, we resized the dataset synthesized in this work to match the size of the target network, while retaining the structure proposed by the existing work. Comparing results with the ground-truth image, in the case of a ReflectanceNet, the image size is 32x32, and you can see the lack of details. In addition, in the case of Half-connected U-Net, the size of result images is the same as the size of results of our method, you can see that it also lacks details comparing with our method's result.
In addition, we are going to show visualization of BRDF parameter estimation results and an environment map in Fig. . You can find that there is less difference between visualized estimated BRDF parameters and ground-truth and an example of an environment map is estimated well. Our approach targets on a general shape, and generates high-resolution images. In contrast, previous approaches only work on a limited object class or low-resolution images with blurry results.

# Conclusion and Future work

We have presented our cGAN based material estimation network with estimating a more precise reflectance map, environment map, and classifying materials. We also proposed our new loss for extracting a more exact reflectance map and tackled the lack of data by using a newly rendered synthetic dataset. We have tested our network and method comparing to other networks and verified that it shows some improvement over the prior methods for the estimating reflectance map itself. It can be used for other applications such as BRDF parameter estimation or environment map estimation and improve results by its better quality.
However, our work have some limitations. Because our work only utilizes the light reflected from the target object, it targets on a single object and material only. Therefore, it is often unclear to guess surrounding illuminations if the target material is diffuse or if observable normals of the target object are limited. To solve this problem, it would be better to think about how to infer the material of an object by obtaining it in a different way, beyond the estimation of the problem through the objects' appearance only.

[^1]: $^{1}$Mingi Lim, Department of Computer Science, Korea Advanced Institute of Science and Technology Email: email@example.com

[^2]: $^{2}$Sung-eui Yoon, Department of Computer Science, Korea Advanced Institute of Science and Technology


abstract: Parton Density Functions (PDFs) and their uncertainties are extremely important topics for both the Tevatron and the LHC. Experiments at the Tevatron can enhance this knowledge not only by constraining the PDF fits, but also by developing and refining the available PDF tools through feed-back from the experiments that are currently analyzing the highest energy hadron collider data available. It is important that the community has standardized tools and methods at its disposal. In this note we summarize briefly the most recent developments of the The Les Houches Accord PDF (LHAPDF), which is the modern replacement for PDFLIB. We also outline and compare the methods of quantifying the impact of PDF uncertainties on physical observables. The PDF weighting method for propagating errors from PDFs to event generator observables is outlined in detail, and example code for using this method with `PYTHIA` is also included.
author: D Bourilkov$^\ddagger$, R C Group$^\ddagger$[^1], and M R Whalley$^\dagger$,
bibliography: TeV4LHC_PDF.bib
title: LHAPDF: PDF Use from the Tevatron to the LHC

# Introduction

The experimental errors in current and future hadron colliders are expected to decrease to a level that will challenge the uncertainties in theoretical calculations. One important component in the prediction of uncertainties at hadron colliders comes from the Parton Density Functions (PDFs) of the (anti)proton.

The highest energy particle colliders in the world currently, and in the near future, collide hadrons. To make predictions of hadron collisions, the parton cross sections must be folded with the parton density functions:

$$\rm \frac{d\ \sigma}{d\; variable}[pp\rightarrow X] \sim \sum_{ij} \, \left(f_{i/p}(x_1)  f_{j/p}(x_2) +(i \leftrightarrow j)\right)\,  \hat{\sigma}
\label{llpdf1}$$

$\rm \hat{\sigma}$ - cross section for the partonic subprocess $ij\rightarrow X$
$\rm x_1$, $\rm x_2$ - parton momentum fractions,
$\rm f_{i/p(\bar{p})}(x_i)$ - probability to find a parton $i$ with momentum fraction $x_i$ in the (anti)proton.
A long standing problem when performing such calculations is to quantify the uncertainty of the results coming from our limited knowledge of the PDFs. Even if the parton cross section $\rm \hat{\sigma}$ is known very precisely, there may be a sizable error on the hadronic cross section $\sigma$ due to the PDF uncertainty.

The Tevatron can contribute to PDF knowledge in many ways that will benefit the experiments at the LHC. First, measurements made by the experiments at FNAL will reduce PDF uncertainties by constraining PDF fits. Perhaps more importantly tools and techniques for propagating PDF uncertainty through to physical observables can be improved and tested at the Tevatron.

Next-to-leading order (NLO) is the first order at which the the normalization of the hard-scattering cross sections has a reasonable uncertainty. Therefore, this is the first order at which PDF uncertainties are usually applied. To date, all PDF uncertainties have been calculated in the context of NLO global analysis. However, useful information can still be obtained from NLO PDF uncertainties with leading order (LO) calculations and parton shower Monte Carlos .

Techniques and tools for calculating PDF uncertainty in the context of LO parton shower Monte Carlos will be the primary topic of this document. Examples are provided employing CTEQ6  error sets from LHAPDF and the parton shower Monte Carlo program `PYTHIA` .

# LHAPDF update

## Recent developments

Historically, the CERN PDFLIB library  has provided a widely used standard FORTRAN interface to PDFs with interpolation grids built into the PDFLIB code itself. However, it was realized that PDFLIB would be increasingly unable to meet the needs of the new generation of PDFs which often involve large numbers of sets ($\approx$`<!-- -->`{=html}20–40) describing the uncertainties on the individual partons from variations in the fitted parameters. As a consequence of this, at the Les Houches meeting in 2001 , the beginnings of a new interface were conceived — the so-called Les Houches Accord PDF (LHAPDF). The LHAGLUE package  plus a unique PDF numbering scheme enables LHAPDF to be used in the same way as PDFLIB, without requiring *any* changes in the `PYTHIA` or `HERWIG` codes. The evolution of LHAPDF (and LHAGLUE) up to summer 2005 is well documented .

Recently, LHAPDF has been further improved. With the release of v4.1 in August of 2005 the installation method has been upgraded to the more conventional `configure; make; make install`. Version 4.2, released in November of 2005, includes the new cteq6AB (variable $\alpha(M_Z)$) PDF sets. It also includes new modifications by the CTEQ group to other cteq code to improve speed. Some minor bugs were also fixed in this version that affected the a02m$\_$nnlo.LHgrid file (previous one was erroneously the same as LO) and SMRSPI code which was wrongly setting `usea` to zero.

A v5 version, with the addition of the option to store PDFs from multiple sets in memory, has been released. This new functionality speeds up the code by making it possible to store PDF results from many sets while only generating a MC sample once without significant loss of speed.

## Consistency checks

As a technical check, cross sections have been computed, as well as errors where appropriate, for all PDF sets included in LHAPDF. 10,000 events are generated for each member of a PDF set for both `HERWIG`  and `PYTHIA` , and at both Tevatron and LHC energies. As this study serves simply as a technical check of the interface, no attempt was made to unfold the true PDF error. The maximum Monte Carlo variance (integration error) in our checks is less than 1 percent. This has not been subtracted and will result in an overestimate of the true PDF uncertainty by a factor $<\sim 1.05$ in our analysis. The results in general show good agreement for most PDFs included in the checks. Overall the consistency is better for Tevatron energies, where we do not have to make large extrapolations to the new energy domain and much broader phase space covered by the LHC.

Two complementary processes are used:

-   Drell–Yan Pairs ($\mu^+\mu^-$): the Drell–Yan process is chosen here to probe the functionality of the quark PDFs included in the LHAPDF package.

-   Higgs Production: the cross section for $gg \rightarrow H$ probes the gluon PDFs, so this channel is complementary to the case considered above.

# PDF uncertainties

As stated above, the need to understand and reduce PDF uncertainties in theoretical predictions for collider physics is of paramount importance. One of the first signs of this necessity was the apparent surplus of high $P_T$ events observed in the inclusive jet cross section in the CDF experiment at FNAL in run I. Subsequent analysis of the PDF uncertainty in this kinematic region indicated that the deviation was within the range of the PDF dominated theoretical uncertainty on the cross section. Indeed, when the full jet data from the Tevatron (including the D0 measurement over the full rapidity range) was included in the global PDF analysis, the enhanced high x gluon preferred by CDF jet data from Run I became the central solution. This was an overwhelming sign that PDF uncertainty needed to be quantified . Below, a short review of one approach to quantify these uncertainties called the Hessian matrix method is given, followed by outlines of two methods used to calculate the PDF uncertainty on physical observables.

## Review of the Hessian Method

Experimental constraints must be incorporated into the uncertainties of parton distribution functions before these uncertainties can be propagated through to predictions of observables. The Hessian Method  both constructs a N Eigenvector Basis of PDFs and provides a method from which uncertainties on observables can be calculated. The first step of the Hessian method is to make a fit to data using N free parameters. The global $\chi ^2$ of this fit is minimized yielding a central or best fit parameter set $S_0$. Next the global $\chi ^2$ is increased to form the Hessian error matrix: $$%\Delta \chi ^{2}=\chi ^{2}-\chi_{0} ^{2}=\sum_{i=1}^N \sum_{j=1}^N H_{ij}(a_{i}-a_{i}^{0})(a_{j}-a_{j}^{0})
\Delta \chi ^{2}=\sum_{i=1}^N \sum_{j=1}^N H_{ij}(a_{i}-a_{i}^{0})(a_{j}-a_{j}^{0})$$

This matrix can then be diagonalized yielding N (20 for CTEQ6) eigenvectors. Each eigenvector probes a direction in PDF parameter space that is a combination of the 20 free parameters used in the global fit. The largest eigenvalues correspond to the best determined directions and the smallest eigenvalues to the worst determined directions in PDF parameter space. For the CTEQ6 error PDF set, there is a factor of roughly one million between the largest and smallest eigenvectors. The eigenvectors are numbered from highest eigenvalue to lowest eigenvalue. Each N eigenvector direction is then varied up and down within tolerance to obtain 2N new parameter sets, $S^{\pm}_i(i=1,..,N)$. These parameter sets each correspond to a member of the PDF set, $F_{i}^{\pm}=F(x,Q;S_{i}^{\pm})$. The PDF library described above, LHAPDF, provides standard access to these PDF sets.

## 'Master' Equations

Although the variations applied in the eigenvector directions are symmetric by construction, this is not always the case for the result of these variations when propagated through to an observable. In general the well constrained directions (low eigenvector numbers) tend to have symmetric positive and negative deviations on either side of the central value of the observable (X$_0$). This can not be counted on in the case of the smaller eigenvalues (larger eigenvector numbers). The 2N+1 members of the PDF set provide 2N+1 results for any observable of interest. Two methods for obtaining a set of results are described in detail below. Once results are obtained they can be used to approximate PDF uncertainty through the use of a 'Master Equation'. Although many versions of these equations can be found in the literature, the type which considers maximal positive and negative variations of the physical observable separately, known as "modified tolerance method", is preferred :

$$\Delta X^{+}_{max}=\sqrt{\sum_{i=1}^N[max(X^{+}_{i}-X_{0},X^{-}_{i}-X_{0},0)]^{2}}$$

$$\Delta X^{-}_{max}=\sqrt{\sum_{i=1}^N[max(X_{0}-X^{+}_{i},X_{0}-X^{-}_{i},0)]^{2}}$$

Other forms of 'Master' equations with their flaws are summarized:

-   $\Delta X_{1}=\frac{1}{2}\sqrt{\sum_{i=1}^N  (X^{+}_{i}-X^{-}_{i})^{2}}$
    This is the original CTEQ 'Master Formula'. It correctly predicts uncertainty on the PDF values since in the PDF basis $X^{+}_{i}$ and $X^{-}_{i}$ are symmetric by construction. However, for physical observables this equation will underestimate the uncertainty if $X^{+}_{i}$ and $X^{-}_{i}$ lie on the same side of $X_{0}$.

-   $\Delta X_{2}=\frac{1}{2}\sqrt{\sum_{i=1}^{2N}  R_i^{2}}$ ($R_{1}= X^{+}_{1}-X_{0}$,$R_{2}= X^{-}_{1}-X_{0}$, $R_{3}= X^{+}_{2}-X_{0}$ ...)
    If $X^{+}_{i}$ and $X^{-}_{i}$ lie on the same side of $X_{0}$ this equation adds contributions from both in quadrature. NOTE: For symmetric and asymmetric deviations, $\Delta X_{1}$ varies from $0\rightarrow \frac{2}{\sqrt{2}} \Delta X_{2}$

-   positive and negative variations based on eigenvector directions
    $\Delta X^{+}=\sqrt{\sum_{i=1}^N(X^{+}_{i}-X_{0})^{2}}$,  $\Delta X^{-}=\sqrt{\sum_{i=1}^N(X^{-}_{i}-X_{0})^{2}}$
    Since the positive and negative directions defined in the PDF eigenvector space are not always related to positive and negative variations on an observable these equations can not be interpreted as positive and negative errors in the general case.

## Summary of Techniques

Two main techniques are currently employed to study the effect of PDF uncertainties of physical observables. Both techniques work with the PDF sets derived from the Hessian method.

### 'Brute Force'

The 'brute force' method simply entails running the MC and obtaining the observable of interest for each PDF in the PDF set. This method is robust, and theoretically correct. Unfortunately, it can require very large CPU time since large statistical samples must be generated in order for the PDF uncertainty to be isolated over statistical variations. This method generally is unrealistic when detector simulation is desired.

Because the effect on the uncertainty of the PDF set members is added in quadrature, the uncertainty is often dominated by only a few members of the error set. In this case, a variation of the 'brute force' method can be applied. Once the eigenvectors that the observable is most sensitive to are determined, MC samples only need to be generated for the members corresponding to the variation of these eigenvalues. This method will always slightly underestimate the true uncertainty.

### PDF Weights

As mentioned above, often it is not possible to generate the desired MC sample many times in order to obtain the uncertainty on the observable due to the PDF. The 'PDF Weights' method solves this problem . The idea is that the PDF contribution to Equation  may be factored out. That is, for each event generated with the central PDF from the set, a PDF weight ($W_{n}^{0}=1,W_{n}^{i}=\frac{f(x_{1},Q;S_{i})f(x_{2},Q;S_{i})}{f(x_{1},Q;S_{0})f(x_{2},Q;S_{0})}$ where $n=1...N_{events},i=1..N_{PDF}$) can be stored for each event. The PDF weight technique can be summarized as follows...

-   Only one MC sample is generated but 2N (e.g. 40) PDF weights are obtained for $$W_{n}^{0}=1,W_{n}^{i}=\frac{f(x_{1},Q;S_{i})f(x_{2},Q;S_{i})}{f(x_{1},Q;S_{0})f(x_{2},Q;S_{0})}$$

    where $n=1...N_{events},i=1..N_{PDF}$

-   Only one run, so kinematics do not change and there is no residual statistical variation in uncertainty.

-   The observable must be weighted on an event by event basis for each PDF of the set. One can either store a ntuple of weights to be used 'offline', or fill a set of weighted histograms (one for each PDF in the set).

The benefits of the weighting technique are twofold. First, only one sample of MC must be generated. Second, since the observable for each PDF member is obtained from the same MC sample there is no residual statistical fluctuation in the estimate of the PDF uncertainty. One concern involving this method is that re-weighting events does not correctly modify the Sudakov form factors. However, the difference in this effect due to varying the PDF was shown to be negligible  . That is, the initial state parton shower created with the central PDF (CTEQ6.1) also accurately represents the parton shower that would be produced by any other PDF in the error set.

The weighting method is only theoretically correct in the limit that all possible initial states are populated. For this reason, it is important that reasonable statistical samples are generated when using this technique. Any analysis which is sensitive to the extreme tails of distributions should use this method with caution.

There are two options for using the PDF weighting technique. One can either store 2N (e.g. 40 for CTEQ) weights for each event, or store $X_{1},X_{2},F_{1},F_{2}$, and $Q^{2}$ and calculate the weights 'offline'. The momentum of the two incoming partons may be obtained from `PYTHIA` via PARI(33) and PARI(34). Flavour types of the 2 initial partons are stored in $F_{1}=MSTI(15)$ and $F_{2}=MSTI(16)$, and the numbering scheme is the same as the one used by LHAPDF, Table , except that the gluon is labeled '21' rather than '0'. The $Q^2$ of the interaction is stored in $Q^{2}=PARI(24)$. In theory, this information and access to LHAPDF is all that is needed to use the PDF weights method. This approach has the additional benefit of enabling the 'offline reweighting' with new PDF sets, which have not been used, or even existed, during the MC generation. We plan to include sample code facilitating the use of PDF weights in future releases of LHAPDF.

```latex
\begin{tabular}{
|p{1.0cm}
*{13}{|p{0.4cm}}
|}
\hline
parton &
$\bar{t}$ &
$\bar{b}$ &
$\bar{c}$ &
$\bar{d}$ &
$\bar{u}$ &
$\bar{d}$ &
g &
d &
u &
s &
c &
b &
t \\
\hline
{\sl n} & -6 & -5 & -4 & -3 & -2 & -1 & 0 & 1 & 2 & 3 & 4 & 5 & 6 \\
\hline
\end{tabular}
```

# Example Studies

## Drell–Yan at the LHC

The Drell–Yan process is chosen as an almost ideal test case involving quark PDFs for the different flavours.

The initial state parton kinematics and flavour contributions are given in Figure  for three regions of invariant mass of the final state lepton pair: $70 < M < 120,\ M > 1000,\ M > 2000$ GeV. As we can observe, they cover very wide range in X and Q$^2$. It is interesting to note that the flavour composition around the Z peak contains important contributions from five flavours, while at high mass the u and d quarks (in ratio 4:1) dominate almost completely.

## Higgs Production in gg$\rightarrow$H at the LHC

This channel is chosen as complementary to the first one and contains only contributions from the gluon PDF. A light Higgs mass of 120 GeV is selected.

## Inclusive Jet Cross Section at the Tevatron and the LHC

As mentioned above, the inclusive jet cross section was one of the first measurements where the need to quantify PDF uncertainty was evident. QCD 2-2 processes are studied for $\hat{P_T}\ >\ 500\ GeV$. The kinematic range probed can be seen in Figure .

## Results

The results for all 3 studies are summarized in Table . The weighting technique produces the same results as the more elaborate 'brute force' approach for all cases.

| Process (method) | $X_{0}$(best fit) | $\Delta X_{1}$ | $\Delta X_{2}$ | $+\Delta X^{+}$ | $-\Delta X^{-}$ | $+\Delta X_{max}^{+}$ | $-\Delta X_{max}^{+}$ |
|:---|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
| DY 70$<$M$<$`<!-- -->`{=html}110 (B.F.) | 1086 | 48 | 42 | +55 | -63 | +51 | -62 |
| DY 70$<$M$<$`<!-- -->`{=html}110 (W) | 1086 | 48 | 42 | +55 | -64 | +51 | -63 |
| DY M$>$`<!-- -->`{=html}1000 (B.F.) | 6.7e-3 | 3.5e-4 | 2.6e-4 | \+ 3.5e-4 | -3.8e-4 | +3.4e-4 | -3.9e-4 |
| DY M$>$`<!-- -->`{=html}1000 (W) | 6.7e-3 | 3.5e-4. | 2.6e-4 | +3.5e-4 | -3.8e-4 | +3.4e-4 | -3.8e-4 |
| DY M$>$`<!-- -->`{=html}2000 (B.F.) | 2.2e-4 | 1.8e-5 | 1.3e-5 | +1.9e-5 | -1.9e-5 | +2.0e-5 | -1.7e-5 |
| DY M$>$`<!-- -->`{=html}2000 (W) | 2.2e-4 | 1.8e-5 | 1.3e-5 | +1.8e-5 | -1.9e-5 | +2.0e-5 | -1.7e-5 |
| $gg\rightarrow H$ (B.F.) | 17 | .94 | .68 | +.82 | -1.1 | +.8 | -1.1 |
| $gg\rightarrow H$ (W) | 17 | .94 | .68 | +.82 | -1.1 | +.8 | -1.1 |
| DJ500 TeV (B.F.) | 0.022 | 0.0068 | 0.0057 | 0.0048 | 0.010 | 0.011 | 0.0042 |
| DJ500 TeV (W) | 0.022 | 0.0068 | 0.0057 | 0.0048 | 0.010 | 0.011 | 0.0042 |
| DJ500 LHC (B.F.) | 880 | 63 | 47 | 56 | 74 | 76 | 53 |
| DJ500 LHC (W) | 880 | 63 | 47 | 57 | 75 | 77 | 53 |

Results for the 3 case studies. The central values of the cross sections in \[pb\] are shown, followed by the estimates of the uncertainties for the different master equations and the 'brute force' (B.F.) and weighting (W) techniques.

# Summary

In this talk new developments of LHAPDF and consistency checks for all PDF sets are described. The approaches to PDF uncertainty analysis are outlined and the modern method of PDF weighting is described in detail and tested in different channels of current interest. Drell-Yan, gluon fusion to Higgs, and high $P_T$ jet production are studied at the Tevatron and LHC energy scales. The methods are in agreement in all cases. Equations for quantifying PDF uncertainty are discussed and the type which relies on maximal positive and negative variations on the observable is considered superior.

## Acknowledgments

The authors would like to thank Joey Huston for encouraging this study. DB wishes to thank the United States National Science Foundation for support from grant NSF 0427110 (UltraLight). CG wishes to thank the US Department of Energy for support from an Outstanding Junior Investigator award under grant DE-FG02-97ER41209. MRW wishes to thank the UK PPARC for support from grant PP/B500590/1.

[^1]: This talk was given by R. C. Group in October 2005 at the TeV4LHC workshop, describing ongoing work performed in collaboration with the other authors.