abstract: Due to the sparse rewards and high degree of environment variation, reinforcement learning approaches such as Deep Deterministic Policy Gradient (DDPG) are plagued by issues of high variance when applied in complex real world environments. We present a new framework for overcoming these issues by incorporating a stochastic switch, allowing an agent to choose between high and low variance policies. The stochastic switch can be jointly trained with the original DDPG in the same framework. In this paper, we demonstrate the power of the framework in a navigation task, where the robot can dynamically choose to learn through exploration, or to use the output of a heuristic controller as guidance. Instead of starting from completely random moves, the navigation capability of a robot can be quickly bootstrapped by several simple independent controllers. The experimental results show that with the aid of stochastic guidance we are able to effectively and efficiently train DDPG navigation policies and achieve significantly better performance than state-of-the-art baselines models.
author: Linhai Xie; Yishu Miao; Sen Wang; Phil Blunsom; Zhihua Wang; Changhao Chen; Andrew Markham; Niki Trigoni
title: Learning with Stochastic Guidance for Navigation

# Introduction

Deep Reinforcement Learning (DRL) has been shown to be extremely effective in mastering complex simulations and artificial tasks, e.g. playing Atari games \[13\] and Go \[17\]. However, DRL's poor sample complexity has limited its application to real world tasks, such as navigating a robot to a target position without crashing.

Deep Deterministic Policy Gradient (DDPG) \[10\] is an actor-critic algorithm that is suitable for such continuous control tasks in principle, but in practise the cost of exploration in complex navigation environments can prove prohibitive.

Since an agent must stochastically explore a long sequence of states during each training episode, high variance becomes the main bottleneck that hinders DDPG from learning effective DRL models. In order to mitigate this issue, conventional architectures generally require a huge number of learning samples, resulting in high computational and environmental costs in practice. In this paper, we propose a new framework that allows an agent to stochastically switch between high variance controllers (e.g. DDPG), and low variance controllers (e.g. simple deterministic controllers), effectively allowing the DDPG component to be quickly bootstrapped instead of starting from completely random moves.

Intuitively, learning is usually easier to be carried out under the guidance from other heuristics. The independent controllers here act as the guidance that are introduced for learning better DDPG policies. In our case, the agent still maintains an independent DDPG module that learns navigation by exploring the environment, but it is able to dynamically switch between learning from exploration or learning from the heuristic controllers. Here, the switching mechanism is constructed as a stochastic function updated by REINFORCE learning signal \[23\] to maximise total reward. Meanwhile, the DDPG component is learned by employing the action selected by the stochastic switch, rather than directly using the output action generated by its policy network. Therefore, the switching mechanism helps DDPG avoid trivial explorations during the early training process, and learns to balance between exploration and heuristic guidance. More interestingly, the DDPG component can be tested in isolation from other controllers, in which case the switch is turned off and the navigation is carried out solely by the DDPG component. Similar to the idea of imitation learning \[3\], the DDPG component is able to learn from the demonstrations given by the guidance (PID and OA (obstacle avoidance) components in our case) and instantly generalise to new situations (which PID and OA could not handle). Here, the guidance can be considered as a positive bias for reducing the variance of gradient estimators, and the model is able to remove this bias after benefiting from it.

For quantitative evaluation, we firstly compare our model with stochastic switch to the vanilla DDPG baseline and deterministic benchmarks for demonstrating the benefits brought by independent controllers. Then the influence of using different independent controllers is investigated, which shows that the framework has strong generalisation ability and it is able to accumulate the benefits from different simple controllers. In addition, we propose two variants of the switch mechanism including a uniformly random switch and an argmax switch for comparison. Finally, we show that the models can abandon the extra controllers when their usage rate declines below a threshold, and it is able to continue the self-learning by only using the DDPG component. For qualitative evaluation, we test our model in a real world scenario. Without further modification, the model trained in simulation is able to be directly transferred to carry out navigation tasks.

In summary, we propose a new framework that leverages the heuristic knowledge provided by independent controllers to bootstrap deep reinforcement learning for robot navigation. Our experiments demonstrate that by incorporating stochastic guidance, we are able to effectively and efficiently train the DDPG navigation policies and achieve significantly better performance than state-of-the-art baseline models. As a simple, robust and easy-to-use framework, it can be a generic method applied to improve many other deep reinforcement learning algorithms and applications.

# Model

Robot navigation task can be defined as a partially observable Markov decision process \[18\] problem, which can be solved by DRL. With observations of world state, a robot needs to decide its action, i.e. a control policy, maximising an accumulative future reward. Since it is ideal for a robot to reach the goal without any crash, the reward function $r_t$ at time $t$ is defined as: $$r_t =
\begin{cases}
    R_{crash}, & \text{if robot crashes}\\
    R_{reach}, & \text{if robot reaches the goal}\\
    (d_{t-1} - d_t)\cos{(\omega_t)} - C, & \text{otherwise}
\end{cases}
\label{eq:reward function}$$

where $R_{crash}$ is a large penalty for collision, $R_{reach}$ is a positive reward for reaching the goal, $d_{t-1}$ and $d_t$ denote the distances between the robot and the goal at two consecutive time steps $t-1$ and $t$, $\omega_t$ represents the rotational speed of the robot at time $t$, and $C$ is a constant time penalty which encourages the robot to approach the goal quickly.

The proposed model consists of three parts: perception, control and stochastic switch, as shown in Fig.. At each time step, the perception part processes an observation and generates a corresponding input representation. Then different controllers can propose candidate actions based on the input representation. Finally, the stochastic switch determines which one of the actions to be carried out.

## Perception

At each time step $t$, the robot observes the state of the world $x_t$, which includes a stack of current and historical geometric observations, its linear and angular velocities and a destination. The geometric observations, which give distances to surrounding objects (depth images in this work), are processed by a convolutional neural network to produce a compressed input representation. It is then concatenated with the velocities and destination for controllers and stochastic switch.

## Control

#### Action

With the observation $x_t$ at time $t$, the robot takes an action $a_t = (a^v_t, a^{\omega}_t) \in \mathscr{A}$, where $a^v_t$ and $a^{\omega}_t$ respectively denotes the expected linear and rotational velocity at time $t$, to navigate. Then it obtains a reward $r_t$ given by the environment for assessing the chosen action and transits to next observation $x_{t+1}$. The goal of our model is to reach a maximum discounted accumulative future reward $R_t = \sum_{t'=t}^{T}\gamma^{t'-t}r_t$, where $\gamma$ is the discount factor and $r_t$ is the reward function in Eq. . In this work, the actions can be determined by the independent controllers and DDPG, establishing a set of candidate actions for the stochastic switch to choose.

#### Independent Controllers

Two independent controllers are introduced to facilitate the learning of DDPG's policies, especially providing reasonable actions in the beginning. One is a proportional-integral-derivative (PID) controller with proportional term \[2\], which derives action from the relative position of the destination $[x_{local}, y_{local}]$ in robot coordinate frame as: $$a_t = K_p \cdot [x_{local}, y_{local}]^T,$$ where $K_p$ is the coefficient for the proportional term. PID controller is one of the most widely used and successful control mechanisms. However, without considering geometric observation, it does not have an obstacle avoidance capability.

The other one is a simple obstacle avoidance (OA) algorithm which can drive the robot without collision. It uses geometric observation to detect and avoid nearby obstacles by controlling the heading direction (rotational speed) of the robot: $$|a^{\omega}| =
\begin{cases}
    a^{\omega}_{max}\cdot\frac{|d_{o} - \beta|}{\beta}, & d_{o} < \beta\\
    0, & \text{otherwise.}\\
\end{cases}
\label{eq:OA controller}$$ where $d_{o}$ is the distance to the closest obstacle, $a^{\omega}_{max}$ represents the largest rotational speed and $\beta$ indicates a pre-defined minimum safety distance. In the case where the distance between the robot and an object is less than the safety distance, i.e., $d_{o} < \beta$, the robot will rotate to avoid collision. These two controllers complement one another to provide candidate actions for stochastic switch. Note that the OA only produces $a^{\omega}$, while the selected $a^v$ is provided by the DDPG controller.

#### DDPG

The main controller of this framework is DDPG, which is an actor-critic approach in DRL \[10\] that simultaneously learns the policy and the action-state value (Q-value) to assess the learnt policy. Although the policy network and the critic network of the DDPG share the same input representation from the perception, the policy network predicts the action, while the critic network estimates the Q-value for current state-action pair. In the learning mechanism of critic network, given the policy $\pi$ which maps states to actions $a_t$, the expected return is $Q^\pi(x_t,a_t) = \mathbb{E}[R_t|x_t,a_t,\pi],$ which can be calculated with the Bellman equation \[22\]: $$Q^\pi(x_t,a_t) = \mathbb{E}[r_t+\gamma\mathbb{E}[Q^\pi(x_{t+1},a_{t+1})|x_t,a_t,\pi].$$ If the policy is deterministic, we can define $a_t=\mu(x_t)$ and the inner expectation can be avoided. Since the outer expectation is independent on policy $\mu$, it becomes an off-policy learning. Then, the objective is to minimise the temporal difference (TD) error: $$\begin{aligned}
    L(\theta^Q)&=\mathbb{E}_{x_t,a_t,r_t,x_{t+1}}[(y - Q(x_t,a_t;\theta^Q))^2], \\
    y &= r_t+\gamma Q(x_{t+1},a_{t+1};\theta^Q)
  \end{aligned}$$ where $\theta^Q$ is a parameter of the critic network. To update the critic network by temporal difference learning \[21\], all learning samples stored in the replay buffer are formulated as $(x_t, a_t, r_t, x_{t+1})$.

The policy network is parameterised by $\theta ^\mu$. During training, the gradients are estimated by applying chain rules to the objective function (expected reward) $J(\theta^{\mu})$ w.r.t the parameters $\theta^\mu$. Generally, in DDPG, the parameters are updated by the gradients computed based on the actions produced by the policy network. However, in our case, we introduce a stochastic switch for choosing the final action from a set of actions proposed by all controllers. Hence, the networks are actually updated by the action decided by the switch network, instead of the $a_t$ produced by the policy network of DDPG.

## Stochastic Switch

The PID controller, OA algorithm and DDPG are three independent sources that produce candidate actions for the switch network to (optimally) select. The switch network is a stochastic deep neural network which consists of a parameterisation network and a multinomial distribution. Conventionally, a softmax layer can be employed to provide the parameter $\theta$ for the multinomial distribution. Here, instead, we apply **stick-breaking construction** \[16, 8, 11\], which is alternative to softmax.

The intuition is to introduce a bias that encourages more usage of the deep reinforcement learning algorithm, such as DDPG in our case. Since our framework is designed to train a robust DDPG component that benefits from the stochastic guidance, we expect it to be used more often than others in this framework so that we are able to get rid of the simple independent controllers after a certain period of training.

It basically transforms the modelling of multinomial probability parameters into the modelling of the logits of binomial probability parameters. In our case ($K=3$ controllers), given the binomial logits $\eta = \text{sigmoid}(\alpha)$, $\xi$ can be generated by two breaks: $\xi_1=\eta_1$, $\xi_2=\eta_2(1-\eta_1)$ and $\xi_3=(1-\eta_2)(1-\eta_1)$. Here $\alpha = f_s(x_t|\theta^s)$ is the unscaled logit from a deep neural network $f_s(\cdot|\theta^s)$ given the input representation $x_t$ and $\theta^s$ is the parameter of the switch network. Generally, the stick breaking function $f_{\text{SB}}$ can be generalised to more breaks $\xi_k = \eta_k \prod\nolimits_{i=1}^{k-1}(1-\eta_i)$ ($1<k<K$). Conditioned on the current observation $x_t$, we are able to construct the stochastic switch policy $s_t \sim \pi^s(s_t|x_t;\theta^s)$ as: $$\begin{aligned}
\label{eq: stick-breaking}
 \xi &= f_{\text{SB}}(\text{sigmoid}(f_s(x_t|\theta^s))) \\
 s_t &\sim  \text{multinomial}(\xi).
\end{aligned}$$ At each time step $t$, the stochastic switch samples a decision $s_t$ and $\theta_1, \theta_2, \theta_3$ corresponds to DDPG, PID and OA. Then, according to the decision $s_t$, the critic network of DDPG takes the final action $a_{s_t} \in \{a_{\text{DDPG}}, a_{\text{PID}}, a_{\text{OA}}\}$ as input and updates the networks accordingly. Meanwhile, the stochastic switch is updated by REINFORCE learning signal so that the switch network is able to dynamically choose to learn through exploration (DDPG), or it can choose to use the output of a heuristic controller (PID or OA) as guidance by observing the environment.

#### REINFORCE Algorithm

Since the gradients cannot be directly back-propagated through the discrete samples, we employ REINFORCE algorithm \[23\] to construct the gradient estimator for the switch network, where the goal is to maximise the total reward $R$ under the switching policy $\pi^s(s_t|x_t;\theta^s)$. Thus the objective function is: $$\mathcal{H} = \mathbb{E}_{p(S; \theta^s)}[R] = \mathbb{E}_{p(S; \theta^s)}\Big[ \sum\nolimits_{t=1}^T \gamma^{t-1} r_t \Big] ,$$ where $S$ is a sequence of decisions $s_1, s_2, ..., s_T$ in an episode, $s_t \sim \pi^s(s_t|x_t;\theta^s)$ is the decision sample at each time step $t$, and $p(S; \theta^s) = \prod_{t=1}^T \pi^s(s_t|x_t;\theta^s)$ is the probability of generating the current decision sequence. Hence, the gradients can be estimated as follows: $$\label{eq:reinforce}
    \dfrac{\partial \mathcal{H} }{\partial \theta^s}=\mathbb{E}_{p(S; \theta^s)}\Big[\dfrac{\partial}{\partial \theta^s}\log p(S;\theta^s) R\Big] \approx
    %\dfrac{1}{N}\sum_{n=1}^N \dfrac{\partial}{\partial \theta^s}\log p(S^n;\theta^s) R^n \\&=
    \dfrac{1}{N}\sum_{n=1}^N\sum_{t=1}^{T_n} \dfrac{\partial}{\partial \theta^s}\log \pi^s(s_t^n|x_t^n;\theta^s)R^n$$ where $N$ is the number of sampled episodes, $T_n$ is the length of the episode $n$, and $R^n$ is the total reward of the episode. It indicates a Monte Carlo based unbiased gradient estimation for updating the switch network.

The introduction of stochastic switch can be considered as an inductive bias for learning to navigate with better action samples. Updated by REINFORCE, the stochastic switch is able to sense the environment, avoid the trivial explorations and select better actions for learning DDPG policies. In addition, as the independent controllers are incorporated via the stochastic switch, the negative influence of the introduced biases from the heuristics is limited. More interestingly, the independent controllers can be turned off in the late training process (the stochastic switch always chooses the output of DDPG as final action) so that the learning of DDPG could further rely on exploration after being bootstrapped by the independent controllers.

#### Variance Reduction

Since the REINFORCE gradient estimator also suffers from the high variance issue, we introduce two control variates \[12\] for alleviating the problem: a centred learning signal $b^c$ (moving average) and an input dependent control variate $b(x)$ respectively. Here, we simply build an MLP (multilayer perceptron) to implement the $b(x)$ conditioned on input $x$. During training, the two control variates are learned by minimising the expectation: $\mathbb{E}_{p(S; \theta^s)}[R-b^c-b(x))]^2$, and the gradients are derived as, $$\label{eq:baseline}
\dfrac{\partial \mathcal{H} }{\partial \theta^s} = \dfrac{1}{N}\sum_{i=1}^N\sum_{t=1}^{T_n} \dfrac{\partial}{\partial \theta^s}\log \pi^s(s_t^n|x_t^n;\theta^s)(R^n-b^c-b(x_t)) \nonumber$$

# Experiments

## Training Environments and Settings

The proposed framework is trained in two different simulators. The first one is a light-weight simulator, *ROS Stage*[^1] (Fig.), in which a large amount of repetitive experiments are conducted for showing the learning curve, demonstrating the improvements brought by stochastic guidance, and comparing to other baseline models. In this simulator, we mount the mobile robot with a laser scanner to provide the geometric information of surroundings. Hence, the convolutional neural network (in Fig. ) is not being used in this case, and the laser scans are directly concatenated with other observation as input representation. By accelerating the simulation time, we obtain the **quantitative evaluation** through a lot of repetitive experiments in *ROS Stage*.

The other one, *ROS Gazebo*[^2] (Fig.), contains a physical engine and can accurately simulate the dynamics of the mobile robot. Thus the model trained in *ROS Gazebo* is directly applied to real world scenario to **qualitatively evaluate** the navigation performance, but it has a larger computational overhead compared to *ROS Stage*. Here, depth images are utilised to observe surroundings, therefore a 3-layer convolutional network (the filters are \[4,4,3,8\], \[4,4,8,16\] and \[4,4,16,32\] respectively) is constructed to provide input representations based on depth images.

In each training episode the robot starts at the origin point with a random heading direction and the destination is randomised within the area beyond obstacles. When the robot collides with an obstacle or reaches the destination, the current episode terminates. The action control frequency is 5Hz and the switching frequency is 1Hz . For all the experiments carried out in *ROS Stage*, the training process lasts for 100k steps and is repeated for 5 times. The averaged learning curves as well as the variance[^3] are illustrated for demonstrating the performance.

As for the hyper-parameters, the hidden layers of critic network and actor network contain 100 ReLU units in each layer, while the output layer of actor network applies tanh and sigmoid respectively for rotational and linear velocity. When updating DDPG parameters, 32 learning samples are randomly sampled from a rank based prioritised experience replay \[15\] as a training batch, and the learning rate for the actor network, the critic network and the stochastic switch are $10^{-4}$, $10^{-3}$ and $10^{-3}$ respectively, and the rest follows \[10\].

## Navigation in Simulated Environment

#### Reinforcement Learning with Stochastic Guidance

Fig. compares the models for demonstrating the benefits brought by learning with stochastic switch. **SGuidance** is our model with stochastic switch that dynamically choose the action from the candidates proposed by the controllers of DDPG, PID and OA. As shown in the figure, **SGuidance** achieves significantly better performance than the **DDPG** baseline. Meanwhile, **DDPG** suffers from the high variance issue according to the wide transparent area around the learning curve, while **SGuidance** is much more stable. This is due to the high complexity of the environment that leads to the highly variant learning samples provided by DDPG, which might lead to trivial explorations. In addition, the stochastic gradient estimator of DDPG applies biased approximation which makes it difficult to guarantee the convergence and stability. By contrast, **SGuidance** is able to benefit from the heuristic simple controllers since the beginning of training procedure instead of starting from completely random moves.

In this experiment, we also plot the rewards of **MoveBase** (without map) and **Oracle** (**MoveBase** with map) for comparison. **MoveBase** package[^4] is a widely used motion planner for mobile robot navigation and is implemented in the ROS package named Navigation Stage, which consists of a local planner \[5, 4\] and a global planner (implemented by Dijkstra or A\* algorithm). The global planner generates an optimal path from the origin to the destination on the global map of the environment, and the local planner dynamically avoids the newly detected obstacles while moving along the optimal path. Hence, we call the **MoveBase** with map **Oracle** in this experiment. As shown in Fig., **DDPG** is able to obtain comparable performance to **MoveBase**. **SGuidance**, however, significantly surpasses the deterministic **MoveBase** model. Even without the access to the global map, **SGuidance** has shown its strong ability to navigate in the environment by just using the geometric information. In addition, we plot the performance of two simple heuristic controllers (OA and PID) for reference. Basically, the simple deterministic controllers can not be applied independently for carrying out navigation task (the accumulative rewards are both under 0). However, when incorporated with DDPG via stochastic switch, it contributes notably for alleviating the high variance issue during the learning of DDPG.

#### Using Different Independent Controllers

This experiment shows the investigation where different independent controllers are incorporated with DDPG via stochastic switch. As illustrated in Fig., **SGuidance (PID + OA)** achieves the best performance when compared to the DDPG with only PID or OA and the DDPG without any independent controllers. Moreover, the contribution of the stochastic switch is greatly enhanced by adding more controllers, which yields more stable learning curves and better navigation performance. Interestingly, PID controller brings more benefits than OA controller in this context, and their benefits could be accumulated with the help of the stochastic switch.

#### Using Different Switching Mechanism

Fig. compares the stochastic switch to other variants of switching mechanism, an uniformly random switch and an argmax switch. The uniform switch assigns uniformed fixed probability to DDPG, PID and OA controllers, while the argmax switch applies biased argmax output instead of stochastically drawing samples from the stochastic switch network. As illustrated in Fig., **SGuidance** has the best performance. **Uniform** switch is not as good as **SGuidance**, but it still contributes remarkably to the navigation performance. The curve of **Argmax** lies in between the other two mechanisms, but has much bigger variance on the total reward. This is because **Argmax** is a biased sampler compared to the others, and the introduced bias in turn damages its final performance since there are less explorations after a certain period of training.

#### Construction of Stochastic Switch Function

In Fig. , the **StickBreaking1** (DDPG, PID, OA) represents the function we applied in the paper and the **StickBreaking2** (DDPG, OA, PID) used an alternative order of the independent controllers. More specifically, according to Eq. , **StickBreaking1** and **StickBreaking2** both set $\eta_1$ with DDPG controller and give different order with PID and OA controller where $\eta_2$ is assinged with the PID controller in **StickBreaking1** but with OA controller in **StickBreaking2**. As shown in the figure, **Softmax** is able to achieve almost adequate performance compared to **StickBreaking1** in terms of the total reward. However, according to the total usage of independent controllers, the DDPG component is being less used in **Softmax** than **StickBreaking1** and **StickBreaking2**. Although the two stick breaking functions have similar total usage of independent controllers, **StickBreaking2** performs slightly worse than **StickBreaking1**, which shows that the order of independent controllers has a small effect on the performance. Hence, the softmax function is a safe choice to construct the stochastic switch function. However, the prior knowledge about the performance of simple controllers could be used to benefit the learning in stick breaking construction. For instance, Fig. shows PID brings more benefit than OA when incorporating with stochastic switch, and Fig. also shows the **StickBreaking1** performs slightly better than **StickBreaking2**.

#### REINFORCE Variance Reduction

Fig. exhibits the performance of DDPG with or without variance reduction techniques. Due to the fact that REINFORCE algorithm also has the high variance issue, we study the benefits brought by the control variates. As shown in Fig., **SGuidance (no CV)** has already improved the vanilla DDPG model significantly. However, by introducing two control variates to reduce the variance of the REINFORCE gradient estimators (Eq.), **SGuidance** is able to further enhance and stabilise the navigation performance, which also illustrates that the high variance issue is certainly influential in the context of deep reinforcement learning.

#### Turning off Independent Controllers

This experiment investigates the property of DDPG with stochastic switch that the trained DDPG policies are able to independently carry out navigation with all the heuristic controllers turned off. In Fig. , the percentage of selecting heuristic controllers by **SGuidance** are demonstrated with dashed lines. Their usage drops quickly and becomes stable after approximately 60k training steps. This is because that the DDPG controller has already reached a comparable policy as other controllers. Therefore we turn off both PID and OA controllers when their usage is beneath 15% to study the performance of isolated DDPG. Instead of abruptly shutting down the controllers, we monotonically diminish the probability of using a proposed action of heuristic controllers to zero within 10k training steps when they are selected by the switch. As the result, turning off the heuristic controllers only slightly affect the navigation performance which supports the independent navigation capability of trained DDPG policies.

## Navigation in Real World Environment

In this experiment, we quantitatively analyse the performance of our model applied in real world environments. The model is trained in a simulated world built by *ROS Gazebo* ( Fig.), and directly transferred into real world scenario without any fine tuning in order to verify the effectiveness and strong generalisation of the model.

A Turtlebot 2 robot mounted with a Kinetic depth camera is used as the mobile platform. Unlike the observation from laser scanner which is simulated in *ROS Stage*, the dimension of state space for using a depth camera is dramatically increased. Therefore, a 3-layer convolutional neural network is employed (as in Fig.) to provide geometric representations. Other inputs, i.e. velocity and goal location, are concatenated with the geometric representation into a dense input representation.

Since the ground truth of the robot locations is not available in the real world environment, we apply the off-the-shelf AMCL ROS package[^5] for providing the estimation of the robot location, and calculating the destination position in the local coordinate frame. In order to improve the localisation accuracy, we record the map of the environment with Gmapping ROS package[^6]. It is worth mentioning that this global map is not used by the navigation component of the model during training or testing. The obstacles are laid out in the room as illustrated in Fig. . The target of this experiment is to employ the learned policy and control the robot to reach several destinations successively without any collision. As shown in Fig., the trajectory of the robot is plotted as the blue curves which indicates that the robot can smoothly avoid all the obstacles and reach each target successfully by only learning in simulation with the proposed stochastic guidance model.

# Related Work

Many works have applied DRL on robotic problems, e.g. navigation \[20, 26, 14, 20\] and manipulation \[6, 25\]. Since most of the robotics problems involve continuous control, policy based approaches such as policy gradient \[19\] or actor-critic method, e.g. DDPG \[10\], are widely used as the conventional approaches. Introducing positive bias is a common approach for alleviating the issue. \[15\] assigns higher weights to the data where the model has less confidence to improve the efficiency of sample usage. \[7\] leverage the concept of information gain when exploring new policies. Unlike above approaches where the bias are tightly merged into the models, our framework incorporates extra knowledge as stochastic guidance without imposing any change to the underlying approach.

Thompson Sampling \[1\] shares the similar spirit of our framework that the model learns to switch among different controllers. The difference is that, instead of explicitly calculating the posterior for updating in Thompson Sampling, our framework directly employs neural networks to construct the latent distributions which are trained jointly with the DDPG component by backpropagation. The advantage is that the switch function can be easily built and conditioned on all of the sensor inputs so that it chooses different controllers according to different contexts/conditions. In addition, the target of our framework is more focused on training a better DDPG component, which is able to benefit from the low-variance gradient estimator due to the better samples generated by the stochastic switch.

In \[9\], Leonetti et al. investigated a low level integration of RL and external controllers where the RL algorithm only explores with feasible actions provided by the planner, these heuristics can not be discarded, both for training and testing. Therefore, the performance of the learner very depends on, if not limited by, the capability of the heuristics. By contrast, in our framework, DDPG can explore the full action space by itself alongside the guidance during the whole training process and can eventually work independently.

# Conclusion

This paper proposes an new framework for effectively incorporating heuristic knowledge to overcome the high variance issue in learning DDPG. The experiments demonstrate that the stochastic switch allows an agent to balance the learning from exploration or heuristics, which significantly bootstraps the performance of navigation that surpasses state-of-the-art baseline models. More interestingly, the DDPG component remains independent and can be tested in isolation from other controllers. By transferring the policies into real world, the robot is able to successfully carry out navigation task, which indicates the robustness and strong generalisation of our proposed framework.

# References

\[1\] Shipra Agrawal and Navin Goyal. Analysis of thompson sampling for the multi-armed banditproblem. InConference on Learning Theory, pages 39–1, 2012.

\[2\] Karl Johan Åström and Tore Hägglund.PID controllers: theory, design, and tuning, volume 2.Instrument society of America Research Triangle Park, NC, 1995.

\[3\] Yan Duan, Marcin Andrychowicz, Bradly Stadie, Jonathan Ho, Jonas Schneider, IlyaSutskever, Pieter Abbeel, and Wojciech Zaremba. One-shot imitation learning.arXiv preprintarXiv:1703.07326, 2017.

\[4\] Dieter Fox, Wolfram Burgard, and Sebastian Thrun. The dynamic window approach to collisionavoidance.IEEE Robotics and Automation Magazine, 4(1):23–33, 1997.

\[5\] Brian P Gerkey and Kurt Konolige. Planning and control in unstructured terrain. InICRAWorkshop on Path Planning on Costmaps, 2008.

\[6\] Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learningfor robotic manipulation with asynchronous off-policy updates. InRobotics and Automation(ICRA), 2017 IEEE International Conference on, pages 3389–3396. IEEE, 2017.

\[7\] Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime:Variational information maximizing exploration. InAdvances in Neural Information ProcessingSystems, pages 1109–1117, 2016.

\[8\] Mohammad Khan, Shakir Mohamed, Benjamin Marlin, and Kevin Murphy. A stick-breakinglikelihood for categorical data analysis with latent gaussian models. InArtificial Intelligenceand Statistics, pages 610–618, 2012.

\[9\] Matteo Leonetti, Luca Iocchi, and Peter Stone. A synthesis of automated planning and rein-forcement learning for efficient, robust decision-making.Artificial Intelligence, 241:103–130,2016.

\[10\] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, YuvalTassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning.International Conference on Learning Representations, 2016.

\[11\] Yishu Miao, Edward Grefenstette, and Phil Blunsom. Discovering discrete latent topics withneural variational inference. InInternational Conference on Machine Learning, pages 2410–2419, 2017.

\[12\] Andriy Mnih and Karol Gregor. Neural variational inference and learning in belief networks.arXiv preprint arXiv:1402.0030, 2014.

\[13\] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc GBellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al.Human-level control through deep reinforcement learning.Nature, 518(7540):529–533, 2015.10

\[14\] Fereshteh Sadeghi and Sergey Levine. (cad)2rl: Real single-image flight without a single realimage.Robotics: Science and Systems, 2017.

\[15\] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay.arXiv preprint arXiv:1511.05952, 2015.

\[16\] Jayaram Sethuraman. A constructive definition of dirichlet priors.Statistica sinica, pages639–650, 1994.

\[17\] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess-che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mas-tering the game of go with deep neural networks and tree search.Nature, 529(7587):484–489,2016.

\[18\] Reid Simmons and Sven Koenig. Probabilistic robot navigation in partially observable en-vironments. InInternational Joint Conference on Artificial Intelligence, volume 95, pages1080–1087, 1995.

\[19\] Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradientmethods for reinforcement learning with function approximation. InAdvances in neuralinformation processing systems, pages 1057–1063, 2000.

\[20\] Lei Tai, Giuseppe Paolo, and Ming Liu. Virtual-to-real deep reinforcement learning: Continuouscontrol of mobile robots for mapless navigation.Intelligent Robots and Systems (IROS), 2017IEEE/RSJ International Conference on, 2017.

\[21\] Gerald Tesauro. Temporal difference learning and td-gammon.Communications of the ACM,38(3):58–68, 1995.

\[22\] Christopher JCH Watkins and Peter Dayan. Q-learning.Machine learning, 8(3-4):279–292,1992.

\[23\] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce-ment learning.Machine learning, 8(3-4):229–256, 1992.

\[24\] Linhai Xie, Sen Wang, Andrew Markham, and Niki Trigoni. Towards monocular vision basedobstacle avoidance through deep reinforcement learning.arXiv preprint arXiv:1706.09829,2017.

\[25\] Ali Yahya, Adrian Li, Mrinal Kalakrishnan, Yevgen Chebotar, and Sergey Levine. Collectiverobot reinforcement learning with distributed asynchronous guided policy search. InIntelligentRobots and Systems (IROS), 2017 IEEE/RSJ International Conference on, pages 79–86. IEEE,2017.

\[26\] Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinav Gupta, Li Fei-Fei, and AliFarhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. InRobotics and Automation (ICRA), 2017 IEEE International Conference on, pages 3357–3364.IEEE, 2017

[^1]: http://wiki.ros.org/stage

[^2]: http://wiki.ros.org/gazebo_ros_pkgs

[^3]: Note that the variance mentioned here is the variance of the smoothed learning curves.

[^4]: http://wiki.ros.org/move_base

[^5]: http://wiki.ros.org/amcl

[^6]: http://wiki.ros.org/gmapping


**Massachusetts Institute of Technology
Department of Electrical Engineering
and Computer Science
** **Proposal for Thesis Research in Partial Fulfillment
of the Requirements for the Degree of
Doctor of Philosophy
**

|                              |                                    |     |
|-----------------------------:|:-----------------------------------|:---:|
|                       Title: | Parallel Processor Architecture    |     |
|                Submitted by: | Peter Nuth                         |     |
|                              | 305 Memorial Drive, 606C           |     |
|                              | Cambridge, MA 02139                |     |
|                          3-3 |                                    |     |
|          Date of Submission: | 2024-10-02                         |     |
| Expected Date of Completion: | September 1990                     |     |
|                  Laboratory: | Artificial Intelligence Laboratory |     |

**Brief Statement of the Problem:**

The proposed research is a study of processor architecture for large scale parallel computer systems. The thesis introduces mechanisms for fast context switching, synchronization between tasks, and run-time binding of variable names to processor memory. Various design tradeoffs are evaluated through simulation of a processor running a typical load. This work contains estimates of the speed and complexity of the different alternatives as implemented in VLSI.

**Doctoral Thesis Supervision Agreement**

|       |                               |
|------:|:------------------------------|
|   To: | Department Graduate Committee |
| From: | Professor William J. Dally    |

The program outlined in the proposal:

|         |                                 |
|--------:|:--------------------------------|
|  Title: | Parallel Processor Architecture |
| Author: | Peter Nuth                      |
|   Date: | 2024-10-02                      |

is adequate for a Doctoral thesis. I believe that appropriate readers for this thesis would be:

|           |                         |
|----------:|:------------------------|
| Reader 1: | Professor Arvind        |
| Reader 2: | Professor Thomas Knight |

Facilities and support for the research outlined in the proposal are available. I am willing to supervise the thesis and evaluate the thesis report.

|     | Signed: |                                               |
|:---:|--------:|:---------------------------------------------:|
| 3-3 |         | Associate Professor of Electrical Engineering |
|     |         |             and Computer Science              |
|     |         |                                               |
|     |   Date: |                                               |
| 3-3 |         |                                               |

Comments:

|      |
|:----:|
|      |
|      |
| 1-1  |
|      |
| 1-1  |
|      |
| 1-1  |

**Doctoral Thesis Reader Agreement**

|       |                               |
|------:|:------------------------------|
|   To: | Department Graduate Committee |
| From: | Professor Arvind              |

The program outlined in the proposal:

|               |                                 |
|--------------:|:--------------------------------|
|        Title: | Parallel Processor Architecture |
|       Author: | Peter Nuth                      |
|         Date: | 2024-10-02                      |
|   Supervisor: | Professor William J. Dally      |
| Other Reader: | Professor Thomas Knight         |

is adequate for a Doctoral thesis. I am willing to aid in guiding the research and in evaluating the thesis report as a reader.

|     | Signed: |                                     |
|:---:|--------:|:-----------------------------------:|
| 3-3 |         | Professor of Electrical Engineering |
|     |         |        and Computer Science         |
|     |         |                                     |
|     |   Date: |                                     |
| 3-3 |         |                                     |

Comments:

|      |
|:----:|
|      |
|      |
| 1-1  |
|      |
| 1-1  |
|      |
| 1-1  |
|      |
| 1-1  |
|      |
| 1-1  |
|      |
| 1-1  |

**Doctoral Thesis Reader Agreement**

|       |                               |
|------:|:------------------------------|
|   To: | Department Graduate Committee |
| From: | Professor Thomas Knight       |

The program outlined in the proposal:

|               |                                 |
|--------------:|:--------------------------------|
|        Title: | Parallel Processor Architecture |
|       Author: | Peter Nuth                      |
|         Date: | 2024-10-02                      |
|   Supervisor: | Professor William J. Dally      |
| Other Reader: | Professor Arvind                |

is adequate for a Doctoral thesis. I am willing to aid in guiding the research and in evaluating the thesis report as a reader.

|     | Signed: |                                               |
|:---:|--------:|:---------------------------------------------:|
| 3-3 |         | Assistant Professor of Electrical Engineering |
|     |         |             and Computer Science              |
|     |         |                                               |
|     |   Date: |                                               |
| 3-3 |         |                                               |

Comments:

|      |
|:----:|
|      |
|      |
| 1-1  |
|      |
| 1-1  |
|      |
| 1-1  |
|      |
| 1-1  |
|      |
| 1-1  |
|      |
| 1-1  |

**Doctoral Thesis Reader Agreement**

|       |                               |
|------:|:------------------------------|
|   To: | Department Graduate Committee |
| From: | Professor William J. Dally    |

The program outlined in the proposal:

|               |                                 |
|--------------:|:--------------------------------|
|        Title: | Parallel Processor Architecture |
|       Author: | Peter Nuth                      |
|         Date: | 2024-10-02                      |
|   Supervisor: | Professor William J. Dally      |
| Other Reader: | Professor Arvind                |
| Other Reader: | Professor Thomas Knight         |

is adequate for a Doctoral thesis. I am willing to aid in guiding the research and in evaluating the thesis report as a reader.

|     | Signed: |                                               |
|:---:|--------:|:---------------------------------------------:|
| 3-3 |         | Associate Professor of Electrical Engineering |
|     |         |             and Computer Science              |
|     |         |                                               |
|     |   Date: |                                               |
| 3-3 |         |                                               |

Comments:

|      |
|:----:|
|      |
|      |
| 1-1  |
|      |
| 1-1  |
|      |
| 1-1  |
|      |
| 1-1  |
|      |
| 1-1  |
|      |
| 1-1  |


abstract: An EMCCD camera, designed from the ground up for extreme faint flux imaging, is presented. CCCP, the CCD Controller for Counting Photons, has been integrated with a CCD97 EMCCD from e2v technologies into a scientific camera at the Laboratoire d'Astrophysique Expérimentale (LAE), Université de Montréal. This new camera achieves sub-electron read-out noise and very low Clock Induced Charge (CIC) levels, which are mandatory for extreme faint flux imaging. It has been characterized in laboratory and used on the Observatoire du Mont Mégantic 1.6-m telescope. The performance of the camera is discussed and experimental data with the first scientific data are presented.
author: Olivier Daigle, Claude Carignan, Jean-Luc Gach, Christian Guillaume, Simon Lessard, Charles-Anthony Fortin and Sébastien Blais-Ouellette
title: Extreme faint flux imaging with an EMCCD

# Introduction

The advent of Electron Multiplying Charge Coupled Devices (EMCCD) allows sub-electron read-out noise to be achieved. However, the multiplication process involved in rendering this low noise level is stochastic. The statistical behaviour of the gain that is generated by the electron multiplying register adds an excess noise factor (ENF) that reaches a value of $2^{1/2}$ at high gains . The effect on the signal-to-noise ratio (SNR) of the system is the same as if the quantum efficiency (QE) of the EMCCD were halved. In this regime, the EMCCD is said to be in Analog Mode (AM) operation.

Some authors proposed offline data processing to lower the impact of the ENF in AM operation. However, one can overcome the ENF completely, without making any assumption on the signal's stability across multiple images, only by considering the pixel binary and by applying a single threshold to the pixel value. The pixel will be considered as having detected a single photon if its value is higher than the threshold and none if it is lower. In this way, the SNR will not be affected by the ENF and the full QE of the EMCCD can be recovered. In this regime, where the EMCCD is said to be in Photon Counting (PC) operation, the highest observable flux rate will be dictated by the rate at which the images are read-out; a frame rate that is too low will induce by losses by coincidence.

However, at a high frame rate, the Clock Induced Charges (CIC) become dominant over the other sources of noise affecting the EMCCD (mainly dark noise). CIC levels in the range of 0.01 – 0.1 were typically measured on a 512$\times$`<!-- -->`{=html}512 CCD97 frame transfer EMCCD from e2v Technologies. Even at a low read-out speed of 1 frame s$\mathrm{^{-1}}$, these CIC levels are at least an order of magnitude higher than the dark noise. Thus, one wanting to do faint flux imaging with an EMCCD is stuck with two conflicting problems: a low frame rate is needed to lower the impact of the CIC whilst a high frame rate is needed if a reasonable dynamic range is to be achieved.

In order to make faint flux imaging efficient with an EMCCD, the CIC must be reduced to a minimum. Some techniques have been proposed to reduce the CIC but until now, no commercially available CCD controller nor commercial cameras were able to implement all of them and get satisfying results. CCCP, the CCD Controller for Counting Photons, has been designed with the aim of reducing the CIC generated when an EMCCD is read out. It is optimized for driving EMCCDs at high speed ($\ge$ 10MHz), but may be used also for driving conventional CCDs (or the conventional output of an EMCCD) at high, moderate, or low speed. This new controller provides an arbitrary clock generator, yielding a timing resolution of $\sim$`<!-- -->`{=html}20ps and a voltage resolution of $\sim$`<!-- -->`{=html}2mV of the overlap of the clocks used to drive the EMCCD. The frequency components of the clocks can be precisely controlled, and the inter-clock capacitance effect of the CCD can be nulled to avoid overshoot and undershoots. Using this controller, CIC levels as low as 0.001 – 0.002 event pixel$\mathrm{^{-1}}$ per frame were measured on the 512$\times$`<!-- -->`{=html}512 CCD97 operating in inverted mode. A CCD97 driven by CCCP was placed at the focus of the FaNTOmM instrument to replace its GaAs photocathode-based Image Photon Counting System (IPCS). In this article, the important aspects of PC and AM operations with an EMCCD under low fluxes are outlined in section . In section , CCCP performance regarding these aspects is presented. Finally, in section , scientific results obtained at the telescope are presented.

# Faint flux imaging with an EMCCD

## The cost of sub-electron read-out noise

The multiplication process that allows an EMCCD to reach sub-electron read-out noise is unfortunately stochastic. Only the *mean* gain is known and it is not possible to know the *exact* gain that was applied to a pixel's charge. This uncertainty on the gain and thus on the determination of the quantity of electrons that were accumulated in the pixel causes errors on the photometric measurements. This uncertainty can be translated in a SNR equation as an ENF, $F$, as follows: $$\label{eqn::noiseFactor}
SNR = \dfrac{S}{\sqrt{F^2 S + \dfrac{\sigma_{real}^2}{G^2}}}.$$ Here, $S$ is the quantity of electrons that were acquired and $G$ the mean gain achieved by the multiplication register. At high gain, $F^2$ reaches a value of 2 and its effect on the SNR is the same as if the QE of the EMCCD was halved .

The effect of the ENF on the output probability of the EM stage of the EMCCD is outlined by figure , left panel. The overlapping output probabilities of different input electrons is the result of the ENF. Its impact on the SNR is shown in the right panel. The plateau at 0.707 of relative SNR is the effect of an ENF of value $\sqrt{2}$. This figure also compares the relative SNR of a conventional CCD ($G = 1$) operated at 2 and 10 electrons of read-out noise.

Thus, for fluxes lower than $\sim$`<!-- -->`{=html}5 photons pixel$\mathrm{^{-1}}$ per image, the EMCCD in AM operation out-performs the low-noise ($\sigma$ = 2 electrons) conventional CCD. This does not take into account the duty cycle loss induced by the low speed readout of the low-noise CCD. Since most of the EMCCDs actually available are of frame transfer type, virtually no integration time is lost due to the read-out process and advantage would advantage would be even greater for the EMCCD compared to the conventional CCD. In AM operation, the maximum observable flux per image is a function of the EM gain of the EMCCD and the integration time should be chosen to avoid saturation.

This ENF affects the SNR only when one wants to measure more than one photon pixel$\mathrm{^{-1}}$. If one assumes that no more than one photon is to be accumulated, it can consider the pixel as being empty if the output value is lower than a given threshold, or filled by one photon if the output value is higher than the threshold. The threshold is determined solely by the real read-out noise of the EMCCD. Typically, a threshold of 5$\sigma$ allows one to avoid counting false events due to the read-out noise (see figure , left panel). Thus, even if the exact gain is still unknown, it has no impact since it does not enter in the equation of the output value. In this operating mode, the ENF, $F$, vanishes by taking a value of 1 .

Counting only one photon pixel$\mathrm{^{-1}}$ does have its drawbacks: at high fluxes, coincidence losses become important (figure , right pannel, continuous and dashed line). The only way of overcoming the coincidence losses is to operate the EMCCD at a higher frame rate. In order to lose no more than 10% of the photons by coincidence losses, the frame rate must be at least 5 times higher than the photon flux. This requires that the EMCCD be operated at high speed (typically $\ge$ 10MHz) in order to allow moderate fluxes to be observed.

## The need for high EM gain

Figure gives a hint of another aspect of the PC operation: the ratio of the gain over the real read-out noise is important. Since PC operation involves applying a threshold that is chosen as a function of the real read-out noise, the EM gain that is applied to the pixel's charge will affect the amount of events that can be counted. Figure shows that for 1 input electron, the highest output probability of the EM register is 1 electron. This causes an appreciable proportion of events ending-up buried in the read-out noise. The proportion of events lost in the read-out noise, $e_l$, which is the proportion of the events that come out of the EMCCD at a value lower than $cut$ electrons, can be estimated by means of the following convolution: $$\label{eqn::gainEffect}
e_l = \dfrac{\displaystyle\sum_{x=0}^{cut}{f(n,\lambda) \ast p(x,n,G)}}{1-f(0,\lambda)},$$ where $f(n, \lambda)$ is the Poissonian probability of having $n$ photons during an integration period under a mean flux of $\lambda$ (in photon/pixel/frame) and $p(x,n,G)$ is the probability of having $x$ output electrons when $n$ electrons are present at the input of the EM stage at a gain of $G$. This probability is defined by $$\label{eqn::emOutputProb}
p(x,n,G) = \dfrac{x^{n-1}e^{-x/G}}{G^{n}(n-1)!}.$$

Figure translates the importance of having a high gain over read-out noise ratio into a detection probability as a function of the $G/\sigma$ ratio. An EMCCD that is operated at an EM gain of 1000 and has a read-out noise of 50 electrons will allow no more than 78% of the photons to be counted. In order to count 90% of the photons, one would have to use an EM gain of 2500 for the same read-out noise. It is the ratio of the gain over the real read-out noise that sets the maximum amount of detected photons. Thus, even if an EMCCD is capable of sub-electron read-out noise, its effect is not completely suppressed. The read-out noise still affect the quality of the data if the EM gain is not high enough.

## The dominance of CIC

When an EMCCD is operated under low fluxes at high gain and high frame rate, a noise source that is usually buried into the read-out noise of a conventional CCD quickly arises: CIC . The CIC are charges that are generated as the photo-electrons are moved across the CCD to be read out. The CIC noise may appear as dark noise except that it has no time component. It is only when the CCD is read out that the CIC occur. Thus, the higher the frame rate, the higher the CIC.

Figure shows the effect of the CIC on the SNR of an EMCCD. One quickly realizes that, at 30 fps, the CIC dominate over the dark noise even for a CIC level as low as 0.0001 electron per pixel per image (top left panel). In AM, at 1 fps, the CIC dominate for levels higher than 0.001 (top right panel). In order to lower the impact of the CIC, one could chose to operate the EMCCD at a lower speed (higher integration time). The effect of this choice is outlined in the bottom panels of the figure (assuming a CIC level of 0.001). It shows that in PC and AM mode, there is little advantage of operating the EMCCD at less than $\sim$`<!-- -->`{=html}0.5 fps. For a given CIC level, the maximum exposure time should be dictated by the time needed for the dark noise to dominate (about twice as high as the CIC). Moreover, the losses due to saturation (in AM) or by coincidence (in PC) are becoming more important, and the rising of the integrating time reduces the dynamic range of the images without providing a gain in SNR at low flux. Thus, there is a minimum frame rate at which the EMCCD should be operated, either in AM or PC. For a lower frame rate, there is no further gain in SNR to achieve at low fluxes and losses occur at high fluxes.

Another problem with the CIC is that it is dependent of the EM gain (see section ). The higher the EM gain, the higher the rate at which the CIC is generated. Thus, in order to reach the high EM gain needed for faint flux imaging (30–50 times the read-out noise), one must really tame the CIC down to low levels.

# CCCP performance

The camera built using CCCP and a CCD97 (hereafter CCCP/CCD97) was used to gather experimental data and to compare the performance of the CCCP controller against an existing, commercial, CCD97 camera (namely the, Andor BI camera, hereafter the Andor camera). The CCCP/CCD97 performance is also presented in absolute numbers. All the data presented, for both cameras, were gathered at a pixel rate of 10 MHz.

## Real and effective read-out noise

The real read-out noise of CCCP/CCD97 was measured at 10MHz of pixel rate using the photon transfer curve method with a faint, constant illumination and a varying integration time. For this technique, the variance of the images is plotted against the mean signal (read for a detailed explanation of the technique). This technique yields both the read-out noise and the reciprocal gain (electrons ADU$\mathrm{^{-1}}$) of the CCD.

Figure shows the measurement for both the real and the effective read-out noise of CCCP /CCD97. The various plots of the left panel show that the real read-out noise is not dependent of the EM gain. Indeed, there is no reason why the read-out noise of the EMCCD could not be affected by the EM gain. The clocking of the High Voltage (HV) clock could induce substrate bounce, EMI or cross-talk and increase the real read-out noise. Moreover, the reciprocal gain of the charge domain (after amplification) of the EMCCD could also change if the output amplifier, for example, was not linear. The good concordance of the three data sets of the left panel shows that the real read-out noise and the reciprocal gain of the EMCCD are stable. One can assume that the effective read-out noise of the CCCP/CCD97 can be calculated by means of $$\sigma_{eff} = \dfrac{\sigma_{G=1}}{G},$$ where $\sigma_{G=1}$ is the read-out noise calculated at an EM gain of 1 and $G$ is the gain at which the EMCCD is operated.

## Clock Induced Charges

In order to measure the CIC generated during a read-out of the EMCCD, many ($\sim 1000$) dark frames were acquired. These frames were exposed for the shortest period of time (to minimize the dark noise) and in total darkness[^1]. Then, the histogram of all the frames is fitted with the EM output probability equation (equation ), assuming that $n$ is always 1. For very low event rates (in this case, $\simeq$`<!-- -->`{=html}0.003 event pixel$\mathrm{^{-1}}$ per frame), this assumption has little effect on the CIC measurement. This gives at the same time the EM gain, $G$, and the mean quantity of events per pixel. The histogram fitting has also the advantage of seeing all the events that are generated in the image area and storage area and in the conventional horizontal register, even the ones buried in the read-out noise. This is mandatory if the CIC levels are to be compared at different EM gains. Obviously, simply counting the events that are at a 5$\sigma$ threshold would yield lower CIC levels for lower gains as more events would end up in the read-out noise (recall figure ). It is beyond the scope of this paper to describe the algorithm in detail[^2]. This code also calculates the mean bias (CIC+dark free) of all the frames and gives the real read-out noise. The bias calculated by the routine can then be used to remove the bias from light frames.

When a zero integration time is used, or when both the image and storage regions of the CCD are flushed prior to the read-out (dump of the lines through the Dump Gate) the dark signal generated during the readout can further be removed by fitting a slope of the mean signal from the first line read to the last one. This yields the dark count rate per read-out line. This allows the CIC to be completely disentangled from the dark noise.

The Andor camera is advertised as having a CIC+dark level of 0.005 electron pixel$\mathrm{^{-1}}$ per image[^3] at an EM gain of 1000 for a 30-ms integration time at -85${}^{\circ}$C. The CIC + dark noise rate measured on such a camera, using the histogram fitting algorithm, is 0.0084. This discrepancy may come in part from the fact that the measurement method used in this article measures all the events instead of counting only the events that are above 5$\sigma$ (which is the way Andor is characterizing its cameras, as stated in their datasheets). This camera is not capable of higher EM gains and it is thus not possible to measure the CIC at higher gains. Obtaining a higher EM gain is just a matter of producing an HV clock that has a higher amplitude. The Andor camera could probably be used at higher EM gain but it is limited by software at 1000. However, as it will be shown in section , the higher the EM gain, the higher the sensitivity of the EM gain to the amplitude of the HV clock. Thus, greater gain must come with greater stability of the HV clock. Moreover, a higher EM gain means a higher CIC level.

Figure shows the level of CIC that was measured on CCCP/CCD97 at different operating temperatures. At an EM gain of 1000, the amount of events attributable to the CIC is of the order of 0.001 electron pixel$\mathrm{^{-1}}$ per image. At an EM gain of 2500, the CIC level reaches 0.0014 – 0.0018. This figure also shows that the CIC is not, or is at most weakly, dependent on the operating temperature. Figure compares the SNR of the CCD97 driven by CCCP and the Andor camera, using the EM gain and CIC levels measured.

## Charge transfer efficiency

Having a high gain, a low CIC and a low real readout noise are not the only factors that are needed to render the faint flux imaging efficient. The Charge Transfer Efficiency (CTE) of the EM register, that is, the efficiency at which the electrons are moved across the EM register, plays an important role. If this CTE is too low, some of the electrons composing a pixel will not be transferred when they should be and they will lag in the following pixels. In images, these lagging electrons will be seen as a tail lagging behind a bright pixel. Figure shows the effect of an EM CTE that is not optimum (left panel, circled events). These leaking pixels are polluting adjacent pixels by raising their value. They can then be counted as pixels being stroked by a photon when they were not. This creates a source of noise.

For means of comparison, one could define a scenario that would allow these *bad pixels* to be identified and counted. For example, one could use dark frames, expected to count only dark and CIC events. If, say, the rate of events in these images is 0.01 event pixel$\mathrm{^{-1}}$ per image, there is, assuming a Poissonian process for the generation of the dark and CIC, only 1% chance that an event would immediately follow another. By counting the quantity of events that are adjacent, it is possible to know the quantity of events that are due to the bad CTE.

In the left panel of figure , the mean CIC+dark rate measured is 0.0084 event pixel$\mathrm{^{-1}}$ per frame[^4]. Thus, no more than 0.84% of the events should be adjacent. However, 5.5% of the events immediately follow one another. The excess in adjacent events (4.6%) must come from the bad CTE. In other words, given any event in an image (dark, CIC, or photo-electron), there is a 4.6% chance that it will be followed by a bad event.

In the left panel of figure , the bad event rate is measured to be 0.46%, where the expected rate is 0.18%. Thus, one can tell that at -85$^\circ$C and at a gain over read-out noise ratio of 22, there are only $\sim$`<!-- -->`{=html}0.3% bad events with the CCCP/CCD97 camera. Thus, even if CCCP's EM gain is higher than that of the Andor camera, the first has a much lower bad event rate, which means that the CTE figure of the EMCCD can be better handled with CCCP.

The deterioration of the CTE increases with both the EM gain and the low temperature of the CCD (figure ). Adjustments to the clocking (clock overlap voltages, clock frequency components, clock high and low levels, etc.) were performed after these measurements were made and better CTE figures are now accomplished, as can be seen in figure . However, the behaviour of the CTE as a function of the temperature and gain is maintained: the bad event rate rises with the EM gain and with lower temperatures.

## Gain stability

The stability of the gain of an EMCCD is important as it will affect the photometric measurements. The gain of the EM register is very sensitive to the variations of the HV clock. The higher the gain, the higher the sensitivity to the HV clock amplitude (figure , bottom panel). At a high gain, a variation of 1 volt in the amplitude of the HV clock represents more than a two-fold variation of the EM gain. This means that, in order to achieve a gain stability of the order of a percent, the HV clock must be stable within $\sim$`<!-- -->`{=html}5 mV. The EM register is also sensitive to the temperature (figure , top panel). In the EM gain regime shown, the temperature must be controlled within $\pm$`<!-- -->`{=html}0.1${}^{\circ}$C in order to achieve a gain stability of $\pm$`<!-- -->`{=html}1%.

### Gain stability over time

The stability of the EM gain of CCCP/CCD97 has been measured by taking dark images continuously at high gain ($G/\sigma \simeq 30$), moderate frame rate (10 fps), and at a temperature of -85${}^{\circ}$C. In total, 30000 images were acquired. Then, the EM gain of these images was determined using the usual algorithm. The algorithm used a window of 400 frames out of the 30000 and slid 400 frames every iteration. Thus, 75 data points were produced, each representing about 40 s. The results are shown in figure , left panel. The error bars in this figure are calculated using $$\label{eqn::errorGain}
e = G\sqrt{\dfrac{2}{n}},$$ where $G$ is the EM gain measured and $n$ is the quantity of events that were used to obtain $G$. The constant $2$ is used to account for the ENF of the EMCCD.

The gain variation over time is thought to be due to the variation of the temperature, rather than the variation of the HV clock amplitude. The temperature controller used to gather these data has an accuracy slightly worse than $\pm$`<!-- -->`{=html}0.1${}^{\circ}$C.

### Gain stability over frame

The EM gain must also be stable through an image or, at least, its variation must be well characterized. EM gain variation may come from the HV clock amplitude which can take some time to stabilize at the beginning of the frame, given that the HV clock is turned off during the integration time. Even if the HV clock is kept running during the integration time, the capacitive coupling between it and the conventional horizontal clocks can induce a variation of the HV amplitude when they are activated and, consequently, a variation in the gain.

The 30000 frames used in section were used once again to measure the gain over the image. The EM gain was determined on a per-line basis rather than on an image basis. Then, lines were binned by 8 to increase the SNR (given equation ). This yields the right panel of figure .

One can see that the HV clock takes about 32 lines to stabilize within the $\pm$`<!-- -->`{=html}1% gain variation. The HV clock was stopped during the exposure time of these frames. This gain variation is stable, which can be taken into account while processing the raw data to yield accurate photometric measurements.

## Experimental SNR

Experimental SNR curves were obtained in the lab. In order to do so, a scene spanning about 3 orders of magnitude in contrast has been observed (figure , bottom panels). Two acquisitions were performed: PC and AM, where the exposure time was set to 0.05s and 0.5s, respectively.

Even though care has been taken to put the lab in total darkness, the background of the scene was still very faintly lit (about 0.02 photon pixel$\mathrm{^{-1}}$ s$\mathrm{^{-1}}$), which explains the background signal. In total, 86000 images with an exposure time of 0.05s and 8600 images with an exposure time of 0.5s were acquired. The same amount of dark frames having the same exposure time were also acquired to remove the CIC+dark component. The dark frames were used to calculate the effective EM gain, by histogram fitting (see section ). The mean bias has been extracted from the dark frames and has been used to subtract the bias of all the light frames.

As for any CCCP data acquisition, the raw data coming out of CCCP during this experiment were stored in fits files during the acquisition and these files are then processed off-line. The resulting SNR curves are presented in figure . The frames where a cosmic ray was detected were simply removed. This accounted for about 0.15% of the frames in PC and for about 1.5% in AM.

### PC processing

The PC frames were processed with a threshold of $5\sigma$. However, the sum of the PC frames, minus the dark frames, gives only the amount of counted photons, $c_p$, whereas one wants to know the sum of incident photons, $f$. There are two possible sources of losses:

-   The threshold, which is responsible for the loss of the events not amplified enough and ending up in the read-out noise;

-   The events that are lost due to coincidence.

The proportion of the events lost due to the threshold, $e_l$, is given by equation if one uses $c_p$ as $\lambda$. The values of $G$ and $cut$ were, in this case, $30\sigma$ and $5\sigma$, respectively. Thus, the correction can be made simply by dividing the $c_p$ by $1-e_l$. This gives the amount of detected photons, $d_p$.

The coincidence losses can be modelled by simple Poissonian statistics. If the amount of incident photons, $f$, is unknown, the probability of having zero photon is known. This is simply $1-d_p$. Thus, $1-d_p$ represents the probability of having zero photon under a flux of $f$, and $f$ can be recovered by means of $$\label{eqn::coincidenceLossesCorrection}
f = -\ln{(1-d_p)}.$$ This correction, however, induces an excess noise that scales as $$F_c = \dfrac{1}{e^{-f}}.$$ At low flux, $F_c$ approaches the value of 1. It is only under moderate fluxes, where coincidence losses become important, that $F_c$ prevails.

Now, one can calculate the effective SNR of the image for every pixel. The SNR is given by $$SNR_{PC} = \dfrac{f}{\sqrt{F_c^2 N}},$$ where $N$ is the variance of the pixel over all the 86000 frames. This SNR curve can then be compared to the curve of a perfect PC system whose noise would be only the shot noise, which is $SNR = \sqrt{f}$. This gives the experimental points shown in figure , left panel. The theoretical curve is given by $$SNR_{PC_t} = \dfrac{S}{\sqrt{(S+D)g}},$$ where $S$ is the incident flux, and $D$ is the mean CIC+dark rate (the mean of all the dark frames). The term $g$ is the excess noise induced by the coincidence loss and its correction that are implied by the PC processing: $$g = \dfrac{S+D}{e^{-(S+D)}}.$$ In this case, $D$ is 0.0023  event pixel$\mathrm{^{-1}}$ per image. Of course, all of these corrections do not take into account the photons lost due to the QE. However, there is nothing CCCP can do about it and the plots are made according to a perfect photon counting system that would have the same QE as the CCD97.

The agreement between the theoretical and the experimental curves for the PC processing is nearly perfect (figure , top left panel). The image resulting from the averaging of all 86000 frames is shown in the bottom left panel of the figure.

### AM processing

The case of the AM processing is simpler. The flux in a pixel is given by dividing its value by the EM gain at which the image was acquired. The mean flux of a pixel, $f$, is simply given by the mean value of the same pixel across all the images minus the dark signal measured for that pixel. The SNR of a pixel is simply given by $$SNR_{AM} = \dfrac{f}{\sqrt{N}},$$ where $N$ is the variance of that pixel through all of the AM frames. When this SNR curve is normalized by that of a perfect photon counting system, this gives the experimental curve shown in figure , right panel.

The theoretical curve of that figure is obtained by taking into account the ENF, $F$, that plagues the EMCCD. Thus, the SNR equation is given by $$SNR_{AM_t} = \dfrac{f}{\sqrt{(f^2+d^2)F^2 + \sigma_{eff}^2}},$$ where $\sigma_{eff}$ is the effective read-out noise. Since the EM gain is high (30$\sigma$), $F^2$ takes the value of 2. The CIC+dark, $d$, is 0.0025  in that plot.

In order to reach very faint fluxes with the AM processing, the bias needs to have an extremely high SNR. There must be as many frames used to build the bias as there are light frames to process. If not, since the same bias is used for all the light frames, the noise of the bias will become the dominant source of noise in the final images. The PC processing is not as sensitive as the AM processing to the SNR of the bias. In PC, the bias's noise will be removed by the threshold applied to the data.

The plots also shows that the EM gain is accurately determined, which is critical for AM processing. The experimental AM plot would shift up or down if the EM gain were underestimated or overestimated, respectively.

### Comments

A result of this experiment that might have struck an attentive reader is the higher than expected CIC+dark rate in PC. On measurements made with histogram fittings (as in figure ), the CIC+dark rate was about 0.0018 for a $G/\sigma$ ratio of $\sim$`<!-- -->`{=html}30. The fitting of the SNR curves needed a CIC+dark level of 0.0023 in order to agree with the experimental data. This is also the mean event rate measured in the dark frames of the PC acquisition. This behaviour is explained through CIC and dark electrons generated into the EM register. This is in agreement with the fact that the mean source of CIC in CCCP/CCD97 comes from the CIC that is generated into the horizontal register . Given that the EM register sole existence is driven by the fact that it generates electrons, it is understandable that it generates CIC too. Dark electrons can also be generated in the EM register. The mean amplification of these CIC and dark electrons varies according to the elements into which the charge is created. The exact amount of CIC that is generated into the EM register is hard to compare at different $G/\sigma$ ratios. At low $G/\sigma$, mostly all of these CIC events will end up in the read-out noise, making it impossible to count them. So, it has been decided not to count them when comparing CIC levels. However, they must be accounted for in the SNR plots.

The effect of these lightly multiplied events will be stronger in the PC data than in the AM data. The PC-processed data see an event as an event, regardless of its multiplication factor. The AM-processed data, however, will see, on average, these events as having less than one input electron. Thus, the CIC+dark level computed in PC will differ from the one computed in AM. As an example, the CIC+dark level of the PC dark frames (0.05s exposure time) of figure is 0.0023. If these dark frames are processed in AM, the mean signal level measured is 0.0015 event/pixel/image, which is closer to the values presented in figure . Given these facts, the threshold used for the PC processing could be raised (6$\sigma$, 7$\sigma$) to avoid counting some of theses events. Of course, the proportion of counted photons would diminish. An optimum threshold, yielding the best SNR, could be calculated.

The image of the low light scene (figure , bottom panel) shows trailing electrons behind the most luminous pixels. The trail is about two orders of magnitude fainter than the mostly lit pixel on its line. This could be caused either by the low horizontal CTE in the EM register or the heating of the multiplication register when many electrons are generated into it. It is unlikely that this trailing happens in the conventional horizontal register as there is very rarely more than one electron per element in the PC image (the highest mean flux being about 0.5 photon pixel$\mathrm{^{-1}}$ per image). Surprisingly, this effect is less visible in the AM image, where more electrons/pixel are gathered, given the 10-fold increase in exposure time. This relation hold for even longer exposure times, as shown in figure , where exposure times of 5 and 15 s frame$\mathrm{^{-1}}$ were used, at the same EM gain (AM processing). These frames also show that these trailing electrons are not the result of an overflow of the horizontal register since it would worsen as the exposure time increases and more electrons are gathered. This phenomenon, however, did not compromise the quality of the scientific data, as discussed in section .

# Scientific results

CCCP/CCD97 was tested at the focal plane of the FaNTOmM integral field spectrograph . FaNTOmM is basically a focal reducer and a narrow band interference filter ($\sim10$Å) coupled to a high resolution ($R > 10000$) Fabry-Perot (FP) interferometer. FaNTOmM falls into the same instrument category as the newer GH$\alpha$FAS . The FaNTOmM instrument is mostly used to map the kinematics of galaxies using the doppler shift of the H$\alpha$  line (such as in ). The H$\alpha$  emission comes from both luminous $\mathrm{H_{II}}$  regions and faint, diffuse H$\alpha$  regions. There are a few strong sky emission lines caused by the OH radicals around H$\alpha$  but there are many dark regions in the spectrum as well. Given the high spectral resolution of these observations, they are mostly read noise limited instead of sky background limited. Moreover, since a FP interferometer is a scanning instrument, it is of great interest to scan the interferometer many times throughout an observation to average the photometric variations of the sky. This requires many short exposures (typically 5-15 seconds between the moves of the interferometer). These kind of observations are perfectly suited for a photon counting camera and this is the reason why FaNTOmM is usually fitted with an IPCS as the imaging device.

The mean drawback of the IPCS is its low QE, which is $\sim20$% for the case of the IPCS of FaNTOmM. However, the IPCS have a very low dark noise, typically of the order of $10^{-5}$ electron pixel$\mathrm{^{-1}}$ s$\mathrm{^{-1}}$. The advantage of the EMCCD is obvious in terms of gain in QE if the CIC of the EMCCD is low enough . The results obtained in lab with CCCP/CCD97 were promising. Engineering telescope time was obtained in September 2008 on the 1.6-m telescope of the Observatoire du mont Mégantic to test CCCP on real-world objects. During this run, the CCCP/CCD97 camera was used at the focal plane of the FaNTOmM instrument, in place of the IPCS.

## Observations

The galaxy was observed during this run. This galaxy has already been observed with FaNTOmM/IPCS through the SINGS H$\alpha$  survey . This observation will allow the comparison of the sensitivity of CCCP/CCD97 with FaNTOmM/IPCS.

The parameters of the observations are presented in table . All the data cubes were processed with the data reduction techniques presented in . The sky emission was subtracted by a sky cube fitting. No spatial smoothing was applied. The Radial Velocity (RV) maps were extracted and then cleaned by an automatic routine that correlates a pixel to the continuum and monochromatic flux-weighted value of the neighbouring ones to determine its validity, based on a maximum deviation. Then, a manual clean-up removes the pixels whose RV has been incorrectly determined. This is usually due to some sky residuals that are polluting the pixel.

```latex
\begin{table*}
\caption{Parameters of the H$\alpha$\, Fabry-Perot observations of \mbox{NGC 7331}.}
\begin{center}
\begin{tabular}{ l  m{0.18\textwidth}  m{0.18\textwidth}  m{0.18\textwidth}}
\hline \hline
Observation name & IPCS & AM & PC \\ \hline
Detector & GaAs IPCS & CCCP/CCD97 & CCCP/CCD97 \\ \hline
Geometry & 512 x 512 & 512 x 512 & 512 x 512 \\ \hline
Pixel size (${}^{\prime\prime}$) & 1.6 & 1.07 & 1.07 \\ \hline
QE at H$\alpha$\, (\%)& $\sim$ 20 & $\sim$ 90 & $\sim$ 90 \\ \hline
$\sigma/G$ ratio & N/A & 30 & 30 \\ \hline
CIC ($\mathrm{\bar{e}}\cdot \mathrm{frame}^{-1}$) & N/A & 0.0015 & 0.0022 \\ \hline
Dark current ($\mathrm{\bar{e}}\cdot \mathrm{s}^{-1}$) & 0.00001 & 0.0012 & 0.0012 \\ \hline
Processing & PC & AM & PC \\ \hline
Date (UT) & 2002-11-03 & 2008-09-11 & 2008-09-11 \\ \hline
Time (UT) & 23:15:00 & 00:30:00 & 04:15:00 \\ \hline
Telescope & OMM 1.6-m & OMM 1.6-m & OMM 1.6-m\\ \hline
Exterior T${}^{\circ}$\, (${}^{\circ}$C)& -12 & 3.5 & 3.5 \\ \hline
Moon age (days)& 27 & 11 & 11 \\ \hline
Moon illumination (\%)& 3 & 90 & 90 \\ \hline
Sky conditions & Mostly clear & Cirrus towards the end of the integration & Cirrus at the beginning of the integration \\ \hline
Mean flux / cycle &
%\includegraphics[width=0.15\textwidth]{ipcsFlux.eps} &
%\includegraphics[width=0.15\textwidth]{cccpFlux.eps} &
%\includegraphics[width=0.15\textwidth]{cccpFluxPc.eps}
\includegraphics[width=0.15\textwidth]{t1a.eps} &
\includegraphics[width=0.15\textwidth]{t1b.eps} &
\includegraphics[width=0.15\textwidth]{t1c.eps}
\\ \hline
Integration time / image (s) & 0.025 & 5 & 0.5\\ \hline
Images / channel & 400 & 1 & 10 \\ \hline
Integration time / channel (s) & 10 & 5 & 5\\ \hline
Number of channels & 48 & 48 & 48 \\ \hline
Number of cycles & 19 & 43 & 41\\ \hline
Total integration time (min)& 152 & 172 & 164 \\ \hline
Filter name & M3 & M3 & M3\\ \hline
Filter $\lambda_{center}$ at outdoor T${}^{\circ}$\, (\AA)& 6578.4 & 6580.9 & 6580.9 \\ \hline
Filter FWHM (\AA)& 15.5 & 15.5 & 15.5 \\ \hline
FP order at H$\alpha$& 765 & 765 & 765\\ \hline
\hline
\end{tabular}
\end{center}
\begin{flushleft}
\end{flushleft}
\label{table::FPobservations}
\end{table*}
```

In table , the plot of the mean flux per cycle is shown. This, coupled to the moon and sky conditions, gives a rough idea of the quality of the data. The moon was absent when the observation with the IPCS was made and the sky was mostly clear. Thus, the slight drop in the flux towards the end of the observation can be explained by absorption. On the other hand, the moon was mostly full (hence the engineering time) during the CCCP/CCD97 observations and there were a few cirrus. The cirrus were lit by the moon and this caused the flux to rise towards the end of the AM and at the beginning of the PC observation. Moreover, the moon's light excites the OH radicals in the upper atmosphere and makes them glow brighter. This causes the sky emission lines around H$\alpha$  to be stronger, which, once subtracted, leaves a noisier background due to shot noise. The same thing happens once the brighter foreground caused by the clouds lit by the moon is subtracted. It thus takes more photons from the galaxy to reach a given SNR in the CCCP/CCD97 observations.

On the other hand the interference filter's transmission also changes from the IPCS to the CCCP/CCD97 observation because of the exterior temperature. Given its systemic velocity of 816 km s$\mathrm{^{-1}}$, 's rest H$\alpha$  wavelength is at 6580Å. Its H$\alpha$  emission was better centred in the CCCP/CCD97 observation. On the IPCS observation, this should benifit the approaching side of the galaxy while weakening the receding side.

## IPCS and CCCP/CCD97 comparisons

On the whole, it is a delicate matter to compare observations made under different environmental and photometric conditions. Nevertheless, the observations should give a first order estimate of how the CCCP/CCD97 compares to the IPCS. The Monochromatic Intensity (MI) maps and the RV maps of all the observations are provided in figures and . One striking observation is that, even tough the integration time is comparable between the IPCS and the CCCP/CCD97 observations, the galaxy's approaching side is barely visible on the MI map of the IPCS. This is reflected in the RV map, which shows only a few sparse pixels from this side of the galaxy. By comparing with the CCCP/CCD97 observations, it is questionable whether these pixels are giving an accurate reading of the rotation velocity of this side of the galaxy. Even tough the observation's conditions were different, it is hardly conceivable that the IPCS observation could have gone deeper than the CCCP/CCD97 observation even in perfect conditions. Moreover, the colder temperature at which the IPCS data were acquired should benifit the approaching (red) side as compared to the CCCP/CCD97 observation.

## Efficiency

The efficiency at which the data were gathered is outlined by figure . This figure shows the pixel's mean intensity distribution in the data cubes superimposed on the expected SNR curve for that flux regime. It shows that all the observations were made in roughly the best conditions for the detectors and their modes of operation. The IPCS mean pixel flux is very low: the IPCS is operated at 40 frames s$\mathrm{^{-1}}$ but does not suffer from CIC. Thus, there is no loss at operating that fast.

## PC versus AM processing

The observing conditions prevailing when the PC and AM data cubes were gathered were not stable enough to provide a convincing comparison. One could see that the background of the MI map of the PC observation is smoother (figure ). The RV map of the AM observation looks "skinny" as compared to the PC one. On this side, it is better to rely on experimental data taken in lab in order to compare both operating modes with their respective integration time (section ).

One comparison that is absolute, however, is the comparison of the PC and AM processing on the data acquired at 0.5 s frame$\mathrm{^{-1}}$ on CCCP/CCD97 (the PC observation). Since there are very few pixels having $\sim$`<!-- -->`{=html}1 photon image$\mathrm{^{-1}}$ (figure ), the effective SNR of the data cube processed in PC should be higher than the one processed in AM. But how does this obvious fact influences the quality of the reduced data of ?

The bottom panels of figures and shows, respectively, the monochromatic intensity and the RV maps of both the PC and AM processing of the PC observation. First, the monochromatic map of the PC processing shows a smoother and darker background, which means that the sky emission has been better removed. This is very important as it will allow the resolving of the radial velocity of fainter regions of the galaxy. Next, the RV map of the PC processing shows a more extended coverage of the signal. This extended coverage is seen as more pixels near the edge of $\mathrm{H_{II}}$  regions and more sparse pixels in diffuse H$\alpha$  regions. Figure enhances these differences by drawing in black the pixels that are exclusive to the PC or AM RV map. There are far more pixels exclusive to the PC RV map than there are for the PC AM map. Thus, as expected, the PC processing yields a better, richer RV map than the AM processing.

# Summary and conclusions

A new camera built using CCCP, a CCD Controller for Counting Photons, and a scientific grade EMCCD CCD97 from e2v technologies was presented. This camera combines the high QE of the CCD with the sub-electron read-out noise of the EMCCD, allowing extreme faint flux imaging at an efficiency that is comparable to a perfect photon counting device. Laboratory experiments showed that the camera behave exactly as predicted by theory, in terms of relative SNR as a function of the flux, down to flux as low as 0.001 photon pixel$\mathrm{^{-1}}$ per image.

It has been shown that the CCCP/CCD97 camera yields very low CIC+dark levels at high EM gain, which represents a significant advance in the sensitivity that an EMCCD camera equipped with a CCD97 can achieve. Moreover, it has been demonstrated that the CTE figure of the EMCCD can be better handled with CCCP. It is beyond the scope of this paper to describe in details the clocking that is applied to the CCD97 to yield the lower CIC and higher CTE. Interested readers are invited to read Daigle et al. (in preparation).

Both PC and AM processing were compared and it has been demonstrated that the PC processing effectively allowed the achievement of a better SNR, for the same data-set. Evidences that the PC processing yields a better SNR than the AM processing, even at 10 times the frame rate, for the same total integration time, was presented. Given that the theory behind the SNR of an EMCCD seems well understood, one could argue that the PC processing should yield a better SNR at low and moderate fluxes, even at a higher frame rate, as compared to AM processing.

The performance of CCCP/CCD97 was compared to that of a GaAs photocathode-based IPCS on an extragalactic target in a shot noise-dominated regime. The achieved SNR of the CCCP/CCD97 observations is superior to the one achieved with the IPCS. Though it is delicate to compare two independent observations taken at different photometric conditions, the results presented could hardly be due solely to the photometric conditions. The CCCP/CCD97 camera must be, at some level, more sensitive than the IPCS, for the flux regime at which the data were acquired.

A new camera that will use the second version of the CCCP controller is under construction at the LAE in Montréal. It will use a larger 1600 $\times$ 1600, non frame transfer, EMCCD from e2v Technologies. This camera will be used as the high resolution camera at the focal plane of the 3D-NTT instrument that is expected to make first light at the end of 2009.

O. Daigle is grateful to the NSERC for funding this study though its Ph. D. thesis. We would like to thank the staff at the Observatoire du mont Mégantic for their helpful support, and the anonymous referee for valuable comments.

*Facilities:* , .

[^1]: For CCCP/CCD97, two frame transfers were done before reading-out, yielding a 0-s integration time. However, dark signal is generated during the read-out.

[^2]: Interested readers are invited to go to <http://www.astro.umontreal.ca/~odaigle/emccd> to read more about it and see the IDL code.

[^3]: In the datasheet available at <https://www.andor.com/download/download_file/?file=L897SS.pdf>

[^4]: The measurements of the *bad events* involved about a thousand dark frames, in addition to the zoomed region of figure .