abstract: One of the most challenges in medical imaging is the lack of data. It is proven that classical data augmentation methods are useful but still limited due to the huge variation in images. Using generative adversarial networks (GAN) is a promising way to address this problem, however, it is challenging to train one model to generate different classes of lesions. In this paper, we propose a deep convolutional conditional generative adversarial network to generate MIP positron emission tomography image (PET) which is a 2D image that represents a 3D volume for fast interpretation, according to different lesions or non lesion (normal). The advantage of our proposed method consists of one model that is capable of generating different classes of lesions trained on a small sample size for each class of lesion, and showing a very promising results. In addition, we show that a walk through a latent space can be used as a tool to evaluate the images generated.
address: ^1^ General Electric Healthcare, Buc, France
  ^2^ LITIS - EA4108 - Quantif, University of Rouen, Rouen, France
  ^3^ Nuclear Medicine Department, Henri Becquerel Center, Rouen, France
bibliography: main.bib
title: RADIOGAN: Deep Convolutional Conditional Generative adversarial Network To Generate PET Images

# Introduction

A generative adversarial network (GAN) is a class of machine learning system invented by Ian Goodfellow in 2014 . Two neural networks a discriminator and a generator compete with each other in a mini-max game. Given a training set, the generator tends to learn how to produce new data with the same statistics as the training set, while the discriminator tries to distinguish between real data and generated one as shown in Fig. 1. Generative methods (in particular, GANs) are currently used in various places for data augmentation . Their potential is vast; they can learn to mimic any distribution of data across any domain: photographs, drawings, music, and prose. However, there's no ground truth data to predict, which make it difficult to measure the model performance, especially in the medical field where the images may contain different lesions. In addition, it is not evident to distinguish a memorizing GAN, which memorize training examples from a generating one that learned meaningful boundary between real and generated images . Walking through a latent space can be used as a tool that can evaluate memorization degree.

Several authors have used GANs to build models for medical imaging synthesis, such as generating computed tomography (CT) from magnetic resonance imaging (MRI) or learning with GANs for chest X-ray classification . In PET images, Ben-Cohen et al. proposed a framework to generate PET images from CT data using a deep convolutional network. However, to our knowledge, there was no study about the generation of different class of lesions with the same model, and neither the verification if the model able to generate a complete new data or its memorizing the images from the training set.

To perform fast lesion classification physicians usually use MIP PET images which are 2D images that represents the 3D images. In this paper, we focus on the generation of MIP PET images. Deep learning methods trained to perform such a task are data hungry and need a lot of data to accomplish this task due the variability in images with different lesions. GANs are a promising tool that can solve this problem by increasing the size of data. However, we are faced with two main challenges when training a GAN: first, each class of lesions presents a small dataset in its own when used separately. Second, it is usually hard to verify if the GAN is generalizing a complete new data or it is just memorizing the dataset. Moreover, medical imaging presents a unique class of difficulties due to the huge variability in images, the presence of lesions in a specific anatomical point, the lack of integration of medical information and other constraints.

In this work, we develop a new GAN architecture, called RadioGAN, to generate FDG-PET images of normal patients and lesions. The idea is to combine different classes of lesions to train a condtional model which can capture not only common features in PET images but also specific ones for each lesion. We address both issues of data generation and class specified data augmentation. Moreover, we propose a framework to evaluate the result of a GAN and to verify if it is generating a complete new images or if it is memorizing the images seen in training by using a walk through a latent space . If a GAN memorizes images, then choosing random seeds in latent space creates basic blends of training images. However, if a GAN generalizes good images, then choosing random seeds produces exciting images that utilize patterns and components from the training dataset but are not simple blend.

# Data

We have one thousand six hundred and six patients (1606) in which six hundred and seventy five (675) are normal (only physiological uptakes) and nine hundred and thirty one (931) with pathological uptakes (cancer). We have four classes of cancer: lymphoma (225), lung cancer (189), oesophagus cancer (97) and head and neck cancer (422). Patient information was anonymized prior for analysis. The voxel size of PET exams is $4.06\times4.06\times2.0$ mm^3^ with a size of 429 x 168 x 168 before preprocessing. The lung cancer and head & neck datasets are from the cancer imaging archive (TCIA) . The images were acquired at the Henri Becquerel Center Institution and the study was approved as a retrospective study.

# METHOD

The object is to generate a specific class of image from one model trained with multiple classes (conditions).

## Data preprocessing

Images were spatially normalized by resampling all the dataset to an isotropic resolution of $2\times2\times2$ mm^3^ using the k-nearest neighbor interpolation algorithm. PET images gray level intensity were normalized to absolute Standardized Uptake Value (SUV) level between \[0 30\] and translated between \[0 1\] to be used in GAN architectures. Maximum intensity Projection (MIP) transformation was used to compress a 3D exam into a 2D representation. Five classes: normal, head & neck cancer, oesophagus cancer, lung cancer and lymphoma are shown in Figure 2.

## Proposed RadioGAN architecture

RadioGAN is inspired from deep convolutional generative adversarial network (DCGAN) which is a class of convolutional neural networks that generate images in an unsupervised learning way and conditional generative adversarial network (CGAN) which is a conditional version of GAN constructed by feeding the GAN the data and the label to condition both the generator and the discriminator .

The design of a GAN can be a challenging part, especially the generator design. The Generator takes as input a random noise and then map it into an image such that the discriminator cannot tell which images came from the dataset (real) and which ones came from the generator (fake).Several studies showed that using Convolutional Layers in the generator network produces better results . To transform a GAN to a DCGAN we follow the steps as in : replace all max pooling with convolutional stride, use transposed convolution for upsampling, eliminate fully connected layers, use of batch normalization except for the output layer for the generator and the input layer of the discriminator, use ReLU in the generator except for the output which uses tanh and LeakyReLU in the discriminator.

Conditional GAN uses image class information as a condition. For example, when we give our Generator a random vector Z, at the same time, we also give the class label Y= \[Normal, Esophagus cancer, head & neck cancer, lung cancer, lymphoma\]. Then the Discriminator will evaluate how well the Generator draws a patient of the corresponding class Label. After that, we use the evaluated error to update the Generator's weights (which includes new weights to accommodate the input Y).

In RadioGAN we take advantage of both DCGAN and CGAN to create a novel architecture DCCGAN (Deep Convolutional Conditional Generative adversarial Network) to generate MIP PET images for different classes of cancer and also for normal patient. The class label is added as an additional input layer for both generator and discriminator. To specify the class of the lesion for the generator, we encoded it as a condition using an embedding layer, which is a dense vector of a fixed size of 50. The five classes of images: lung cancer, esophagus cancer, head & neck cancer, lymphoma and normal are mapped to a different 50 element vector representation. Then, the output is followed by a Dense layer with a linear activation. The Dense layer has 15 neurons to match the 5 x 3 feature map activation of the generator model, and added as a second channel. For the discriminator, a new Embedding layer of a size of 50 is added, followed by a Dense layer to scale up to image dimension with a linear activation and added to the input image as a second channel as shown in Fig. 3.

The discriminator is an eight layers deep convolutional neural network in which each layer is composed of a convolution with a filter of 3 X 3 and a stride of 2 followed by a LeakyReLU and a Dropout with no pooling, and a BatchNormalization with a momentum of 0.8 except for the first and the two last layers. The last layer is a Dense layer with a Sigmoid function to classify the image for real or fake. The generator takes a random noise Z and passed to a Dense layer, followed by a Reshape and a BatchNormalization of a momentum of 0.8. Then a series of transposed convolution followed by a Batchnormalization are applied except for the last layer which is a convolutional layer with one channel and a tanh activation function. Then, the adversarial network is extended to a conditional model by conditioning both the discriminator and the generator on the lesions, normal labels.

We have trained our model for 300 epochs and used the binary crossentropy loss and Adam optimizer with a learning rate of 0.0002 and beta1 of 0.5. Accuracy is used as the metric for the discriminator.

# Results

## Evaluation of the memorization vs generalisation

Once the model is obtained, we can input a random vector (seed) of length 100 into the Generator and we get a patient image. If we input seed Z1 = \[1, 0, 0...0\] for example, we may get a patient with oesophagus cancer image and if we input seed Z10 = \[10, 0, 0...0\] we may obtain a normal patient image. What happens when we input the seeds between Z1 and Z10 ? For example Z2 = \[2, 0, 0...0\], Z3 = \[3, 0, 0...0\], … , Z9 = \[9, 0, 0...0\] ? A high quality generator must always output realistic images, so we should not see a half patient, a patient with 3 legs or a head instead of the stomach. Instead, we should see a sequence of realistic patient images that slowly transform from oesophagus cancer to normal. Latent walks can help us to distinguish simple memorizing GANs from complex generalizing GANs. In Fig. 4 an example of images generated with a standard CGAN which tends to memorize, we can see the image framed in red is not realistic. While in Fig. 5 an example of images generated with our proposed RADIOGAN which tends to generate realistic new images and not only blending them.

# Discussion and Conclusion

We have developed an end-to-end convolutional conditional generative adversarial network to generate MIP PET images. The advantage of this strategy is to train one model which can capture common features, but also specific ones for each lesion and for normal class. The integration of class lesions and normal class information is an interesting and challenging approach due to the difficulty to evaluate a GAN model. We have approached this problem through a walk in the latent space to evaluate the quality of the images generated.

Generative adversarial networks are a very promising tool to tackle the lack of data in the medical imaging field, and it can be also used to provide an advantage for other problems such as domain adaptation where data come from different distribution. RadioGAN shows promising results and can be extended to whole 3D body images to generate PET images from PET for data augmentation or from another modality such as CT for data synthesis. These results can be confirmed on a larger database.

# Acknowledgments

The authors acknowledge financial support from the Canceropole Nord-Ouest for emerging projects
(https://www.canceropole-nordouest.org/).


abstract: Urban imagery usually serves as forensic analysis and by design is available for incident mitigation. As more imagery collected, it is harder to narrow down to certain frames among thousands of video clips to a specific incident. A real-time, proactive surveillance system is desirable, which could instantly detect dubious personnel, identify suspicious activities, or raise momentous alerts. The recent proliferation of the edge computing paradigm allows more data-intensive tasks to be accomplished by smart edge devices with lightweight but powerful algorithms. This paper presents a forensic surveillance strategy by introducing an Instant Suspicious Activity identiFication at the Edge (I-SAFE) using fuzzy decision making. A fuzzy control system is proposed to mimic the decision-making process of a security officer. Decisions are made based on video features extracted by a lightweight Deep Machine Learning (DML) model. Based on the requirements from the first-line law enforcement officers, several features are selected and fuzzified to cope with the state of uncertainty that exists in the officers' decision-making process. Using features in the edge hierarchy minimizes the communication delay such that instant alerting is achieved. Additionally, leveraging the Microservices architecture, the I-SAFE scheme possesses good scalability given the increasing complexities at the network edge. Implemented as an edge-based application and tested using exemplary and various labeled dataset surveillance videos, the I-SAFE scheme raises alerts by identifying the suspicious activity in an average of 0.002 seconds. Compared to four other state-of-the-art methods over two other data sets, the experimental study verified the superiority of the I-SAFE decentralized method.
author:
bibliography: L-CNN.bib
title: I-SAFE: Instant Suspicious Activity identiFication at the Edge using Fuzzy Decision Making

# Introduction

Providing safety and well-being of the residents who live in populated cities is a rising challenge. City planners answered the security challenge by adding more cameras to enhance ubiquitous surveillance . The capability of real-time activities monitoring enables faster reaction for first responders in cases of emergencies. For instance, when a security agent sees the live footage and identifies a problem, quick actions can be taken. However, it is very difficult, if not impossible, for a security agent to focus on one of so many cameras when an event happens. In fact, most surveillance video streams are normally used as a forensics for diagnostics, lessons learned, and preparation for future events. Likewise, it takes long time to investigate information from thousands of video clips. The situation could be worse when the footage has been deleted due to limited storage space. To utilize the limited storage space more efficiently, many video surveillance systems integrate motion sensors. The cameras do not capture videos unless they are triggered by the motion sensors' detection of certain movements.

Currently, smart security cameras use intelligence to detect, classify and recognize objects of interest to determine which video clips to retain . These advances accelerate the decision making for the operator and also classify the data for later forensic analysis. Recently, machine learning (ML) models are adopted to detect anomalous behaviours by identifying certain bio-mechanical movements , but suffer from high false positive rates from inadequate training.

Edge computing is recognized as a promising solution to tackle the challenges in today's ubiquitously deployed video surveillance systems , . Migrating computing power to the edge allows more intelligence at each edge node such that on-site or near-site data processing becomes feasible, which consequently enables real-time object detection , tracking , and feature abstraction at the edge. Presently, it is still challenging to recognize the activities based on the features and identify suspicious behaviors.

Recently, researchers introduced a lightweight Convolutional Neural Network (L-CNN) for real-time object detection focusing on a primary object of interest (humans) and a hybrid KERnelized Machine learning and Artificial Networks (Kerman) algorithm for object tracking at the edge. The Kerman algorithm analyzes each frame of the video stream and extracts movement features and models patterns.

In this paper, we introduce an Instant Suspicious Activity identiFication at the Edge (I-SAFE) using a fuzzy decision making engine. A set of contextualized detectors are created by considering the features in a spatio-temporal context , which include the location of the camera and time of the day. Then, a fuzzy model with five membership functions is proposed to decide whether or not the behavior or activity of each of the people in the frame are suspicious. The fuzzy model was generated with inputs from campus police officers. Their knowledge and experience are integrated in the rule set, which uses the fuzzified contextualized features. The experimental study using real-world surveillance video streams verifies the I-SAFE scheme nominates suspicious activities in average of 0.002 seconds.

The contributions of this paper are as follows:

-   A lightweight smart safety system which is decentralized and real-time which detects humans and rises an individual based alarm in case certain factors are seen in the behavior;

-   A data experiment leveraging domain expert features selection that best describe the behavior elements of each pedestrian to support edge paradigm constraints;

-   A noise resistant fuzzy control module that upon receiving image features determines the intentions of each pedestrian in the frame; and

-   A comprehensive experimental study that verified the effectiveness of the I-SAFE decentralized method by comparing it to four other state-of-the-art methods over two other data sets.

The rest of this paper is structured as follows. In Section , the background knowledge of the problem and fuzzy systems are introduced. Section presents a novel smart surveillance architecture to support the proposed I-SAFE scheme. Section explains the feature extraction method and contextualization. The fuzzy model of the I-SAFE scheme is introduced in Section . The experimental results are presented in Section . Finally, Section provides conclusions.

# Background and Related Work

## Human-in-Loop Surveillance Systems

The surveillance community is aware of the growing demand for human resources to interpret data such as live video streams . The ubiquitous deployment of networked static and mobile cameras creates a huge amount of video data that is being transmitted to data centers for analysis and atomize the process . Many automated object detection algorithms have been investigated using ML  and statistical analysis  approaches that are implemented at the server side of the surveillance system.

There are also efforts made to promote operators' awareness by leveraging context , providing query languages , re-configuring the networked cameras , utilizing event-driven visualization , and mapping conventional real-time images to 3D camera images . Lack of scalability is still a challenge in traditional human-in-the-loop solutions to meet the demand of real-time surveillance.

## Safety Modeling and Anomaly Detection

Anomaly detection can have various definitions. For example, some researchers define anomaly as the most rare state in a sequence , . In case of a video, the algorithm selects the most rare frames in the sequence. For example, detecting a person or vehicle not in a normal behavior for pattern of life/anomaly detection (POL/AD) . In the Appearance and Motion DeepNet (AMDN) framework, several classifiers work in parallel to detect whether or not there is an object in a frame , such as a car or bike in the park . An integrated pipeline incorporates the output of object trajectory analysis and pixel-based analysis for abnormal behavior inference . It aims at better automation in the surveillance system with an algorithm that robustly captures activities such as: loitering, fighting and passing cars. Although POL/AD is comprehensive, the approach is unfortunately too expensive to be implemented on fog nodes which host edge units' data streams. Labeling partial video segments rather than bounding boxes in video frames, anomaly analysis segments the video where an action such as moving, stealing, or incident , . Video range labeling using a refined Recurrent Neural Network (RNN) also translates to a more accurate rare instance detection along with outputs of the bounding box around the anomalous object .

Loitering detection is selected as the case study to implement and test the I-SAFE framework. Loitering is moving back and forth around a centralized spot. Hence stopping, starting, and returning to the same spot are obvious indicators. Although loitering and move-top-move actions look similar, having a model that differentiates between the motivations is helpful to distinguish them accurately. A spatio-temporal clustering method based on the pedestrian speed is adopted for classification . It uses some features for motions detection and alarm raising, which suffers an overall low performance. An unsupervised dynamic sparse coding approach method was suggested for unusual event detection using an atomically learned event dictionary . This approach shows the unusual scene which loitering may be one, but does not concern the security aspect. Finally, the loitering problem was tried using a Markov random field (MRF) and generally seeks rare occurrence anomaly detection.

Two-dimensional human pose estimation in still images and videos has been explored , where both top-down and bottom-up  approaches have been proposed. The research community also retrieved the spatial configuration of humans by matching the holistic human shape , aggregating poses from segmentations, or using contours to model the shape of human body parts . Recently, many new ML models are introduced that can detect and connect human body parts to detect human pose , . These approaches suffer from the huge computation burden of image analysis for each and every object in the frame; and consequently, the computing time may be longer than one second in a crowded scene . Besides the long delay to recognize the pose of an object, the detected pose is to be compared to predefined "normal poses" to detect anomalous ones, such that the detection accuracy highly depends on the quality of the training set. If the training set fails to provide a sufficiently large number of "normal pose" exemplars, the system could suffer a high false positive rate. To reduce the need for a complete training data set, we seek a model to estimate between representative CNN extracted examples using fuzzy logic.

## Fuzzy Controller

Fuzzy sets and fuzzy logic together make fuzzy controllers – which is an attractive and promising method to deal with uncertainties. While the concept of fuzzy mathematics and its use is not new, it is specifically useful when methodologies are too complex for analysis by conventional quantitative techniques or when the available sources of information are interpreted in a qualitatively inaccurate or uncertain way .

Fuzzy logic, which is the formal operator on which fuzzy logic control is based upon, emulates human thinking and natural language process rather than the traditional Boolean logic . It is an effective tool for capturing the approximate, inexact nature of the real world. Viewed as a semantic classifier, the underlying part of the fuzzy logic controller (FLC) is a set of linguistic rules related by the dual concepts of fuzzy implication and the compositional rule of inference. The FLC provides the automation technique which can convert the linguistic control strategy based on expert knowledge into a stand-alone control system that is robust to environment and measurement noise . When the FLC is applied to domains for which it is trained, reports provide evidence of FLC superior results when compared to those obtained by conventional control methods. Hence, fuzzy logic control is considered as a step toward bridging conventional precise mathematical control to human-like decision making .

Because of the high uncertainty and complexity in describing the human activities, fuzzy decision making is appealing to address suspicious human activity detection in public safety surveillance systems.

A comprehensive survey for video classification and anomaly detection for video analytics was given in . Some works are focused on specific human behavior classification . Both methods used the context of the data to show human decision making and behavior prediction, instead of anomalous behavior. Researchers have also tried to reach better accuracy for human behavior prediction by combining the fuzzy logic and a Hidden Markov Model (HMM) .

# I-SAFE System Architecture

In a well performing safety system, humans are detected and their movement and behavior is closely watched to draw a conclusion of their intent. This process may be performed by the human agents or a smart system. The proposed I-SAFE system tries to mimic the same decision-making logic as the human agent. Figure presents the decentralized smart surveillance system architecture proposed to enable the I-SAFE. Within the architecture, the functionality of I-SAFE scheme can be considered as three steps:

-   *Video Feature Extraction*: Based on the design of the decision making algorithm, features from the video are extracted. These features present the movement of the humans as objects of interest in the frame. A lightweight CNN that feeds an online tracker, gives the position of the objects in each processed frame. In finding the domain-selected features for decision making, I-SAFE calculates the relative speed of each human in the frame and also the relative movement direction.

-   *Feature Contextualization*: Obviously, the context of the data can support the decision. Experiences from law-enforcement officers confirm that time and geo-location are factors which have an undeniable affect on the outcome whether an officer engages in for further investigation. Realizing this importance, I-SAFE incorporates contextual information to the features before feeding them to the decision making algorithm.

-   *Fuzzy Decision Making*: As one of the more robust approaches to control an environment, a fuzzy system armed with a complete set of rules and comprehensive membership functions (please refer to Section for details) can make a powerful tool for decision making. The fuzzy relations between inputs and outputs enhances system robustness to noise. This effect is amplified by the object tracker and the feature extraction imperfections that lead to added noise to position-related features.

As illustrated in Fig. , the I-SAFE system utilizes edge camera units. The edge device is either a basic surveillance camera with a Single Board Computer (SBC) mounted, like a Raspberry Pi board, or a smart camera with integrated computing resources. The edge node processes the video frames and extracts the features for decision making purposes . Under the framework of I-SAFE, although the live feed is available for the human operators in the control room, it is not recommended due to the high traffic volume, and it is impractical to expect any human operator to identify useful information from hundreds of real-time video streams. Only the extracted features that are essential for decision making are immediately transferred to guarantee low network communication workload. Exploiting a web service, the camera video stream can be stopped in case of no requests. Additionally, data context is not added to features at the edge, because of the repetition that is involved leading to undesired overhead in feature communication.

The feature stream is outsourced to a fog device located closest. A fog node can be a smart-phone, laptop, or a desktop, which is more powerful than the edge device and is deployed near the source of data. The reason for this outsourcing is simply the limitations of the edge node. After video processing, more calculations could drastically impact performance as verified by our experimental study reported in Section .

Adopting the *decentralized approach*, the I-SAFE scheme possesses several advantages over the traditional cloud-based services. In terms of the system architecture design, the network manager is eliminated that usually becomes a bottleneck of performance as the number of nodes increases. According to the capacity of a fog node, a number of edge servers are assigned to it at system setup. And the operator may have access to the real-time stream from the edge or the decisions made by the fog services.

Figure shows that the access to the edge is managed by a private *blockchain access control* (BAC) protocol. The access authentication is conducted in network setup phase and the smart contract is enforced in the blockchain network. To ensure the whole platform is scalable and easy-to-upgrade, both the video processing and security management functions are implemented using microservices architecture. Each microservice is placed inside of a docker container with all requirements necessary for ease of distribution. Due to the limited space, the rationales, architecture, implementation, and the performance evaluation are not presented in this paper. Interested readers are referred to two papers for more information, one details the microservices based surveillance platform and the other is focused on the blockchain enabled security mechanisms .

# Feature Preparations

In the I-SAFE scheme, dynamic data is analyzed to prepare features for the fuzzy engine with the following steps: feature generation, selection, extraction, and contextualization, where the classifier works best if the features produce most divergence between classes. Deep Learning (DL) models, which auto-define the features, need labeled training dataset that are unfortunately not available for security use-case. Thus, selected features based on subject matter expert (SME) are presented in this section. Moreover, context supports robust operations by including environmental, societal, and cultural information.

## Feature Selection

Looking at recent publications for anomaly detection, there are many different features that could be of importance. However, this work utilizes law enforcement officers to have a better understanding of what features should be of interest. Based on their input, loitering in odd hours or places, indicates high chance of misbehavior. Although there are other clues such as the appearance, clothing, and certain smells that may also draw an officer attention, in order to minimize bias or profiling individuals and avoiding extracting sensitive private information from the individual, the system only considers the pattern of movement after tracking the general figure of a human body. In the future, gesture and important body keypoints can be added to the tracking module for more accurate description of the activities.

The pattern of individual's movement is an important factor in the decision making procedure of an officer. Actually, many ML models can learn from the dynamic time-series data to model patterns and detect the anomalies . In a 2-D RGB picture of a random scenario, the pattern of movement may be confusing as depth information is also presented in up or down movements. In addition, studies show that movement patterns are context specific . For example, a person who is walking to his/her car in a parking lot may behave differently from a person who is walking into his/her dorm room. Thus, comparing the patterns and directions generalize scenarios leads to high false positive rate. Furthermore, mapping the pattern of movement is time consuming and resource demanding. Thus, it is not affordable for resource limited edge or fog devices for each person. Instead, indicators of the movement are chosen because of their generalization to all scenes.

A strategy is employed to utilize the number of speed and direction changes, in order to have the moving pattern. The more number of changes indicates the higher probability of loitering. Again according to the law officers who watched pedestrian movements closely on a daily basis, a person who has a known destination in mind is likely to walk straight an at certain reasonable pace. Turning around in an area or changing position without an apparent destination should raise an alarm. The other benefit of this strategy is that there is no need for extracting complex pastern routes for each person, rather the indicators. The difference in calculation time is more noticeable when there are more than three objects are present in the frame.

There are two complimentary features. *Standing at one location for a long time* may be an indicator of loitering. Nonetheless, this feature should be used in context. In addition, if there are more than one person in the scene, it is less likely to be of suspicious. Thus, the *number of the people* in the frame is also considered as a feature.

## Video Feature Extraction

The human detection uses a Convolutional Neural Network (CNN). If a human as the object of interest is detected, the object's bounding box coordination resides in a queue. In each upcoming frame, the queue is updated with the tracker bounding box prediction. Once every several frame when the CNN is applied to the input frame, not only it will add any newly detected objects to the queue for tracking, but it also checks the object placement in each previous bounding box. If the Intersection Over Union (IOU) is smaller than a threshold which is set by the administrator, it will add the person as a new object and delete the old bounding box and information related to it.

The online tracker should give an accurate estimation of the position of each object, otherwise the extracted features lead to inaccurate classification. The Kerman hybrid tracker maintains tracking and supports track hand-off to improve tracking accuracy. By extracting the coordination of the object in the frame and comparing it to the previously collected information, we can obtain the indicators and features for the decision making. Post processing algorithm of the frame after reception of each bounding box is presented in Fig. . The feature set for each frame is then transmitted to the fog unit to be contextualized and change detection determination.

It should be mentioned that the accuracy of the I-SAFE system incorporates algorithms that detect and track the human as the object of interest. Although the algorithms used for human detection and tracking at the edge utilize deep learning architectures and methods, the performance accuracy is not 100% which means that the camera may lose the object of interest and so the features used in the fog node will be disrupted. The I-SAFE framework has a disunified architecture where the performance can be improved by integrating new algorithms.

## Feature Contextualization

The first step toward a dynamic data analytics is to consider all of the features that explain the input data including feature generation method. Feature generation is learned from previous analysis with salient feature selection. In case of human-oriented public safety surveillance, features are directly extracted from the video frames (e.g., intensities, lines, shapes) as well as other external factors (e.g., camera placement, lighting conditions, and scene content). The procedure of putting these features together with the factors will be referred to as *feature contextualization*.

This paper focuses on university campus surveillance and attempts generalization. During the normal operating hours of a building on campus; many students, faculty, staff, and other personnel may be detected in the scene. While it is normal to observe many people during the day, it is abnormal if many people appear after 11:30 pm. In the case of the abnormality detection, the contextual data of the time of day assists in decision making. According to the police, the time of day determines officer vigilance in monitoring attention to a gathering. What context features should be selected and how common they are utilized in cases of surveillance are determined based on the suggestions from our campus police officers. The edge device that hosts the data extraction from the video and prepares the feature list for each frame, cannot handle the contextualization of features due to the resource constraints. The video processing task makes use of most of the computational power while the rest is allocated to the transmission and security modules. Moreover, the context, such as spatiotemporal and geo-location are usually repeated data that sending them with each frame creates overhead. Therefore, the video features are extracted by the edge camera and sent to the fog node for contextualization. There are three features that are added to the camera data during the contextualization phase.

The first is the *time of the day*. The importance of this feature may differ from one location to the other, but it is considered as one of the most important factors to make a decision in security systems. The second factor is the *geo-location of the camera*. Cameras installed indoors should have a different set of thresholds for decision making than the outdoor cameras. Just as the accessibility and space use-case varies, normal behavior changes too. Additionally, cameras that are installed outside of a bank should have less tolerance for detection of a human being after hours and should raise an immediate alarm in such a case. Thus, the *security level* of the building where the camera is mounted is the third context we consider.

# Fuzzy Model

## Rationale

The decision-making process for the safety surveillance system is based on a fuzzy control system model. Although the momentum in FLCs systems is lost due to absence of experts in many challenges, the fuzzy method remains one of the best methods for systems with high noise levels. In surveillance, the law officers can act as the experts and their opinion shall be used for system operation. The officers take months or even years to develop an innate sense of behavior analysis for a certain location of their duty.

A DNN training that can do classification with proper accuracy requires many training sets both negative and positive examples and while negative data is easy to acquire, positive samples are harder to gather. Even after labeling, if the dataset does not contain all scenarios, the result does not cover the whole input space which leads to undetected events.

In case of the campus safety system, there are many campus police officers who have spent time in the domain of the interest and they know what to expect from the crowd. Their experience is used for creating a series of rules that are implemented in the fuzzy model to detect the anomalies. On the other hand, general purpose classifiers with unsupervised learning methods do not offer high accuracy and suffer form noise distortion, trivial solutions, and collapsing of features in deeper models .

With the contextualized features, a fuzzy control system is introduced to mimic how the police officers make decisions and to obtain a semantic output amenable to human operators. Unlike mathematical probability analysis, the fuzzy-based models are not based on numbers, but *semantic classes*. A FLC maps the input sensor measurements to linguistic labels, which are a description of the input. The fuzzy system affords the mapping of operator knowledge into a decision making model.

For anomalous activity detection, the officer performs linguistic-type reasoning in his/her mind and reaches a conclusion of a behavior being normal or abnormal, instead of giving a numeric description of the observation. To mimic this cognitive behavior, the fuzzy model gives a linguistic output which is the classification label. This output can be translated to a number based on the defuzzification formulation that is consumed by digital computers. The results are reported to the police department with the amount of attention (i.e., based on a confidence, credibility, or reliability estimate) needed to assess a specific scene.

## Fuzzy Model

The feature-set for each of the objects of interest is sent to the fuzzy logic controller at the fog node where it contextualize and fed the FLC.

The first step to realizing the fuzzy model is fuzzification, which translates the features to a fuzzy value. For any set $X$, membership functions represent the fuzzy subsets of it. For an element $x$, the fuzzy subset $A$ corresponding value is denoted by $\mu_A(x)$ as shown as Eq. ():

$$A \{x , \mu_A(x) ; x \in X\}
 \label{eq:A}$$

In order to fuzzify the measurements (the contextualized features in this case), each measurement is compared to its respected subsets $\mu_{A_i}(x)$ as shown in Eq. (), which creates the linguistic variables that are used in the rules. Linguistic variables are in contrast to normal variables where each variable presents a range of meanings such as cold, medium, hot.

$$w^x_i \max_x[\min(A^\prime(x), \mu_{A_i}(x))]
 \label{eq:w}$$

where each measurement is considered as a range shown by $A^{\prime}(x)$. If $x$ is only one number as the measurement, then $A^\prime(x)$ becomes only one number too. Note, subscript $i$ represents each of the fuzzifiers (membership functions).

If two sets $X$ and $Y$ are considered in a fuzzy system, based on each fuzzifier in each set for each reading, linguistic variables $w^x_i$ and $w^y_i$ are calculated. The linguistic variables are used in the rule set to calculate the results ($C^\prime_i(z)$) for each rule. The Minimum value between the resulting Premise $W_i$ and each fuzzy membership functions for the output $C_i(z)$ yields Eq. ():

$$\begin{aligned}
 W_i  \min[w^x_i, w^y_i] (AND_{Operand})\nonumber\\
 W_i  \max[w^x_i, w^y_i] (OR_{Operand})\nonumber\\
 C^\prime_i(z) = \min[W_i, C_i(z)]
 \label{eq:rule}
\end{aligned}$$

## Membership Functions

According to Eq. (), the linguistic variable is mapped to an interval \[0, 1\] which can be inferred as a credibility analysis. If the credibility of a subset is less than $0.5$, then the linguistic variable is not reliable enough. This yields to subsets (membership functions) that should cover each set so that at no point $x$, the credibility of aligning a feature to a set falls below $0.5$. This in return yields that the output results of the system can have higher credibility as the inputs are more reliably fuzzified. In addition, it is very important to consider the best membership functions to cover the entire set. Having the wrong candidate as the membership function can lower the credibility of that set faster or the membership function will not cover the desired area in the set.

Figure presents the fuzzy-relation types, where the first five sets (Fig. a - e) are the input membership functions and the last one (f) is the output.

In Fig. the x-axis shows variable $x$ in and the y-axis represents the credibility of membership functions that $x$ belongs to as it changes. In part (d) for example, having five people in the frame, means about $1.0$ confidence in "medium activity" and $0.27$ confidence in "high activity". Then Eq. () determines that having five people in the frame means "medium activity" and it is considered for linguistic variable of set: "NumPpl".

Three membership functions are considered for each feature interval and five membership functions are used to describe all possible levels of suspicious behavior with high accuracy. As illustrated in Fig. , the following contextualized features are presented to the fuzzy system, thus the input is fuzzified and the output is made based on rules:

-   \(a\) The time of the day, which is the hour ranging from \[0:00 to 24:00\];

-   \(b\) The number of the times an object in a frame changes the speed \[0 30\]. If a person walks for a long time in the frame and changes directions many times showing not having a clear destination, there is a good chance of loitering;

-   \(c\) The time that a human object stays in a frame, normally a person walks out of the frame in several seconds if they are walking at normal speed \[0 30\] seconds;

-   \(d\) The number of people in the frame under processing \[0 40\]; and

-   \(e\) The number of the times an object in a frame changes the direction \[0 30\].

The boundaries of each set are designed to handle almost every possible scenario, but in case of outliers, the fuzzy system can handle noise and out of range values very well. The key is the fuzzification process. During the calculation phase, if the values are outliers (out of the set scope), the fuzzification still maps the linguistic variable that is most closest to the edge of the set limit. If the fuzzy system fails to align measurements to set values, the human operator will receive an acknowledgement that the inputs are erroneous and alert the operator to observe the situation in the videos. Reported in an error log, the output is also set to zero so no alarm is raised.

It is noticeable that in the models of Fig. , each set is covered with at least one of the membership functions for any given input, and there is no point on the x-axis that at least one membership function is above $0.5$ of the nominal value. Which implies that the decisions of set classification are made with high credibility.

Having a membership function that has a salient value for a long range translates to higher share and higher power of corresponding linguistic variable. All membership function shapes are chosen keeping this in mind. And is why in the malicious behavior membership functions just one point is the highest confidence in that result so no bias toward one behavior is not added.

In a completely defined fuzzy control system, all possible combinations of the fuzzy linguistic variables should be considered in the rule set. However, there is a exponential relationship between the number of rules and the processing time. Hence, the rule set for I-SAFE is designed to cover all scenarios of interest while utilizing a combination of the features.

## Producing the Output

Figure shows how the features are combined in rules to reach a conclusion. The $Com$ column shows the combination between the linguistic conditions (features). Only logical Intersection ($AND$) and Union ($OR$) are used, and the parenthesis show which logical operation should be executed first between the conditions. Since the speed change and direction change variables are subject to high measurement noise, the I-SAFE system employs the OR operator between them to reduce their impact based on the number of rules using them.

Based on the five linguistic variables respective to each fuzzy set for every contextualized features, the system gives the suspicious activity fuzzy probability. After defuzzification, the geo-location of the camera and building security level can be considered to give an appropriate threshold to raise the alarm. Such that the camera position and building security are considered as the two final features.

Once again, the expert's experience or knowledge enforces the rules to control the environment. The rules may be different corresponding to different conditions. The expert is asked to generalize rules and features that are vital in the decision-making process. Note that each membership function boundary that is used in the input and output generations in Fig. is based on a camera that is installed on a hallway in a campus building. If the area under security surveillance needs more supervision, the administrator can change the fuzzifiers subject to tolerate less activity and/or raise the alarm sooner in certain times. All changes are done at setup and no more adjustments are needed.

The rule set used in I-SAFE system emulates the calculations that an officer performs before approaching a suspicious object. As shown in the fuzzy sets, the number of people in a frame is an important feature. The highest attention is drawn to scenarios where only few people are present at night. Based on the law enforcement experience, the videos that have only one person in frames that are the most important. Therefore, the video footage with one or two people are considered as of high interest. With increase in the number, the interest goes down based on the double sign membership functions. Another key area is the amount of time that the object is present in the location. As the time expands, the probability that the object is loitering goes higher with respect to its set membership functions.

## Suspicious Score

The last step is to defuzzify the results of the rules and translate it to a number between 0 to 100%. A threshold decides whether or not the output should raise an alarm to make operators aware of an activity. Then the operator will make decision for further actions. The $C^\prime_i(z)$ can be defuzzified using Eq. ( ):

$$F^{-1}(c) = \frac{\int (W_i\mu_c(x)) }{\int \mu_c(x)}
 \label{eq:defuzzy}$$

where the $\mu_c$ is the membership function of the output.

The I-SAFE system is able to draw attention to the scene where anomalous or suspicious activities are determined, but it is the human operator that makes the decision of action. The fuzzy model is implemented on the fog level devices and it is easy to access and reconfigure parameters through a single cloud node, for a batch of edge units connected to the fog, if the operator chooses so.

# Experimental Results

A proof-of-concept prototype of the I-SAFE scheme has been implemented and tested using real-world surveillance video streams. The experimental results are encouraging that the design goals are achieved to provide a secure, agile, and fast surveillance system for safety monitoring. The I-SAFE system detects the activity successfully in an average of 0.002 seconds after features are pre-processed.

## System Setup

The prototype consists of both edge and fog layer function units. At the edge, human detection and tracking are accomplished using lightweight L-CNN and Kerman algorithm. Features are created using the indicators and other methods explained previously. The edge functions are hosted by a Tinker Board with 1.8 GHz ARM-based RK3288 SoC and 2 GB LPDDR3 dual-channel. The Tinker Board is placed behind the camera, in this sense, the camera can be considered as the sensor and the edge device is the Tinker that connects to the sensor through Local Area Network (LAN). The features are sent to a laptop PC running Ubuntu 16.04 operating system as a fog node, where the contextualization and fuzzy decision making of the I-SAFE scheme is located. The PC has a 7th generation Intel core i7 processor @3.1GHz and 32 GB of RAM. The wireless connection between the fog and edge is through wireless LAN (WLAN) with 100Mbits/s.

## Threshold Setting

Suspicious activity can be interpreted in different ways; but the goal was to provide a machine-level triage of the situation for cueing operators to human abnormal behavior. For example, the abnormal behavior could point to a bicyclist in a crowded place where everyone else is walking . In another attempt a certain possible pose of a human is a sign of abnormal behavior . The challenge is to determine an ontology of activities that would alert and operator to potential abnormal activity. In this paper, we consider a campus environment, where simply put, the students are unlikely to loiter around in parking lots or in hallways in late hours. The system is designed in such a way that it will require certain thresholds to be met before raising an alarm.

It is important to note the separation of the feature map and decision making algorithm, makes the project more suitable for edge computing paradigm where outsourcing the process to higher link in the hierarchy is indisputable.

Figure is a scenario where two people are walking at their normal pace and they will exit the frame when reaching the end of the hallway. The algorithm follows both objects and outputs the abnormality score corresponding to the measured likelihood of suspicious activity respective to each individual.

Figure compares the malicious scores of these cases. The x-axis is the time an object in the frame in second, and the y-axis is the defuzzified suspicious score. The red line is the score of the case of single person walking at 11:00 am, where the object walked through the corridor in about 100 seconds. The score is in a reasonable range showing no suspicious behavior. In contract, the blue curve is the score of the single person walking at 3:00 am and stays in the area for long time. The suspicious score is rising as time goes by as the blue curve starts at a higher value at time zero than the red curve does, because the scheme considers walking at 3:00 am is more suspicious. As the time of movement increases, the normal activity's score rises slower because of the other parameters that besides the time shows suspicious activity, however the blue line has higher jumps in the score.

Figure does not indicate the threshold to raise the alarm. It can be set conveniently by the system administrator based on the experience of building usage. In another words, setting a reasonable threshold needs a statistical analysis of the distribution of activity scores corresponding to the behavior patterns in this building.

## Performance Evaluation

Figure shows the delay due to the data transfer from edge to the fog. Three scenarios are compared: no encryption, AES (Advanced Encryption Standard) encryption, and AES+RSA (Rivest-Shamir-Adleman) encryption. The AES+RSA has handshake and establishment of connection based on the RSA and the rest of the data transaction is based on AES, which is better in terms of low latency on resource limited devices. It is noticeable that the delay does not have substantial impact to the real-time performance of the system. Since five to eight frames per second is the speed at which the edge device can process the input frames. In addition, with the increment of the number of detected human objects, the feature file being transmitted for each frame gets bigger and also the communication takes more time. As more data is transmitted there are spikes in the transmission times due to unstable network connection. Two scenarios are included in Fig. , one where there are between 0 to 2 objects in the frame and another where we have 6 to 10 objects in each frame for comparison of delay as the files get bigger.

Figure compares the difference between processing the fuzzy model at the edge or at the fog level. The total time shown in Fig. includes the time for data contextualization and the fuzzy control system results. As shown in Fig. , the communication time is much shorter than the time required in decision making process. So the communication time is neglected in Fig. . Note how the edge devices struggles to have around 1.5 (FPS). The same operation takes about 0.002 second on the fog node. Figure justifies outsourcing the fuzzy decision making function to the fog node along with showing the heavy load on the edge devices. Processing a frame and generating a decision in 0.002 seconds in average for human activity detection meets real-time requirements. Considering the velocity of pedestrians, a person cannot move much in 0.1 seconds (processing rate of the current video processing applied at edge) giving ample time for a security response.

In the future, with the introduction of more powerful edge devices, the whole process may be executed at the edge. Figure is generated based on a scenario of two people in the frame. Intuitively, if more people are in the image frame, longer delay is expected both for video processing and decision making.

Figure presents some occasions where the detection and tracking algorithms failed. As explained before, this leads to a lower accuracy of the decision making. Figure includes three instances. Part (a) of this figure shows the object of interest (person in white) at the beginning, who is closer to the camera and is walking past the other person. The tracking algorithm stops following the person in white and stays with the person in red as the red one becomes closer. The detection algorithm, however, detects the person in white again and deletes the other bounding box. Unfortunately, the suspicious score data gathered for the person in the white shirt is lost. Figure (b) shows a scenario where the person in red is far away from the camera and the system failed to detect them. Figure (c) is when the detection algorithm detects only one person instead of two that are in the frame. While this problems exists, it happens very seldom over numerous trails. Finally, Fig. (d) shows a very challenging case, when one object blocks the other, as there is no way to have both two people detected. Each of these issues can be mitigated through intelligent design such as more cameras and enhanced trackers, of which the general operation is scalable.

We also ran the I-SAFE on two publicly available video detests, namely Adam subway entrance and more than 3 hours of video from a mall security camera. One point which in these datasets is very noticeable, is that the original videos have $\times5$ the number of frames compared to what is provided. The main reason for this down sampling goes back to the slow human motion compared to fast cameras of today. Figure shows some of the instances that the I-SAFE detects people and assigns a score to them. In Fig. , the bounding boxes around the humans are shown with the corresponding loitering score around each object. Notice how objects that get further away from the camera are not detected, which is the result of inadequate pixel density for detection or in some congested cases low detection resolution. It is worth highlighting that in Fig. , the blue boxes are the tracker output while the green boxes show the detection algorithm that checks the frames for new objects with the frequency of 1 in 5 frames.

It can be seen in these samples pictures in Fig. , that the camera needs to train to a person with abnormal behavior (here is a man walking on the phone carrying a cardboard piece with different colors on it, for camera detection) for more traditional detection methods.Noted that the proposed I-SAFE does not require training when deployed as camera placement supports the threshold parameters that the operator needs to recalculate.

Finally, Table compares the results of the I-SAFE to the ground-truth and other models for abnormal loitering detection as reported in each paper and shows comparable results. Although the small number of the loitering cases are detected with other algorithms, the I-SAFE achieves these results while minimizing the delay and network overhead in a decentralized edge computing paradigm environment. Observing the scores for the videos in the dataset, we concluded that a threshold of 60% is best to show the abnormal activity using an average CPU of 68.3% on a single thread and 96 MB of memory. Of course, with a higher threshold the system gives less False Positives.

Although these examples are used for model performance analysis, they carry very limited number of positive examples to have an otherwise machine-learning solution to them.

A closer look at Table and Fig. shows that the human activity recognition heavily depends on how accurately it can detect and track each individual. In the mall video segment, because of the frame complexity and the object partial or complete collision, tracking may be interrupted and data may be lost. The advantage of decoupling feature extraction from decision making is that we can use the same fuzzy model with future more accurate video processing techniques.

| **Detection Model** | **Sub. Ent.** |       | **Mall 1** |       | **Mall 2** |       |
|:-------------------:|:-------------:|:-----:|:----------:|:-----:|:----------:|:-----:|
|         2-7         |      TP       |  FP   |     TP     |  FP   |     TP     |  FP   |
|    Ground Truth     |      14       |   0   |     4      |   0   |     4      |   0   |
|     Adam et al.     |      13       | **4** |   **4**    | **1** |   **4**    |   3   |
|     Zhao et al.     |    **14**     |   5   |     NA     |  NA   |     NA     |  NA   |
|    Cocsar et al.    |    **14**     | **4** |     NA     |  NA   |     NA     |  NA   |
|     Kim et al.      |      13       |   6   |     NA     |  NA   |     NA     |  NA   |
|     **I-SAFE**      |      13       | **4** |     3      |   2   |   **4**    | **1** |

Loitering Score in different video samples (TP: True Positive, FP: False Positive, NA: Not Available).

# Conclusions

The smart surveillance system for public safety should be able to detect suspicious people or activities in realtime. Based on the lightweight human object detection and tracking algorithms previously reported, this paper advances proactive surveillance system design by proposing I-SAFE, an instant suspicious activity identification in the edge paradigm using CNN feature extraction and fuzzy decision making. The algorithms to extract features from incoming video stream are implemented on an edge device, which efficiently reduces the communication overhead and enables outsourcing the decision making process to the fog level. The fog device contextualizes the features, fuses the seven features with a fuzzy logic control system, and provides decision making. The rules and features adopted are chosen under the guidance of campus police officers. A proof-of-concept prototype of the I-SAFE scheme has been implemented and tested using real-world surveillance video streams.

Our on-going efforts consider two directions: (1) Adding features to the tracking and classification algorithms to detect gesture for more accurate decision making; and (2) enhancing the lightweight detection and tracking algorithms to tackle the challenging situations shown in Fig. .

# ACKNOWLEDGEMENTS

The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the United States Air Force.


author: Eugenio Carretta, Valentina D'Orazi, Raffaele G. Gratton,; Sara Lucatello
title: A sequence of nitrogen-rich very red giants in the globular cluster NGC 1851[^1]

# Introduction

In the variegated landscape traced by multiple stellar populations in globular clusters (GCs; see Gratton, Sneden & Carretta 2004 and Gratton, Carretta & Bragaglia 2012a for recent reviews that summarize decades of studies on this topic), some objects present a further level of complexity.

This is the case of the globular cluster NGC 1851. Less massive than GCs where multiple main sequences were discovered for the first time (such as $\omega$ Cen, Anderson 1998; and NGC 2808, D'Antona et al. 2005), this cluster contained evidence of multi-modality, starting with the bimodal distribution of stars on its horizontal branch (HB; Walker 1992) and following with the finding of a split subgiant branch (SGB). The origin of this feature is still debated, which possibly invokes explanations of age or CNO-content differences (Milone et al. 2008, 2009, Cassisi et al. 2008, Gratton et al. 2012b). Beside these features, a split in the red giant branch (RGB) was recently detected by Han et al. (2009) with $UVI$ Johnson broadband photometry and by Lee et al. (2009a) using narrow band Ca II and Strömgren photometry. Side by side to a main RGB, another sequence of cluster red giant stars, less populated, is seen in these studies, and it recently was shown to be connected to the faint SGB by Lardo et al. (2012), using Strömgren photometry.

Three effects may produce a redder sequence of red giants in a stellar population:

1.  The metal content (in particular the abundance of the $\alpha$-elements) is higher in the redder sequence. This implies an increase in the number of free electrons, and as consequence of the opacity, giving a redder RGB.

2.  The relative age of the two sequences is different with the reddest one being younger, which causes the RGB to be redder and less steep.

3.  Enhanced N, along with C, may result in a redder sequence.

Lee et al (2009b) inserted NGC 1851 in the sample used to devise an enrichment scenario for multiple populations in GCs where supernovae (SNe) also do contribute to the pollution of the intracluster gas. This scenario is ruled out in most clusters, due to the constancy of their $\alpha$-element content (Carretta et al. 2010a), but we recently found a contribution by core-collapse SNe in a fraction of RGB stars (Carretta et al. 2011) from the abundance analysis of Ca and other $\alpha$-elements in NGC 1851. Moreover, a spread in \[Fe/H\][^2], which was small but exceeding the observational errors, was detected in the same analysis, confirming earlier results by Yong and Grundahl (2008) and Carretta et al. (2010b). Metal-rich and metal-poor giants also show a different radial distribution with more metal-rich stars being less centrally concentrated (see Figure 6 in Carretta et al. 2011); however, these stars are not segregated exclusively along the reddest RGB sequence.

A spread in the total CNO sum was found by Yong et al. (2009) in a limited sample of four RGB stars in NGC 1851. Enhanced CNO abundances in a fraction of stars were also proposed by Cassisi et al. (2008) as a possible explanation for the split SGB. The implication is that small mass AGB stars (those experiencing episodes of third dredge-up) also contribute to the budget of nuclearly processed matter to construct a second generation of stars in NGC 1851. Thus, a fraction of stars may be found to be younger than the other. However, Villanova et al. (2010) challenged this hypothesis in part by finding that the RGB stars on the two sequences do not show any significant difference in the total C+N+O sum but only in their content of $s-$process elements, in particular, for Ba. On the other hand, variations of $s-$process elements like Zr and La were found to be correlated to light elements (like Al) by Yong and Grundahl (2008): this again calls for contribution from thermally pulsing AGB stars of low initial masses.

Using a large set of elemental abundances for a large sample of giants, we recently proposed a classification scheme that divides the RGB stars in NGC 1851 into two populations using a combination of Fe and Ba (Carretta et al. 2011). Our data show that stars are nicely segregated along the split RGB according to their Ba abundances with Ba-poor stars located mostly (but not only) on the bluest side of the RGB, whereas only Ba-rich stars are found on the less populated reddest sequence. This result puts on a more statistically robust base similar findings by Villanova et al. (2010). We recently find evidence of a statistically significant correlation between Al and Ba abundances in the more metal-rich component of RGB stars (Carretta et al. 2012a).

Moreover, using the comparison of Strömgren colours and synthetic spectra computed with varying C and N abundances, we demonstrated that stars on the reddest sequence are very good candidates to be C-rich and N-rich, while stars that are C-rich and N-poor (i.e. O-poor) likely lie side-by-side on the bluest sequence with N-poor and C-normal stars (see figures 23 and 28 in Carretta et al. 2011).

The existence of a Na-O anticorrelation in each population suggests that the classical chain of events (first generation, pollution, second generation, marking the birth of a genuine GC, as proposed in Carretta et al. 2010c) was present in each component. In turn, this may suggest a merger of two formerly individual clusters into an ancestral galaxy or two different, yet spatially close regions of star formation.

Villanova et al. (2010) found a different N content in the two group of stars with different Ba abundances. Campbell et al. (2012) recently studied the CN bandstrengths of a sample of 21 AGB stars and 17 RGB stars in NGC 1851; however, they were selected on the $V,B-V$ color-magnitude diagram, where the red and blue RGB sequence cannot be separated. In their two samples they found a quadrimodal distribution of CN strengths with the more CN-weak populations coupled with low Ba abundances and the more CN-strong peaks of the distributions associated with high Ba values.

Apart from these limited samples of RGB stars, no extensive spectroscopic analysis of the N abundance of objects on the reddest sequence was made up to now. With the present paper, we intend to fill this gap, deriving N estimates for about 60 red giants and providing another constraint for a better understanding of the complex star formation in NGC 1851.

# Observations and analysis

The same observational data was recently used by us (Carretta et al. 2012a) to derive Al abundances for a large sample of RGB stars in NGC 1851: two exposures of 600 sec each made with FLAMES-GIRAFFE mounted at VLT-UT2 and the high-resolution grating HR21 (R=17,300, spectral range from 8484 Å to 9001 Å). We refer to Carretta et al. (2012a) for details on the treatment of spectra (flat-fielding, calibration in wavelength, and sky subtraction). As before, atmospheric parameters (as well as abundances of Fe, O, Na, Mg, Ba etc.) were adopted from the highly homogeneous analysis by Carretta et al. (2011). The present sample is restricted to a magnitude range between $V\sim 17$ and $V\sim 15$.

In Carretta et al. (2012a), only a single feature was used to derive the N abundance, which, assuming \[C/Fe\]=0, reproduces the CN band strength to account for a possible contamination of Al lines. However, N lines are ubiquitous in the observed spectral range, and instead of treating them purely as contaminant, we can use them to derive N abundances for a large number of giants in NGC 1851. Our approach closely follows the one adopted in Carretta et al. (2012b) for the globular cluster M 4, which has a metallicity \[Fe/H\]$=-1.20$ dex (Carretta et al. 2009) similar to that of NGC 1851 (\[Fe/H\]$=-1.16$ dex). However, we remind the reader that the proprietary spectra taken in the case of M 4 were acquired to maximise the signal around the Al lines, whereas the spectra with HR21 retrieved from the public ESO archive for NGC 1851 were observed in a public survey designed for other purposes and do not have an optimal S/N ratio for the task of abundance analysis for elements like N.

To deal with this issue, we used an improved version of the procedure adopted in Carretta et al. (2012a). Briefly, 18 spectral regions dominated by CN bands and 8 regions that are apparently line-free were selected on a master spectrum obtained by summing up the ten spectra of giants with the best S/N value. The list of these regions is given in Carretta et al. (2012b). For each CN feature, we measured the average flux within the in-line region, and this was normalized using a weighted reference continuum where the weights are equal to the width of the continuum regions. The N abundance from each feature was then obtained by comparing the normalized flux with the fluxes measured in the same way on three synthetic spectra computed using the Kurucz (1993) grid of model atmospheres (with the overshooting option switched on). The line list is described in Carretta et al. (2012a), and the synthetic spectra were computed with the package ROSA (Gratton 1988) by assuming \[C/Fe\]=0.0 dex, three evenly spaced values of nitrogen, \[N/Fe\]=-0.5, +0.0, and +0.5 dex, and the appropriate atmospheric parameters for each star.

Because of the non-optimal quality of the spectra, the final abundances of N were obtained after applying a severe k$\sigma$-clipping at 1.5 $\sigma$ to the average abundance from individual features in each star. On average, only ten features survived this culling process. The average abundance ratio we found is \[N/Fe\]$=-0.082 \pm 0.037$ dex with rms=0.294 dex from 62 stars.

As explained in Carretta et al. (2012a), an error due to flux measurement can be estimated for abundances derived through this procedure. This is obtained by estimating photometric errors from the S/N ratio of the spectra and from the width within each of the reference continuum and in-line regions (see details in Carretta et al. 2012a). The N abundances, derived by summing the original value of the CN band strength and its error, allow us to estimate the error associated to N abundances due to errors in the flux measurements. When we applied this procedure to a random CN feature used to estimate the N abundance, we found an average error of $+0.120 \pm 0.028$ dex from 62 stars in NGC 1851, which is a conservative estimate since the final N abundances rest on approximatively ten features per star. Actual star-to-star errors (those really relevant in the problem under scrutiny here) are then about 0.04 dex. Internal errors imputable to errors in the adopted atmospheric parameters have a similar impact, since star-to-star errors in effective temperature, gravity, model abundance and microturbulent velocity are only 4 K, 0.041 dex, 0.034 dex, and 0.16 km/s, respectively, as estimated in Carretta et al. (2011). The total error expected in N abundance due to these uncertainties is 0.137 dex. However, these uncertainties have a negligible impact on our results, since they affect stars on both sequences.

Unfortunately, we have no stars in common with the small sample studied by Yong et al. (2009). In our sample, there are two stars in common with the analysis of Villanova et al. (2010), one in their Ba-poor group (star 35999) and the other in their Ba-rich group (star 29203). Taking into account the C abundance (assumed, in our case; derived, in Villanova et al. 2010), we found a difference (in the sense Villanova minus us) of 0.30 dex in \[N/Fe\] for star 35999 and only 0.10 dex for star 29203. The comparison is not conclusive, owing to the different features and methods adopted in the two studies and resting on two objects only. For the aim of this analysis, the comparison of relative N abundances along the two sequences of the RGB, it is, however, the internal self-consistency that actually matters.

# Results and discussion

The derived values of the \[N/Fe\] ratios for the 62 stars in our sample are listed in Table 1, and their distribution is shown in Fig. .

As a working hypothesis, we defined the giants having \[N/Fe\]$\leq -0.15$ dex as N-poor stars and those with \[N/Fe\]$>-0.15$ dex as N-rich giants . The separating value simply corresponds to the dip in the histogram in Fig. .

These two groups are indicated with different symbols in Fig. , where stars in our sample are plotted in the $y,u-y$ colour-magnitude diagram (CMD) using the unpublished Strömgren photometry kindly provided to us by Jae-Woo Lee (private communication). From this CMD it is clear that all stars on the less populated, redder sequence are N-rich apart from a single outlier (possibly due to a combination of uncertainties in the photometry and/or N abundance). The bulk of giants, however, lie on the bluest RGB, which consists of a mix of N-rich and N-poor stars.

To represent this finding in a more quantitative way, we traced by eye the mean ridge line passing through the bluest sequence in the $y,u-y$ plane, using the N-poor stars as reference. Then, for each star, we computed the colour differences between the $u-y$ observed colour and the colour of the mean ridge line at the same magnitude $y$. The distributions of these residuals are plotted in Fig.  separately for the groups of N-rich (lower panel) and N-poor (upper panel) giants.

The two distributions are clearly different, as confirmed by a statistical Kolmogorov-Smirnov test (see Fig. ): the two populations have a very low probability (0.0052) of being extracted from the same parent sample. Thus, the main conclusion of the present study is that the redder, less conspicous, secondary sequence on the RGB of NGC 1851 seems to be populated only by N-rich stars. The N-poor stars are almost completely segregated on the bluest part of the RGB with a fraction of N-rich giants.

There are six stars in common between our analysis and the sample of 17 RGB stars analyzed by Campbell et al. (2012). Using the Strömgren photometry available to us for these 6 objects, we see that the two stars with the highest values of $\delta$S(3839) (stars 41113 and 32112 in Table 1 of Campbell et al. 2010) lie on the redder RGB sequence. The other four stars (46228,44414,47385, and 44939) are located on the blue sequence and all have lower values of $\delta$S(3839).

In our study, we presented the first spectroscopic derivation of N abundances of giants in this cluster made for such a large number of stars on the RGB, which nicely supports the results by Villanova et al. (2010) and Campbell et al. (2012) based on much more limited samples. While we cannot state anything on the total C+N+O sum with not knowing the C abundance of the individual objects, we are able to spectroscopically ascertain that stars are located on the redder sequence in NGC 1851 because they are N-rich.

Moreover, the present work strongly confirms the findings by Carretta et al. (2011) based on an independent approach. Simply from Strömgren photometry and explorative spectral synthesis of a C-rich/N-rich and of a C-rich/N-poor typical giant in NGC 1851 from Carretta et al. (2011), we concluded that stars on the redder sequence were "good candidate to be C-rich and N-rich (hence O-poor). On the other hand, the C-rich and N-poor stars (i.e., O-rich) may not be separated from the other N-poor and C-normal stars in none of the Strömgren indices" (Carretta et al. 2011). *In the present paper, we were able to confirm this statement by deriving directly the N abundances of 62 RGB stars.*

For 60 stars in our sample we have the values of the $hk$ index, kindly provided by Jae-Woo Lee (private communication). The mean value of the index for stars on the redder sequence is +0.927 ($rms=0.104$), whereas we found a mean value of +0.799 ($rms=0.114$) for stars with normal N abundances. This difference can be compared to the Full Width Half-Maximum (FWHM) of the RGB in NGC 1851 in $hk$ found by Lee et al. (2009b): 0.182 with a measurement error of 0.013 (their supplementary Table 3). A direct comparison to the N abundances found for BHB stars by Gratton et al. (2012c) is not easy because different abundance indicators (high-excitation lines) were used in that study, and non-LTE corrections are likely not negligible. Moreover, this comparison is hampered by the lack of knowledge of the absolute C values for our sample, where N abundances are derived from CN features. Finally, we found no evidence of radial variations in N abundances in the present sample.

Recently, Lardo et al. (2012) showed that the red sequence on the RGB of NGC 1851 is photometrically connected to the faint SGB discovered by Milone et al. (2008). This result, when coupled with evidence from spectroscopy in the present and previous studies (e.g. see Yong et al. 2009 and Villanova et al. 2010 for a different view), gives support to the suggestion by Cassisi et al. (2008) that an increased amount of CNO (a factor about two) in stars of the faint SGB may explain the difference between the split sequences.

It is also well assessed (Villanova et al. 2010, Carretta et al. 2011, Campbell et al. 2012) that only Ba-rich stars are found on the redder sequence of the RGB with Ba-poor stars mostly (but not exclusively) confined to the more populated bluest part of the giant branch. This close similarity with the pattern found here for N abundances points towards the action of first generation polluters which experience episodes of third dredge-up, stars with initial mass in the range 1.5-3 $M_\odot$. Moreover, Gratton et al. (2012b) investigated the abundance pattern of stars belonging to the two SGBs in NGC 1851; they found that stars on the faint SGB are Sr and Ba-rich, while the group on the bright SGB show a low content of these elements. These findings point out that the observed spread of $s-$process elements, currently traced from the SGB up to the RGB, is real in this cluster.

On the other hand, the correlation between proton-capture elements (like Na and Al) and the amount of $s-$process elements found by us (Carretta et al. 2011, 2012a) and other studies (Yong and Grundahl 2008; Villanova et al. 2010) presents a problem related to the different evolutionary times of different classes of first generation polluters. As discussed by Carretta et al. (2011), if the polluters were AGB stars, those producing proton-capture elements, such as Al, are in the mass range between 4 and 8 $M_\odot$. This leaves a time delay of about 200 Myr before the effects of mass loss from low-mass AGB stars (producing the bulk of $s-$process and enhanced CNO sum) may be incorporated in the gas forming stars of the second generation.

A possible solution of this puzzle is if the two sources of pollution are decoupled in origin. In other words, suppose that the gas enriched in $s-$process elements by low-mass AGB stars is not forged by stars of the forming proto-cluster. This gas may, for example, be collected in the potential well of a star-forming region immersed into a larger dwarf galaxy where matter leftover by a generation of low-mass AGB stars slowly inflows towards the denser proto-cluster regions. Here, this gas "unprocessed" by first generation cluster stars may combine with nuclearly processed ejecta and eventually form stars of a second generation that are rich in both Al and N and $s-$process elements. Such a variant to the first-generation-pollution-second generation sequence, where part of the pollution is not due to the same cluster stars, was originally introduced by Bekki et al. (2007).

As an alternative, if fast-rotating massive stars (Decressin et al. 2007) are responsible for the enrichment in proton-capture elements like Na and Al, then the low-mass AGB stars within the first generation of cluster stars may contribute to the enrichment without any particular problem that is related to the age difference between intermediate and low-mass AGB stars.

```latex
\begin{table*}
\centering
\caption[]{Abundance ratios [N/Fe] for RGB stars in NGC~1851.}
\begin{tabular}{lccrlccr}
\hline
star  & [N/Fe]  & rms  & nr$^1$ & star & [N/Fe]  & rms  & nr$^1$ \\
      & (dex)   & (dex)&     &      & (dex)   & (dex)&    \\
\hline
  13618 & $-$0.223 &0.165 &14 &  32256 & $-$0.195 &0.134 &11 \\
  14827 &   +0.032 &0.424 &10 &  35750 & $-$0.033 &0.215 &10 \\
  16120 & $-$0.214 &0.106 &11 &  35999 & $-$0.031 &0.125 & 7 \\
  20189 & $-$0.064 &0.231 & 9 &  36292 &   +0.508 &0.415 & 8 \\
  20426 & $-$0.473 &0.250 &11 &  38484 & $-$0.201 &0.236 &14 \\
  20653 &   +0.227 &0.119 & 9 &  38818 &   +0.026 &0.222 & 8 \\
  20922 & $-$0.170 &0.025 & 8 &  39364 & $-$0.204 &0.255 &16 \\
  21453 &   +0.052 &0.040 & 8 &  40300 & $-$0.106 &0.084 &11 \\
  21830 &   +0.086 &0.280 &11 &  40615 & $-$0.035 &0.033 & 7 \\
  22360 & $-$0.146 &0.243 &11 &  41113 & $-$0.054 &0.076 & 8 \\
  22588 & $-$0.417 &0.060 & 7 &  41855 & $-$0.387 &0.500 & 9 \\
  22813 & $-$0.310 &0.158 &12 &  43528 &   +0.389 &0.397 &11 \\
  23647 & $-$0.078 &0.109 &10 &  44224 &   +0.092 &0.665 &12 \\
  23765 &   +0.161 &0.081 &11 &  44414 & $-$0.196 &0.085 & 9 \\
  25037 & $-$0.265 &0.409 &13 &  44939 & $-$0.264 &0.060 &11 \\
  25497 &   +0.173 &0.190 & 8 &  45006 & $-$0.575 &0.418 &13 \\
  25678 & $-$0.709 &0.699 &11 &  45090 & $-$0.197 &0.256 &13 \\
  25799 & $-$0.570 &0.278 &10 &  45413 &   +0.300 &0.172 &11 \\
  26532 &   +0.178 &0.135 & 7 &  46228 & $-$0.195 &0.146 &12 \\
  26552 &   +0.035 &0.123 &10 &  46657 &   +0.063 &0.449 &12 \\
  26880 &   +0.424 &0.344 & 8 &  46958 & $-$0.462 &0.124 &10 \\
  27491 & $-$0.402 &0.089 & 7 &  47385 & $-$0.428 &0.127 &10 \\
  28116 &   +0.195 &0.068 & 7 &  47795 &   +0.380 &0.198 &13 \\
  29203 &   +0.313 &0.161 &13 &  48085 & $-$0.248 &0.037 &10 \\
  29470 &   +0.493 &0.124 & 9 &  48277 &   +0.212 &0.238 & 8 \\
  30286 &   +0.150 &0.116 & 8 &  48388 &   +0.063 &0.261 &14 \\
  31284 & $-$0.186 &0.116 &11 &  49965 &   +0.177 &0.133 & 8 \\
  31399 &   +0.317 &0.247 & 9 &  50876 & $-$0.670 &0.132 & 8 \\
  31463 & $-$0.301 &0.390 &10 &  50973 & $-$0.330 &0.071 & 9 \\
  31520 & $-$0.002 &0.162 & 9 &  51311 & $-$0.225 &0.207 &10 \\
  32112 &   +0.049 &0.137 &13 &  52570 & $-$0.639 &0.700 &11 \\
\hline
\end{tabular}
\label{t:abu}
\begin{list}{}{}
\item[1-] number of CN features used to derive [N/Fe]
\end{list}
\end{table*}
```

[^1]: Based on observations collected at ESO telescopes under programme 188.B-3002

[^2]: We adopt the usual spectroscopic notation, which is for any given species X, \[X\]= $\log{\epsilon(X)_{\rm star}} - \log{\epsilon(X)_\odot}$ and $\log{\epsilon(X)}=\log{(N_{\rm X}/N_{\rm H})}+12.0$ for absolute number density abundances.