Semantic Driven Multi-Camera Pedestrian Detection
Alejandro López-Cifuentes Marcos Escudero-Viñolo Jesús Bescós Pablo Carballeira










Alejandro López-Cifuentes (Corresponding Author) Universidad Autónoma de Madrid. Video Processing and Understanding Lab (VPULab).    alejandro.lopezc@uam.es    Marcos Escudero-Viñolo Universidad Autónoma de Madrid. Video Processing and Understanding Lab (VPULab). Email: marcos.escudero@uam.es Jesús Bescós Universidad Autónoma de Madrid.  Video Processing and Understanding Lab (VPULab). E-mail: j.bescos@uam.es Pablo Carballeira Universidad Autónoma de Madrid. Video Processing and Understanding Lab (VPULab). E-mail: pablo.carballeira@uam.es




López-Cifuentes et. al.

Semantic Driven Multi-Camera Pedestrian Detection
    Alejandro López-Cifuentes, Marcos Escudero-Viñolo, Jesús Bescós, and Pablo Carballeira
    
==========================================================================================




In the current worldwide situation, pedestrian detection has reemerged as a pivotal tool for intelligent video-based systems aiming to solve tasks such as pedestrian tracking, social distancing monitoring or pedestrian mass counting. Pedestrian detection methods, even the top performing ones, are highly sensitive to occlusions among pedestrians, which dramatically degrades their performance in crowded scenarios. The generalization of multi-camera set-ups permits to better confront occlusions by combining information from different viewpoints. In this paper, we present a multi-camera approach to globally combine pedestrian detections leveraging automatically extracted scene context. Contrarily to the majority of the methods of the state-of-the-art, the proposed approach is scene-agnostic, not requiring a tailored adaptation to the target scenario—e.g., via fine-tunning. This noteworthy attribute does not require ad hoc training with labelled data, expediting the deployment of the proposed method in real-world situations. Context information, obtained via semantic segmentation, is used 1) to automatically generate a common Area of Interest for the scene and all the cameras, avoiding the usual need of manually defining it; and 2) to obtain detections for each camera by solving a global optimization problem that maximizes coherence of detections both in each 2D image and in the 3D scene. This process yields tightly-fitted bounding boxes that circumvent occlusions or miss-detections. Experimental results on five publicly available datasets show that the proposed approach outperforms state-of-the-art multi-camera pedestrian detectors, even some specifically trained on the target scenario, signifying the versatility and robustness of the proposed method without requiring ad-hoc annotations nor human-guided configuration.















§ INTRODUCTION
 
In the current worldwide situation, pedestrian detection has reemerged as a pivotal tool for intelligent video-based systems aiming to solve tasks such as pedestrian tracking, social distancing monitoring or pedestrian mass counting. Automatic people detection is generally considered a solid and mature technology able to operate with nearly human accuracy in generic scenarios <cit.>. However, the handling of severe-occlusions is still a major challenge <cit.>. Occlusions occur due to the projection of the 3D objects onto a 2D image plane. Although recent deep-learning based methods are able to cope with partial occlusions, the detection process fails when only a small part or no part of the person is visible. To cope with severe-occlusions, a potential solution is the use of additional cameras: if they are adequately positioned, the different points of view might allow for disambiguation.




Disambiguation is generally achieved by projecting every camera's detections on a common reference plane. The ground plane is usually the preferred option as it constitutes a common reference in which people's height can be disregarded. Per-camera detections can then be combined on the ground plane to refine and complete pedestrian detection. However, there are several challenges to be addressed during this combination or fusion process. Among the striking ones are: the convenience to define common visibility areas where cameras' views overlap, and how to cope with camera calibration errors and persons’ self-occlusions. See Figure <ref> for visual examples of these challenges, which we detail below:

In multi-camera approaches a common strategy is to define an operational area or Area Of Interest 𝒜𝒪ℐ on the ground plane. This area represents the overlapping field-of-view of all the involved cameras. It can be used to reduce the impact of calibration errors in the process and to generally ease the fusion of per-camera detections. This area is generally manually defined for each scenario, precluding the automation of the process.

Scene calibration is a well-known task <cit.> which can be performed either manually or using automatic calibration methods based on image cues. In both cases, small perturbations in the calibration process may cause uncertainty in the fusion of the detections on the ground plane. The impact of calibration errors increases with the distance to the camera: generally, calibration is more accurate for pixels belonging to objects close to the camera.

Self-occlusions are caused by the intrinsic three-dimensional nature of people, resulting in the occlusion of some human parts by some others. If the visible parts are different for different cameras and these are used to project a person location on the ground plane, the cameras' projections will diverge, hindering their fusion.

To cope with these challenges, in this paper we present a multi-camera pedestrian detection method which is driven by semantic information automatically extracted from the 2D images and transferred to the 3D ground plane, and includes the following novel contributions:

  
  * A novel approach to globally combine pedestrian detections in a multi-camera scenario by creating connected components in a graph representation of detections. 
  
  * An height-adaptive optimization algorithm which uses semantic cues to globally refine the location and size of people detections by aggregating information from all the cameras.

The proposed method is applied over an operational area in the ground plane, which is automatically defined  by an adaptation of the method described in <cit.>.

Experimental results on public datasets (PETS 2009 <cit.>, EPFL RLC <cit.>, EPFL Terrace <cit.> and EPFL Wildtrack <cit.>) prove that the proposed method: (1) outperforms state-of-the-art monocular pedestrian detectors <cit.>, (2) outperforms state-of-the art scene-agnostic multi-camera detection approaches; and (3) results in a performance comparable, and even better, to deep-learning multi-camera detection approaches trained and fine-tuned to the target scenario, while not requiring neither a manually annotated operational area nor a specific training on that scenario.

The rest of the paper is organized as follows: Section <ref> reviews the State of the Art, Section <ref> describes the proposed method, Section <ref> presents and discusses experimental results and Section <ref> concludes the paper.








§ RELATED WORK
 
Multi-camera people detection faces the combination, fusion and refinement of visual cues from several individual cameras to obtain more people locations.
A common pathway in existing approaches starts by defining an operational area, either manually or, as we propose, based on a semantic segmentation. Then, approaches combining detections using a common reference plane, usually follow a three-stages strategy: (1) extract detections on each camera frame, (2) project detections onto the common plane and (3) combine detections and back-project them to the individual views to obtain per-camera people detections. Finally, obtained detections are sometimes post-processed to further refine their localization.



 §.§ Definition of the operational area
 
Some approaches <cit.> rely on manually annotated operational areas where evaluation is performed. An advantage of these ad hoc areas is that the impact of camera calibration errors is limited and controlled. Besides, these areas are defined to maximize the overlapping between the field of view of the involved cameras. However, the manual annotation of these operational areas hinders the generalization of people detection approaches. Our previous work in this domain <cit.> resulted in an automatic method for the cooperative extraction of operational areas in scenarios recorded with multiple moving cameras: semantic evidences from different junctures, cameras, and points-of-view are spatio-temporally aligned on a common ground plane and are used to automatically define an operational area or Area of Interest (𝒜𝒪ℐ).



 §.§ Semantic Segmentation
 
Semantic segmentation is the task of assigning a unique object label to every pixel of an image. During the last years, top performing strategies evolved from the seminal Fully Convolutional Network scheme <cit.> and the use of dilated convolutions <cit.>. For instance, Zhao et al. <cit.> proposed to implicitly use contextual information by including relationships between different labels—e.g. an airplane is likely to be on a runway or flying in the sky but not on the water. These relationships reduce the inner complexity of datasets with large sets of labels, generally improving performance. Lately, the development and use of new backbones for feature extraction has benefited the task. Zhang et al. <cit.> proposed a new ResNet modification called ResNeSt that uses channel-wise attention to capture cross-feature interactions and learn diverse object representations. Similarly, Tao et al. <cit.> proposed the use of a hierarchical attention to combine multi-scale predictions, increasing the performance on the small object instances, as those in PASCAL VOC dataset <cit.>.



 §.§ Monocular people detection
 
As stated in Section <ref>, automatic monocular pedestrian detection is considered a mature technology able to obtain accurate results in a broad range of scenarios. Well established object detectors based on CNNs as Faster-RCNN <cit.> and YOLOv3 <cit.>, have demonstrated their reliability during the last years. Adapting their core schemes, recent approaches have further increased their performance. Specifically, YOLOv3 has been improved by both decreasing the complexity of the model through new architecture designs <cit.> and by efficient model scaling <cit.>.

Alternatively, novel detectors—also based on CNNs, have been proposed. Tan et al. <cit.> proposed a weighted bi-directional feature pyramid network allowing easy and fast multi-scale feature fusion, and obtaining a new family of detectors called EfficientDet, that achieved a new state-of-the-art performance in the COCO dataset <cit.>. Alternatively, Zhu et al. <cit.> proposed to use attention in the form of deformable transformers to also obtain state-of-the-art results.

Nevertheless, even though the most recent works have demonstrated really high performances, in scenarios with severe-occlusions the performance of these algorithms decreases.



 §.§ Projection of per-camera detections
 
Multi-camera pedestrian detection is fundamentally based on the projection of monocular detections onto a common reference plane. Projection is typically achieved either by using calibrated camera models that relate any 2D image point with a corresponding referenced 3D world direction <cit.>, or by relying on homographic transformations that project image pixels to a specific 3D plane <cit.>. In both cases, the ground plane, where people is usually standing on, is chosen as reference for simplicity reasons.



 §.§ Fusion and refinement of per-camera detections
 
Fusion and refinement approaches can be mainly divided into three different groups depending on how global detections are obtained. The first group encompasses geometrical methods, which combine detections based on the geometrical intersections between image cues. The second group embraces probabilistic methods, that combine detections via optimization frameworks and statistical modeling of the image cues. The third group is composed of solutions based on the ability of deep learning architectures to model occlusions and achieve accurate pedestrian detection at scene level. 

Regarding geometrical methods, detections are combined by projecting foreground masks to the ground plane in a multi-view scenario: the intersection of foreground regions leads to pedestrian detection <cit.>. Accuracy can be increased by projecting the middle vertical axis of pedestrians, leading to a more accurate intersection on the ground plane and, therefore, to a better estimation of the pedestrian's position <cit.>. Following the same hypothesis,  the use of a space occupancy grid to combine silhouette cues has been proposed: each ground pixel is considered as an occupancy sensor and observations are then used to infer pedestrian detection <cit.>. All of these approaches outperform single-camera pedestrian detection algorithms by the use of ground-plane homography projections. Nevertheless, the evaluation of foreground intersections in crowded spaces may lead to the appearance of phantoms or false detections. To handle this problem, the general multi-camera homography framework has been extended by using additional parallel planes to the ground plane <cit.>. The intersection of the image cues with these parallel planes is expected to suppress these phantoms. Similarly, parallel planes can be also used to create a full 3D reconstruction of pedestrians, that can then be back-projected to each of the camera views, improving monocular pedestrian detection <cit.>. Finally, Lima et al.<cit.> replicates a preliminar version of the method proposed in this paper, which is available as a preprint <cit.>, with the addition of people re-identification features to guide the fusion of per-camera detections.

Among probabilistic methods, an interesting example is the use of a multi-view model shaped by a Bayesian network to model the relationships between occlusions <cit.>. Detections are here assumed to be images of either pedestrians or phantoms, the former differentiated from the latter by inference on the network.

Recent approaches are focused on deep learning methods. The combination of CNNs and Conditional Random Fields (CRF) can be used to explicitly model ambiguities in crowded scenes <cit.>. High-order CRF terms are used to model potential occlusions, providing robust pedestrian detection.  Alternatively, multi-view detection can be handled by an end-to-end deep learning method based on an occlusion-aware model for monocular pedestrian detection and a multi-view fusion architecture <cit.>.



 §.§ Improving detection's localization
 
Algorithms in all of these groups require accurate scene calibration: small calibration errors can produce inaccurate projections and back-projections which may contravene key assumptions of the methods. These errors may lead to misaligned detections, hindering their later use. To cope with this problematic, one can rely on an Height-Adaptive Projection (HAP) procedure in which a gradient descent process is used to find both the optimal pedestrian's height and location on the ground-plane by maximizing the alignment of their back-projections with foreground masks on each camera <cit.>.    











§ PROPOSED PEDESTRIAN DETECTION METHOD
 
The proposed method is depicted in Figure <ref>. First, state-of-the-art algorithms for monocular pedestrian detection and semantic segmentation are used to extract people detections and the semantic cues for each camera respectively. These cues drive the automatic definition of the 𝒜𝒪ℐ, and detections outside this area are discarded. Surviving per-camera detections are combined to obtain global 3D detections by establishing rules and constraints on a disconnected graph. These detections are back-projected to their original camera views in order to further refine their location and height estimates.







 §.§ Preliminaries
 
Monocular Pedestrian Detection: is performed using a state-of-the-art detector. In order to avoid a potential height-bias, we ignore the height and width of the detected bounding boxes, i.e. the j^th pedestrian detection at camera k is just represented by the middle point of the base of its bounding box: 𝐩_j,k= (x,y,1)^T, in homogeneous coordinates[we use common notation, upper case to denote 3D points/coordinates and lower case to denote 2D camera plane points/coordinates.].


Semantic Segmentation: is performed using a state-of-the-art semantic segmentation algorithm. The method is used to label each image pixel 𝐩_k for every camera k and every frame n: l_n(𝐩_k) = 𝑠_𝑖, where 𝑠_𝑖 is one of the L pre-trained semantic classes: S= {𝑠_𝑖}, where i ∈ [1,L], i.e. floor, building, wall... Figure <ref> depicts examples of semantic labels for selected camera frames of the Terrace Dataset <cit.>.


Projection of People Detections: Let ℋ_k be the homography matrix that transforms points from the image plane of camera k to the world ground-plane. The j^th detection of camera k, 𝐩_j,k=(x,y,1)' is projected onto the ground plane by: 



    𝐏_j,k = ℋ_k ×𝐩_j,k = (X, Y, T)',

which corresponds to a (X=X/T,Y=Y/T,Z=0)' 3D point of the ground plane.




 §.§ Pedestrian Semantic Filtering
 




  §.§.§ Automatic Definition of the 𝒜𝒪ℐ

To obtain a semantic partition of the ground-plane, an adaptation of <cit.> for static-camera scenarios is carried out. We first project every image pixel 𝐩_k via ℋ_k. Every projected point 𝐏_k inherits the semantic label assigned to 𝐩_k: 

    l_n(𝐏_k) = l_n(𝐩_k) = s_i ∈ S.

Thereby, a semantic locus—a ground-plane semantic partition, is obtained for each camera. The extent of each locus is defined by the image support, and missing points inside the locus are completed by nearest-neighbor interpolation.

In order to globally reduce the impact of moving objects and segmentation errors, we propose to temporally aggregate each locus along several frames. In a set of T loci obtained for T consecutive frames, a given point on the ground plane 𝐏_k is labeled with T semantic labels, which may be different owing to inaccuracies in the semantic segmentation or to the presence of moving objects. A single temporally-smoothed label l̅_n(𝐏_k) is obtained as the mode value of this set. Examples of these per-camera obtained smoothed loci are included in the first four-columns of Figure <ref>.

We propose to combine these loci to define the 𝒜𝒪ℐ. The definition of the 𝒜𝒪ℐ is scenario-dependent but can be generalized by defining a set 𝒢 of ground-related semantic classes: floor, grass, pavement, etc. The operational area 𝒜𝒪ℐ is obtained as the union of the projected pixels from any camera which are labeled with any class in 𝒢:


    𝒜𝒪ℐ = ⋃_k^K 𝐏_k,   s.t.  l̅_n(𝐏_k) ∈𝒢.


An example of a so-obtained 𝒜𝒪ℐ is included in the right-most column of Figure <ref>.







  §.§.§ Detection Filtering

Projected detections 𝐏_j,k lying outside the operational area, 𝐏_j,k∉𝒜𝒪ℐ, are filtered out and so, discarded for forthcoming stages.




 §.§ Fusion of Multi-Camera Detections
 
We propose a geometrical approach to combine detections on the ground-plane. Every camera single detection is considered a vertex of a disconnected graph located in the reference plane. Vertices are then joined generating connected components C_m, each representing a joint 3D global detection. The whole fusion process is summarized in Figure <ref>. The conditions that shall be satisfied to join two vertices or detections, 𝐏_j,k and 𝐏_j',k', are:





  
  * That vertices in a connected component are close enough. The l_2-norm between any two vertices in C_m shall be smaller than a predefined distance R_1: ‖𝐏_j,k, 𝐏_j',k'‖ _2≤ R_1 (Figure <ref> (a)). R_1 may be fixed in the interval between 2.5 and 3.5 with no influence in the results. We experimentally set R_1 = 3 meters to: 1) reduce the computational cost of the final stage (see below) assuming that vertices separated R_1 do not belong to the same object and 2) protect against calibration errors, assuming that they are not larger than R_1.
  
  * That vertices in a connected component come from different cameras. This condition prevents the joining of two different detections from the same camera which are near in the ground plane. (Figure <ref> (b))

To avoid ambiguities, the creation of connected components is performed in order, according to the spatial position of the detections: those with a lower module are combined first.

The outcome of the fusion process for K cameras is a set of M connected components { C_m,   m= 1,...,M }, each containing K_m detections: | C_m | = K_m ≤ K, where K_m < K when a person is occluded or not detected in one or more cameras.

As each connected component is assumed to represent a single person, an initial ground-position of the person 𝐏_m^𝒢 = (X_m, Y_m, Z_m=0)^T is obtained by simply computing the arithmetic mean of all the detections in the C_m connected components (Figure <ref> (c)).




 §.§ Semantic-Driven Back-Projection
 
To obtain correctly positioned detections, i.e. visually precise detections, in each camera, ground-plane detections need to be back-projected to each camera and 2D bounding-boxes enclosing pedestrians need to be outlined based on these projections.





  §.§.§ The problem of back-projecting 3D detections

Let 𝐏_m be an orthogonal line segment to the ground plane which represents the detected pedestrian and extends from the detection 𝐏_m^𝒢 to a 3D point h_m meters above. Using the camera calibration parameters, the segment 𝐏_m can be back-projected onto camera k. This back-projection defines a 2D line segment 𝐩_m,k, which extends between 𝐩_m,k and 𝐩_m,k + η⃗ (see Figure <ref> (a)).

We propose to create 2D bounding-boxes around these back-projected 2D line segments. To this aim, each segment is used as the vertical middle axis of its associated 2D bounding-box 𝐛_m,k. For simplicity, the width of 𝐛_m,k is made proportional to its height. Due to pedestrian self-occlusion, calibration errors and the uncertainty on the pedestrians' height, this back-projection process results in misaligned bounding-boxes (see Figure <ref> (a)), hindering their later use and degrading camera-wise performance. 

To handle this problematic, we define an iterative method which aims to globally optimize the alignment between all 3D detections and their respective views or back-projections in all cameras. This method is based on the idea proposed in <cit.>. While the referenced method is guided by a foreground-segmentation, we instead propose to use a cost-function driven by the set of pedestrian-labeled pixels Ω_k from the semantic segmentation (e.g. see person label in Figure <ref>). Next, we detail the full process for the sake of reproducibility. 




  §.§.§ Method Overview

As a 3D detection 𝐏_m, with height h_m, inevitably results in misaligned back-projected 2D detections, the proposed method tries to adapt the 3D detection segment to each camera, generating a set of 3D detection segments, 𝐏_m,k, for each 3D detection and iteratively modifying their positions and height to maximize 2D detections' alignment with the semantic segmentation masks, while constraining all the segments to have the same final height h'_m (as they are all projections of a same pedestrian) and to be located sufficiently close to each other. This process is not performed independently for each 3D detection but jointly and iteratively for all 3D detections. Observe that the joint nature of the optimization problem for all 3D detections is a key step as pedestrian pixels Ω_k may contain segmentations from more than one pedestrian.

For each 3D segment 𝐏_m, the method starts by initializing (i.e., iteration i=0) the per-camera adapted segments:


    𝐏^(  i=0 )_m,k = 𝐏_m   ,  k=1...K.




  §.§.§ Iterative steepest-ascent algorithm

For each 3D segment, let 𝒫^( i )_k = {𝐏^( i)_m,k,   m=1...M } be the set of adapted detections to camera k at iteration i, and let ℙ^( i)= {𝒫^( i)_k,   k=1...K } be the set of camera-adapted segments for all cameras at the same iteration.

The optimization process aims to find ℙ^*, the solution to the constrained optimization problem:



    ℙ^*    = arg max_ℙ Ψ(ℙ), s.t ‖𝐏_m, 𝐏_m,k‖_2≤ R_2  ∀ (m, k),

where R_2 defines the maximum distance between 3D projections of a single pedestrian, which we set to twice the average width of the human body, i.e. 1 meter, to forestall the effect of nearby pedestrians in the image plane. Performed experiments suggest that variations in R_2 value have no significant influence on the results. 

Ψ(ℙ) is defined as the cost function to maximize and is based on the alignment of the back-projected bounding boxes with the set of pedestrian-labeled pixels in each camera: Ω_k. The cost function considers the information from all the cameras. 



    Ψ(ℙ^( i))=-∑_k=1^K ∑_𝐩 γ(𝐩,Ω_k)  Φ(𝐩,ℙ^( i))/|F_k|,

where γ(𝐩,Ω_k) is a weight for pixel 𝐩: ω for pedestrian and ω/3 for non pedestrian pixels, ω=1 in our setup, |F_k| is the number of pixels in the camera image plane and Φ(𝐩,ℙ) is the loss function of pixel 𝐩 with respect to ℙ:



    Φ(𝐩,ℙ^{ i }) = 
        ∏_m|𝐩∈𝐛^( i)_m,k (1-1/d_m,k))         ,   if  l_k(𝐩) ∈Ω_k
    
    
              1 - ∏_m|𝐩∈𝐛^( i)_m,k (1-1/d_m,k))     ,   if  l_k(𝐩)  ∉Ω_k,

where d_m,k is the distance from 𝐩 to the vertical middle axis 𝐩^( i)_m,k of the back-projected bounding box 𝐛^( i)_m,k.

At each iteration i, the set of camera-adapted segments is moved towards the direction of maximum increment:


    ℙ^( i) = ℙ^( i-1) + τ_i∇Ψ(ℙ^( i-1)),

where τ_i ∈ℝ_+ is the gradient-step that makes ℙ^( i)≥ℙ^( i-1). This gradient-step is initialized, τ_0 = 5, and updated following a decrease schedule of 50% every 3 iterations, to ease convergence. The gradient ∇Ψ(ℙ^( i)) in the i-th iteration is approximated by Forward Difference Approximation:

    ∇Ψ(ℙ^( i)) = Ψ(ℙ^( i)) - Ψ(ℙ^( i) - ϵ)/ϵ.


The algorithm continues until convergence is reached or the R_2-constrain is violated.







§ RESULTS
 
This section addresses the evaluation of the proposed method. To this aim, we first describe the evaluation framework; then, in the ablation studies, we measure the performance improvement of each of the method's stages; finally, we finish by comparing our approach with alternative state-of-the-art approaches in classic and recent multi-camera datasets.



 §.§ Evaluation Framework





  §.§.§ Datasets

Results are obtained by evaluating the proposed method over five scenarios extracted from four publicly available multi-camera datasets in which cameras are calibrated and temporally synchronized:



  * EPFL Terrace <cit.>: Generally used in the state-of-the-art to evaluate multi-camera approaches. It consists of a 5000 frames sequence per camera showing up to eight people walking on a terrace captured by four different cameras. All the cameras record a close-up view of the scene.

  * EPFL RLC <cit.>: Consists of an indoor sequence of 2000 frames per camera recorded in the EPFL Rolex Learning Center using three static HD cameras with overlapping field of views. All these cameras represent close-up views of the scene.

  * EPFL Wildtrack <cit.>: A challenging multi-camera dataset which has been explicitly designed to evaluate deep learning approaches. It has been recorded with 7 HD cameras with overlapping fields of view. Pedestrian annotations for 400 frames are provided. All of them are used to define the evaluation set used in this paper.

  * PETS 2009 <cit.>: The most used video sequences from this widely used benchmark dataset have been chosen.


  * PETS 2009 S2 L1, which contains 795 frames recorded by eight different cameras of a medium density crowd—in this evaluation, we have just selected 4 of these cameras: view 1 (far field view) and views 5, 6 and 8 (close-up views). 

  * PETS 2009 City Center (CC), recorded only using two far-field view cameras with around 1 minute of annotated recording (400 frames per camera).






Table <ref> contains a comparative description of these datasets including the type of data and annotations provided, as well as a subjective indication of their complexity for the pedestrian detection task.





  §.§.§ Performance Indicators

To obtain quantitative performance statistics according to an experiment-based evaluation criterion the following state-of-the-art performance indicators have been selected: Precision (P), Recall (R), F-Score (F-S), Area Under the Curve (AUC), N-MODA (N-A) and N-MODP (N-P) <cit.>. To globally assess performance, a single value for each statistic and each configuration is provided by averaging per-camera ones.




 §.§ System Setup

A common setup has been used for all the presented results. Faster-RCNN <cit.>, YOLOv3 <cit.> and EfficientDet-D7 <cit.> are used as baseline algorithms to obtain monocular pedestrian detections. The three object detectors are pre-trained on the COCO dataset <cit.> and we do not fine-tune nor adapt them to any of the faced scenarios. For the semantic segmentation, the Pyramid Scene Parsing Network (PSP-Net) <cit.>, pre-trained on the ADE20K dataset <cit.> (L=150, has been selected considering a trade-off between performance and efficiency.)

In the Pedestrian Semantic Filtering stage, all frames in each sequence are used for temporal and spatial semantic aggregation, i.e. T=N. For the Semantic-Driven Back-Projection stage, the initial height estimation h_m has been set to an average pedestrian height of 1.7m. Besides, for all the datasets, convergence in the iterative steepest-ascent algorithm has been reached before or at the 8^th iteration.




 §.§ Results Overview

The evaluation has been performed carrying out two different studies:



  * The Ablation Studies aim to gauge the impact of the different stages in the performance of the proposed approach. To this end, the following versions of the proposed method are compared:

    
  * “Baseline (Faster-RCNN, YOLOv3 and EfficientDet-D7)”, provides reference results of monocamera pedestrian detectors.
    
  * “Baseline + Filtering (Filt)” is a simplified version of our method which aims to independently evaluate the effect of the proposed automatic 𝒜𝒪ℐ computation obtained by the “Pedestrian Semantic Filtering" stage.
    
  * “Baseline + Filtering (Filt) + Fusion (Fus) + Back-Projection (BP)” is the full version of the proposed method, which additionally evaluates the “Fusion of Multi-Camera Detections" and “Semantic-Driven Back-Projection" stages.

Ablation Studies are conducted on four of the described datasets: Terrace, PETS 2009 S2 L1, PETS 2009 CC and RLC.


  * State-of-the-art Comparison results analyze the proposed method with respect to several non deep-learning state-of-the-art multi-camera pedestrian detectors on the same four scenarios used in the Ablation Studies. Additionally, the method is compared with novel deep-learning methods on the Wildtrack dataset.





 §.§ Ablation Studies



  §.§.§ Evaluation Criterion

The availability of bounding-box annotations permits to use the classic performance criterion <cit.>: a detection is considered a TP one if the Intersection Over Union (IoU) with a ground-truth bounding-box is higher than 0.5.



  §.§.§ Results

Table <ref> agglutinates the method's performance on a per-stage basis. Qualitative examples of automatically generated 𝒜𝒪ℐs and algorithm results are depicted in Figure <ref> and Figure <ref> respectively. A visual example of the limitations of the Semantic-Driven Back-Projection stage is included in Figure <ref>. 











  §.§.§ Discussion

Table <ref> shows that filtering-out detections using automatically generated 𝒜𝒪ℐs (Baseline + Filtering) improves the performance of all the baselines for datasets where the ground-plane area does not cover the whole image representation, i.e. datasets containing close-up views of the scene as EPFL Terrace and RLC. In these datasets, our precise 𝒜𝒪ℐs reduce phantom detections obtained by the baseline detectors. Although 𝒜𝒪ℐs are automatically computed, they are more precise (tighter to real scene edges) than those defined in the dataset. 

Overall, in the EPFL Terrace dataset, the performance of Faster-RCNN + Filtering improves Faster-RCNN by 2.44% and 2.82% in terms of AUC and N-MODA respectively. YOLOv3 + Filtering presents relative increments over YOLOv3 baseline of 1.20% and 1.31% for AUC and N-MODA respectively. Finally, EfficientDet + Filtering also overcomes its baseline results by a 7.04% for N-MODA.

For the EPFL RLC dataset with the proposed 𝒜𝒪ℐ, Faster-RCNN is improved by a 5.13% regarding AUC and by a 17.24% concerning N-MODA. For YOLOv3, relative increments of a 6.25% and a 11.86% in terms of AUC and N-MODA are achieved. EfficientDet gains relative increments of a 3.65% and a 11.47% for AUC and N-MODA.




The proposed Filtering stage does not improve baselines' performance for those datasets in which the ground-plane dominates the scene, i.e. those recorded with far-field view cameras as both scenarios from PETS 2009. In these cases, although the baseline pedestrian detectors may create phantom detections, those lie inside the proposed 𝒜𝒪ℐ and no false-pedestrians are suppressed. However, as depicted in Figure <ref>, the automatically obtained 𝒜𝒪ℐs are larger and more precise than the original operational areas in the datasets, thereby obtaining a more realistic and exhaustive evaluation. Furthermore, observe how the proposed generation method effectively handles multi-class ground partitions as in the PETS 2009 dataset, where the proposed 𝒜𝒪ℐ encompasses road, grass, pavement and side-walks classes enabling a high adaptability to unseen scenarios (see Figure <ref> right).




Table <ref> also shows that the complete method (Baseline + Filtering + Fusion + Back-Projection) notably improves Faster-RCNN baseline's performance, mainly in scenarios with heavy occlusions, i.e. EPFL Terrace and EPFL RLC (See Table <ref> for details). Specifically, for the EPFL Terrace dataset results are relatively increased a 6.14%, a 7.14% and a 16.90% in terms of AUC, F-Score, and N-MODA respectively, whereas relative improvements are of a 5.19%—in AUC, a 5.13%—in F-Score terms—and a 20.69% in N-MODA, for the EPFL RLC dataset. 

For YOLOv3 and EfficientDet detectors a similar analysis arises. In scenarios where heavy occlusions are present—EPFL Terrace and RLC datasets, performance is increased. 
For the EPFL Terrace Dataset, relative increments of a 2.38%, a 2.29% and a 11.84% are obtained when using YOLOv3 detections in terms of  AUC, F-Score and N-MODA respectively. In the case of EfficientDet, increments of a 4.87%, a 5.95% and a 16.90% are obtained with respect to the same metrics. For the EPFL RLC dataset, the improvement increases to a 6.25%, a 6.41% and a 15.25% for YOLOv3, whereas a 1.21%, a 3.75% and a 13.11% relative increase is obtained for EfficientDet in terms of AUC, F-Score and N-MODA respectively.

For both PETS scenarios the performance of the EfficientDet mono-camera detector is saturated (97% F-Score). The specific characteristics of this dataset: low pedestrian density over a wide space, low level of occlusions and a high point of view due to cameras being hanged up in streetlights (see Table <ref> and Figure <ref>), turns it in the least complex dataset among those analysed. The generation of new false positive detections and the optimization process related problems ((Figure <ref>)) may lead to a slight decrease when saturated baselines are used in low complex datasets. Leaving these specific situations aside, the benefits of the proposed method are evident if one accounts for both performance indicators and qualitative results (see Table <ref> and Figure <ref> respectively): the proposed multi-camera detection approach is able to cope with partial, severe and complete occlusions by combining detections from all the cameras through the proposed semantic-guided process leading to an increase of all the reported metrics.

Focusing specifically on the Semantic-Driven Back-Projection process, results in Figure <ref> depict highly tight pedestrian bounding boxes, independently from people's height, self-occlusions and calibration problems, suggesting that the optimization process is able to automatically adapt bounding-boxes by jointly estimating pedestrian heights and world positions. Results in Table <ref> corroborate this observation. Semantic-Driven Back-Projection leads to a higher overlap between detections and ground-truth annotations: in terms of the N-MODP metric, the proposed method achieves relative improvements of a 4.05% for EPFL Terrace, a 3.95% for both PETS 2009 S2 L1 and PETS 2009 CC and a 1.45% for the RLC dataset when Faster-RCNN is used as the baseline detector. When YOLOv3 is used as the baseline detector, our method achieves a N-MOPD increment of a 4.10% for EPFL Terrace whereas the N-MODP metric remains stable for PETS 2009 S2 L1, PETS 2009 CC and RLC datasets, suggesting that YOLOv3 individual performance for these datasets is already heaped. A similar result arises when using EfficientDet detector which by default is highly tight to pedestrians. Results are increased only for PETS CC dataset by a 13.63% while manteined for the rest of the datasets. It is important to remind that, even tough the N-MODP metrics are sometimes slightly reduced or maintained, without the proposed Semantic-Driven Back-Projection process the back-projected bounding boxes and ground-truth would be misaligned (see Figure <ref>) decreasing the performance in terms of all the accuracy metrics of the proposed method.

Nevertheless, the optimization cost function aims to maximize the 2D detections’ alignment with the semantic segmentation masks, leading to a bias towards wider pedestrians by design, a situation that may result sometimes into wrong relocations of the back-projected bounding-boxes. Figure <ref> shows an example of this case: notice the erring behaviour in Camera 1 when there is an extreme overlapping.




 §.§ State-of-the-art Comparison



  §.§.§ Evaluation Criterion

The same criterion used in the Ablation Studies applies for the Terrace, PETS and RLC datasets. However, in the Wildtrack dataset, as the ground-truth is provided via detections on the world ground plane (i.e., no bounding-boxes are provided), the evaluation criterion is different. Specifically, a detection is considered a TP if it lies at most r = 0.5m to a ground-truth annotated point <cit.>. This radius roughly corresponds to the average width of the human body. Due to the absence of bounding-boxes, for this dataset the Semantic-Driven Back-Projection stage is not included.




  §.§.§ State-of-the-art Algorithms

The following multi-camera algorithms have been selected to carry out the comparison:


  * POM <cit.>. This algorithm proposes to estimate the marginal probabilities of pedestrians at every location inside an 𝒜𝒪ℐ. It is based on a preliminary background subtraction stage.

  * POM-CNN <cit.>. An upgraded version of POM in which the background subtraction stage is performed based on an encoder-decoder CNN architecture.

  * MvBN+HAP <cit.>. Relies on a multi-view Bayesian network model (MvBN) to obtain pedestrian locations on the ground plane. Detections are then refined by a Height-Adaptive Projection method (HAP) based on an optimization framework similar to the one proposed in this paper, but driven by background-subtraction cues.

  * RCNN-Projected <cit.>. The bottom of bounding-boxes obtained thorough per-camera CNN detectors are projected onto ground-plane, where 3D proximity is used to cluster detections.

  * Deep-Occlusion <cit.> is an hybrid method which combines a CNN trained on the Wildtrack dataset and a Conditional Random Fields (CRF) method to incorporate information on the geometry and calibration of the scene.

  * DeepMCD <cit.> is an end-to-end deep learning approach based on different architectures and training scenarios:


  * Pre-DeepMCD: a GoogleNet <cit.> architecture trained on the PETS dataset.

  * Top-DeepMCD: a GoogleNet <cit.> architecture trained on the Wildtrack dataset.

  * ResNet-DeepMCD: a ResNet-18 <cit.> architecture trained on the Wildtrack dataset.

  * DenseNet-DeepMCD: a DenseNet-121 <cit.> architecture trained on the Wildtrack dataset.















  §.§.§ Results

Table <ref> includes performance indicators for the proposed method compared with multi-camera algorithms POM <cit.> and MvBN+HAP <cit.> on the Terrace, PETS and RLC scenarios (results for the compared methods are extracted from <cit.>). Table <ref> compares the performance of the proposed approach against deep-learning methods, some of them explicitly trained with data from the Wildtrack dataset (which we denote as Fine-Tuned) and others trained with data from other datasets or not even trained (which we denote as not Fine-Tuned). Performance indicators for these methods are extracted from <cit.>. In addition, qualitative results for the Wildtrack dataset are presented in Figure <ref>, including obtained detections in camera frames, global detections on the ground-plane and the automatically computed 𝒜𝒪ℐ.



  §.§.§ Discussion

Results in Table <ref> show that the proposed approach (Baseline + Filtering + Fusion + Back-Projection), either with Faster-RCNN, YOLOv3 or EfficentDet baseline, outperforms the MvBN + HAP and the POM-CNN methods in terms of N-MODA metric. The proposed method obtains better results in terms of N-MODA which, precisely, measures detection accuracy along the whole video sequences. Best results are obtained when EfficientDet is used to extract mono-camera detections. Specifically, N-MODA is increased a 1.21% for EPFL Terrace and a 1.14% for both PETS 2009 S2 L1 and CC. Moreover, it obtains the higher performance on the heavily occluded RLC dataset. Besides, N-MODP results, i.e. the overlapping between detections and ground-truth, are better than those obtained by the HAP method <cit.>. This suggests that our use of semantic segmentation masks instead of foreground masks (HAP method) benefits the optimization process. Relative increments in N-MODP performance of a 5.48% for EPFL Terrace, a 3.95% for PETS 2009 S2 L1 and a 1.28% for PETS 2009 CC support this assumption.

Presented results in Table <ref> suggest that the proposed method is able to obtain reliable pedestrian detections in a variety of scenarios with a variety of pedestrian detection algorithms in terms of performance.

Finally, results on the Wildtrack dataset (Table <ref>), indicate that the proposed method, operating on detections from a Faster-RCNN, a YOLOv3 or a EfficientDet model, is able to outperform deep-learning approaches that have not been specifically trained using Wildtrack data and use manually annotated detection constrains. Our method, using EfficientDet detections, improves a 45.45% respect to Pre-DeepMCD—the second ranked, which is an end-to-end deep learning architecture trained on the PETS dataset. However, algorithms explicitly trained on data from the Wildtrack dataset, i.e., DenseNet-DeepMCD, ResNet-DeepMCD, Top-DeepMCD, and Deep-Occlusion, outperform the proposed method, in our opinion for two main reasons:


  
  * First, the qualitative results presented in Figure <ref> suggest that results in Table <ref> are highly biased by the authors' manually annotated area. The proposed method obtains a broader 𝒜𝒪ℐ (Figure <ref>, green area) than the one provided by the authors (Figure <ref>, red area). Although the automatically obtained 𝒜𝒪ℐ seems to be better fitted to the ground floor in the scene than the manually annotated one, the performance of our method decreases because ground-truth data is reported only on the manually annotated area. Thereby, our true positive detections out of this area result in false positives in the statistics (see Figure <ref>, cameras 1 and 4).
  
  
  * Second, they learn their occlusion modeling and their inference ground occupancy probabilistic models specifically on the Wildtrack scenario using samples from the dataset. This training, as any fine-tuning procedure in Deep Neural Networks, is highly effective, as indicated by the increase in performance resulting from the use of the same architecture but adapted for the Wildtrack scenario (compare results of Pre-DeepMCD and Top-DeepMCD). This training requires the use of human-annotated detections in each scenario, hindering the scalability of these solutions and its application to the real world. The proposed approach, on the other hand, performs equally without the need of being adapted for every target scenario reported in this paper. 


Respect to the first issue, i.e. the effect of using the proposed automatically extracted 𝒜𝒪ℐ instead of the one provided by the dataset and used by the rest of the methods, in order to obtain a fairer comparison, we have included in Table <ref> the result of the proposed method evaluated on the authors' 𝒜𝒪ℐ using our top ranked method, i.e. using EfficientDet baseline. As it can be observed, when using the authors' 𝒜𝒪ℐ for evaluation, the proposed method outperforms TOP-DeepMCD by a 8.33% and DenseNet-DeepMCD by a 3.17% ranking the third best method on the Wildtrack dataset without requiring a dataset specific fine-tunning stage as the two above it. In addition, performance with respect to GMC-3D <cit.>, which replicates the previous version of the proposed method with the addition of Person Re-Identification features is increased a 17.85 %.

On average, and contrary to state-of-the-art approaches, the proposed method adapts to different target scenarios without needing a separate training stage for each situation, with the consequent reduction of computational resources and time, and neither requiring a manually annotated area of interest.







§ CONCLUSIONS
 
This paper describes a novel approach to perform pedestrian detection in a multi-camera recorded scenario. First, the adapted strategies for the temporal and spatial aggregation of semantic cues, along with homography projections, are used to obtain an estimation of the ground-plane. Through this process, a broader, accurate and role-annotated Area of Interest (𝒜𝒪ℐ) is automatically defined. Per-camera detections, obtained by a state-of-the-art detector, are projected to the reference plane, and those laying outside the obtained 𝒜𝒪ℐ are filtered-out. A fusion approach based on creating connected components on a graph representation of the detections is used to combine per-camera detections yielding global pedestrian detection. Then, a Semantic-Driven Back-Projection method handles occlusions and uses semantic cues to globally refine the location and size of the back-projected detections by aggregating information from all the cameras. Results on a broad set of scenarios confirm that the method outperforms every other compared multi-camera not deep-learning method and also every deep-learning method not adapted to the target dataset, even with different baseline algorithms. The proposed method performs close to scenario-tailored methods, but without their training stage, which highly hinders their straight use in new scenarios. In overall, results suggest that the proposed approach is able to obtain accurate, robust, tight-to-object and generic pedestrian detection in varied scenarios, included crowded ones.


This study has been partially supported by the Spanish Government through its TEC2017-88169-R MobiNetVideo project. 














spmpsci      

   



















