Abstract
this sensing has to be done without requiring people to wear on silicon. In this paper we describe the details of our design provide feature information and ultra-low power operation.
and experimentally determine suitable parameters for the proposed histogram. The resulting histogram and counting algoOur previous work in [2] provided an initial evaluation rithm are implemented and tested on a set of iMote2 camera of new address-event (AE) imager architectures and a model sensor nodes deployed in our lab.
for emulating this architecture on wireless sensor nodes. In this paper, we present and evaluate a design for localizing 1. INTRODUCTION and counting people in indoor spaces with a set of wide-angle camera sensor nodes mounted to the ceiling, facing down. The integration of wireless sensor networks and camera techOur design targets the architecture presented in [2] and localnologies is rapidly pacing towards a new generation of lowizes and counts people using a histogram derived from mocost, low-power camera sensing nodes. To enable a large tion and size information. The resulting algorithm was imnumber of ubiquitous applications and services, these wireplemented on a camera sensor node and is currently deployed less camera nodes should be able to recognize, count and on a home testbed for assisted living. The main contribution track humans, preferably anonymously as they move inside of the work described here is the design and evaluation of a buildings. The BehaviorScope project at Yale [?] uses such lightweight histogram-based method for counting and localizinformation to infer people behaviors, with assisted living as ing people using motion and size information and the fact that the application focus. Human locations collected from a sen-this can be applied on the motion outputs of an address-event sor network deployed inside a house are processed together image sensor. It is implemented on a real testbed currently
INTRODUCTION
and counting people in indoor spaces with a set of wide-angle camera sensor nodes mounted to the ceiling, facing down. The integration of wireless sensor networks and camera techOur design targets the architecture presented in [2] and localnologies is rapidly pacing towards a new generation of lowizes and counts people using a histogram derived from mocost, low-power camera sensing nodes. To enable a large tion and size information. The resulting algorithm was imnumber of ubiquitous applications and services, these wireplemented on a camera sensor node and is currently deployed less camera nodes should be able to recognize, count and on a home testbed for assisted living. The main contribution track humans, preferably anonymously as they move inside of the work described here is the design and evaluation of a buildings. The BehaviorScope project at Yale [?] uses such lightweight histogram-based method for counting and localizinformation to infer people behaviors, with assisted living as ing people using motion and size information and the fact that the application focus. Human locations collected from a sen-this can be applied on the motion outputs of an address-event sor network deployed inside a house are processed together image sensor. It is implemented on a real testbed currently with building map information to recognize the activities of deployed inside a house for an assisted living application. the house inhabitants. The locations of people in time and space during the course of the day provide a set of macro-
The rest of the paper is organized as follows. Section 2 gestures that are parsed by a framework of Hierarchical Probprovides some background to the problem and surveys the reabilistic Context Free Grammars into a set of predefined aclated work. Sections 3 and 4 outline our approach and detivities [1] .
scribe the details of the motion histogram. Section 5 exAssisted living as well as other similar applications in plains how a sensor node uses the histogram to localize and security, workplace safety and entertainment share a comcount and Section 6 presents our experimental results. Video mon sensing requirement. The sensor network should cover demonstrations of our experiments are available at http://www. large indoor spaces localizing and counting people as they eng.yale.edu/enalab/behaviorscope/counting.htm. Secmove about the space. Furthermore, for practical purposes, tion 7 concludes the paper. in Figure 1 Other, simpler adaptative background-modeling techniques, blob, a further pass has to be performed where blobs that are such as continuously averaging frames, have the opposite efclose enough are merged together. The end result is that it is fect of counting people that seldom move as a background common to merge blobs that do not belong together, as well objects. as to separate blobs that compose the same object, as in [5] .
Other research in counting untagged people includes the This has the additional effect of adding uncertainty to the louse a number of different types of sensors and techniques. [9] cations that are extracted from the blob. Some have attempted uses an array of PIR sensors in arranged in a line to detect the to obtain more precise locations from each blob by employing number of people through a flight of stairs. In [10], the center the distance transform [6] rather than center-of-mass or foot scanlines of an image frame is used for a similar effect. These estimation, but that approach fails for fragmented blobs.
approaches are used to detect people that pass through a conMoreover, there is the problem of maintaining and updatfined area, and do not adapt well to open spaces. Moreover, ing the background model. This is a necessary process due since each person is only detected at entrances and exits, any to the presence of a series of change factors in a stream of error at detection time will propagate indefinitely. frames, among which are: (1) Natural oscillations in pixel Another approach [11] utilizes multiple cameras with largeintensity; (2) Gradual changes in lighting, such as those im-ly overlapping field-of-views to get information about the 2-posed by the movement of the sun; (3) Presence of repetitive dimensional cross-section of a room, its objects and its occubackground motion, such as waving foliage; (4) Changes in pants. The algorithm provides good location precision, but position of staticobjects, such as furnilture, requires the use of multiple cameras to achieve that when Adaptive background-modeling approaches are computacovering even a small room. What is more, their approach tionally expensive, sometimes modeling each Single pixel as demands precise calibration of inrinSiC an extrinSiC camera a mixture of GaussianLs [7] or with a separate Kalmnan filter param[eters. In the settinLg of assisted-living situationls, how- [8] . Many of these approaches require the field of view to be ever, the largest issue is that the computed cross-section fails [5] . In that paper, the authors describe their tracking system for assisted The primary goal of the motion histogram is to determine the living. Their background model takes into consideration only probable location of each person given the coordinates of the the pixel intensity oscillations, and would fail in a less conmoving pixels in each frame. The value of each histogram bin trolled environment. More importantly, their system does not corresponds to the number of foreground pixels in a unique handle rooms larger than the single stereo-pair's field-of-views. area of the image. In Figure 1 , bin b is associated to the set In [12] , binary edge-detected images are used in a peo-of pixels in the blue square on the top-left side of the image. and horizontal directions. The bin size is calculated from the expected image size of a human, so that, in optimal condiIn light of the afore-mentioned problems with background tions, a person in the field of view of the overhead camera differencing, we choose to shift the emphasis away from back-is entirely covered by a single bin, and partially covered by ground subtraction into a different paradigm. The intuition neighboring bins. In typical operation, though, people may behind this is that humans can recognize and count other huspan multiple bins (when they extend their arms, for exammans based on shape, size and movement. The background ple), but the algorithm described here still holds.
differencing approach attempts to extract and operate on mainIf the bin areas on the left side of Figure 1 are square with ly the first two types of information. We choose to focus on width w, and if the smallest distance between bin centers is 6, the latter two, while at the same time simplifying them by then g can be defined as introducing a set of constraints on the deployment and the en- The relLationLs p and h need not be as trivialL as these, and our movemen an size criteriaan extract teir discretephys-better results may beextracted from irregular bin saswe shal icatl loctJionUl from1 our measJurementsl.
descrbellUater. figure (Figure 5b ) produces
In the case where each bin maps to equal, but shifted areas very distinct images for each of the people depending on their in the image, the histogram can be seen as the result of the distance from the center axis of the camera: people near the cross-correlation of the image with a human model. In the center of the image appear as seen from the top, while those simplest case, this model is a square, as in our discussion so at the edges are seen diagonally from the side. Hence, the far. Another possibility is to utilize a more complex func-square histogram bins yield good results for subjects near the tion as a kernel, such as a multivariate Gaussian distribution. center of the image, where there is an approximate top-view, For the other types of models considered later in this paper, but not so much as people wander toward the image edges. the histogramn-producinLg operatioln mnay no lLonLger be a crossAccounltinlg for this a human model is derived from a 3-correlatioln, in that the kernelshap ma vary with its position. diesinl objet that is the prjete back ito the mg is, the prediction Pti is matched to the observed peak qtj if and We utilize for this purpose an algorithm similar to [3] due only if qtj is the peak that is closest to Pti and Pti is the preto its speed. The major difference is that it is adapted for diction that is closest to qtj. This can be found by looking at use with the histogram. At each time instant t we wish to the distance matrix D to see whether dij the minimum value find the list of detected people Pt= {Pti Y , based on obof row i and column j at the same time. All peaks and predicserved histogram peaks Qt= {qtj }7o and predicted loca-tions that are not matched at this stage are further investigated tions Pt = {Pti},0n The variable nt Pt is, thus, the in the stages below. number of detected people at time t, while mt Qt Q is the False Negatives and Person Leaving Given that the number of peaks in that instant. As a nota bene on notation, histograms are built from frame differencing data, the peaks for the remainder of the paper the ith element of a list (A) is tend to shorten and disappear when a person stops moving, denoted with the same letter as the list, but in lower case, and causing a false negative. This does not mean that the person its index i appended to the subscript (ai).
is outside the field-of-view, so the person should continue beEach detected (and predicted) person is represented by a ing accounted for. In order to resolve this scenario, at each feature vector (Sti, hti, bti, vti, ati), denoting the values for new frame we construct the list St of predicted stops, conthe peak position, peak height, peak breadth, velocity and ac-taining the people whose motion is under a certain threshold. celeration, respectively. Note that the height corresponds to This can be measured by watching the height component of the value of the histogram bin not the height of the detected the histogram peak. In the next frame, any person that has person. Position, height and breadth are acquired directly not been matched but who can be found in St is believed to from a single histogram, while velocity and acceleration are have stayed at the same location. Those who aren't present in be computed by following the changes in position over time.
St are discarded, as they either left the field-of-view or were Note that sti, vti and ati are 2-dimensional vectors, with x-an uncaught false positive. Additionally, peaks at the hisand y-axis components. Meanwhile, peaks are represented togram edges were allowed to disappear even if their height as a vector containing only the features that can be extracted was small. where each weight Qx is a scalar weight chosen such that frame will they be included in the detected list PT+ This the distance between the most different peaks is 1. With this does away with transient peaks while still counting the new definition, the best match is the one with distance closest to peaks which are more stable, allowing new people to enter 0. A distance matrix D is computed, recording the distance the covered area. between each prediction and each peak.
Merging and Splitting It is common for image seg- bers of people.
mentation and blobbing algorithms to count two or more foreerrors plateaus around the range w = 30cm to 50cm, then ground objects as a single one. The same is true with the sharply rises. Meanwhile, Figure 7 shows a much different histogram algorithm described here. Figure 7 is the result of varying while holding steven41@example.net. We have implemented the motion histogram and counting alBoth plots were generated using the 3-dimensional bin model, gorithm in a sensor network composed of multiple Intel iMote2 with a height of 170cTn. Due to space limitations, the plots sensor nodes. The nodes are suited with a custom-built camerafor different nrumbers of people have beenL combined, and the board (Figure 8 ) that containLs the OmniVision 0V7649 imnumbJer of errors (y axiLs) was therefore norma lzed. Tere iLs ager. Te nodes acquire images at 32 x 240 resolution, lno plLot for when the room is empty because there were lno downsamuplLe them to 80 x 60, thenL run the algorithmn defalse positives in that situation, so the plLot is all zeros. scribed in Figure 4 . The m[appings g and h are precomuputed From the first plot ( Figure 6 ) one finds that the number of and kept in the node's memory for fast operation. Each peak is recorded along with time of detection, and sent through Fig. 9 . Composite of images from all 6 nodes, with 3 people the radio once the send buffer is full. This entire process rein the scene (blue circles). The numbers on the circles are peats approximately every 110ms, allowing a frame rate of each person's temporary ID. The correct count is shown in just over 8 frames per second. The packets are processed by the top-left corner. the base node, which reports the person-count to the gateway computer, along with the peak data for visualization purposes.
The nodes are placed on the testbed structure on the ceil-42% of all detections. We believe there is room for improveing of our lab, where they are a single hop away from their ment in those conditions, by utilizing a better tracker. The base. Given the lower ceiling height at the lab (240cm) and histogram achieves its best precision at the center two-thirds the presence of cubicle walls, 6 nodes are required to cover of the image. This is clear when people walk closely and sidethe entire area. In this configuration, each node has an active by-side near the edges, which causes an occlusion to occur. field-of-view of approximately 3m x 2m. The node posiNear the center, the maximal accuracy (15cm) was achieved tions are chosen to minimize field-of-view overlaps, and the on all runs of the parallel-walking tests. Additionally, on the images they acquire are cropped until the overlap is virtually runs where the two people crossed paths, the tracking algozero. This way, we attempt avoid most of the correspondence rithm was able to keep the correct count and locations regardissues to focus on the histogram performance. However, since less of distance from the image center. the node time-synchronization protocol we utilize gives us a For the next experiment, people walked around the testbed measured discrepancy of 187ms, and given that the nodes acthrough every node's field-of-views, at times standing still quire pictures in an unsynchronized manner, the peak detecfor a few seconds, and other times changing the position of tion timestamp has a significant margin of error. This unoffice chairs. During the experimental runs where a single certainty in the timing between nodes makes it possible for person was present, the network correctly counted the numphysical inconsistencies to occur, such as a person apparently ber of people 89.5% of the time. The majority of the errors being in two places at the same time. That is, the corresponwere false positives that occurred at field-of-view overlaps, dence problem reappears in a different form. The details of due to the timing issues already discussed. For two people, timing have not been entirely worked out, and are a possible that number drops to 82.48%, and 79.8% for three. This small direction for future work. Our preliminary results show good decrease in accuracy follows an increase in packet drop rates, accuracy regardless. which reached 6.6% when all three people were in the lab. We tested the histogram positional accuracy by having Packet loss is expected to increase with the number of peotwo people walk toward one-another and meet at the center of ple, given that all nodes will be attemting to transmit data the image. This was captured by a single node. The histogram simultaneously. We are currently working on a more apt meswas able to differentiate distances of up to 15cm 100% of the saging and routing scheme. None of the errors were caused time. This is the maximum possible resolution, given that by the change in furniture placement. A close examination of d = 15cm. This resolution greatly suits the assisted-living the data has shown that the sensing at each node functioned scenario, where the main interest is in the logical spacial locorrectly. The discrepancies were due to the time synchrocation (such as "on the sofa", or "by the stove"), instead of nization that resulted in double counting. exact coordinates. The same test was performed for locations increasingly farther from the center. The result was the same 7. CONCLUSION for distances up to im from the image center. At that distance, although the histogram at times produced a single peak We have developed a lightweight, online people-counter utifor both people, the tracking/counting algorithm was able to lizing a novel, AB-friendly motion-histogram. The histogram disambiguate them. At the farthest position where one fully is robust to pixel intensity fluctuations, gradual lighting changes covered by the camera (1.5m), the algorithm missed around and furniture repositioning. Abrupt alterations in lighting may,

{'paper_id': '16544644', 'title': 'Lightweight People Counting and Localizing in Indoor Spaces Using Camera Sensor Nodes', 's2_url': 'https://api.semanticscholar.org/CorpusID:16544644', 'source': 'S2ORC'}