text: An Overlay Architecture for High Quality VoIP Streams Yair Amir, Member, IEEE, ACM, Claudiu Danilov, Stuart Goose, David Hedqvist, and Andreas Terzis, Member, IEEE, ACM Abstract— The cost savings and novel features associated with This paper describes an overlay architecture that can be Voice over IP (VoIP) are driving its adoption by service providers. deployed without any changes to the underlying network Unfortunately, the Internet's best effort service model provides infrastructure to address the problems that happen during no quality of service guarantees. Because low latency and jitter is the key requirement for supporting high quality interactive these intervals of network loss and failures. The overlay conversations, VoIP applications use UDP to transfer data, maintains high packet delivery ratio even under high loss, thereby subjecting themselves to quality degradations caused by while adding minimal overhead under low, or no-loss condi- packet loss and network failures. tions. The system is based on several observations regarding In this paper we describe an architecture to improve the VoIP applications. First, we notice that it is often possible to performance of such VoIP applications. Two protocols are used for localized packet loss recovery and rapid rerouting in the event recover packets even within the tight delay budget of VoIP. of network failures. The protocols are deployed on the nodes of While many VoIP streams are subject to large latencies that an application-level overlay network and require no changes to prohibit timely end-to-end recovery, it is possible to perform the underlying infrastructure. Experimental results indicate that recovery for many short links that are in the order of 30 msec. the architecture and protocols can be combined to yield voice Our second observation is that by dividing long end-to-end quality on par with the PSTN. paths into multiple shorter overlay links, an overlay network architecture can localize packet recovery at the overlay hop I. I NTRODUCTION on which the loss occured. Thus, even for VoIP streams with A end-to-end latencies considerably larger than 30 msec, the vast LTHOUGH subscribers are accustomed to the consis- majority of the packet losses can be rapidly recovered on the tent voice quality and high reliability provided by the shorter overlay hop on which they were dropped. Our third traditional Public Switched Telephone Network (PSTN), the observation is that the overlays allow the deployment of a promise of a single converged IP network to carry voice and routing algorithm that maximizes packet delivery probability data – and the cost savings therein – motivates the adoption while conforming to VoIP delay requirements, thereby improv- of voice-over-IP (VoIP) technologies. However, customers ing voice quality. expect VoIP to meet and exceed the standard of quality long The contribution of this paper is a custom overlay network offered by the Public Switched Telephone Network (PSTN). architecture targeted at VoIP by judiciously combining two Providing IP solutions that meet the stringent requirements complementary mechanisms: First, a real-time1 packet recov- of high quality and reliable voice communication poses a ery protocol that immediately delivers newly received packets, non-trivial challenge. Delays of 100-150 msec and above in a way similar to UDP, but this protocol attempts to recover are detectable by humans and can impair the interactivity of missing packets. Recovery is attempted only once, and only if conversations. By comparison, humans are far less tolerant of a packet is likely to arrive at the destination within the VoIP audio degradation than of video degradation. Hence, to meet delay constraint. This protocol is deployed on every overlay these requirements it is crucial to minimize primarily the net- link. Second, a routing protocol that chooses overlay paths work latency and secondarily packet loss as much as possible. using a synthetic metric which combines the measured latency To minimize latency, contemporary VoIP solutions use UDP and loss rate of each overlay link. as the transport protocol. However this has the potential to The architecture was implemented as part of the Spines [4] expose VoIP packets to packet losses and equipment failures. open source overlay network platform. The behavior of our Although the Internet can offer reasonable quality (relatively protocols was evaluated under controlled network conditions low loss and good stability) for the majority of VoIP streams, on the Emulab [5] testbed and directly in the Internet on it has been shown [1], [2], [3] that it remains vulnerable to the Planetlab [6] distributed testbed. The performance of the occasional bursts of high loss and link failures that preclude proposed routing metric was evaluated through extensive simu- it from delivering the constant, high quality service demanded lations, comparing it to other metrics, on thousands of random for telephony. topologies with various loss and delay link characteristics. We Y. Amir, C. Danilov and A. Terzis are with the Johns Hopkins University, show that by leveraging our overlay for disseminating VoIP Department of Computer Science, Baltimore, Maryland. Email: {yairamir, streams, the loss rate of the communication can be drastically claudiu, email@example.com S. Goose and D. Hedqvist are with Siemens Corporate 1 Our definition of real-time refers to timely recovery of packets on short Research, Inc. Princeton, New Jersey. Email: {stuart.goose, overlay links. Protocols such as RTP, RTCP, that do not recover packets, work email@example.com independently of our protocols and benefit of our higher packet delivery ratio. reduced. For example, for a packet loss rate of 5%, our system In our experiments we used a well-understood, widely de- can usually recover within the latency constraints all but 0.5% ployed and good quality codec, the standard ITU-T G.711 [7], of the packets. This leads to a commensurate increase in the combined with its PLC mechanism [8]. The G.711 codec we voice quality of the calls. Our results reveal that using a used samples the audio signal at a rate of 8kHz and partitions standard voice codec, we could achieve PSTN voice quality the data stream into 20 msec frames, thus sending 160 byte despite loss rates of up to 7%. Our routing metric improves packets at a rate of 50 packets/sec. overall performance by selecting paths that optimize packet VoIP quality is evaluated using an objective method de- delivery ratio. It outperforms routing schemes using latency, scribed in ITU-T recommendation P.862 [9], known as Percep- loss and hop-count as routing metrics, especially in networks tual Evaluation of Speech Quality (PESQ). The PESQ score with high latency in which delay constraints becomes more is estimated by processing both the input reference and the stringent. degraded output speech signal, similarly to the human auditory When using overlay routing on general purpose computers system. The PESQ score ranks speech signals on a scale from - issues such as process scheduling and application processing 0.5 (worst) to 4.5 (best), where 4.0 is the desired PSTN quality. overhead need to be addressed. We show that when overlay routers are run with high priority, application-level routing adds a relatively small delay of under 0.15 msec, even on B. Internet loss characteristics loaded computers where 10 processes constantly compete for Packets are lost in the Internet due to congestion, routing CPU at the same time. anomalies and physical errors, although the percentage of The rest of the paper is organized as follows: In Sec- physical errors is very small at the core of the network. Paxson tion II we present the motivation and background of our in [2] studied the loss rate for a number of Internet paths work. In Section III we introduce our overlay architecture. and found that it ranged from 0.6% to 5.2%. Furthermore We present and evaluate our protocols in Section IV. The in that study and a follow-up [10], Paxson discovered that routing limitations of overlay networks and how they can loss processes can be modeled as spikes where loss occurs be addressed in real systems are described in Section V. according to a two-state process, where the states are either Section VI discusses how we can integrate our approach in "packets not lost" or "packets lost". According to the same the current VoIP infrastructure. Section VII presents related studies, most loss spikes are very short-lived (95% are 220 work, and Section VIII concludes our paper. msec or shorter) but outage duration spans several orders of magnitude and in some cases the duration can be modeled by II. BACKGROUND a Pareto distribution. In a more recent study, Andersen et al in [3] confirmed Paxson's earlier results but showed that the A. Voice over IP average loss rate for their measurements in 2003 was a low There are several steps involved in sending voice commu- 0.42%. Most of the time, the 20-minute average loss rates were nication over the Internet. First, the analog audio signal is close to zero; over 95% of the samples had a 0% loss rate. On encoded at a sampling frequency compatible with the human the other hand, during the worst one-hour period monitored, voice (8KHz in our case). The resulting data is partitioned the average loss rate was over 13%. An important finding in into frames representing signal evolution over a specified time [3] is that the conditional probability that a second packet is period. Each frame is then encapsulated into a packet and lost given that the first packet was lost was 72% for packets sent using a transport protocol (usually UDP) towards the sent back-to-back and 66% for packets sent with a 10-msec destination. The receiver of a VoIP communication decodes delay, confirming the results in [10]. the received frames and converts them back into analog audio In addition to link errors and equipment failures, the other signal. major factor contributing to packet losses in Internet is delayed Unlike media streaming, VoIP communication is interactive, convergence of the existing routing protocols. Labovitz et i.e. participants are both speakers and listeners at the same al [11] use a combination of measurements and analysis to time. In this respect, delays higher than 100-150 msec can show that inter-domain routes in the Internet may take tens of greatly impair the interactivity of conversations, and therefore minutes to reach a consistent view of the network topology delayed packets are usually dropped by the receiver codec. after a fault. They found that during this period of delayed Voice quality can be adversely affected by a number of convergence, end-to-end communication is adversely affected. factors including latency, jitter, node or link failures, and by In [12], Labovitz et al. find that 10% of all considered routes the variability of these parameters. The combined impact, as were available less than 95% of the time and that less than perceived by the end-users, is that voice quality is reduced 35% of all routes were available more than 99.99% of the at random. Contemporary VoIP codecs use a buffer at the time. In a follow-up study [13], Chandra et al showed that 5% receiver side to compensate for shortly delayed packets, and of all failures last more than 2 hours and that failure durations use forward error correction (FEC) or packet loss concealment are heavy-tailed and can last as long as 20 hours before being (PLC) mechanisms to ameliorate the effect of packet loss or repaired. In a related study performed in 2003, Andersen et excessive delay. The error correction mechanisms usually add al [3] showed that while some paths are responsible for a redundancy overhead to the network traffic and have limited large number of failures, the majority of the observed Internet ability to recover from bursty or sudden loss increase in the paths had some level of instability. In a different analysis [14], network. Boutermans et al study the effect of link failures on VoIP 4.5 4.5 Uniform Uniform 25% burst 25% burst 50% burst 50% burst 75% burst 75% burst 4 4 UDP PESQ - 5 percentile UDP PESQ - Average 3.5 3.5 3 3 2.5 2.5 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 loss rate (%) loss rate (%) Fig. 1. Network loss - Average PESQ Fig. 2. Network loss - 5 percentile PESQ performance and show that a single network failure between We can see that on average, the G.711 codec can handle up to two nodes located at San Francisco (CA) and Reston (VA) 1% loss rate, while keeping a PESQ score higher than 4.0 (the affected network traffic for 50 minutes with several periods of expected PSTN quality level). Burstiness does not play a major down-time, of which one lasted 12 minutes. All these statistics role until the loss rate is relatively high, when the voice quality indicate the Internet today is not ready to support high quality is anyway low. However, given the regular expectancy of high voice service as we are going to show in the following section. quality phone calls, we also need to analyze the most adversely affected voice streams. Figure 2 presents the 5 percentile of C. Voice quality degradation with loss the above measurements. We can see that for the most affected We evaluated the effect of losses that mimic realistic Internet streams burstiness does have a significant impact, and even at patterns on VoIP quality. Our evaluation is based on the 0.5% loss rate the G.711 codec cannot provide PSTN standard standardized PESQ measure. To do so, we instantiated a voice quality. At 0.5% loss and 75% burstiness the PESQ score network with various levels of loss and burstiness (we define dropped to 3.69. burstiness as the conditional probability of loosing a packet Considering the fact that current loss rate measurements in when the previous packet was lost) in the Emulab [5] testbed, the Internet average at about 0.42% with an average burstiness and measured the quality degradation when sending a VoIP of 72%, and that occasionally loss can be even much higher, stream on that network. these experiments show that new solutions are required to Emulab is a testing environment that allows deployment of improve the quality of VoIP traffic if it is to compete with networks with given characteristics composed of real com- the existing PSTN. puters running Linux, connected through real routers and switches. Link latency, capacity and loss2 are emulated using III. A N OVERLAY A RCHITECTURE additional computers that delay packets or drop them with certain probability or if their rate exceeds the requested link Overlay networks allow easy deployment of new services, capacity. All the Emulab machines are also directly connected as they allow full control over the protocols running between through a local area network through which they are managed participating nodes. While the Internet provides generic com- and can be accessed from the Internet. On this local area munication solutions, an overlay network usually has limited network we constantly monitored the clock synchronization scope and therefore can deploy application aware protocols between the computers involved in our experiments and accu- that would otherwise be impractical to deploy directly in rately adjusted our one-way latency measurements. the larger Internet. Most overlay networks, including ours, We used the G.711 codec with PLC to transfer a 5 minute run above the networking infrastructure, and therefore do not audio file using UDP over the lossy network, repeating each require any change in the network routers or IP protocols. experiment for 20 times. The network had a 50 msec delay and The use of overlay networks usually comes with a price, 10 Mbps capacity, enough to emulate a trans-continental long- partially due to the management overhead of the overlay, but distance call over a wide area network. We finally decoded mostly due to the sub-optimal placement of the overlay routers the audio file at the destination, divided it into 12 second in the physical network topology. However, overlays are small intervals corresponding to normal conversation sentences, and compared to the global underlying network, and therefore compared each sentence interval with the original to generate protocols that exploit the relatively limited size and scope its PESQ score. of overlays not only can overcome these drawbacks, but can Figure 1 shows the average PESQ score of all the sentence actually offer better performance to end-user applications. intervals as a function of loss rate and burstiness of the link. The overlay handles traffic for multiple connections in var- 2 Emulab cannot set conditional loss probability on the links. For burstiness ious directions, such that every connection can have a limited experiments we dropped packets with conditional probability at the application number of intermediate overlay hops. Previous work [15] [16] level, before processing them. shows that increasing the number of nodes in an overlay returns a descriptor that can be used for sending and receiving data. A spines sendto() call resembles the regular sendto(), and a spines recvfrom() resembles the regular recvfrom(), with similar parameters. Virtual Ports are only defined in the context of an overlay node, and have no relation to the actual operating system ports. Spines nodes connect to each other using virtual links form- ing the overlay network. Spines offers a number of protocols on each virtual link, including a best effort service, a TCP-fair Application reliable protocol [15] and a real time recovery protocol [17]. Each overlay node pings its direct neighbors periodically to check the link status and latency. Round trip time measure- Spines daemon ments are smoothed by computing an exponential weighting Fig. 3. A Spines overlay average. Spines nodes add a link specific sequence number on every data packet sent between two neighboring overlay follows a diminishing return function. At some early point, nodes. The receiving overlay node uses this sequence number the benefit of adding additional nodes to an existing overlay to estimate the loss rate of overlay links. Upon receiving is small, and is overcome by the overhead associated with a a number of packets out of order (in our implementation, larger overlay. Therefore we opt for having a relatively small three), the node decides that a loss event happened. The overlay (tens to hundreds of nodes) with direct link latencies loss rate is computed by averaging the number of packets in the order of milliseconds or tens of milliseconds. received between two subsequent loss events over the last L loss events (in our implementation L = 50). This way, the loss estimate converges relatively fast when loss rate increases A. Spines (fewer packets will be received between two loss events), but Spines [4] is an open source overlay network that allows is conservative in switching to temporarily low-loss overlay easy deployment and testing of overlay protocols. It runs in links. Based on link loss and latency, a cost for each link is user space, does not need special privileges or kernel modifi- computed as described in Section IV-B and propagated through cations, and encapsulates packets on top of UDP. Spines offers the overlay network by an incremental link-state protocol that a two-level hierarchy in which applications (clients) connect uses reliable flooding mechanism. to the closest overlay node, and this node is responsible for The control traffic required for maintaining the overlay forwarding and delivering data to the final destination through network is small compared to the overall data traffic, consisting the overlay network. The benefit of this hierarchy is that in our implementation of periodical hello messages and link it limits the size of the overlay network, thus reducing the updates. One 32 byte hello message is sent every second by amount of control traffic exchanged. An example of a Spines each of the two end-nodes of a direct link. A single link update overlay network is shown in Figure 3. Overlay nodes act both is propagated to all the nodes in the overlay through a reliable as servers (accepting connections from various applications) flooding algorithm only in case of a network change, such as and as routers (forwarding packets towards clients connected a variation in the estimation of delay or loss rate of the link, to other overlay nodes). Applications may reside either locally or when a link or node goes down. On the initial state transfer with the Spines nodes or on machines different than the over- the entire routing table is transfered. In this case, as well as lay node they connect to. In this paper we focus only on the when multiple network events happen simultaneously, multiple overlay protocols and in all of our experiments, applications link updates are aggregated, so that a regular Ethernet packet reside either on the same machine as their overlay node, or on can carry between 60 and 90 distinct updates, depending on a different machine connected through a local area network the sparsity of the network. In the current implementation, to the overlay node. We expect that the overlay nodes are Spines scales to up to several hundred overlay nodes, and connected through high latency wide area links. up to a thousand clients per node. Spines detects network In order to connect to a Spines overlay node, applications failures through its hello protocol in under 10 seconds. Our use a library that enables UDP and TCP communication experiments on the Planetlab network show that reducing the between the application and the selected Spines node. The failure detection interval to less than 10 seconds increases the API offered by the Spines library closely resembles the Unix chance of route instability. socket interface, and therefore it is easy to port any application to use Spines. We describe in Section VI the necessary steps to IV. P ROTOCOLS FOR INCREASING VO IP QUALITY adapt current VoIP applications to use Spines. Each application is uniquely identified by the IP address of the overlay node it Traditional VoIP systems use UDP to transfer data, exposing connects to, and by an ID given at that node, which we call the audio channels to packet loss. One of the main reasons Virtual Port. Spines provides both reliable and best-effort com- for not using end-to-end retransmissions is that lost packets, munication between end applications, using the applications' even when recovered end-to-end from the source, are not node IP address and the Virtual Port in a way similar to TCP likely to arrive in time for the receiver to play them. Overlay and UDP. Similar to the socket() call, a spines socket() function networks break end-to-end streams into several hops, and even 4.5 4.5 Uniform Uniform 25% burst 25% burst 4 50% burst 4 50% burst 75% burst 75% burst 3.5 3.5 3 3 RT protocol loss (%) RT protocol loss (%) 2.5 2.5 2 2 1.5 1.5 1 1 0.5 0.5 0 0 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 Link loss (%) Per-link link loss (%) Fig. 4. Real-time recovery loss - 1 link Fig. 5. Real-time loss recovery - 2 concatenated links though an overlay path may be longer than the direct Internet lost. For a link with independent loss rate p in both directions, path between the two end-nodes, each individual overlay hop the above sequence of events will happen with probability usually has smaller latency, thus allowing localized recovery p · (1 − p) · p = p2 − p3 . Another significant case is when the on lossy overlay links. retransmission request does arrive, but the retransmission itself is lost, which can happen with probability p·(1−p)·(1−p)·p = A. Real-time recovery protocol p2 −2p3 +p4 . Other types of events, that involve multiple data packets lost can happen, but their probability of occurrence Our overlay links run a real-time protocol that recovers is negligible. We approximate the loss rate of our real-time packets only if there is a chance to deliver them in time, and protocol by 2p2 − 3p3 , assuming that the loss probability on forward packets even out of order to the next hop. We describe the link is uniform3. our real time recovery protocol next: The delay distribution of packets follows a step curve, such • Each overlay node keeps a circular packet buffer per out- that for a link with one way delay T and loss rate p, (1 − p) going link, maintaining packets sent within a time equal fraction of packets arrive in time T , (p − 2p2 + 3p3 ) are to the maximum delay supported by the audio codec. retransmitted and arrive in time 3T + ∆, where ∆ is the time Packets are dropped out of the buffer when they expire, or it takes the receiver to detect a loss, and (2p2 − 3p3 ) of the when the circular buffer is full. In our implementation, the packets will be lost by the real time recovery protocol. For circular buffer can maintain 1000 VoIP packets, totaling a path that includes multiple links, the delay of the packets 160 KBytes. will have a composed distribution given by the combination • Intermediate nodes forward packets as they are received, of delay distributions of each link of the path. The time ∆ it even out of order. takes the receiver to trigger a retransmission request depends • Upon detecting a loss on one of its overlay links, a on the inter-arrival time of the packets (the receiver needs node asks the upstream node for the missed packets. A to receive a packet to know that it lost the previous one) retransmission request for a packet is only sent once. and on the number of out of order packets that the protocol We only use negative acknowledgments, thus limiting the can tolerate. For a single VoIP stream, packets usually carry amount of traffic when no packets are lost. 20 msec of audio, so they arrive at relatively large intervals. • When an overlay node receives a retransmission request In our overlay approach, we aggregate multiple voice streams it checks in its circular buffer, and if it has the packet within a single real-time connection on each link. The overlay it resends it, otherwise it does nothing. A token bucket link protocol handles packets much more often than a single mechanism regulates the maximum ratio between the VoIP stream, and therefore the inter-arrival time of the packets number of retransmissions and the number of original is much smaller. Standard TCP protocol needs three packets packets sent. This way we limit the number of retrans- out of order before triggering a loss. Since latency is crucial for missions on lossy links. VoIP applications, and as packet reordering is a rare event [18], • If a node receives the same packet twice (say because it in our experiments we trigger a retransmission request after was requested as a loss, but then both the original and receiving the first out of order packet. the retransmission arrive), only the first instance of the We implemented the real time protocol in the Spines overlay packet will be forwarded towards the destination. network platform and evaluated its behavior by running Spines The protocol does not involve timeouts and never blocks on Emulab. Figure 4 shows the loss rate of the real time for recovery of a packet. On the other hand, this is not a fully recovery protocol on a 10 msec link with various levels of loss reliable protocol and some of the packets will be lost in case and burstiness, and Figure 5 shows the combined loss for two the first retransmission attempt fails. Such events can appear when a packet is lost, the next packet arrives and triggers a 3 In many cases, the loss rate probability may not be uniform. Later in the retransmission request, but the retransmission request is also paper, we investigate the impact of burstiness on our protocols. 90 90 Uniform Uniform 80 25% burst 80 50% burst 70 75% burst 70 60 60 Delay (milliseconds) Delay (milliseconds) 50 50 40 40 30 30 20 20 10 10 0 0 80 82 84 86 88 90 92 94 96 98 100 80 82 84 86 88 90 92 94 96 98 100 Packets (%) Packets (%) Fig. 6. Delay distribution - 1 link, 5% loss Fig. 7. Delay distribution - 2 concatenated links, 5% loss each concatenated 10 msec links that experience the same amount of loss and burstiness, in both directions, running Spines with … … the real-time protocol on each link. For each experiment, we 10 Mbps 10 Mbps 10 Mbps 10 Mbps 10 Mbps generate traffic representing the equivalent of 10 VoIP streams A 10 ms B 10 ms C 10 ms D 10 ms E 10 ms F for a total of two million packets of 160 bytes each, and then average loss rate was computed. We can see that the level Fig. 8. Spines network - 5 links of burstiness on the link does not affect the loss rate of the real-time protocol. The real-time loss rate follows a quadratic curve that matches our 2p2 − 3p3 estimate. For example, for a In order to evaluate the effect of local recovery on voice single link with 5% loss rate, applying the real-time protocol quality we ran the same experiment depicted in Figure 1 and reduces the loss rate by a factor of 10, to about 0.5% regardless Figure 2 on top of a Spines overlay network. We divided the of burstiness, which yields an acceptable PESQ score (see 50 msec network into 5 concatenated 10 msec links as shown Figure 1). in Figure 8, ran Spines with the real-time protocol on each link, For the single 10 msec link experiment with 5% loss rate, and sent 10 VoIP streams in parallel from node A to node F . the packet delay distribution is presented in Figure 6. As We generated losses with different levels of burstiness on the expected, 95% of the packets arrive at the destination in about middle link C − D and set the network latency threshold for 10 milliseconds. Most of the losses are recovered, showing the G.711 codec to be 100 msec. a total latency of 30 msec plus an additional delay due to Figure 9 presents the average PESQ score of all the sentence the inter-arrival time of the packets required for the receiver segments in the G.711 streams using Spines, and compares it to detect the loss, and about 0.5% of the packets could not with the results obtained when sending over UDP directly. be recovered. Lost packets are shown as no delay measured, Since most of the packets are received in time to be decoded between 99.5 and 100%. at the receiver, we can see that when using Spines, regardless In the case of uniform loss probability the delay of the of burstiness, the G.711 codec can sustain on average even recovered packets is almost constant. However, when the link network losses of 7-8% with PSTN voice quality. experiences loss bursts, multiple packets are likely to be lost in As discussed earlier, users of telephony services expect high a row, and therefore it takes longer for the receiver to detect quality service. Therefore, in addition to average performance the loss. The increase of the interval ∆ (the time it takes characteristic, it is important to look at the performance of to detect a loss) results in a higher delay for the recovered the worst cases. Figure 10 shows the quality of the worst 5 packets. Obviously, the higher the burstiness, the higher the percentile sentence intervals. We can see that the codec can chance for consecutive losses, and we can see that the packet handle up to 3.5% losses with PSTN quality and even for the delay is mostly affected at 75% burstiness. worst 5% sentence segments, the burstiness did not play a Figure 7 shows the delay distribution for the two-link major role in the voice quality. network, where both links experience 5% uniform distribution loss rate. As in the single link experiment, most of the losses are recovered, with the exception of 1% of the packets. We B. Real time routing for audio notice, however, a small fraction of packets (slightly less than The real time protocol recovers most of the missed packets 0.25%) that are lost and recovered on both links, and that in case of occasional, or even sustained periods of high loss, arrive with a latency of about 66 msec. This was expected to but if the problem persists, we would like to adjust the overlay happen with the compound probability of loss on each link, routing to avoid problematic network paths. pc = 0.05 · 0.05. Burstiness results for the two-link network, Given the packet delay distribution and the loss rate of the not plotted in the figure, follow the same pattern as shown in soft real-time protocol on each overlay link, the problem is Figure 6. how to find the overlay path between a pair of source and 4.5 4.5 Spines Uniform Spines 25% burst Spines 50% burst Spines 75% burst 4 4 UDP Uniform UDP 75% burst RT PESQ - 5 percentile PESQ - Average 3.5 3.5 Spines Uniform 3 Spines 25% burst 3 Spines 50% burst Spines 75% burst UDP Uniform UDP 75% burst 2.5 2.5 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 loss rate (%) loss rate (%) Fig. 9. Real-Time protocol - Average PESQ Fig. 10. Real-Time protocol - 5 percentile PESQ 10 ms, 2% loss several links, our approximation for the total expected delay B will then be the sum of the expected delay of each individual link. We call this metric expected latency cost function. A D ? E We evaluated the performance of the expected latency based routing and compared it with several other cost metrics C described below. We used the BRITE [19] topology generator to create random topologies using the Waxman model, where 20 ms, 1% loss the probability to create a link depends on the distance between the nodes. We chose this model because it generates mostly Fig. 11. Two-metric routing decision short links that fit our goal for localized recovery. We assigned random uniform loss from 0% to 5% on half of the links of each topology, selected randomly. Since routing decisions destination, for which the packet delay distribution maximizes are done at relatively large intervals (10 seconds or more), the number of packets that arrive within a certain delay, so the short term burstiness of the losses does not influence our that the audio codec can play them. The problem is not trivial, cost metric, and therefore we only looked at the long term and deals with a two metric routing optimizer. For example, in loss average in this experiment. We considered every node Figure 11, assuming a maximum delay threshold for the audio generated by BRITE to be an overlay node, and every link codec to be 100 msec, if we try to find the best path from to be an overlay edge. For each topology we determined the node A to node E, even in the simple case where we do not nodes defining the diameter of the network (the two nodes for recover packets, we cannot determine which partial path from which the shortest latency path is longest), and determined the node A to node D is better (maximizes the number of packets routing path between them given by different cost metrics. arriving at E within 100 msec) without knowing the latency By adjusting the size of the plane in which BRITE generates of the link D-E. On the other hand, computing all the possible topologies, networks with different diameters are created. For paths with their delay distribution and choosing the best one each network diameter we generated 1000 different topologies is prohibitively expensive as the network size increases. and evaluated the packet delivery ratio between the network However, if we can approximate the cost of each link by diameter nodes when running the real-time protocol on the a metric dependent on the link's latency and loss rate, taking links of the network, using different routing metrics. Note that into account the characteristics of our real-time protocol and while we adjusted the size of the network plane, the placement the requirements of VoIP, we can use this metric in a regular of the nodes and individual link latencies was chosen randomly shortest path algorithm with reasonable performance results. by BRITE. Figure 12 shows the average delivery ratio for Our approach is to consider that packets lost on a link actually network topologies with 15 nodes and 30 links, and Figure 13 arrive, but with a delay Tmax bigger than the threshold of the shows the delivery ratio for network topologies with 100 nodes audio codec, so that they will be discarded at the receiver. and 200 links. For a link with direct latency T and loss rate p, Then, the packet delay distribution of a link will be a three considering an audio codec threshold Tmax = 100 msec and step curve defined by the percentage of packets that are not the packet inter-arrival time ∆ = 2 msec, the cost metrics used lost (arriving in time T ), the percentage of packets that are are computed as follows: lost and recovered (arriving in 3T + ∆), and the percentage of • Expected latency: Cost = (1 − p) · T + (p − 2p + 3p ) · 2 3 packets missed by the real-time protocol (considered to arrive 2 3 (3T + ∆) + (2p − 3p ) · Tmax after Tmax ). The area below the distribution curve represents • Hop distance: Cost = 1 the expected delay of the packets on that link, given by the • Link latency: Cost = T formula: Texp = (1−p)·T +(p−2p2 +3p3 )·(3T +∆)+(2p2 − • Loss rate: Cost = −log(1 − p) 3p3 ) · Tmax. Since latency is additive, for a path consisting of • Greedy optimizer: We used a modified Dijkstra algorithm 100 100 99.5 99.5 99 99 Avg. delivery ratio (%) Avg. delivery ratio (%) 98.5 98.5 98 98 97.5 97.5 expected latency expected latency 97 hop distance 97 hop distance latency latency loss rate loss rate 96.5 greedy optimizer 96.5 greedy optimizer best route 96 96 0 10 20 30 40 50 60 70 0 10 20 30 40 50 60 70 Network diameter (ms) Network diameter (ms) Fig. 12. Comparing routing metrics - 15 node networks Fig. 13. Comparing routing metrics - 100 node networks that, at each iteration, computes the delay distribution of network congestion, and the likelihood of routing instability the selected partial paths and chooses the one with the and oscillations. In our experiments on Planetlab we noticed maximum delivery ratio. that allowing consecutive route changes due to a single link • Best route: All the possible paths and their delay dis- to happen faster than 10 seconds apart, can create occasional tributions were computed, and out of these the best one instability that leads to VoIP quality degradation. Therefore, in was selected. Obviously, this operation is very expensive, the current implementation, Spines considers a link to be down mainly because of the memory limitation of storing all if two neighboring nodes cannot communicate through it for combinations of delay distributions . Using a computer more than 10 seconds. Moreover, for links that are available with 2GB memory we could not compute the best route but change their network characteristics (latency, loss rate), for networks with more than 16 nodes. Spines requires at least 30 seconds between two consecutive As expected, for small diameter networks the loss-based cost updates of the same link. routing achieves very good results, as the delay of the links is less relevant. With the increase in the network diameter, the latency-based routing achieves better results. At high latencies, V. D EPLOYMENT C ONSIDERATIONS FOR OVERLAY the packet recovery becomes less important than the risk N ETWORKS of choosing a highly delayed path, with latency more than the codec threshold. The expected latency routing achieves lower delivery ratio than the loss-based routing for small Running overlay node software in user level gives us great diameter networks, but behaves consistently better than the flexibility and usability, but comes at the expense of packet latency-based routing. The slight drop in delivery ratio for processing through the entire networking stack, and process low diameter networks is causing just a small change in VoIP scheduling on the machines running the overlay nodes. quality (see Figures 1 and 2), while the robustness at high Executing overlay network functionality on loaded comput- network delays is very important. Interestingly, the greedy ers naturally degrades the performance of the overlay system. optimizer fails at high latency networks, mainly due to wrong This degradation is critical especially for latency sensitive routing decisions taken early in the incremental algorithm, VoIP streams. For example, if the overlay daemon runs as a without considering the full topology. user level process on a computer that has other CPU intensive This experiment shows that the expected latency metric, processes, it is common for the overlay network system not while being comparable with other routing metrics for small to be scheduled for several hundred milliseconds, and even diameter networks achieves better routing in high latency seconds, which of course, is not useful for VoIP. In fact, networks, exactly where we need it the most. our experience with another messaging system, the Spread While an efficient link cost metric is essential for finding toolkit [20] that is commonly deployed on large websites, routes that provide high quality VoIP, it is important to shows that on heavily loaded web servers a process may be trigger fast routing decisions that avoid links that suddenly scheduled only after eight seconds. It is common practice on fail, or experience high levels of congestion. Internet studies such systems to assign the messaging system higher priority mentioned in Section II show that it can take up to minutes, (real time priority in Linux). Note that messaging systems or tens of minutes until the Internet routing converges in consume negligible amounts of processing when they do not case of a failed link. Since overlay networks are smaller need to send or receive packets. Therefore, increasing their compared to the Internet, they can afford closer monitoring priority does not affect performance of other applications of the networking resources and faster decisions in case the unless packets - that they probably need, anyway - are sent or network conditions change, in the order of seconds or tens received. For a VoIP service it is reasonable to expect that the of seconds. The dynamic of the routing decisions is a trade- overlay nodes will be well provisioned in terms of CPU and off between how fast the system can detect link failures or networking capabilities. loaded computer by running a simple while(1) infinite loop program on the middle node B, and repeated the above … … 100 Mbps LAN 100 Mbps LAN experiment. Running Spines with the same priority as the A B C loop program, when forwarding a single voice stream we achieved a very high packet delay average of 74.815 msec, Fig. 14. Spines network - 2 links and the maximum packet delay was 154.468 msec. When competing with 4 loop programs in parallel, with the same 0.4 priority as Spines, the average packet delay for a single stream Spines UDP went up even more to 298.354 msec (about 900 times more 0.35 than without CPU competing applications), and the maximum 0.3 packet delay was 604.917 msec. Obviously, such delays are not Average packet latency (ms) 0.25 suitable for VoIP. However, when we set real-time priority for the Spines process, the high CPU load did not influence our 0.2 performance. Even when competing with 10 loop programs 0.15 and a load of 200 streams, the average packet delay was a 0.1 low 0.315 msec and the maximum packet delay measured was 0.469 msec. 0.05 Our conclusion is that the performance of overlay routing 0 20 40 60 80 100 120 140 160 180 200 at the application level can be highly affected by the load VoIP streams on the supporting computers. However, this overhead can be easily reduced to negligible amounts compared to wide area Fig. 15. Spines forwarding delay latencies simply by increasing the priority of the overlay node applications. This should not be a problem in general, as the overlay node applications do not use CPU unless they send A. Routing performance in Spines packets, and if they do need to send packets, in a system It is interesting to evaluate the overhead of running the dependent on real-time packet arrival, the overlay nodes should overlay nodes as regular applications in the user space, and not be delayed. how the routing performance is affected by the amount of traffic or load on the computers. We deployed Spines in Emulab on a three node network as shown in Figure 14, B. Case study: Planetlab where the middle node B had two network interfaces and Planetlab [6] is a large distributed testbed of computers was directly connected to nodes A and C through local area connected through the public Internet. Currently Planetlab links. All the computers used were Intel Pentium III 850MHz has 583 nodes at 275 sites. Unlike Emulab, which has a machines. The one-way latency between node A and node reservation mechanism that completely allocates computers to C measured sending 160 byte packets and adjusted with the a particular experiment, Planetlab is a shared environment. clock difference between the nodes was 0.189 msec. Experiments run within slices created on several (or all) We ran a Spines instance on each node, using the real- Planetlab computers. Each slice acts as a virtual machine time protocol on the links A − B and B − C. Then we on each computer and shares the resources with other users' sent a varying number of voice streams in parallel (from 1 to slices. In the current incarnation of Planetlab it is not possible 200 streams), consisting of 20000 packets of 160 bytes each, to set real-time process priorities, and the available CPU time from node A to node C using Spines. We measured the one depends on the applications that run at the same time. Due to way latency of each packet, adjusted by the clock difference the high CPU load on the machines, Planetlab losses happen between machines A and C. When forwarding 200 streams, mostly at the overlay nodes, happen in large bursts (tens to the middle node B running Spines showed an average CPU hundreds of packets lost in a row), and are mainly caused by load of about 40%. However, the sending node A, on which delays in scheduling the receiver application while competing both Spines and our sending application were running, reached with other processes (note that due to the stringent delay a maximum 100% CPU utilization. requiremnts of VoIP applications, packets that arrive late are Figure 15 shows the average latency of packets forwarded discarded and are therefore equivalent to lost packets). As through Spines as the number of parallel streams increases such, we believe that Planetlab, in its current configuration, from 1 to 200, and compares it to the base network latency is not suitable for deploying VoIP applications. measured with UDP probes. The standard deviation of all the However, it is useful to evaluate the behavior of our overlay measurements was a very low 0.012, and the highest single protocols on an Internet testbed with realistic latency char- packet latency measured, which happened when we sent 200 acteristics. In a first experiment we deployed Spines on 32 streams in parallel, was 0.963 msec. What we see is that Planetlab nodes that we found to be synchronized to under regardless of the number of streams, the three Spines nodes 20 msec, and that were located at 26 sites, shown in Table I, add a very small delay totaling about 0.15 msec due to user- within the North American continent. The maximum delay level processing and overlay routing. between any two sites was 47 msec, and appeared between We evaluated the routing performance of Spines on a CPU MIT and Internet2 - Denver. Note that our overlay protocols TABLE I P LANETLAB SITES - C ONTINENTAL US Canarie - Calgary Canarie - Halifax Canarie - Toronto Harvard University Internet2 - Atlanta Internet2 - Chicago Internet2 - Denver Internet2 - Indianapolis Internet2 - Washington Johns Hopkins Information Security Institute Massachusetts Institute of Technology New York University Northeastern University CCIS Princeton Purdue Rutgers University University of Florida - ACIS Lab University of Georgia University of Illinois at Urbana-Champaign University of Kansas University of Maryland University of Pennsylvania University of Rochester University of Wisconsin Washington University in St Louis Wayne State University 100 100 90 90 Spines Spines 80 UDP 80 UDP 70 70 CDF: Streams (%) CDF: Streams (%) 60 60 50 50 40 40 30 30 20 20 10 10 0 0 0 2 4 6 8 10 12 14 16 18 20 0 2 4 6 8 10 12 14 16 18 20 Lost packets (%) Missed packets (%) Fig. 16. Planetlab US - percentage of streams that lost at least a certain Fig. 17. Planetlab US - percentage of streams that missed at least a certain percentage of packets percentage of packets do not require clock synchronization, however we do need sender and receiver at any time during the experiment was 512 synchronized clocks in our experiments to evaluate the number Kbps. of packets arriving within a certain delay constraint. In order For each UDP and Spines stream we counted the number to compensate for the 20 msec clock difference between the of lost packets and the number of late packets (arriving at nodes we set the codec threshold at 120 msec. destination after more than 120 msec). Based on these we We created a Spines overlay network consisting of a fully computed the packets missed by the codec as the sum of connected graph, such that each of the 32 overlay nodes had a both lost and late packets. Figure 16 presents the cumulative direct overlay link to all other nodes. Based on the latency and distribution function (CDF) of the lost packets over all streams loss rate measured on each link, Spines selected the routing (i.e. the percentage of streams that lost less than a certain paths that minimize the expected latency metric presented in percentage of packets). We first notice that about 50% of Section IV-B. the UDP streams experienced only the 5% loss we injected, We then sent traffic consisting of 4 voice streams from each the rest of the streams being highly affected by underlying node to all the other nodes, in pairs, such that at any time there Planetlab loss. In contrast, the large majority of Spines streams was only one node sending packets to only one receiver node. were able to recover most of the losses. As the underlying Each voice stream was sending a total of 2000 packets. At the loss increases, for about 5% of the streams, Spines could not same time, and between the same pair of nodes, we sent equal improve the delivery ratio over UDP. We believe this is due traffic consisting of 4 voice streams directly via UDP, using to the loss nature of highly CPU loaded Planetlab nodes - in point to point connections between sender and receiver. This long bursts and happening at the end nodes. In Figure 17 we way, dynamic variations of loss and latency in the network present the CDF of missed packets that could not be decoded affected equally the UDP streams and the streams sent through by the voice codec, either because they were lost or because the Spines overlay. In addition to the underlying Planetlab they arrived in more than 120 msec. Even though our testbed losses, we generated 5% additional loss on each overlay link was confined to the North American Continent, with delays and also on each UDP stream by dropping packets at each well below the 120 msec codec threshold, we see that many node, just before processing them. As we ran our experiment of the packets received by the UDP streams arrive late; the on 32 nodes, with 8 voice streams running simultaneously number of missed packets being considerably higher than the between each pair of nodes, there were a total of 992 sender- number of lost packets. Spines streams were able to deliver receiver combinations, and the combined traffic between the more packets than UDP in time in almost all cases, until the 100 Spines by Spines through the node at University of Washington, to 90 UDP which both sites had better connectivity compared to their 80 direct link. This result indicates the ability of Spines to adapt 70 to the topology and find alternate routes that improve VoIP CDF - packets (%) 60 quality. We also notice in Figure 18 that when Spines could 50 not improve latency by routing through alternate routes, the 40 overhead of using Spines nodes was minimal. Both UDP and 30 Spines packets arrived with almost identical latency. 20 10 VI. I NTEGRATION WITH THE CURRENT INFRASTRUCTURE 0 0 50 100 150 200 250 300 350 Given the large installed base of VoIP end clients and the Latency (ms) even larger planned future deployments it is imperative that our Fig. 18. Planetlab Inter-Continental - percentage of packets that arrive earlier system integrates seamlessly with the existing infrastructure. than a certain latency We explain what are the necessary steps to achieve this in the rest of this section. The first component of the integration has to do with how loss rate became very high (20%) and CPU scheduling delays VoIP clients are able to find their closest Spines server. We equally affected Spines and UDP streams. assume that each domain that wants to take advantage of the benefits provided by our system will deploy a Spines node TABLE II as part of their infrastructure. In this case VoIP clients can P LANETLAB SITES - I NTER -C ONTINENTAL use DNS SRV [21] records to locate the Spines node that 1. University of Washington (USA) is serving their administrative domain. This DNS query will 2. RNP - Rio Grande do Sul (Brazil) return the IP address of (at least one) Spines node that can 3. Swedish Institute of Computer Science (Sweden) serve as their proxy in the Spines overlay. Once this node 4. KAIST (Korea) 5. Monash University - DSSE (Australia) is found the VoIP clients can communicate with it using the interface we described in Section III-A. Then, the VoIP clients have to direct media traffic to flow through the Spines network We evaluated the benefit and robustness of our protocols on rather than directly over UDP. large networks that span multiple continents through a second We have two proposed solutions for this issue: one that experiment. We selected five sites in Planetlab, each one on requires changes to the end-clients and one that does not. We a different continent. The sites are shown in Table II. The begin with the solution that requires "Spines-enabled" clients. network latency between the sites was fairly large, ranging In the current architecture, the Session Initiation Protocol (SIP) between 59 ms (University of Washington to KAIST) and [22] is used as a signaling protocol so that the two communica- 290 ms (RNP to KAIST). Given the high latency between the tion endpoints can negotiate the session parameters, including sites, the VoIP quality is heavily affected, and any reduction the IP address and ports that each client is waiting to receive in packet delay is critical for improving the interactivity of media traffic on. A Spines-enabled VoIP client announces its conversation. capability using the parameter negotiation feature that is part We connected all sites in a Spines overlay network where of SIP, within the initial INVITE SIP message. The VoIP client each node had direct links to all other nodes, and used the includes in the same INVITE message the IP address and the expected latency metric for routing inside Spines. We sent Virtual Port that its Spines server is waiting to receive media VoIP traffic from every site to all other sites, in pairs, both traffic for the client. If the peering VoIP client is also Spines- using UDP and Spines, similarly to the previous experiment. enabled it will reply positively and include in its reply the Between every pair of nodes, Spines and UDP streams were address and Virtual Port at its own server. On the other hand, run simultaneously. As the latency was relatively high, the if the peer is not Spines-enabled it will return an error code Spines real-time recovery protocol was not able to recover indicating to the session initiator that it will have to revert to a any of the lost packets, and therefore we observed almost "legacy" session. If the SIP negotiation was successful using identical loss rates for Spines and UDP streams. However, Spines, each source will send media traffic through its local Spines was able to improve packet latency by using indirect Spines server towards the Spines server indicated by the peer routes that offered lower latency compared to the end-to-end client. As the traffic is forwarded through the overlay network, path provided by Internet routing. the egress Spines node will finally deliver it to the destination Figure 18 shows the CDF of the packet latency for all VoIP client. streams. First, we can see that fewer packets have high latency RTP [23] and RTCP data is sent seamlessly through the when using Spines, compared with UDP. The most significant Spines network, offering the end clients information about the latency improvement happens for the slowest 10% of the network conditions along the overlay path they use. packets, which achieve a latency of about 190 ms when using While this first solution is architecturally pure, it requires Spines, compared with about 290 ms when using UDP. The changes to the end clients which may not be initially possible. reason is that streams between RNP to KAIST were routed In this case, we propose to use a solution similar to the NAT-traversal in SIP [24]. Specifically, Spines nodes will be to specific application characteristics such as the codec used or required to intercept SIP INVITE messages and change the type of PLC. Since the approach uses end-to-end traffic analy- IP address and ports to point to themselves rather than to the sis to optimize performance, it is complementary to our work. VoIP clients. This way all the media traffic will flow through We believe that it can transparently use our Spines overlay the Spines network which will eventually deliver it to the end- architecture between the APS gateways to improve end-to-end hosts. packet delivery ratio by exploiting our multi-hop routing and local recovery mechanisms. Finally, OverQoS [31] is probably closest to our work, as it proposes an overlay link protocol VII. R ELATED WORK that uses both retransmissions and FEC to provide loss and Our goal in this work is to reduce the effect of Internet throughput guarantees. OverQoS depends on the existence of losses on the quality of VoIP traffic. We do so by using an an external overlay system (the authors suggest RON as an overlay network that attempts to quickly recover lost packets option) to provide path selection and overlay forwarding. In by using limited hop-by-hop retransmissions and an adaptive this respect, our system can use OverQos as a plug-in module routing algorithm to avoid persistently lossy links. In this as an alternative to our real-time recovery protocol presented in respect our work is related with techniques that try to reduce Section IV-A, probably with the necessary modifications that the loss rate of underlying Internet paths and with other work take into account the special requirements of voice traffic. in overlay networks. Multi Protocol Label Switching (MPLS) [25] has been VIII. C ONCLUSION recently proposed as a way to improve the performance of In this paper we have shown that current network conditions underlying network. This is done by pre-allocating resources can significantly affect VoIP quality, and we proposed a de- across Internet paths (LSPs in MPLS parlance) and forwarding ployable solution that can overcome many of these conditions. packets across these paths. Our system is network agnostic and Our solution uses the open source Spines overlay network therefore does not depend on MPLS, but it can leverage any to segment end-to-end paths into shorter overlay hops and reduction in loss rate offered by MPLS. At the same time, attempts to recover lost packets using limited hop-by-hop MPLS will not eliminate route and link failures or packet retransmissions. Spines includes an adaptive routing algorithm loss. Since it runs at a higher level, our overlay network can that avoids chronically lossy paths in favor of paths that will continue to forward packets avoiding failed network paths. deliver the maximum number of voice packets within the Forward Error Correction (FEC) schemes [26] have also been predefined time budget. We evaluated our algorithms and our proposed as a method of reducing the effective loss rate system implementation in controlled networking environments of lossy links. These schemes work by adding redundant in Emulab, on the Internet using the Planetlab testbed, and information and sending it along with the original data, based through extensive simulations on various random topologies. on the feedback estimate of loss rate given by RTCP, such Our results show that Spines can be very effective in masking that in case of a loss, the original information (or part of it) the effects of packet loss, thus offering high quality VoIP even can be recreated. Most of the VoIP solutions today (including at loss rates higher than those measured in the Internet today. the G.711 codec we use in this paper) use some form of FEC to ameliorate the effect of loss. Given the occasional R EFERENCES bursty loss pattern of the Internet, many times the FEC [1] A. Markopoulou, F. A. Tobagi, and M. J. Karam, "Assessing the mechanisms are slow in estimating the current loss rate, and quality of voice communication over internet backbones," IEEE/ACM Transactions On Networking, vol. 11, no. 5, pp. 747–760, October 2003. therefore we believe that localized retransmissions are required [2] V. Paxson, "End-to-End Packet Dynamics," IEEE/ACM Transactions on for maintaining voice quality. Moreover, since our approach Networking, vol. 7, no. 3, pp. 277–292, 1999. increases the packet delivery ratio, FEC mechanisms will [3] D. G. Andersen, A. C. Snoeren, and H. Balakrishnan, "Best-Path vs. Multi-Path Overlay Routing," in Proceedings of IMC 2003, Oct. 2003. notice a considerable reduction in loss, and therefore reduce [4] "The Spines Overlay Network," http://www.spines.org. their redundancy overhead. [5] B. White, J. Lepreau, L. Stoller, R. Ricci, S. Guruprasad, M. Newbold, Overlay networks have emerged as an increasingly growing M. Hibler, C. Barb, and A. Joglekar, "An integrated experimental environment for distributed systems and networks," in Proc. of the Fifth field over the last few years, motivated mainly by the need to Symposium on Operating Systems Design and Implementation. Boston, implement new services not supported by the current Internet MA: USENIX Association, Dec. 2002, pp. 255–270. infrastructure. Some of the pioneers of overlay network sys- [6] L. Peterson, D. Culler, T. Anderson, and T. Roscoe, "A Blueprint for Introducing Disruptive Technology into the Internet," in Proceedings of tems are X-Bone [27] and RON [16], which provide robust the 1st Workshop on Hot Topics in Networks (HotNets-I), Oct. 2002. routing around Internet path failures. Other overlay networks [7] "ITU-T Recommendation G.711: Pulse code modulation (PCM) focus on multicast and multimedia conferencing [28][29]. of voice frequencies," http://www.itu.int/rec/recommendation. asp?type=items&lang=E&parent=T-REC-G.711-198811-I. Our work uses the same basic architecture of an overlay [8] "ITU-T Recommendation G.711 appendix I: A high quality low- network but it is optimized towards the specific problems of complexity algorithm for packet loss concealment with G.711," http:// VoIP traffic. In [30], Tao et al present a practical solution www.itu.int/rec/recommendation.asp?type=items&lang=E&parent=T- REC-G.711-199909-I!AppI. to improve VoIP quality by using application driven path- [9] "ITU-T Recommendation P.862: Perceptual evaluation of switching gateways (APS) located at the edge of the access speech quality (PESQ)," http://www.itu.int/rec/recommendation. networks, between the end-hosts and the Internet. The APS asp?type=items&lang=e&parent=T-REC-P.862-200102-I. [10] Y. Zhang, N. Duffield, V. Paxson, and S. Shenker, "On the Constancy gateways constantly monitor end-to-end network conditions of Internet Path Properties," in Proceedings ACM SIGCOMM Internet over multiple paths and choose the the optimal path according Measurement Workshop, Nov. 2001. [11] C. Labovitz, A. Ahuja, and F. Jahanian, "Delayed Internet Convergence," Claudiu Danilov is an Assistant Research Scien- in Proceedings of SIGCOMM 2000, Aug. 2000. tist in the Department of Computer Science, Johns [12] C. Labovitz, C. Malan, and F. Jahanian, "Internet Routing Instability," Hopkins University. He received the BS degree in IEEE/ACM Transactions on Networking, vol. 5, no. 6, pp. 515–526, Computer Science in 1995 from Politehnica Univer- 1998. sity of Bucharest, and the MSE and PhD degrees in [13] B. Chandra, M. Dahlin, L. Gao, and A. Nayate, "End-to-End WAN Computer Science from The Johns Hopkins Univer- Service Availability," in Proceedings of 3rd USISTS, Mar. 2001. sity in 2000 and 2004. His research interests include [14] C. Boutremans, G. Iannaccone, and C. Diot, "Impact of link failures on distributed systems, survivable messaging systems voip performance," in NOSSDAV 2002. and network protocols. He is a creator of the Spines [15] Y. Amir and C. Danilov, "Reliable communication in overlay networks," overlay network platform and the SMesh wireless in Proceedings of the DSN 2003, June 2003, pp. 511–520. mesh network. [16] D. Andersen, H. Balakrishnan, F. Kaashoek, and R. Morris, "Resilient overlay networks," in Proc. of the 18th Symposium on Operating Systems Principles, Oct. 2001, pp. 131–145. [17] Y. Amir, C. Danilov, S. Goose, D. Hedqvist, and A. Terzis, "1-800- OVERLAYS: Using Overlay Networks to Improve VoIP Quality," in the Proceedings of the International Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV), June 2005, pp. 51–56. [18] G. Iannaccone, S. Jaiswal, and C. Diot, "Packet reordering inside the Stuart Goose has B.Sc. (1993) and Ph.D. (1997) Sprint backbone," Sprintlab, Tech. Rep. TR01-ATL-062917, June 2001. degrees in computer science both from the Univer- [19] A. Medina, A. Lakhina, I. Matta, and J. Byers, "BRITE: An approach to sity of Southampton, United Kingdom. universal topology generation," in International Workshop on Modeling, He held a Post Doctoral position at the University Analysis and Simulation of Computer and Telecommunications Systems of Southampton. He then joined Siemens Corporate - MASCOTS '01, August 2001. Research Inc. in Princeton, New Jersey, USA, hold- [20] "The Spread Toolkit," http://www.spread.org. ing various positions in the Multimedia Technology [21] A. Gulbrandsen, P. Vixie, and L. Esibov, "A DNS RR for specifying the Department where he led a research group exploring location of services (DNS SRV)," RFC 2782, Feb. 2000. and applying various aspects of Internet, mobility, [22] J. Rosenberg, H. Schulzrinne, G. Camarillo, A. Johnston, J. Peterson, multimedia, speech, and audio technologies. His R. Sparks, M. Handley, and E. Schooler, "SIP: Session Initiation current position is Director of Venture Technology Protocol," RFC 3261, June 2002. at Siemens Technology-To-Business Center in Berkeley, California, USA. He [23] H. Schulzrinne, S. Casner, R. Frederick, and V. Jacobson, "RTP: A scouts for disruptive technologies from universities and startups, runs projects transport protocol for real-time applications," IETF, RFC 1889, Jan. to validate the technical and business merit of technologies, and, if successful, 1996. [Online]. Available: http://www.ietf.org/rfc/rfc1889.txt the technologies are transferred to the relevant product lines within Siemens. [24] F. Thernelius, "SIP, NAT and Firewalls," Master's thesis, Department of Dr. Goose has been an author on over 40 refereed technical publications. Teleinformatics, Kungl Tekniska Hgskolan, May 2000. He serves as program committee member and reviewer for IEEE Interna- [25] E. Rosen, A. Viswanathan, and R. Callon, "Multiprotocol Label Switch- tional Conference on Distributed Multimedia Systems and IEEE International ing Architecture," RFC 3031, Jan 2001. Conference Multimedia Expo. [26] J.-C. Bolot, S. Fosse-Parisis, and D. F. Towsley, "Adaptive FEC-based error control for internet telephony," in INFOCOM (3), 1999, pp. 1453– 1460. [27] J. Touch and S. Hotz, "The x-bone," in Third Global Internet Mini- Conference at Globecom '98, Nov. 1998. [28] S. Banerjee, B. Bhattacharjee, and C. Kommareddy, "Scalable applica- tion layer multicast," in Proc. of ACM SIGCOMM, 2002. [29] Y. hua Chu, S. G. Rao, S. Seshan, and H. Zhang, "Enabling conferencing David Hedqvist is a student of computer science at applications on the internet using an overlay multicast architecture," in Chalmers University of Technology in Gothenburg, ACM SIGCOMM 2001. ACM, Aug. 2001. Sweden. He specializes in software development [30] S. Tao, K. Xu, A. Estepa, T. Fei, L. Gao, R. Guerin, J. Kurose, and algorithms. His work on voice over IP began D. Towsley, and Z.-L. Zhang, "Improving VoIP Quality Through Path during an internship at Siemens Corporate Research Switching," in INFOCOM 2005, Mar. 2005. in Princeton, NJ and this is also the subject of his [31] L. Subramanian, I. Stoica, H. Balakrishnan, and R. Katz, "OverQoS: An Masters thesis. He will receive his MSc. in 2006. Overlay Based Architecture for Enhancing Internet QoS," in USENIX NSDI '04, Mar. 2004. Yair Amir is a Professor in the Department of Com- puter Science, Johns Hopkins University where he served as Assistant Professor since 1995, Associate Professor since 2000, and Professor since 2004. He holds a BS (1985) and MS (1990) degrees from Andreas Terzis is an Assistant Professor in the the Technion, Israel Institute of Technology, and a Department of Computer Science at Johns Hopkins PhD degree (1995) from the Hebrew University of University where he joined the faculty in January Jerusalem. Prior to his PhD, he gained extensive 2003. Andreas received his Ph.D. in computer sci- experience building C3I systems. He is a creator of ence from UCLA in 2000. His research interests are the Spread and Secure Spread messaging toolkits, in the areas of network services, network security the Backhand and Wackamole clustering projects, and wireless sensor networks. He is a member of the Spines overlay network platform, and the SMesh wireless mesh network. the IEEE and ACM SIGCOMM. He has been a member of the program committees of the IEEE International Conference on Distributed Computing Systems (1999, 2002, 2005, 2006), the ACM Conference on Principles of Distributed Computing in 2001, and the International Conference on Dependable Systems and Networks (2001, 2003, 2005). He is a member of the ACM and the IEEE Computer Society.


text:
Sidney Rochell Lowe
Position: Point Guard and Shooting Guard ▪ Shoots: Right
Height: 6-0 ▪ Weight: 195 lbs.
Born: January 21, 1960 in Washington, District of Columbia
High School: DeMatha Catholic in Hyattsville, Maryland
College: North Carolina State University
Draft: Chicago Bulls, 2nd round (1st pick, 25th overall), 1983 NBA Draft
NBA Debut: October 28, 1983
As Coach: 5 Yrs, 79-228, .257 W-L% (Full Record)
Other SR Links: College Basketball at Sports-Reference.com
Like this page? Sponsor it for $10. Your message will replace this ad.

subset: commoncrawl


text:
I tried making this recipe for a client this weekend, the cake tasted great though in my opinion but did anyone find the cake a more spongy texture?
post #106 of 108
12/23/13 at 8:55am
How true. Surely he meant exactly what he said. Besides he uses box mixes in the bakery, correct.


text:
Famous Model Photographers
Famous photographers - the greatest photographers of all time, Terry richardson (1965) famous for: modern "straight photography" as a fashion photographer, richardson's work has appeared in gq, rolling stone, vanity fair. Famous photographers - buzzle, Famous photographers it could be a hobby to some, a profession to others, or just simply an obsession. photography has inspired and stimulated the minds of many.. Famous photographers >> best photographers, Pinhole photography workshop with ilan wolf, known internationally for his work done with the ancient technique of the camera obscura is a rara avis..
Top fashion photographers directory. world famous fashion, Top fashion photographers directory: portfolio, photography, websites of famous fashion photographers. Famous models - supermodels - pop monk, Profiles of famous models is a collection of biographies, supermodels, famous model photos and models information.. Most famous photographers ← famous photographers' lists, Nan goldin (1953- ) born in washington d.c. one of the last decade's most influential photographers, nan goldin turned to photography after her sister's suicide.
Terry richardson facts & biography | famous-photographers.com, Terry richardson was born on august 14, 1965, in new york city. his father was a schizophrenic fashion photographer and his mother was also a fashion photographer. Fine art photography: work of 24 famous photographers, When a picture matches the creative vision of an artist taking the shot, the photograph is considered fine art. here is the work of 24 famous photographers.. Taltopia - casting calls, art & photography contest, A social networking platform that connects aspiring talent with fans and industry professionals..


text:
Sprint counters AT&T double data promo with its own deal
Not to be outdone by its rivals, Sprint on Wednesday announced a limited time promotion for more data. Available today and through October 31, Sprint Family Share Pack plans offer double the allotted data. Those on 32GB, 40GB, and 80GB plans will receive 60GB, 80GB, and 120GB, respectively. ...Read More


text:
All you wild-eyed rovers who (we're guessing) poured Jameson on your Lucky Charms this morning, then boarded our subway train, bound for glory—we salute you. That giant green top hat looks AMAZING. But seriously, it's not even lunchtime. Pace yourselves. Watch some videos. Drink some water.
Our weekly roundup of GQ-approved standout looks from every red carpet, paparazzo spotting, and big-ticket event across the globe... more >


text:
Find records of Sophie Andrones
Find birth, death, marriage, military, and other records of Sophie Andrones at Ancestry.com.Search for records
Biography & Family History Add details
Sophie Andrones (1901 - 1980) was born on March 10, 1901. She was born into the Andrones family.
There are no known spouses or children on file for Sophie. Sophie died in August 1980 at 79 years of age.
Sophie Andrones's last known residence is at Hackensack, Bergen County, New Jersey.
The data on this page comes from the official record for Sophie Andrones in the SSDI (United States Social Security Master Death Index). What is the SSDI?
This look at Sophie Andrones's history is maintained by the AncientFaces community Contribute any information you have to connect to the people who know her. Some things we would like to know about Sophie's genealogy:
- Where was Sophie born?
- Where did Sophie die?
- Who was Sophies family?
- What did Sophie look like? Add some photos of their life.
Family Tree & Genealogy Add / Edit People
You can add or remove people from this genealogy by clicking here.
History of Sophie Andrones Add details
- Given name
- Sophie
- Surnames
- Andrones family history
- Last Known Residence
- Hackensack, Bergen County, New Jersey
- Born
- Death
- Gender
- Female
Do you know more about Sophie Andrones?
You can add details to this page and share Sophie's story.
Obituary Add details
There is no info regarding Sophie Andrones's obituary records. You can add obituary information about Sophie Andrones by clicking "Add Details".
Discuss Sophie
Share your memories below with the rest of Sophie's family. We'll notify anyone else who is searching for Sophie of your message.
Find records of Sophie Andrones
Find birth, death, marriage, military, and other records of Sophie Andrones at Ancestry.com.Search for records


text:
Most Impressive Preseason Performers so Far
by Curt Popejoy
Even though we aren't very far into the NFL preseason football, there have already been some exceptional individual performances... Read More »
How Much Should Giants Really Expect from Jason Pierre-Paul in 2014?
by Kevin Boilard
When it comes to former All-Pro defensive end Jason Pierre-Paul , the New York Giants are hoping for the best. But are they prepared for the worst? Pierre-Paul presents himself as the Giants... Read More »
Steps to Get Giants Pass Rush Back on Track
by Kevin Boilard
Pundits will tell you the NFL is a passer's league. If that's the case, then the next most important player—after the quarterback—is the one directly trying to stop him: the pass-rusher... Read More »
JPP Is Key to 2014 Giants
by Brad Gagnon
You're familiar with the popular adage about not being able to win in the modern-day NFL without a good quarterback, but the same philosophy basically holds true with pass-rushers... Read More »
Can Healthy, Lighter JPP Carry NYG?
by Gary Davenport
The 2013 season was a disaster for defensive end Jason Pierre-Paul and the New York Giants . The Giants lost their first six games. Pierre-Paul managed only a pair of sacks in the worst season of ... Read More »


text:
Message From The Author
Book Title:
Genre: Regency Period
Try our Advanced Book SearchHelp
Advanced Book Search
Search books by title, genre, publication month, publication year, and rating or search by any combination of these options (i.e. all Mysteries published in January 2001 with 4.5 rating).
If you want to search for a name or phase, include quotation marks around your search term (example: "Deborah Smith")
Visitor login is required to post a review and comment on the blog and other interactive features on the site. Use your same username and password to register for the RT Forums.
Book Title:
Genre: Regency Period
A Passionate Performance
Take chances. Make mistakes. Get messy." That's the motto of Ms. Frizzle, one of my children's favorite characters from PBS's wonderful "The Magic School Bus" series. And it pretty well sums up my new writing thrust. After several years of writing Regencies and historicals as Eileen Winwood, I'm now using my real name and taking some fun risks with the Regency format.
Regency fans have probably heard warnings about the genre: The market is tight and publishers are cutting back because, while Regency readers are a devoted group, they don't occupy a large segment of the market. Etc., etc. On the other hand, you can't help but look at the success of movies like "Sense and Sensibility," and "Emma" and realize that there is something truly compelling about the Regency era for all audiences.
If you're a fan of historicals or contemporaries, medievals or time-travel, suspense or sci-fi, I urge you to give Regencies a try. The characters laugh, cry and make love with the best of them, and there is a wonderful tension between the rigid standards of behavior for lords and ladies and the passion that works its messy will on them as they discover romance.
Ah, passion. It can wreak havoc with expectations and rules and propriety. Which is why my latest release is entitled A PASSIONATE PERFORMANCE.
Lord Linton, a flamboyant master of illusion as cunning as he is handsome, is out to capture his father's killer. Taking a page from the bard, he decides theater is the thing to catch the villain. But when he hires an impoverished miss to masquerade as his mistress, even Linton cannot envision the extraordinary performance that awaits.
A little magic and a lot of sparks catch Linton and the feisty Sarah unawares, and I'm hoping they'll do the same for those of you venturing into your first-or 31st-Regency. My next Signet, THE PERFECT BRIDE, features matchmaking ghosts and a warrior hero who loses his heart to his betrothed's spinster chaperon. Magic? Ghosts? Warriors? Not traditional Regency fare, you say? Well, as Ms. Frizzle would say, "Take chances. Pick up a Regency." Starting with A PASSIONATE PERFORMANCE, of course! You may write to me c/o Dutton Signet, 375 Hudson St., New York, NY 10014.
RT Book Reviews © 2009 - 2014. All rights reserved.
More than 250 new books reviewed in every issue!
Subscribe: Print and Digital
Get RT's latest news and book picks, plus information on contests, videos and convention updates delivered to your inbox every month!
|Julie Garwood
Garwood shares some Sweet Talk.
Read the Interview >>
|Alma Katsu
Learn all about thnew author's paranormal trilogy.
Read the Interview >>
|Jeaniene Frost
Go inside this author's paranormal tale, Once Burned.
Read the Interview >>
Jude Deveraux
Deveraux shares the details about her latest romance.
Allison Brennan
Learn all about the new series Brennan is kicking off with her returning characters, FBI agent in training Lucy Kincaid and security expert Sean Rogan!
CJ Lyons
This author wraps up her Angels of Mercy series.
Author Interview: Meljean Brook
Meljean Brook on her award-winning book, Riveted.
Author Interview: Susane Colasanti
Colasanti discusses soulmates and summer love.
Author Interview: Delphine Dryden
Delphine Dryden talks kink and geekiness.
Download the ratings for all the new books reviewed in the October issue, conveniently formatted on easily printable pages.


text:
|By PlatformPCPS3Xbox 360Wii U3DSPS VitaAndroidiPhoneiPadOther HardwareEditor's Choiceby Author
|By LatestThe best and worst s : May Returns Bayonetta 2
10.0Fantasy Life
8.0Pokémon Art Academy
7.0Shantae and the Pirate's Curse
7.5Sid Meier's Civilization: Beyond...
9.0A City Sleeps
5.5The Legend of Korra
7.0Samurai Warriors 4
8.0Disney Fantasia: Music Evolved
8.0Fairy Fencer F
5.0More reviews
|By GenreActionAdventureFightersFree-to-playMMOMusicPlatformShootersSportsRPGStrategyMore genres
This morning, Dale told you about the sound test mode in Super Smash Bros. for 3DS and how it'll let you listen to the music even while your system is closed in sleep mode. Nintendo has also come out with a list of musicians ...
|Jordan Devore
Here it is. Formerly rumored and now in butt kickin' video form, Margaret is making her way to Persona 4 Ultimax Arena's roster. Free-for-the-first-week DLC characters Adachi and Marie also got English trailers, below, ...
|Steven Hansen
Yatagarasu Attack on Cataclysm is no longer targeting a summer launch, publisher Nyu Media revealed today. The crowdfunded fighting game now aims to launch sometime this autumn. The word came along with an announcement regar...
|Kyle MacGregor
First there was a fish playing Pokémon over a Twitch stream like it was a human or something, and now we've got two fish fighting each other in Super Street Fighter II Turbo. If you're new to this phenomenon, the tank...
|Jordan Devore
If you're a fan of six-button fight pads and find yourself wanting a new one for your PS4, you're in luck -- Arc System Works is releasing a pad for Guilty Gear Xrd SIGN. It features a full layout with the R1 and R2 keys as f...
|Chris Carter
The latest update from Super Smash Bros boss Sakurai details Samus and her new outfits. Blue and orange shorts and a small top, both of which were seen at the end of Metroid: Fusion and Metroid: Zero Suit, respectively. Here's what Sakurai had to say:
"Looking at the number of days we have left for development, it would be an impossible task to create this... That's what I told my staff. But thanks to the determination of her female designer, these Zero Suit outfits got completed in time. From the ending of Metroid: Zero Mission, here's Samus in shorts!"
Interesting that Sakurai felt compelled to specify that the designer was one of his female staff members. Personally I would have loved an alt based on her Metroid: Fusion suit. Anyway, news about a videogame characters new outfits. Because videogames.view full story + comments
Harajuku's LHP fashion shop is displaying a line of Persona 4 Arena Ultimax-themed apparel and accessories, and it's all extremely expensive (6,000 yen/$58.50 for a t-shirt) and pretty ugly at that. Dengeki brings us a caval...
|Brittany Vincent
I should have known this all along, but I was holding out hope that it would actually be a new character. That "mysterious character" in Dragon Ball Xenoverse is a custom creation. So, in essence, it's "you." You'll be able t...
|Brittany Vincent
Gaze upon that glory above. It's beautiful, isn't it? I have always been reluctant to upgrade to a 3DS XL, but this might just be what kicks it over the edge. Nintendo of Europe announced this bundle on its Twitter earlier. This handheld comes with a digital copy of Super Smash Bros., so players can get their Nintendo fight on and silently let everybody around them know it.
|Darren Nakamura
Season 2 of Killer Instinct begins on October 15, 2014. Fans can expect new fighters, enhanced game modes, and a revamped interface. The Ultra Edition will cost $39.99 and will include eight fighters starting with TJ Combo an...
|Hamza CTZ Aziz
I had ever intention to just play the Wii U version of Super Smash Bros. I think I'm going to have to check out the 3DS version now though after seeing there's going to be a Game Boy stage. Sakurai even threw in a "ba-d...
|Hamza CTZ Aziz
A few months ago, Darren covered an inventive fighting game by the name of Lethal League. I thought it looked cool back then and have only become more into the concept over time. Essentially, two to four players try to take ...
|Jordan Devore
Kano is back in the Mortal Kombat universe with all sorts of cybernetic abilities, like a mood heart in his chest that changes colors with his fighting style and whether he is wistful, embarrassed, or horny. He's like a PlayStation 4 controller with that light bar.
|Steven Hansen
Under Night In-Birth Exe:Late will strike Europe next year, NIS America has announced. The PlayStation 3 fighting game comes from Japanese indie studio French Bread, the team behind the Melty Blood series. The title was published in Japan earlier this year by BlazBlue developer Arc System works, and recently announced for a North American release by Aksys Games.
|Kyle MacGregor
Arcana Heart 3: Love Max is connecting with North America on September 23, localization house Aksys Games has announced. The all-girl fighter from Ark System Works and Examu will then gear up to strike Europe later this year, courtesy of NIS America. According to both publishers, the title will be distributed physically at retail as well as digitally via PlayStation Network.
|Kyle MacGregor
Remember that new Godzilla game that everyone seems to think will fail? I'm still interested, because Godzilla. And since the Blue Öyster Cult saw fit to write a song about Godzilla, everyone knows you should pay attenti...
|Brittany Vincent


text:
Posted Sep 01, 2012By
Photos from Day 6 of the U.S. Open tennis tournament in New York on Saturday, September 1, 2012.
[insertSlideshowjava xml="http://director.denverpost.com/images.php?album=1654" api="dpmc"]
The Photos: San Francisco Giants faced the Kansas City Royals in World Series Game 4 at AT&T Park in San Francisco,
The University of Colorado hosted UCLA at Folsom Field in Boulder on Saturday afternoon, October 25, 2014.
The Mt. SAC Cross Country Invitational on the campus of Mt. SAC in Walnut, Calif., Saturday, Oct. 25, 2014. E-Mail
Photos from across the greater Los Angeles area during Week 8 of the high school football season. Games from Long B
Friday, October 24, 2014.
West Covina defeats Los Altos 21-0 during a prep football game at West Covina High School in West Covina, Calif., F


text:
At North Beach we nurture and empower all students to reach their full potential as learners and citizens.
North Beach Thanks 2014
News & Upcoming Events
Mon. Nov. 10: Annual Giving Campaign BEGINS!
Tues. Nov. 11: Veterans Day, no school
Fri. Nov. 14: Movie Night - The Lego Movie, Cafeteria, 6:30pm
Mon. Nov. 17: Coffee with Principal Cox, Cafeteria, 9:45am
Wed. Nov. 19: Picture Retakes
Fri. Nov. 21: Last Day of Annual Giving Campaign
Mon.-Wed. Nov. 24-26: Parent/Teacher Conferences, no school
Thurs.-Fri. Nov. 27-28: Thanksgiving Holiday
Staff Spotlight
Sidney Gill - Kindergarten Teacher
Ms. Gill taught in many schools in Seattle Public Schools from 1982 until 2006. She taught in special education classrooms for 8 years and then in regular classrooms at many elementary grade levels. She returned to Seattle in June of 2013 after teaching for 5 years in an international school in Dalian, China and then for 2 years in Sofia, Bulgaria. In September of 2012, Ms. Gill became a grandmother and was so anxious to return to life in Seattle, so she could be in her granddaughter's life. "It has been a joy to return to life in the United States and to be embraced by my daughters, my son, and my dear friends. It is also an incredible gift to be back in Seattle Public Schools. I am thrilled to be a North Beach Seal and look forward to being a part of this wonderful school community. I look forward to every day with the children and hearing the sound of joyful learning."
New Faces at North Beach
North Beach PTA Calendar


text:
Bioderma • Sensibio H2O (aka Crealine H20 Solution micellaire) • Cleansers
(622 reviews)
|Average Rating.
|Would buy this product again.
diana10301 on 10/15/2014 10:35:00 PM more reviews by diana10301
Age: 30-35 Skin: Normal, Fair-Medium, Warm Hair: Blond, Straight, Medium Eyes: Brown
Love this as a toner and eye make up remover. For the evenings I still use more deeper cleansers, since I don't feel this water removes make up completaly, maybe because I tend to put a lot on me during the day. But in the lazy mornings I like to use it as water alternative, it does not dry the skin and makes me feel refreshed.
I tried quite a few micellaire waters and so far this is my favourite. The best deal you can get in a France, anywhere else it can be hard to find and quite expensive.
Was this review helpful to you? Yes No
hellodarling88 on 10/15/2014 10:09:00 PM more reviews by hellodarling88
Age: 25-29 Skin: Combination, Medium Hair: Brown Eyes: Brown
Don't expect anything drastic. It's more like your skin but better type of products.
Disliked it. Felt it removed most of my make up and left remnants floating on the surface of my skin somehow.
Divine! Removed any tightness after cleansing and somewhat relaxes my pores after a long day under make up. It does something for sensitive skin unlike most products that just don't do anything to aggravate it. I don't use a cotton pad & just smooth it over my skin like an essence seeing as I already double cleanse. But when I do use a cotton pad, it does pick up any light traces of make up I missed.
A definite repurchase.
Almost through my bottle and trying to get my stewardess friends to buy it back to Malaysia for me :)
Was this review helpful to you? Yes No
Lydia84 on 10/15/2014 3:54:00 PM more reviews by Lydia84
Age: 30-35 Skin: Combination, Fair-Medium Hair: Blond Eyes: Blue
Great product! Wouldn't be without this now. Even great on eye make up! I normally feel I have sensitive skin around my eyes however this product doesn't cause me any discomfort! Love it!!
Was this review helpful to you? Yes No
maryr624 on 10/15/2014 8:43:00 AM more reviews by maryr624
Age: 25-29 Skin: Combination, Fair Hair: Brunette Eyes: Brown
LOVE this product. There's no going back. I use it as a makeup remover and face cleanser 1-2 times a day. Leaves your skin feeling smooth and clean. No stickiness, no dryness, nothing. Well worth the price!
Was this review helpful to you? Yes No
aizmaskas on 10/11/2014 7:19:00 AM more reviews by aizmaskas
Age: 25-29 Skin: Acne-prone, Fair Hair: Brunette, Straight, Medium Eyes: Other
This is the best eye makeup remover for me. I don't use it to clean my facial makeup because it's too expensive for that. It doesn't smell and doesn't sting my eyes. The only downside is that it doesn't take off waterproof mascara.
1 out of 1 people found this review helpful. Was this review helpful to you? Yes No
marianieto830 on 10/8/2014 7:45:00 AM more reviews by marianieto830
Age: 30-35 Skin: Other Hair: Other Eyes: Other
My all time favorite! I have tried sooo many cleansing products, but this one is just pure wonder for your skin. It hydrates it, adds glow, leaves it soft, doesn't irritate it, removes makeup effortlessly (even water-proof), smells gentle and nice and ... it's just marvelous. I can't brag enough about it, seriously! The only minus is that the big packaging (500ml) that has this "pump mechanism" is not very nice to use, because when you press it the product comes out under a lot of pressure (obviously) and splashes out (not only on your cotton pad, as it has to be). But the products is so great, that I don't really care about this minus of the packaging. Love it!
1 out of 1 people found this review helpful. Was this review helpful to you? Yes No
makeupjunkie707 on 10/5/2014 12:19:00 AM more reviews by makeupjunkie707
Age: 36-43 Skin: Very Oily, Fair-Medium Hair: Blond Eyes: Blue
HG! It's the best waterless cleanser I've ever tried. Super gentle with no scent. Too bad it's so expensive and hard to get.
Was this review helpful to you? Yes No
mikaylatabone on 9/29/2014 11:05:00 AM more reviews by mikaylatabone
Age: 18 & Under Skin: Combination, Tan Hair: Brunette Eyes: Brown
Since discovering this product last year, it hasn't left my skincare routine. And nor will it ever!
Wonderful product, it performs to its purpose and removes all signs of makeup, even the waterproof mascara. All you need is a reasonable amount on a cotton pad, swipe gently and all make up is gone in a flash.
Would definitely recommend purchasing this, don't be fooled by the price, it's worth every cent.
1 out of 1 people found this review helpful. Was this review helpful to you? Yes No
Mimiyu23 on 9/29/2014 10:47:00 AM more reviews by Mimiyu23
Age: 19-24 Skin: Combination, Fair-Medium Hair: Brown Eyes: Black
Really gentle and effective for makeup touch ups and removing face makeup! But i don't like it for removing my eye makeup specially waterproof mascara.
Was this review helpful to you? Yes No
SarahGD on 9/25/2014 11:19:00 AM more reviews by SarahGD
Age: 30-35 Skin: Dry, Fair Hair: Brunette Eyes: Blue
I really like this cleanser. I normally use an oil cleanser but if I;m a little lazy in the evening this is a great alternative. Your skin feels really clean after, but def need to go over your face twice. And also wipe off the moisture it leaves on your skin before moisturising. I had very sensitive skin around my eyes and it did not irritate it at all.
Top Rated Cleansers
- 82 reviews
LUSH Let the Good Times Roll64 reviews
Aura Cacia Grapeseed Oil81 reviews


text:
Indian Springs Apartments2177 W Jasper St, Broken Arrow, OK 74011 View On Map
For rates, availability, and other information fill out the availability request form and a representative from Indian Springs Apartments will contact you shortly
Indian Springs Apartments is located in the Tulsa Area in the 74011 zip code. View this apartment on a map.
Back to Top