text:
A distributed computing system can be operated in a fault tolerant manner using a set of computing devices. A set of computing devices can tolerate a number of failures by implementing identical replicas of a state machine and selecting proposals. The set of computing devices participating in the distributed computing system by hosting replicas can be modified by adding or removing a computing device from the set, or by specifying particular computing devices for participation. Changing the participating computing devices in the set increases fault tolerance by replacing defective devices with operational devices, or by increasing the amount of redundancy in the system.
1. (canceled)
2. In a fault-tolerant distributed computing system comprising a first set of computing devices, the computing devices in the first set determining operations to be performed by a state machine as a sequence of steps, a method of allowing a second set of computing devices to determine operations in the sequence of steps, the method comprising:
specifying, by some computing device in the first set, the second set of computing devices;
designating one step to be the last step for which the first set determines an operation to be performed by the state machine; and
determining, by the computing devices in the second set, operations to be performed by the state machine in steps subsequent to the designated step.
3. The method of
verifying, by one or more computing devices in the second set, the authenticity of one or more computing devices in the first set.
4. The method of
5. The method of
6. In a fault-tolerant distributed computing system comprising a first set of computing devices, each computing device in the first set executing a replica of a state machine, operations to be performed on the state machine being proposed by a leader of the first set, a method of creating a second set of computing devices, each computing device in the second set executing a replica of the state machine, the method comprising:
requesting, by some computing device in the first set, to create the second set;
agreeing, among a quorum of the devices in the first set, that the computing devices comprising the second set will execute the state machine after a given step;
proposing, by the leader of the first set, a sufficient number of null operations for the state machine in order to arrive at the given step.
7. The method of
8. The method of
9. The method of
verifying, by one or more computing devices in the second set, the authenticity of one or more computing devices in the first set.
10. The method of
11. In a fault-tolerant distributed computing system comprising a plurality of computing devices, a first set of the computing devices determining operations to be performed by a state machine in a first sequence of steps, and a second set of the computing devices determining operations to be performed later by the state machine in a second sequence of steps, where the first sequence and second sequence are mutually exclusive, a computer-readable medium including computer-executable instructions for execution on a first computing device, the computer-executable instructions facilitating performance by:
an execution module for executing operations in steps of the state machine; and
a first agreement module for coordinating with other computing devices in the first set to determine operations in the first sequence.
12. The computer-readable medium of
a second agreement module for coordinating with other computing devices in the second set to determine operations in the second sequence.
13. The computer-readable medium of
14. The computer-readable medium of
15. The computer-readable medium of
16. The computer-readable medium of
17. The computer-readable medium of
18. The computer-readable medium of
19. In a fault-tolerant distributed computing system comprising a plurality of computing devices, a first set of the computing devices determining operations to be performed by a state machine in a first sequence of steps, and a second set of the computing devices determining operations to be performed by the state machine in a second sequence of steps, where the first sequence and second sequence are mutually exclusive and where the first sequence of steps precedes the second sequence of steps, a computing device comprising:
an execution module for executing operations of the state machine; and
a first agreement module for coordinating with other computing devices in the first set to determine operations in the first sequence.
20. The computing device of
a second agreement module for coordinating with other computing devices in the second set to determine operations in the second sequence.
21. The computing device of
22. The computing device of
23. The computing device of
24. The computing device of
25. The computing device of
26. The computing device of
This invention relates generally to distributed computing and, more particularly, relates to fault tolerant distributed computing.
An advantage of distributed systems is the ability to continue to operate in the face of physical difficulties that would cripple a single, monolithic computing device. Such difficulties could include: sustained power outages, inclement weather, flooding, terrorist activity, and the like.
To compensate for the increased risk that individual member computing devices may become disconnected from the network, turned off, suffer a system malfunction, or otherwise become unusable, redundancy can be used to allow the distributed computing system to remain operational. Thus, the information stored or process executed on any one computing device can be redundantly stored on additional computing devices, allowing the information to remain accessible, even if one of the computing devices fails.
A distributed computing system can practice complete redundancy, in which every device within the system performs identical tasks and stores identical information. Such a system can allow users to continue to perform useful operations even if almost half of the devices should fail. Alternatively, such a system can be used to allow multiple copies of the same information to be distributed throughout a geographic region. For example, a multi-national corporation can establish a world-wide distributed computing system.
However, distributed computing systems can be difficult to maintain due to the complexity of properly ensuring that the individual devices comprising the system perform identical operations in the same order. To facilitate this often difficult task, a state machine approach is often used to coordinate activity among the individual devices. A state machine can be described by a set of states, a set of commands, a set of responses, and client commands that link a response/state pair to each command/state pair. A state machine can execute a command by changing its state and producing a response. Thus, a state machine can be completely described by its current state and the action it is about to perform.
The current state of a state machine is, therefore, dependent upon its previous state, the commands performed since then, and the order in which those commands were performed. To maintain synchronization between two or more state machines, a common initial state can be established, and each state machine can, beginning with the initial state, execute the identical commands in the identical order. Therefore, to synchronize one state machine to another, a determination of the commands performed by the other state machine needs to be made. The problem of synchronization, therefore, becomes a problem of determining the order of the commands performed, or, more specifically, determining the particular command performed for a given step.
One mechanism for determining which command is to be performed for a given step is known as the Paxos algorithm. In the Paxos algorithm, any of the individual devices can act as a leader and seek to propose a given client command for execution by every device in the system. Every such proposal can be sent with a proposal number to more easily track the proposals. Such proposal numbers need not bear any relation to the particular step for which the devices are attempting to agree upon a command to perform. Initially, the leader can suggest a proposal number for a proposal the leader intends to submit. Each of the remaining devices can then respond to the leader's suggestion of a proposal number with an indication of the last proposal they voted for, or an indication that they have not voted for any proposals. If, through the various responses, the leader does not learn of any other proposals that were voted for by the devices, the leader can propose that a given client command be executed by the devices, using the proposal number suggested in the earlier message. Each device can, at that stage, determine whether to vote for the action or reject it. A device should only reject an action if it has responded to another leader's suggestion of a higher proposal number. If a sufficient number of devices, known as a quorum, vote for the proposal, the proposed action is said to have been agreed upon, and each device performs the action and can transmit the results. In such a manner, each of the devices can perform actions in the same order, maintaining the same state among all of the devices.
Generally, the Paxos algorithm can be thought of in two phases, with an initial phase that allows a leader to learn of prior proposals that were voted on by the devices, as described above, and a second phase in which the leader can propose client commands for execution. Once the leader has learned of prior proposals, it need not continually repeat the first phase. Instead, the leader can continually repeat the second phase, proposing a series of client commands that can be executed by the distributed computing system in multiple steps. In such a manner, while each client command performed by the distributed computing system for each step can be thought of as one instance of the Paxos algorithm, the leader need not wait for the devices to vote on a proposed client command for a given step before proposing another client command for the next step.
The distributed computing system, as a whole, can be modeled as a state machine. Thus, a distributed computing system implementing complete redundancy can have each of the devices replicate the state of the overall system, so that each device hosts its own �replica� of the same state machine. Such a system requires that each replica maintain the same state. If some replicas believe that one client command was executed, while a second group of replicas believes that a different client command was executed, the overall system no longer operates as a single state machine. To avoid such a situation, a majority of the replicas can be generally required to select a proposed client command for execution by the system. Because any two groups of replicas, each having a majority, shares at least one replica, mechanisms, such as the Paxos algorithm, can be implemented that rely on the at least one common replica to prevent two groups, each containing a majority of replicas, from selecting different proposed client commands.
However, a system that is expected to run for an extended period of time must deal with devices becoming permanently retired or otherwise permanently failed. The system should therefore provide a means of swapping out some devices and replacing them with others (i.e., changing the set of state machine replicas) so that the system's tolerance of future failures is restored to its original level. Otherwise, once a quorum of failures occur, the system is no longer fault tolerant. Replacing failed devices restores the fault tolerance of the system to tolerate new failures. Additionally, the system should allow for the addition of new devices to increase the number of tolerable device failures and thereby provide greater fault tolerance. The original Paxos algorithm gives a simple description of how the system can decide to change the set of state machine replicas participating in the protocol. But the approach presented by the original Paxos algorithm does not address various special circumstances that can arise during a change in the replica set.
A distributed computing system can be operated in a fault tolerant manner using a set of computing devices. A set of computing devices can tolerate a number of failures by implementing identical replicas of a state machine and selecting proposals using a Paxos algorithm. Embodiments of the invention allow the set of computing devices participating in the distributed computing system to be modified by adding or removing a computing device from the set, or by specifying particular computing devices for participation. Changing the participating computing devices in the set increases fault tolerance by replacing defective devices with operational devices, or by increasing the amount of redundancy in the system.
Therefore, in one embodiment of the present invention, a first set of computing devices is responsible for determining a sequence of steps to be performed by a state machine that is replicated among the devices. A second set of devices can be defined, removing the responsibility from the first set, and assuming the responsibility for determining steps in the sequence of steps to be performed by the state machine. The range of steps for which the first set is responsible is mutually exclusive with the range of steps for which the second set is responsible.
In another embodiment of the present invention, a leader of a first set of computing devices responsible for determining a sequence of steps to be performed by a state machine learns that a new set of computing devices has been chosen to succeed its set. The new set assumes responsibility for determining a sequence of steps beginning with a specified step, and the leader of the first set causes null operations to be determined as the steps to be performed in steps prior to the specified step.
In a further embodiment of the present invention, computing devices running replicas of an identical state machine host an execution module and one or more agreement modules. Each device's execution module executes steps of the replicated state machine in sequential order. Each device has an agreement module corresponding to each set of replicas in which it is a member. Agreement modules of different replica sets on the same machine each communicate with the single execution module on that machine to provide it with chosen operations for the sequence of steps.
Although the description herein focuses primarily on the operation of computing devices in a distributed computing system, it will be appreciated that the description is equally applicable to processes running on a single computing device, such as on separate processors or in separate memory spaces. Thus, additional embodiments include the operation of the modified Paxos algorithm in multiple processor environments, whether the multiple processors are physically located in one or more computing devices, and in multiple virtual machine environment, whether the multiple virtual machines are being executed by one or more computing devices. Additional features and advantages of the invention will be made apparent from the following detailed description of illustrative embodiments which proceeds with reference to the accompanying figures.
While the appended claims set forth the features of the present invention with particularity, the invention, together with its objects and advantages, may be best understood from the following detailed description taken in conjunction with the accompanying drawings of which:
A distributed computing system can comprise a number of individual personal computing devices, server computing devices, or other devices that have sufficient processor and storage abilities to participate in the system. The distributed computing system can aggregate the abilities of its constituent computing devices to either provide for greatly increased processing capabilities and storage space, or to implement redundancy, allowing multiple devices to provide access to the same information. Thus, one common usage for a distributed computing system is the aggregation of the processing capabilities and storage space of many different personal computing devices attached to a common network. Such a distributed computing system can maintain information regarding the system, such as which devices are currently part of the system and on which device a given set of information is stored. This information can be necessary for the devices to aggregate their capabilities and storage space and, as a result, each device may contain a copy. Synchronization of the information among the devices of the system can be facilitated through a state machine approach as described below.
Alternatively, an increasingly common usage for distributed computing systems is that of a network server that can act as a central storage repository for various forms of information. Such a distributed system seeks to replicate the central store on all of its constituent devices so that every client seeking to communicate with the central storage can find a convenient and efficient device with which to communicate. Furthermore, because of the distributed nature of the system, local events such as power outages, floods, political unrest, and the like may only affect a few computing devices, allowing the overall system to continue to operate properly and provide access to information and other services to clients.
Such a distributed computing system can be thought of as a state machine, with the future state of the machine defined by the current state and the action to be taken. Each constituent device of the distributed computing system can then independently execute the state machine of the overall system. The state-machine approach can be implemented asynchronously; so, precise synchrony across the constituent devices need not be maintained and synchronization between the devices can be achieved by setting an initial state for all of the devices and subsequently executing the same functions in the same order. A common method for maintaining synchronization is to allow the constituent devices of the distributed computing system to all agree upon the next function before executing that function, and to maintain a list of the functions that were executed. In such a manner, every device can have the same state and if a device fails it need only determine the last function it executed, identify, from the list, any functions that have been agreed upon since that last function, and execute those functions.
A distributed computing system acting as a server can be especially useful for serving a large amount of information to a diverse set of clients, such as a central database for a multi-national corporation, or a popular World Wide Web site. In such situations, a large number of clients can request information from the distributed computing system acting as a server. By implementing the server functionality across multiple devices, the server as a whole is far less prone to failure due to the increased redundancy.
One mechanism by which the constituent computing devices can agree upon the next function to execute is known as the Paxos algorithm. In the Paxos algorithm, as will be described further below, any device can act as a leader and transmit a suggestion for a proposal number to other devices within the distributed computing system. The other devices can respond with either an indication of the proposal having the largest proposal number for which that device has already voted or an indication that the device has not voted for any previous proposals. Once the leader receives the responses from the other devices, it can determine which function to propose and request a vote for a proposed function. Each device will vote for the proposal unless it has, at some time prior to the requested vote, responded to a suggestion for a higher proposal number. If a quorum of devices votes for the proposal, then the proposal is accepted, and the leader can transmit a message to all of the devices requesting that they execute the agreed upon function.
The Paxos algorithm, however, like any other consensus algorithm, requires a large number of computing devices to tolerate failures. Specifically, to tolerate F number of failures, the algorithm requires a distributed computing system comprising at least 2 F+1 number of computing devices. Of these devices, only a simple majority is needed to select commands and continue proper operation of the system. The remaining devices can remain unused until one of the devices selecting commands and operating the system fails.
Should a computing device permanently fail, however, it is desirable to remove it from the set of 2 F+1 devices, and to replace it with a new device, in order that the Paxos algorithm will be able to tolerate F additional failures.
Distributed Computing Environment
Turning to the drawings, wherein like reference numerals refer to like elements, the invention is illustrated as being implemented by a distributed computing system, such as the exemplary distributed computing system 10 shown in
Although not required, the invention will be described in the general context of computer-executable instructions, such as program modules, being executed by a computing device. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform particular tasks or implement particular abstract data types. Moreover, those skilled in the art will appreciate that the invention may be practiced with many different computing devices, including hand-held devices, multi-processor systems, microprocessor based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, and the like. As described above, the invention may also be practiced in distributed computing environments, such as distributed computing system 10, where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.
Turning to
The invention may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. In distributed computing environments, tasks can be performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
Components of computer device 100 may include, but are not limited to, a processing unit 120, a system memory 130, and a system bus 121 that couples various system components including the system memory to the processing unit 120. The system bus 121 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Associate (VESA) local bus, and Peripheral Component Interconnect (PCI) bus also known as Mezzanine bus. Furthermore, the processing unit 120 can contain one or more physical processors.
Computing device 100 typically includes a variety of computer readable media. Computer readable media can be any available media that can be accessed by computing device 100 and includes both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by computing device 100. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term �modulated data signal� means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of the any of the above should also be included within the scope of computer readable media.
The system memory 130 includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM) 131 and random access memory (RAM) 132. A basic input/output system 133 (BIOS), containing the basic routines that help to transfer information between elements within computer 110, such as during start-up, is typically stored in ROM 131. RAM 132 typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit 120. By way of example, and not limitation,
The computing device 100 may also include other removable/non-removable, volatile/nonvolatile computer storage media. By way of example only,
The drives and their associated computer storage media discussed above and illustrated in
The computing device 100 can operate in a networked environment, such as that shown in
When used in a networking environment, the computing device 100 is connected to the general network connection 171 through a network interface or adapter 170, which can be a wired or wireless network interface card, a modem, or similar networking device. In a networked environment, program modules depicted relative to the computing device 100, or portions thereof, may be stored in the remote memory storage device. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used.
In the description that follows, the invention will be described with reference to acts and symbolic representations of operations that are performed by one or more computing devices, unless indicated otherwise. As such, it will be understood that such acts and operations, which are at times referred to as being computer-executed, include the manipulation by the processing unit of the computing device of electrical signals representing data in a structured form. This manipulation transforms the data or maintains it at locations in the memory system of the computing device, which reconfigures or otherwise alters the operation of the computing device in a manner well understood by those skilled in the art. The data structures where data is maintained are physical locations of the memory that have particular properties defined by the format of the data. However, while the invention is being described in the foregoing context, it is not meant to be limiting as those of skill in the art will appreciate that various of the acts and operation described hereinafter may also be implemented in hardware.
In accordance with the invention, a distributed computing system can implement a fault tolerant algorithm while allowing devices to be added and removed from the system. The Paxos algorithm, which will be described in greater detail below, can provide a mechanism of implementing a distributed computing system that can tolerate a certain number of faults provided that more than twice that number of computing devices are used. Mathematically, if the variable �F� represents the number of faults that can be tolerated, a distributed computing system of at least 2 F+1 computing devices can implement the Paxos algorithm by attempting to run identical replicas of a state machine. Of these 2 F+1 computing devices, any collection of at least a majority of them, in other words any collection of at least F+1 devices, can be a quorum and can select proposals in a fault-tolerant manner.
If no computing devices fail, a collection of 2 F+1 computing devices can implement the state machine according to the Paxos algorithm. If one or more of the computing devices fail, one or more of the remaining F computing devices can be used to enable the system to continue to use the Paxos algorithm and operate in a fault-tolerant manner. However, if a computing device fails permanently, it may be desirable to either remove it from the collection of 2 F+1 computing devices, or to add a new computing device to the collection of 2 F+1 computing devices.
According to one embodiment contemplated by the present invention, an initial set of computing devices is used to provide a fault-tolerant distributed computing system. When one or more of the computing devices fails, the device can be removed from the initial set, and a new computing device may be added to the set. Consequently, the operation of the fault tolerant system can modify the set of computing devices upon which the fault-tolerant distributed computing system operates.
State Machines
In a distributed environment, such as distributed system 10 illustrated in
The individual devices that comprise the distributed computing system can each execute the state machine of the system. The devices can, therefore, be coordinated by determining an initial state and then executing the same functions in the same order from then on. A device can be synchronized by simply determining the last function the device executed, locating that function in an ordered list of functions executed by other devices, and then directing the device to perform the functions from the ordered list that the device has not yet performed. Such a state machine approach was initially proposed in the article �Time, Clocks, and the Ordering of Events in a Distributed System,� by Leslie Lamport published in The Communications of the ACM, Volume 21, Number 7, July 1978, the contents of which are hereby incorporated by reference in their entirety.
Paxos Algorithm
By using a state machine approach, the synchronization of the constituent devices 11 through 15 of the distributed computing system 10, shown in
For clarity, several new or alternative terms and concepts are now presented in describing the Paxos algorithm as used in embodiments of the present invention. With reference to
The Paxos algorithm ensures that no two AMs ever choose different operations for the same slot by letting one replica be a �leader� and only allowing the leader to propose operations that can be selected for a slot. However, the leader may fail, so there is preferably a protocol for electing a new leader. Through the election protocol, the new leader learns enough about the actions of previous leaders that it will not make conflicting choices.
The �term of office� of a leader is called a �view�, and each view is uniquely identified by a view identifier (ID), which is equivalent to the �proposal number� used in classic Paxos descriptions. That is, for each particular view, a single leader proposes all operations to be performed by the state machine during that view. Each replica keeps track of its current view and will accept operation proposals only from that view's leader. A view ID consists of a view counter and an initiating replica identifier; thus, if multiple replicas initiate views at the same time, they will start different views. In the example of
A leader remembers the last slot for which it proposed an operation. When a client sends it a new request, it proposes that this request be the operation in the slot following the last slot by sending a PROPOSED message to all replicas. When a replica receives a PROPOSED message from the leader of its current view, it prepares it by recording it in a local log. Then, it sends the leader a PREPARED message. When the leader has received PREPARED messages from a quorum (generally including itself), it chooses the operation, sending a CHOSEN message to all replicas. In other words, it promises no conflicting operation will ever be chosen and thus any EM may execute it.
After the leader proposes an operation, there are network and processing delays before the operation will be executed. During this time, the leader may propose further operations, to overlap the work of multiple stages of the process via pipelining.
In embodiments of the invention, each state machine makes periodic copies of its state on stable storage so the algorithm can prevent replicas' logs from growing indefinitely. Such a copy is called a �checkpoint�, and the slot of a checkpoint is defined to be the slot of the last operation executed before the copy was made. The slot of a replica's last on-disk checkpoint is called its �local stable point.�
Through periodic messages, the leader keeps track of replicas' local stable points and keeps them informed of the collective stable point, which is the qth highest local stable point, where q is the size of a quorum (e.g., F+1). The significance of this value is that because eventually a quorum of replicas will be functional, there will always eventually be a checkpoint available reflecting the first c operations, where c is the collective stable point. Since some replica will be able to supply state reflecting the results of all operations in slots less than or equal to the collective stable point, the operation choices for these slots need not be remembered. Thus, in an embodiment of the invention, replicas discard PREPARE entries for these slots, preventing their logs from growing indefinitely, in a process called �log truncation.� To prevent any replica's log from exceeding a fixed or variable maximum log size, a leader does not propose an operation for a slot until the collective stable point is within a maximum log size of that slot.
The process by which a new leader is elected and learns enough about previous views to ensure it does not propose conflicting operations is called �view change.� As an overview, if any replica learns, by means outside the scope of this overview, that the leader may have failed, it may initiate a view change. It chooses a new view ID greater than its current view ID and transmits VIEW-CHANGE messages to all replicas. A replica receiving a VIEW-CHANGE message enters this new view unless it is already in a higher numbered view. It replies to the initiator with a VC-REPLY message describing the prepare entries in its log. Once the view change initiator has VC-REPLY messages from a quorum, it chooses a leader for the new view and supplies it these messages. The new leader uses these messages to learn enough about previous views to ensure it makes no conflicting proposals. The view change process is described more fully in the earlier cited references of Lamport. In the example of
In embodiments of the invention, through periodic messages, the leader tells each replica how many consecutive operations have been chosen. An AM may thus hear that the next operation needed by its EM has been chosen. However, the AM may not know what operation was chosen for this slot, perhaps because it was offline when the leader sent the PROPOSED and CHOSEN messages for that operation. In this case, the AM asks its peer AMs if any of them has this operation in its cache of recently chosen operations. An AM keeps a cache of recently chosen operations which includes operations from its log, but additionally may include operations that had been locally truncated from its log. If an AM has the operation in its cache, it provides it to the requesting peer AM, obviating the need for the requesting AM to perform a full state transfer. If no peer can supply this operation, and the AM learns that the leader discarded the operation due to log truncation, the replica's EM cannot proceed. At this point, it fetches state from a more up-to-date replica.
Modified Paxos Algorithm Using Change of Replica Sets
As can be seen from the above description, it may not be necessary for each of the computing devices to participate in an instance of the algorithm by hosting a replica of the common state machine in use. In other words, some devices may not initially be members of the state machine replica set in use, such as device D 304 in the example of
A general description of an approach to changing replica sets is given as follows: The state machine contains, as part of its state, the set of replicas responsible for the service. There is a constant pipeline depth �α� for holding a number of pending operations such that a change to the replica set takes effect after α operations. In other words, if the replica set is changed in the nth slot, the new replica set allows for up to α pending operations by choosing operations only in slots n+α and higher. A large α is desirable since α limits the depth of operation pipelining: the leader cannot propose operation n+α until it knows the replica set responsible for that operation, and it may not know this until it executes operation n.
Embodiments of the invention modify the classic Paxos model to have each view be replica-set-specific, i.e., refer only to a fixed replica set. In this way, replica sets do not interfere with each other and views are simpler to manage because they refer to a fixed replica set. To ensure views are replica-set-specific, embodiments of the invention extend the view ID description to include an �epoch� value, described below. Each device runs a separate agreement module (AM), labeled by epoch, for each replica set in which it is currently an active member. If a device is in multiple replica sets, its multiple AMs operate independently and concurrently, each in its own epoch, and therefore in its own view.
An example of the extended view ID is shown with reference to
An example of the use of changing replica sets with replica-set-specific views, as used in an embodiment of the invention, is now described with reference to
By referring to a fixed replica set, views are easy to manage in embodiments of the invention. View changes, for example, involve a clearly defined set of participants: all AMs in the replica set. These are the AMs to which a view change initiator sends VIEW-CHANGE messages, and VC-REPLY messages from a majority of these constitute a quorum of replies sufficient for the next leader to proceed. Furthermore, it is clear when an AM is responsible for view change initiation and when it is eligible to be a leader: it is created with those responsibilities and has them until it is destroyed. Additionally, a leader knows exactly with which AMs it needs to exchange periodic messages.
Turning attention to
To further facilitate replica-set-specific views, embodiments of the invention allow a leader to empty its proposal pipeline if it is aware that a new replica set has been chosen. Such leaders are known as �lame duck� leaders. Emptying the proposal pipeline is useful in scenarios such as the following: A leader of replica set Rm-1 proposes the last operation it is responsible for, rm-1, and starts redirecting clients to the new replica set. It then crashes and the subsequent view change loses all the old leader's recent proposals, so the last slot for which the new leader proposed is less than rm-1. Since all active clients have been redirected to the new replica set, they will not submit requests to the new primary of the old replica set, and no operation will get proposed for operation rm-1. This means the state machine will not make progress, since only operations in slots rm or higher can get chosen, and they cannot be executed until operation rm-1 is executed. To avoid this scenario, embodiments of the invention contemplate the following. When a leader learns that another replica set will take over at operation rm, but the last slot for which it proposed an operation is less than rm-1, it proposes null operations for each slot between the last proposed slot and rm. This requires at most α-1 null operations because the last proposed slot is at least the number of operations the machine has executed, which is at least rm-α. This process of emptying the pipeline ensures that execution of the state machine eventually reaches slot rm.
To further facilitate replica-set-specific views, embodiments of the invention discard old, unneeded AMs. Once the collective stable point of Rm becomes at least rm-1, then Rm is �established.� That is, a quorum of its members have started, and by definition of collective stable point eventually one of them will be able to provide a checkpoint with slot at least rm-1. This obviates the need for any information about operations less than rm, i.e., all operations that previous replica sets (i.e., R0, R1, . . . , Rm-1) are responsible for choosing. Thus, when Rm is established, all previous replica sets are deemed �defunct�, i.e., no longer needed. Any machine learning that it has defunct AMs may destroy its defunct AMs, including their logs. If a machine destroys its only running AM, it may destroy its EM as well; it is no longer an active participant in the service.
To further facilitate replica-set-specific views, in embodiments of the invention a machine keeps only a small amount of non-volatile state for destroyed AMs: the highest epoch among destroyed local AMs (�MaxDefunct�), and the replica set corresponding to the epoch succeeding it (RMaxDefunct+1). It supplies these to any client or remote AM that tries to initiate communication with a destroyed AM. A client hearing about a destroyed AM starts communicating with the succeeding replica set, and an AM hearing of a defunct peer knows to destroy itself.
Embodiments of the invention provide a mechanism for AMs of a replica set to ensure that its successor replica set gets established, and for the AMs to eventually learn of this establishment so they can be destroyed. When an AM of replica set Rm-1 learns that its EM has executed operation rm-1, i.e., the last operation for which this AM's replica set is responsible, it creates a thread that periodically sends an IS-SUCCESSOR-UP-TO-DATE message to each AM in the new replica set. Such a message asks if the recipient's local stable point is at least rm-1 and, if not, offers to supply it a checkpoint at that slot or later. It does not transmit such messages to AMs that have already responded affirmatively. Eventually, it receives affirmative responses from a quorum of the new replica set and thereby learns that Rm is established. At this point, it knows it is defunct and destroys itself.
Because every AM in Rm-1 sends IS-SUCCESSOR-UP-TO-DATE messages, eventually one of them establishes Rm. The responsibility of sending IS-SUCCESSOR-UP-TO-DATE messages could be assigned to only the leader of Rm-1. Alternatively, the responsibility of sending IS-SUCCESSOR-UP-TO-DATE messages is assigned to all AMs, rather than to only the leader of Rm-1. This allows a new replica to obtain its initial checkpoint sooner if the EM of the old replica set's leader is behind. Also, it spreads the load of providing checkpoints among the AMs of the old replica set rather than placing the entire burden on the leader. An additional reason for assigning the responsibility to all AMs pertains to an optimization using certificate chains, described below.
An EM uses the offer to supply a checkpoint implicit in an IS-SUCCESSOR-UP-TO-DATE message in the following scenario. Suppose an EM needs to acquire the next operation from a collocated AM, but no collocated AM is responsible for this operation. For instance, in
If only IS-SUCCESSOR-UP-TO-DATE messages could create new AMs, the new AMs could not begin until an EM executed operation rm-1. As an alternative optimization to let AMs start sooner, embodiments of the invention allow an EM to send a JOIN message to each new replica when it executes operation rm-α, which creates the set. A machine receiving a JOIN message will start an AM for the new replica set. JOIN messages are not acknowledged or retransmitted, since they are only an optimization.
Embodiments of the invention create and use �successor information� about later-established replica sets. When the EM executes an operation that creates replica set Rm, it creates successor information to be used by the local AM for replica set Rm-1. This information includes the first operation number its successor is responsible for, rm, and the identities of the succeeding replicas. The AM needs rm so if it is ever leader it will not propose an operation numbered greater than or equal to rm. It needs the identities of succeeding replicas so it can later send IS-SUCCESSOR-UP-TO-DATE messages.
It is possible that rather than executing the operation rm-α that creates replica set Rm, the EM receives a state transfer that enables it to skip that operation. Thus, after completing a state fetch, the EM checks to see if it now knows successor information for any of its collocated AMs, and, if so, distributes it to them. To achieve this, the state includes rm and Rm even for old replica sets. A state machine can retain this information indefinitely. Alternatively, the state machine may eventually discard such information in the following way. Recall that a leader will not propose an operation for a slot until its replica set's collective stable point is within a maximum log size of that slot. Thus, when the state machine executes an operation in a slot that is a maximum log size away from rm-1, it can infer that the collective stable point of replica set Rm is at least rm-1 and thus replica sets with epochs less than or equal to m-1 are defunct. So, it may discard information for replica sets with epochs numbered m-1 or less, remembering only the number of the latest replica set it knows to be defunct. Thus, it may not be able to supply certain AMs with successor information, but for those AMs the EM can tell them to destroy themselves.
Embodiments of the invention use �certificate chains� to indicate that devices in a replica set are legitimate. A machine receiving a message saying it is part of a new replica set may need to verify that the sender is not lying. Normally trust is not an issue in fail-stop consensus. When replica sets change, however, machines exchanging such a message are not yet in the same fail-stop consensus group. A machine in the initial replica set knows it belongs there via a service certificate, i.e., a certificate signed by a trusted certification authority (CA) specifying the initial replica set. Because the CA may be offline during a replica set change, embodiments of the invention use replicas to attest to new replica sets.
A certificate chain is a sequence of certificates. Each certificate attests to a certain replica set. The first certificate in the chain is a service certificate, attesting to the initial replica set R0. Each other certificate is signed by a member of a preceding replica set. With reference to
If a machine receives a certificate chain showing it is in a particular replica set, it can be sure this is so. The new AM saves this as its origin chain. For AMs in R0, the service certificate itself is an origin chain. When an AM needs to tell a member of its successor replica set to create an AM, it signs a certificate describing the new replica set, appends it to its origin chain, and sends this as the new AM's origin chain.
Embodiments of the invention use two optional optimizations to help keep origin chains short. First, if one replica set is attested to by some certificate, and the replica set contains the signer of some subsequent certificate in the chain, then all intermediate certificates in the chain between the two can be removed. Second, an AM can replace its origin chain if it receives a shorter one from another predecessor. For instance, again with reference to
Embodiments of the invention provide an interface by which a state machine can change the replica set. A state machine, while executing an operation, may invoke a change_replicas( ) function, specifying an array of machines to be included in the new replica set.
Embodiments of the invention use policies to determine when replica sets should change. In some embodiments, administrators explicitly manage replica sets by directing a client to submit a replica set change request. The state machine ignores requests lacking evidence of administrative authority. An administrator might use an explicit request, for example, to permanently retire a machine or to deploy new machines for additional fault tolerance.
In some embodiments, replica set changes are performed automatically, in order to cope with persistent machine failures without requiring an administrator. Each EM sends a special client request to the service at a regular interval, e.g., every five minutes, reporting itself alive. The state machine keeps track of extra state consisting of the last time each EM reported itself alive. This state is completely determined by operations, and not by local clocks on each replica, so each operation includes a timestamp assigned by the leader, and each operation is said to execute at a certain �time.� Additional details on executing operations at a given time are discussed in Adya, et al., �FARSITE: Federated, available, and reliable storage for an incompletely trusted environment,� in Proc. 5 th OSDI, pp. 1-14, Boston, Mass., December 2002, which is hereby incorporated by reference in its entirety for all that it teaches without exclusion to any part thereof. At a given interval, e.g., every hour, the state machine checks whether the oldest last-alive time among replicas is more than an hour old. If so, it assumes the corresponding device is persistently failed, and calls the change_replicas( ) function to replace it in the replica set.
The state machine also deterministically chooses a new device to be the replacement. Devices willing to be replicas periodically send special client requests to the service. Such a request contains a certificate from an accepted authority authorizing the sender to participate in the fail-stop service. The state includes a mapping from potential replicas to timestamps when those machines last made such a request. To choose a new machine, the state machine makes a pseudorandom but deterministic choice among those machines whose timestamps are sufficiently recent.
In view of the many possible embodiments to which the principles of this invention may be applied, it should be recognized that the embodiments described herein with respect to the drawing figures are meant to be illustrative only and should not be taken as limiting the scope of invention. For example, those of skill in the art will recognize that some elements of the illustrated embodiments shown in software may be implemented in hardware and vice versa or that the illustrated embodiments can be modified in arrangement and detail without departing from the spirit of the invention. Therefore, the invention as described herein contemplates all such embodiments as may come within the scope of the following claims and equivalents thereof.
Referenced by
Legal Events
Original Image


text:
Wake Forest Football is now nine practices through the spring season after completing another practice on a beautiful Tuesday morning at the Doc Martin Football Complex.
Ron Wellman discusses Wake Forest's future football opponents in his monthly column in Gold Rush.
Defensive Coordinator Mike Elko gives us an update of the defense's progress during the spring season.
Wake Forest head football coach Dave Clawson put the Demon Deacon football team through a 90-minute scrimmage on Saturday morning at BB&T Field.
Final Stats (PDF) | Photo Gallery
|There are no upcoming events.

subset: commoncrawl


text:
Dashahas itter of puppies is here. - 3 girl and 4 boys .AKC registered, all shots and deworming will be done in due time. Tails and dew claw job is done. Each pup is checked by the vet. Our pups are very social Please email to [email removed] for more info and videos. Check ...
Mar 29 -
New York City :
Staten Island - Dogs & puppies
Olivia has itter of puppies is here. - 1 girl and 4 boys .AKC registered, all shots and deworming will be done in due time. Tails and dew claw job is done. Each pup is checked by the vet. Our pups are very social and good with little kids an other home pets. Please email to ...
AKC Cocker Spaniel Puppies, Born March 20, 2015 Parents on the premises. Champion Bloodlines. Buff, Black and Black and Tan. Will be vet checked, first shots and deworming. Ready to go May 20,2015. Asking $700.00. Call 401-231-6743
Mar 28 -
Rhode Island :
Johnston - Dogs & puppies
2female chocolate cockers born feb1 ready for new homes April 1 have 1st shots and wormed these girls are gorgeous
Mar 23 -
Albany :
Fly Creek - Dogs & puppies
Mar 22 -
Baltimore :
Baltimore - Dogs & puppies
Please call me for information on this puppy... She is a white and tan can do some tricks...
Mar 21 -
Montpelier :
Lebanon - Dogs & puppies
Buff, chocolate, black, red/white puppies. Some older pups available. AKC, Crate trained, shipping available$400.00. Raised in the home, very sweet!
This ad was posted with the eBay Classifieds mobile app.
Mar 21 -
Springfield (MA) :
Three Rivers - Dogs & puppies
Dasha gave birth to 7 puppies. 4 girls and 3 boys.
Please visit our Facebook page and the website nyccockersplace.webs.com for details and respond if interested to reserve a puppy.
Mar 12 -
New York City :
Staten Island - Dogs & puppies
My cocker spaniel, Baby is a 5 yr. old American cocker spaniel. We bread her with my sons 4 yr. old Toy Poodle, H.D. (Humpty Dumpty). Baby had 5 puppies on Feb. 17th., 2015. The Black and White one is a boy. Baby is a really sweet natured dog. Friendly with everyone and ...
Mar 10 -
Richmond :
Fredericksburg - Dogs & puppies
Adorable silver cocker spaniel puppies. Born 01/30/2015. Championed Sired and vet checked. Accepting Deposits. Ready to go for Easter!!
Mar 10 -
Rhode Island :
Warren - Dogs & puppies
3 Year old Cocker Spaniel Male available for adoption. Looking for a new loving home for him. He is intact, AKC registered and microchipped. He has also proven father to a litter of puppies. He is very obedient, loving and knows sit, stay, back, leave it, drop it, down, off, ...
Mar 7 -
Washington DC :
Quantico - Dogs & puppies
These adorable puppies will be available Sunday afternoon. Mom is a Bichon Frise and dad is a Cocker Spaniel. Brown pup is a male, and two others are females. Will have vet records indicating vet wellness exam and first round of shots. Born January 10, 2015. Located ...
Mar 6 -
Hagerstown-Martinsburg :
Cross Junction - Dogs & puppies
I have one Cocker Spaniel puppy available for sale. He is 13 weeks old born Nov. 30th. He is up to date on shots and has a health certificate and ACA paperwork.
He was raised in a home with other pets and small children. Very playful and cute.
Call or text 717-977-9426
Mar 3 -
Dutch Country PA :
Strasburg - Dogs & puppies
I am selling my just under 9 week old cocker spaniel puppy.. She can already sit and lie down.. And is paper training and outdoor training. I am allergic to her fur.will only sell to a family home with references.
Feb 27 -
Brattleboro :
Claremont - Dogs & puppies
A male cocker spaniel puppy that is just adorable, very loveable and playful. Raised in the house with children. He is adorable black and white.
Feb 24 -
Charlottesville :
Lignum - Dogs & puppies
Does Fido need a home to stay while your on vacation and away? ! If your dog can not come with you then board him with us! Your dog is guaranteed to be in loving hands! Located in Lodi, NJ we are a cage-free boarding facility. When boarded with us your dog(s) have absolute ...
Feb 16 -
Northern NJ :
Lodi - Animal services
Each sweet baby has Up-To-Date vaccines, vet health check and 3 rounds of deworming. They are home birth and raised with us and our 4 children. We start paper/wee pad training with the "Misty System" very early on to help ensure potty training adaptation within the first ...
Feb 13 -
New York City :
New York - Dogs & puppies
The puppies are 2 months old. The puppies have the following coats, Buff color and Parti color. We only have 2 males and 2 females left.
The puppies come with a blanket, a chew toy, and a ball. They also come with purbreed regristration papers. We also deliver in local areas ...
Feb 12 -
New York City :
New York - Dogs & puppies
I am a established breeder. I have been breeding for 27 years.
I raise quality family raised puppies.
You can go to my web site for additional information.
Feb 12 -
Hartford :
Meriden - Dogs & puppies
**AKC Puppies**
Has Champion blood line on mothers side.
Tails are docked and dew claws are removed, will be vet checked and be dewormed prior to leaving. Ready just in time for Easter..
Please call or email for more info or photos.
Feb 9 -
Harrisburg :
Snow Shoe - Dogs & puppies
Beautiful puppies born on oct 9th, there three months old looking for good homes, one male left, already have there first and second set off shots, parents are on the premises, looking for good homes, I can be reached at 443-528-2138 thank you
Feb 6 -
Baltimore :
Brooklyn - Dogs & puppies
We'll be sending text message alerts for your subscription to


text:
Roles to serve as consultant for United Way
Linda Roles, former executive director of the United Way, has moved into a consulting role with the organization.
In this role, she will support the current executive director, Jessica Boyer, and her staff in this phase of the executive director succession plan.
Roles served as executive director of United Way of Douglas and Pope Counties from July 1998 through June 2010.
Are you a newspaper subscriber but you don't have a Digital Access account yet? https://secure.forumcomm.com/?publisher_ID=40&event=subscriber/lookup.
You will need your subscription account number and phone number. Not sure if you have an account? Email us at email@example.com and we can help you.


text:
Miami's Top 5: What you should know Thursday
1. Miami Beach voters became the first in Florida to call for the decriminalization of marijuana for medical use in a Tuesday vote that gives a glimpse of statewide support for the issue.
2. A friendship is gone, and Jonathan Martin is undergoing treatment for what has made Richie Incognito a national villain, but Dolphins players side more with Incognito.
3. On the same night the city of Miami Gardens was responding to a rash of recent shootings in the area, an 11-year-old was transported to the hospital after being shot.
4. For the first time since the 1970s, the number of Americans who get their TV from cable or satellite is falling.
5. Five hours have been set aside for Thursday's "stand your ground" hearing in the Florida House, where changes to the law, including its repeal, will be considered.
Miami-Dade County
Follow Us
Oops, you haven't selected any newsletters. Please check the box next to one or more of our email newsletters and submit again.
Oops, you didn't provide a valid email address. Please double-check the email field and submit again.
Join the Discussion
Miami Herald is pleased to provide this opportunity to share information, experiences and observations about what's in the news. Some of the comments may be reprinted elsewhere on the site or in the newspaper. We encourage lively, open debate on the issues of the day, and ask that you refrain from profanity, hate speech, personal comments and remarks that are off point. Thank you for taking the time to offer your thoughts.


text:
that Scott Klein will be indicted and sent to jail within the next 24 mmonths.
This prediction was brought to you by the same fool who says he shorts $400,000 blocks of stocks.
I think your avatar fits you perfectly. I think you need to alert everyone what an idiot you are. My guess is that your other avatar is spencer_big. Do I have it right? You betcha!
Hope so! What a bunch of scum!


text:
The credits for this production have not yet been completed or verified.
The King and I
Pre-opening information; subject to change.
Vivian Beaumont Theatre, (4/16/2015 - )
Category: Musical, Comedy, Drama, Revival, Broadway
Setting: On the docks and in and around the Royal Palace in Bangkok. The 1860s.
music by Richard Rodgers; lyrics by Oscar Hammerstein II


text:
I have a problem with RabbitMQ 2.8.2 server. After one or two days usage I receive Disk Space warning from RabbitMQ and the only solution I found is to clear directory /var/lib/rabbitmq/mnesia/rabbit@linux-3blg/msg_store_transient and restart RabbitMQ. I use rather huge messages in my program 1-50MB may be problem is there, but I really need stability. Does anybody knows the solution?
Take the 2-minute tour ×
Stack Overflow is a question and answer site for professional and enthusiast programmers. It's 100% free, no registration required.
This means that messages are not being acknowledged and consumed. It might help to make use of the Time-To-Live Extensions. See http://www.rabbitmq.com/ttl.html#per-queue-message-ttl and set an expiration for messages.


text:
Please enter a valid mileage
Want to save money? Check out the 2009 Chrysler Sebring
By KBB.com Editors
Accounting for more than one in every four passenger cars sold, the mid-size sedan market remains the most popular in the country. Currently, the segment consists of the class leaders, the Honda Accord and Toyota Camry, and a host of "category alternatives," some of which are more compelling than others. Lately, though, it seems almost every new or redesigned entry shows up to the party with a combination of attributes attractive enough to merit consideration. Chrysler's entry, the Sebring Sedan, has not lived up to expectations, partly due to stiff domestic competition from both the Ford Fusion and Chevrolet Malibu, but also because of perceived weaknesses in the areas of interior quality and engine noise, both of which Chrysler addressed in 2009.
If you like the balanced nature of a mid-size sedan but want or need some of the standard in-cabin goodies traditionally optional on other makes, the Sebring might be just the car for which you've been waiting. Unique styling also sets the Sebring apart from the crowd.
The Sebring sedan's overall sense of refinement doesn't match the Honda Accord's or Toyota Camry's, or even that of some domestic rivals like the Chevrolet Malibu. Also, electronic stability control is optional, a feature most competitors now list as standard equipment.
For 2010, the grooves are removed from the Sebring's hood, while new wheels and instrument panel gauge cluster mark the other changes for 2010.
Driving Impressions Although the 2010 Sebring is a vast improvement over Chrysler's previous mid-size entries of late, the car still falls short of the standards set by Honda, Toyota and even Hyundai....And on the list of things we do like about the Sebring, its ride and handling characteristics aren't at the top. Compared to a direct competitor like the Chevrolet Malibu, for instance, the Sebring is neither as comfortable on the highway nor as eager on a winding road. Furthermore, the cabin isn't as quiet and the powertrains aren't as smooth. On all those fronts the Sebring is far beyond acceptable, however, especially for anyone attracted more by its styling and amenities.
Uconnect tunes
This optional hard drive-based system includes navigation with real-time traffic information, MP3/WMA music and JPEG image uploading, a high-speed USB 2.0 port, auxiliary audio input, voice recognition, voice message recording and playback, Bluetooth hands-free cell phone connectivity and more.
Tilt and Telescoping steering wheel
This adjustable steering wheel makes finding a comfortable driving position a breeze, no matter your size.
With a passenger cabin defined by clean, contemporary styling and a generous helping of metallic-looking trim (plus an abundance of average-grade plastics), the Sebring might not be as warm or welcoming as some of its competitors, but it's comfortable and attractive nonetheless. Complementing a host of available in-cabin technologies is a fold-flat passenger's seat that can make laptop work easier (or help accommodate longer cargo). Leather seating is complemented by a touch of tortoise shell-like trim and brushed silver accents.Exterior
Before its introduction, many figured the new Sebring would show up looking something like a scaled-down version of the larger Chrysler 300 Sedan that has been such a big hit for the automaker. According to company officials, however, that car's bold look just didn't downsize well. Instead, the Sebring adopts the general styling direction that now defines more than half the vehicles in the Chrysler lineup, a look that includes the grille and headlamps that made their debut on the Chrysler Crossfire two-seater. Limited trim-level models include body-colored door handles, available fog lamps and dual chrome exhaust outlets.
The Sebring Touring includes a six-CD/DVD/MP3/SIRIUS Satellite Radio sound system with auxiliary input jack, air conditioning, power windows/locks/mirrors, remote keyless entry, cruise control, four-way adjustable steering wheel, delayed-off headlamps and cloth seats. V6 trims add 18-inch wheels and more optional equipment.
Options for the Sebring include a power sunroof, heated front seats, remote start, traction control and electronic stability control. V6 models offer 18-inch wheels, navigation with Boston Acoustics audio and the Electronic Convenience Group that adds automatic temperature control, auto-dimming rearview mirror, remote start, fog lamps and a tire pressure monitor.
The Sebring's base four-cylinder engine delivers highway mileage of up to 30 miles per gallon, while the range-topping 3.5-liter V6 offers 235 horsepower. The 3.5-liter benefits from a six-speed automatic transmission with Auto Stick manual-shift capability, while the 2.4-liter engine is matched to a traditional four-speed automatic.
2.4-liter in-line 4
173 horsepower @ 6000 rpm
166 lb.-ft. of torque @ 4400 rpm
EPA city/highway fuel economy: 21/30 (Sedan), 20/29 (Convertible)
3.5-liter V6
235 horsepower @ 6400 rpm
232 lb.-ft. of torque @ 4000 rpm
EPA city/highway fuel economy: 16/27 (FWD)
To compare current market prices, check out KBB.com's car classifieds to view the new and used Chrysler Sebring vehicles for sale near you.
Based on 304 Ratings for the 2007 - 2010 models.Review this car
By jimmy "the Greek" on Monday, March 23, 2015
I own this car - My approximate mileage is 67,000overall rating 7 of 10rating details
Pros: "solid car"
Cons: "suspension clunks @ struts"
Likely to recommend this car? (1-10): 5
"Solid packake but could have been better. The gear ratios are way off, you don't need 1st or 2nd at all. Torque converter locks up 15mph too soon. Its sad the design engineers don't understand what drivers want in a sporty coupe. No exhaust tone at all. Convertable top uses up 75% of trunk. Rear seat doesn't have enough leg room for my 2 short grandsons. Engine electronics are very good. Dash is normally unreadable during daylight hours. Automatic shift in manual mode is awkward. Left to right vs fwd to rear manual control makes no sense."
Was this review helpful to you? yes | no | report abuse
By Panga on Sunday, March 22, 2015
I own this car - My approximate mileage is 30,800overall rating 7 of 10rating details
Pros: "Top operation, air conditioning, and power windows"
Cons: "Plastics take a beating in FL sun, garage helps."
Likely to recommend this car? (1-10): 8
"I bought the newest, cheapest convertible I could get when I moved to Florida. I share the same beautiful sunshine and fresh air enjoyed by drivers of Bentley convertibles. Convertibles are a passion for me, having owned my first in 1967. The Sebring is everything I could have expected in an inexpensive convertible. The new state-of-the-art convertible operating system is a huge factor in the enjoyment of this car (the top can actually be operated using the key chain). This is the same system used on all upscale convertibles and, for some reason this has escaped the notice of engineers at Ford & Chevy. Were it not for this system, I would actually prefer a Mustang convertible."
Was this review helpful to you? yes | no | report abuse
By Kemal on Thursday, December 04, 2014
I owned and sold this caroverall rating 1 of 10rating details
Likely to recommend this car? (1-10): 1
"Car has been trouble ever since I bought it. It had only 16,000 miles on it when purchased from a private individual. Has 44,000 miles on. It now. Had electrical problems, brakes replaced at 27,000, #3 cylinder leaks at 38,000, struts are knocking all the time, Uconnect works sometimes."
12 people out of 29 found this review helpful
Was this review helpful to you? yes | no | report abuse
By CARL on Tuesday, November 25, 2014
I own this car - My approximate mileage is 81,200overall rating 10 of 10rating details
Pros: "Fuel MILAGE AND RELIABILITY"
Cons: "SMALL ENGINE"
Likely to recommend this car? (1-10): 10
6 people out of 11 found this review helpful
Was this review helpful to you? yes | no | report abuse
By bond on Sunday, November 16, 2014
I own this car - My approximate mileage is 122,000overall rating 8 of 10rating details
Pros: "fun fun fun top down unbeleiveable"
Cons: "I dont have any none"
Likely to recommend this car? (1-10): 10
"this is a great car yes its priced for us folks who cant afford a the more expensive vehicles, she looks fantastic and is always clean. has great suspension ride is smooth, I get rave reviews from my neighbors as to how beautiful she looks I have her tinted even been looking to add a new black out grille everyone stares at her she looks that great. A joy to drive I see so many on the road, I have upgraded by adding drilled/slotted rotors/ceramic pads for better stopping, done all of the proper maintenance even have a two year contract for oil changes with the dealers Chrysler/Dodge dealership I have had to change only one part she doesn't leak, the vinyl top is the original and as long I continue to condition the top it looks brand new"
6 people out of 9 found this review helpful
Was this review helpful to you? yes | no | report abuse
By Sorry owner on Tuesday, September 02, 2014
I own this car - My approximate mileage is 32,500overall rating 1 of 10rating details
Pros: "None"
Cons: "Brake system, transmission, tire pressure sensor"
Likely to recommend this car? (1-10): 1
"One mechanical problem after another. It has been off-road more than on. Non-drivable for most of its life. Wish I never bought it."
6 people out of 13 found this review helpful
Was this review helpful to you? yes | no | report abuse


text:
Do guys like straddling?
In my mind, I think straddling a guy while you are kissing or just alone together is one of the single sexiest turn-ons, ever. I can imagine what I... Show More
Most Helpful Opinion
Was this helpful? Yes
Have an opinion?
What Guys Said 3
Most men simply love being straddled by women - their are sites dedicated to this enjoymen. My personal love is feeling a womens nice soft ass on my chest , looking up at her beautiful body and pretty face above me...yes definely a turn on for most men. As long as both parites enjoy...have fun ..:)
What Girls Said 5
Click "Show More" for your mentions
What They Said On Facebook


text:
Are you interested in public art opportunities for the St. Elizabeths East Campus? We want your ideas and input! Please attend the 2nd Arts Working Group meeting on Tuesday, April 30, 2013 from 6:30pm-7:30pm being held in the DC Commission on the Arts & Humanities' Multipurpose room @ 200 I (Eye) Street, SE.
Each center provides career counseling, resume assistance, job placement and much more.
Tweets by @ONEDOES_ONECITY
Download forms, file online or check your refund status at the Taxpayer Service Center.


text:
Hot Topics
About Us / Contact Us
Activities & Curriculum
Activities for Outcome-Based Learning
Arts & Crafts
Music for Learning
Recommended Reading
Topics In Early Childhood Education
Art and Creativity in
Early Childhood Education
Teaching Peace with Elyse
Ideas and Activities for Indoor and Outdoor Play
The Reading Corner
Teaching Children with Special Needs
How to Get a School Grant
Earlychildhood NEWS Blog
Job Sharing Board
State Licensing Requirements
ProSolutions CEUs
today to our
monthly newsletter!
Home Learning Environments: A Blueprint for More Exciting Places and Spaces
You serve as an influential role model for your child as she learns about her world. It is fun for her to imitate you and copy your daily activities while she gains new skills and practices some old ones.
Read More
All About Dinosaurs!
Keep your curriculum fresh and exciting with thematic activities for infants, toddlers, preschoolers, and school-age children! Every month features a different new theme and new activities are added each week.
Learn More
Lucky Leprechaun Treasure Box
Little leprechauns can create their own magic box to guard lucky treasures on St. Patrick's Day!
Find Out How
© 2008 Excelligence Learning Corporation, All Rights Reserved.


text:
DENVER -- Colorado Avalanche goalie Semyon Varlamov has recovered from a hip injury that caused him to miss three games, so the team on Monday reassigned rookie goalie Sami Aittokallio to the Lake Erie Monsters of the American Hockey League.
Varlamov will dress as Jean-Sebastien Giguere's backup Monday against the Columbus Blue Jackets. Center Chuck Kobasew will replace Mark Olver on the fourth line. Kobasew missed eight games with a back injury.
For the Blue Jackets, Brandon Dubinsky will move from left wing to center on the second line, replacing injured center Artem Anisimov. Anisimov is listed as day-to-day after sustaining a head injury on a hit from the Minnesota Wild's Charlie Coyle in the first period Saturday in the Blue Jackets' 3-2 shootout win. Blake Comeau, who was a healthy scratch the past two games, will play left wing on the fourth line.
Here are the projected lineups for Monday's game:


text:
Bare minerals makeup
Hi, I have been reading the forum about bare minerals makeup. I own a silk flower/makeup boutique and decided to try bare minerals before putting it in my shop. I started with Bare Escentuals and absolutely loved it. I hate liquid foundation, especially in the summer, and liked the idea of a powder instead. I have been wearing it for about 1-1/2 years now and will NEVER go back to liquid makeup. My husband is a doctor and has an aesthetic clinic and recommends minerals makeup to his patients after chemical peels, etc. You MUST try this if you haven't already, as I tell my clients, "You will wonder where you have been all your life". Good luck!
I thought BE was good too when I want the powder look. Have you tried adding camellia oil to it? It's a really easy way to give you a dewy glow. You can take your minerals and sprinkle them in one hand, add a couple of drops of camelliabotanic.com and apply. It makes your skin feel like silk. I like both BE minerals and the nourish brand. I agree, will never go back to the old foundation, PLUS it ages your skin.
Expensive! I want to look good & not go Broke! I switched to, "Elegant Minerals."
I was ecstatic to find mineral makeup. I can't say enough.
But, "Bare Escentuals" is really expensive.
They always have the commercials running, so you know they add that into the cost of the product. Ouch!!
I wanted to look good and not go broke doing it!
I found, "Elegant Minerals" using the yahoo search engine.
They put sifters on all of their jars and that makes it easier to control the powder.
I've gotten a mineral makeup through the mail and the makeup ends all over the place when you open the jar.
Elegant Minerals put sifter seals their foundation jars.
This keeps the powder from getting into the lid during shipping.
They've always answered my questions really fast. And all the jars are factory sealed too.
I wanted the same high quality product at an affordable price.
Elegant Minerals works better for me and my pocketbook!
Here is a link that might be useful: Elegant Minerals Makeup


text:
Added on: 2008-08-13 Listened: 19K timesin: expressions in French, learning French
Pronunciation in French [fr]
- Pronunciation by MystOgriff (Female from France)
- Pronunciation by spl0uf (Male from France)
- Pronunciation by hendialys (Female from France)
- Pronunciation by Clador06 (Male from France)
Could you do it better? different accent? Pronounce ménage à trois in French
Accents & languages on maps
- Random word: bien achalandé
by Yandex Translate English translation:
ménage à trois = threesome
Not satisfied?
Modify word:
· Add languages
· Add categories
Report word:
Is there anything wrong with this word?
Please report it here.


text:
Crazy Racing Crash Comes ThisClose to Hitting Spectators
There is definitely such a thing as getting too close to the action.
There is definitely such a thing as getting too close to the action.
Fulfilling the childhood dreams of kids everywhere, 5-year-old leukemia patient JP Gibson was signed to a one-day contract by his favorite sports team, the Utah Jazz.
The hurdles. They can be daunting.
They're kind od tall and have the ability to really trip you up and injure you.
This woman, who we'll just assume was attempting them for the first time ever (because who could be that bad at them?!), found that out the hard way.
Fights at NFL games are as much a part of the experience as touchdowns and overpriced beer, but this one is particularly frightening.
There is no more feel-good story in sports right now than the Kansas City Royals. And you're about to like them even more.
This week's episode features appearances by TNA star DJ-Z, Ring of Honor TV star Bobby Fish and central NY wrestler Jay Freddie. Check out this week's episode
After watching this video I feel like we are about 2 years away from playing video games with real humans. I'm starting to think that movie "Gamer" with Gerard Butler was less science fiction and more of a documentary
Motivational speeches should be reserved for people who are loud, outgoing and a little crazy.
Chris Farley taught us this in his "Livin' In a Van Down by the River" skit from NBC's "Saturday Night Live" many years ago.
Matthew McConaughey is pretty much the polar opposite of Farley.
I was instructed that I needed to watch this video and then buy a barbie jeep and give this a try.
Check out these two old school Greenbay Packers fans walking around with the most explicit jerseys ever.
Meet Tim Knoll.
This video proves that you're never too young to start doing trick plays to win a game.


text:
Cell Phones and Technology from The Hills 2008-04-15 11:13:46
The Hills Tech Quiz: Episodes 24 — "Girls Night Out"
by Tech
Like us on Facebook
Although this week's episode was called "Girls Night Out," it seemed as though it was all work and no play with Lauren and Whitney doing double duty for People's Revolution and with Heidi being back in Bolthouse. Thankfully the girls got a break from the computer lab and spent more time on mending friendships then causing unnecessary drama. . . . excluding Spencer drama, naturally. Now for the fun part — the Hills tech quiz! Get started below!
Photos courtesy of MTV
The Hills Tech Quiz: Episodes 24 — "Girls Night Out"
What was Lauren's responsibility during the People's Revolution photo shoot?
Kelly Cutrone was carrying around a ____________ laptop during the shoot.
What item does Spencer tell Stephanie is going to take up even more room in her apartment?
What color of lighting is Kelly Cutrone debating about for the fashion show?
How many times was Lauren shown on camera with her cell phone?
Who said, "Hi, OMG, oh my god"?
Read More TV Tech QuizTech QuizWhitney PortSpencer PrattHeidi MontagExamCelebrity GadgetsLauren ConradThe Hills
Heidi Montag
Whitney Port
Spencer Pratt
Lauren Conrad
kaykayelle 6 years
I was all primed to answer that Heidi had a Dell monitor on her desk and no such question was asked. :(
sabrina27 6 years
5/6 best ever at the geek quizzes. Lauren is going to do great working for that badass boss Kelly Cutrone.
LOVErickii 6 years
=) me too caligirl101
LOVErickii 6 years
=) me too caligirl101
caligirl101 6 years
100% sweet.
caligirl101 6 years
100% sweet.
KadBunny 6 years
As much as I love Lauren and Whitney I kinda hate how they're probably getting these awesome jobs because of their connections to MTV; I wonder if they'd get hired even without the cameras
darlene 6 years
4/6 thats good
darlene 6 years
4/6 thats good
ilanac13 6 years
i feel like i obviously wasn't paying enough attention to this episode since i got 2 wrong. oh well...at least i know the right color lighting.
hollitosway 6 years
I think it's kinda lame how Whitney always does the big moves like going to Paris and getting a better job and Lauren just jumps on her bandwagon. I feel like Whitney must somehow be frustrated by that!
SkinnyMarie 6 years
I just watch it while on my lunch break. fun times. I am really shocked at Laurens work ethic. It seems like she actually does something instead of going "yeaaaahhh" and stare like heidi
SkinnyMarie 6 years
I just watch it while on my lunch break. fun times. I am really shocked at Laurens work ethic. It seems like she actually does something instead of going "yeaaaahhh" and stare like heidi
Tres-Chic-NY 6 years
OMG Lauren wasn't rockin' a cell phone this ep. And another thing I've noticed, they finally had to bleep some one. Heidi and Spencer for cursing. I don't think I've ever heard someone curse on the hills
karisaamy 6 years
6/6 and I usually do the worst at the tech quiz
Tech 6 years
Tech 6 years
bellaressa 6 years
bellaressa 6 years
TidalWave 6 years
hm. There's a discussion about people being "too busy" to hang out with friends, yet everyone seems to have time to watch tv :) I don't know how you guys do it!
Link Love
- Find out what Tim Cook's plan for his fortune is — Newser
- How to enable WiFi calling on your iPhone — TheUnlockr
- This viral video is being developed by a film studio — Co.Create
- A Girl Scout invention wants to help the paralyzed — Not Impossible
- The stats of women in the tech world are actually declining — HuffPost Tech
- 12 Tumblr posts only Tolkien fans will appreciate — BuzzFeed Geeky
- How to get noticed — maybe even hired — over email — POPSUGAR Smart Living


text: Web exclusive! Unleash your inner fashion goddess with this hot high-heeled sandal. Price: Price pending Just in time for summer, Dr. Scholl's brings you the "Band" Casual Sandal. With a sassy four-inch heel and strappy suedette upper, this edgy sandal is bursting with attitude. Strap in with the adjustable buckle closure and prepare to turn heads when you hit the mall or the boardwalk. You'll never find another pair of heeled women's sandals so stylish and comfortable all at once. Featured in Dark Denim 4" heel 1" platform Fabric-covered TPR outsole Imported More


text:
Print or e-mail your favorites and plan your perfect visit to NYC today.
Learn more about the five boroughs that make up New York City with our complete guide.
Discover what to do, where to stay and what to see in New York City with our planning tool.
NYC is the place for inquisitive minds looking for new things to see, hear, taste and learn.
Enjoy Shakespeare in the Park, Fourth of July fireworks, the US Open and more.
Download your free copy or order a guide online today.
Wireless Internet is available at locations throughout NYC—including many parks and libraries.
Get tips, maps and everything else you'll need during your stay.
These NYC-centric mobile apps put the City at your fingertips.
NYC Comedy Week – 2-for-1 Tickets to New York Comedy Shows
March 23–29, celebrate the show's final season with a two-course lunch or two cocktails for $19.69.
Stay in New York City and save with these offers.
New York City is made up of five boroughs: the Bronx, Brooklyn, Manhattan, Queens and Staten Island. Each one has enough attractions—and enough personality—to be a city all its own. Learn more about them with this guide.
Working from an 1882 brownstone, Harlem, Your Way! specializes in custom tours for groups and individuals. But the company also has standard, albeit unusual, walking and bus tours. Go by foot to a Champagne Jazz Safari to a Harlem cabaret, attend a gospel church on Sunday followed by a soulful gospel breakfast, browse the African Vendors' Market for bargains, tour gorgeous Harlem brownstones or board a bus to explore ethnic neighborhoods in Brooklyn and the Bronx.