text:
|Publication number
|Publication type
|Application number
|Publication date
|Mar 27, 2003
|Filing date
|Jul 15, 2002
|Priority date
|Jul 16, 2001
|Also published as
|Publication number
|Sang Cha, Juchang Lee, Kihong Kim
|Original Assignee
|Cha Sang K., Juchang Lee, Kihong Kim
|Export Citation
|BiBTeX, EndNote, RefMan
|Patent Citations (5), Referenced by (95), Classifications (16), Legal Events (6)
|External Links: USPTO, USPTO Assignment, Espacenet
[0001] This application claims the benefit of co-pending U.S. Provisional Application Ser. No. 60/305,956, filed Jul. 16, 2001, entitled "Parallel Logging and Restart Method and System Based on Physical Logging in Main-Memory Transaction Processing System," and co-pending U.S. Provisional Application Ser. No. 60/305,947, filed Jul. 16, 2001, entitled "Parallel Logging and Restart Method and System Based on Physical Logging in Disk-Based Transaction Processing System."
[0002] 1. Technical Field
[0003] This invention in general relates to database systems and methods. More specifically, the present invention relates to parallelized redo-only logging and recovery for highly available main-memory database systems.
[0004] 2. Description of Related Art
[0005] A main-memory database management system (MM-DBMS) keeps a database in main memory to take advantage of the continuously improving the price/density ratio of available memory chips. This memory-centered database architecture simplifies the DBMS and enables the MM-DBMS to better exploit the hardware computing power, such as high-speed L2 cache memory, than a disk-resident DBMS (DR-DBMS) where the database is kept in the disk. For database designers and application developers, the simplicity of a DBMS translates to the ease of optimizing the performance of the overall database systems and their applications.
[0006] While the benefit of the MM-DBMS has well been perceived for read-oriented transactions, the MM-DBMS can also achieve a higher performance than the DR-DBMS in update transactions because updates in the MM-DBMS incur only sequential disk accesses for appending the update logs to the end of the log file and occasionally checkpointing the updated database pages to the backup copy resident in the disk.
[0007] Logging is essential for MM-DBMS to recover a consistent database state in case of a system failure. The recovery involves first loading the backup database in memory and then replaying the log in the serialization order. Checkpointing helps throw away the old portion of the log file and thus shorten the log replay time. Between these two types of run-time disk accesses, the logging performance is more critical than the recovery performance. If an MM-DBMS relies on a single log device in favor of the simplicity of enforcing the serialization order during log replay, its update throughput during logging is bound by the contention on a single log buffer and the I/O bandwidth of the log device.
[0008] To address the problem of this bottleneck, multiple log disks for parallel logging has been used. However, a na�ve parallel logging scheme pays the cost of merging log records distributed over multiple log disks in the serialization order during recovery. To overcome this problem, Lee et al proposed the so-called differential logging that exploits a full degree of parallelism both in logging and recovery. See Juchang Lee, Kihong Kim, and Sang K. Cha, "Differential Logging: A Commutative and Associative Logging Scheme for Highly Parallel Main Memory Database," Proceedings of ICDE Conference, pp. 173-182, 2001.
[0009] The differential logging scheme uses a bit-wise XOR operation, both associative and commutative, for a redo operation as well as an undo operation so that the log records, each of which contains the bit-wise XOR difference between the after and before images, can be replayed in a manner independent of their serialization order during recovery. Such order independence enables distribution of log records to an arbitrary number of log disks, leading to almost linear scale-up of the update throughput during logging until it is bound by either the CPU power or the I/O bandwidth.
[0010] Not only the logging time, but also the recovery time can also be scaled down proportionally by replaying the log records in each log disk independently in a single pass. Even the process of loading the backup database partitioned over multiple disks may proceed in parallel along with the process of replaying the logs once the main memory database is initialized with zeros. In addition to the benefit of parallel execution, the differential logging scheme also reduces the log volume to almost half compared to the conventional redo/undo logging.
[0011] Similarly, in the area of non-differential logging, there is also a need for an efficient logging scheme that can exploit massive parallelism.
[0012] It is an object of the present invention to provide an efficient logging and recovery scheme based on non-differential logging that can be used to recover a consistent transaction processing system after a failure occurs.
[0013] It is another object of the present invention to provide a logging and recovery scheme where massively parallel operations are possible.
[0014] The above-mentioned and other objects are achieved by the present invention that uses physical logging and selective replaying of log records based on an update sequence number for representing the sequence of updates to the database. The update sequence number may be a global sequence number (GSN) representing the sequence of updates to the entire database, a transaction sequence number (TSN) representing the sequence of transactions logged, or a slot sequence number (SSN) for representing the sequence of updates to a particular slot of the database. A preferred embodiment is called "parallel redo-only logging (PROL)" combining physical logging and selective replay of log records using private buffering.
[0015] Since the order of replaying log records doesn't matter in physical logging, parallel operations for recovery are possible. Since the physical logging does not depend on the state of the object to which the log records are applied, the present invention makes it easy to construct a log-based hot standby system. The performance evaluation of the present invention on a 4-way multiprocessor system shows that the PROL scheme outperforms the non-parallelized redo-only logging scheme.
[0016]FIG. 1 is a block diagram of the architecture of a highly parallel and available MM-DBMS of the present invention.
[0017]FIG. 2 is a block diagram of a highly parallel recovery scheme of the present invention.
[0018]FIG. 3 is an illustration where the portions of a database covered by two log records overlapping each other.
[0019]FIG. 4A is the structure of an update log record structure in a preferred embodiment using SSN-based PROL.
[0020]FIG. 4B is the organization of a page stored in main memory in SSN-based PROL.
[0021]FIG. 5 is a block diagram of the logging system architecture of the present invention using SSN-based PROL.
[0022]FIG. 6 is a block diagram of the PROL-based hot standby system architecture of the present invention.
[0023]FIG. 7 is a graph of the logging throughput with varying number of log disks.
[0024]FIG. 8A is a graph of the logging throughput with varying slot sizes.
[0025]FIG. 8B is a graph of the logging throughput in a 4-way multiprocessor platform during logging with the abort ratio varying from 0% to 20%.
[0026]FIG. 9 is a graph comparing the logging throughput of the stand-alone server with that of the master server that ships SSN-based PROL log records to the slave server.
[0027]FIG. 10 is a graph of the log processing time in seconds with varying number of log disks.
[0028]FIG. 11A is a graph of the log processing time with varying slot sizes in a 4-way multiprocessor platform.
[0029]FIG. 11B is a graph of the log processing time when 1.2 M transactions are replayed in a 4-way multiprocessor platform.
[0030]FIG. 12A is a graph of the measured restart time with two log disks and two backup disks.
[0031]FIG. 12B is a graph of the measured restart time with four log disks and four backup disks.
[0032] A. Architecture of Highly Parallel and Available MM DBMS
[0033] Highly Parallel Logging
[0034]FIG. 1 shows the architecture of a highly parallel and available MM-DBMS of the present invention, where database functionalities are implemented with a primary database 100, a data dictionary 101 for storing metadata about the structure of the database, and a number of database managers, such as a memory manager 102 for managing allocation of space and data structures in main memory, a lock manager 103 for managing a locking table used for concurrency control among transactions executing concurrently, a transaction manager 104 for ensuring that the database remains in a consistent state despite a system failure, a replication manager 105 for managing a remote backup copy in case of environmental disasters, a recovery manager 106 for recovering a consistent database state in case of a system failure, a checkpoint manager 107, and a log manager 108.
[0035] The log manager 108 is responsible for logging updates to the database 100 by generating log records. Each log record contains the physical image of the updated portion of the database. Logging is done in parallel by distributing log records to multiple log disks such as 109 and 110. In the present invention, log records may be preferably partitioned into multiple disks by transaction IDs (TIDs).
[0036] The checkpoint manager 107 is responsible for checkpointing, the process of making backup copies of the entire database from time to time. In the present invention, checkpointing may be done in parallel by partitioning each backup copy into multiple backup disks such as 111 and 112. A preferred system may maintain two backup copies based on the ping-pong checkpointing scheme. The locations of backup databases and important log records such as ones recording the beginning and the end of checkpointing are kept in a log anchor 113. In case of a system failure, a consistent state of the primary database 100 is recovered from the log and the most recent backup database.
[0037] The primary database 100 preferably consists of a number of fixed-size segments, where a segment is the basic unit of memory allocation. A segment is divided into fixed-size pages, and a page is again divided into fixed-size slots. Although the size of a slot is fixed within a page, it may vary by pages. A record is basically stored in one slot while a variable-length record is handled by linking multiple fixed-size slots. The size of a slot is determined when formatting a page in order to allocate it to a table. When a page becomes free by deletion of a record, it is added to the global free page list. The preferred embodiment uses fine-granular, slot-level locking and logging.
[0038] Highly Parallel Recovery Operations
[0039]FIG. 2 shows a highly parallel recovery scheme of the present invention where the recovery time can be scaled down proportionally as well by replaying the log records in each log disk independently in a single pass.
[0040] A primary database 200 is located in main memory is reconstructed in case of a system failure by reading and playing backup database stored in back disk such as 202 and 203 through buffers such as 206 and 207 in a recovery manager 201, and reading and playing log records stored in log disks such as 204 and 205 through buffers such as 208 and 209 in the recovery manager 201.
[0041] Since most of the conventional logging schemes require replaying log records by the serialization order, the achievable degree of parallelism during recovery is limited. Suppose that log records are distributed to multiple disks by transaction IDs (TIDs). Then, the log records stored in multiple disks should be merged in the serialization order before replaying. Or suppose that the database is partitioned and log records are distributed to log disks by that partitioning. This resource-partitioned parallel logging does not scale up well when updates are skewed to certain partitions or transactions span multiple partitions.
[0042] To overcome this limitation of conventional logging schemes, order-independent log replay has been explored by Lee et al based on differential logging. See Juchang Lee, Kihong Kim, and Sang K. Cha, "Differential Logging: A Commutative and Associative Logging Scheme for Highly Parallel Main Memory Database," Proceedings of ICDE Conference, pp. 173-182, 2001.
[0043] Order independence of log replaying enables the MM-DBMS to distribute log records without restriction to an arbitrary number of log disks, leading to an almost linear scale-up of the update throughput during logging until it is bound by either the CPU power or the IO bandwidth. The recovery time can be scaled down proportionally as well by replaying the log records in each log disk independently in a single pass. Even loading of the backup database, which may be partitioned over multiple disks for further parallelism, can proceed in parallel with replaying log records.
[0044] Theorem 1. (Order Independence of Commutative and Associative Log Replay)
[0045] Suppose that a redo operation (O ρ L) and an undo operation (O υ L) are commutative and associative for a slot O and a log record L. Then, given the initial state O0 of O and a set of log records {L1 |1≦i≦m}, the final state Om of O can be recovered from the initial state O0 by redoing the log records in an arbitrary order, and the initial state O0 can be recovered from the final state Om by undoing the log records in an arbitrary order.
[0046] Proof. The final state Om can be obtained by redoing the log records in their generation order, namely Om=O0 ρ L1 ρ L2 ρ . . . ρ Lm. Suppose that the log records are redone in the order of Lk(1), Lk(2), . . . , Lk(m), where k(i)ε{1, 2, . . . , m} and k(i)!=k(j) for all i and j, i !=j. Then, O0 ρ Lk(1) ρ Lk(2) ρ . . . ρ Lk(m)=O0 ρ L1 ρ L2 ρ . . . ρ Lm=Om.
[0047] Conversely, the initial state O0 can be obtained by O0=Om υ Lm υ Lm-1 υ . . . υ L1. Suppose that the log records are undone in the order of Lk(1), Lk(2), . . . , Lk(m), where k(i) ε {1, 2, . . . , m} and k(i) !=k(j) for all i and j, i !=j. Then, O0 υ Lk(1) υ Lk(2) υ . . . υ Lk(m)=Om υ Lm υ Lm-1 υ . . . υ L1=O0. ▪
[0048] As shown in Theorem 1, any logging scheme with commutative and associative redo and undo operations can replay log records in an arbitrary order. One such scheme is the so-called differential logging, which logs the bit-wise XOR difference between the after and the before images and replays log records by applying the same XOR operation.
[0049] The present invention presents alternative ways of implementing the associative and commutative redo and undo operations based on physical logging so that the order of replaying log records doesn't matter.
[0050] Log-based Management of Hot Standby
[0051] In a hot standby configuration, one server acting as a master server is being actively used to run a database, while another server acting as a slave server is in standby mode ready to take over if there is an operating system or hardware failure involving the first server. Typically, a message called "heartbeat" is passed between the two servers to monitor the working condition of each other server.
[0052] For high availability, the management of a hot standby system with the 1-safe scheme may be used, where a transaction can commit before successfully sending its log records to the slave server. In this scheme, the slave server continuously replays log records received from the master server. If no heartbeat message arrives from the master server for a given period, the slave server automatically takes over after aborting all the transactions alive. When the failed master server restarts, it first asks the latest reflected log record ID (identification) to the taken-over server. Then, it collects the transactions committed in the log located after that log record and sends all the collected log records to the taken-over server. Finally, it receives the log records generated during its down time from the taken-over master server and replays them. The preferred embodiment uses the 1-safe scheme for simplicity, but those skilled in the art would appreciate similar schemes can be used for other replication schemes.
[0053] B. Selective Log Replay
[0054] The present invention replays log records selectively to fully exploit the parallelism. The selective log replay is based on the observation that the consistent state of a slot can be recovered by replaying the latest log record for the slot when using physical logging. Replaying intermediate log records doesn't matter because the last log record replayed overrides all the log records replayed thus far for a given slot. To make selective log replaying possible, it is necessary to maintain an update sequence number or timestamp of a given slot to be that of the last log record replayed for that slot during the recovery time. Then, the log records whose update sequence number is smaller than the update sequence number of a slot may be safely discarded.
[0055] Definition 1. (Selective Redo and Undo Operations)
[0056] Suppose that sn(O) or sn(L) denotes the update sequence number of a slot O or a log record L. Then, the selective redo operation (O ρ L) and selective undo operation (O μ L) for a slot O and a log record L are defined by
[0057] Theorem 2. (Order Independence of the Selective Log Replay Operations)
[0058] If ρ and μ are defined by the Definition 1, it enables order independent log replay.
[0059] Proof. Suppose that sn(L1)<sn(L2)<sn(L3) for log records L1, L2, and L3. Then,
[0060] (O ρ L1) ρ L2=L1 ρ L2=L2, and (O ρ L2) ρ L1=L2 ρ L1=L2
[0061] ∴ (O ρ L1) ρ L2=(O ρ L2) ρ L1
[0062] ((O ρ L1) ρ L2) ρ L3=(L1 ρ L2) ρ L3=L2 ρ L3=L3, and ((O ρ L2) ρ L3) ρ L1=L3
[0063] ∴ ((O ρ L1) ρ L2) ρ L3=((O ρ L2) ρ L3) ρ L1
[0064] Therefore, ρ is commutative and associative operator. The same holds for the μ.
[0065] By the Theorem 1, ρ and μ enable the order independent log replay. ▪
[0066] Although this selective log replay can be applied to any kind of physical logging including redo-undo logging or redo-only logging, redo-only logging is preferable in favor of the reduced log volume, comparable to that of the differential logging.
[0067] Update Sequence Numbers
[0068] There are several choices for the update sequence number. A simple choice is to timestamp log records when the precision of timestamps is enough to distinguish any two updates on the same slot. An obvious alternative is to maintain a single, global sequence number with Algorithm 1 for each update. However, using a single counter incurs a contention on the counter during the parallel logging. Even during the recovery, a single, global hash table has to be built to maintain the update sequence number for the slots that are replayed by the encountered log records. Since each log loader at recovery accesses the global hash table, it incurs the contention.
[0069] Algorithm 1. Global Sequence Number (GSN)
[0070] 1. Acquire the latch for the global counter
[0071] 2. Increase the counter by 1 and save the current value of it
[0072] 3. Release the acquired latch
[0073] 4. Return the saved counter value
[0074] In order to lessen the contention on the global counter and the global hash table, it is possible to partition the database and maintains a counter for each partition. Since each partition has the space for its own counter, the global hash table need not be built during the recovery. The size of a partition can be chosen arbitrarily from the entire database to a slot. As the size of a partition decreases, the contention will be lessened but the space overhead for the local counters will grow. Thus, the choice will be guided by a tradeoff between space needed for the local counters and the contention incurred by global counter.
[0075] Optimization 1. When a counter covers a smaller portion of the database than a transaction lock does, access to the counters does not require any latching.
[0076] For example, since transactional locking is generally performed at the slot or higher level, maintaining a counter for each slot allows a transaction to access the counters without any latching as in Algorithm 2. Note that only one transaction can update a slot at a given moment, and the same for a counter.
[0077] Algorithm 2. Slot Sequence Number (SSN)
[0078] 1. Increase the counter for the given slot by 1
[0079] 2. Return the current value of the counter
[0080] Even with the global sequence number, there is room for reducing the lock contention, especially when a transaction involves a number of updates. If the log records of a transaction are stored consecutively on disk and the transaction generates only a log record for the slot updated multiple times, assignment of the global sequence number can be deferred until the commit time as shown in Algorithm 3.
[0081] Algorithm 3. Transaction Sequence Number (TSN)
[0082] 1. If this is the first call by the given transaction, invoke Algorithm 1 and save the return value
[0083] 2. Return the saved counter value
[0084] According to these two optimization rules, slot sequence number (SSN) and the transaction sequence number (TSN) are used. It is also assumed that the slot-level locking and the so-called private log buffering, which guarantees that the log records of a transaction are stored consecutively on disk and the transaction generates only a log record for the slot updated multiple times.
[0085] Handling Limitation of Selective Log Replay
[0086] There may be a situation where two log records cover either the same portion of the database or non-overlapping portions of the database. FIG. 3 shows such a situation where situation shown in FIG. 3, where the portions of database covered by the log records L1 and L2 overlap each other. Suppose that L1 is encountered after L2 has been replayed. Then, L1 is discarded because it has a smaller update sequence number than L2, and this may result in the loss of information recorded in the shaded area of L1.
[0087] Such a situation does not occur when logging is done at the page level, because the size of a page is fixed in most of database systems. However, it may occur when the lot-level logging assumed. A page is formatted with a desirable slot size when it is popped out of the free page list. Thus, the slot size in a page can vary with time, and this leads to the situation shown in FIG. 3. This situation should be handled carefully, especially when storing an update sequence number within a slot. If the slot size at the time of log record creation is different from the slot size at the time of replay, it is impossible to compare the update sequence number of a log record to that of a slot.
[0088] To address the above situation, another sequence number named PVN(page version number) may be introduced. Residing in the header of each page, the PVN is increased by one when a page is formatted with a different slot size. When creating a log record, the PVN value of the corresponding page is copied to the log record. During recovery, a log record with a smaller PVN value than the corresponding page is discarded.
[0089] Another limitation of selective replay is that timestamps need to be reset in order to avoid their overflow. Resetting timestamps should be performed in an action-consistent state, followed by action-consistent checkpointing. Fortunately, the action-consistent checkpointing is needed rarely although it interferes with user transactions
[0090] C. Parallel Redo Only (PROL) Implemntation
[0091] Private Log Buffering
[0092] A preferred embodiment of the present invention uses parallel redo only logging ("PROL") that combines redo-only logging followed by selective reply. To implement PROL, a private log buffering method is used. The private log buffering method maintains the redo and undo log records of active transactions in their own private memory space constituting one or more private buffers. When a transaction commits, only the redo records are flushed together with a commit log record. The undo records of a transaction are used for aborting the transaction, but they are not written to disk. In this way, only the redo records need to be written to the disk.
[0093] Such private log buffering method has the following advantages over the contrasted public log buffering, which writes both redo and undo records to a public log buffer once they are generated. Note that the private log buffering also needs a public log buffer, but it is accessed only when the transaction commits.
[0094] When a transaction aborts, there is no need to write its log records on the disk except when a checkpointing is performed. But, in the public log buffering, since the aborted transaction's log record may be already stored on the log disk, the so-called compensation logging is needed to reduce the amount of undo operations during the post-crash recovery.
[0095] If a transaction involves a number of updates, the private log buffering method reduces the contention on the public log buffer.
[0096] Since the log records of different transactions are intermixed on the disk in the public log buffering, it is not easy to collect the log records of a loser transaction. But, since the private log buffering method stores the log records of a transaction consecutively on the disk in the private log buffering, only the tail in the log file may be inspected.
[0097] The private log buffering method requires more memory space than the public log buffering method. To reduce such memory overhead under in-place update scheme, PROL does not maintain after images of the redo log record in the private log buffer when an update takes place. Instead, the after images are collected and directed to the public log buffer when the transaction commits. For an active transaction, the modified private log buffering in PROL requires the space for only the undo log records and some information needed for gathering the after images. Another advantage of this modification is to require only a log record when a transaction updates the same slot multiple times.
[0098] The modified private log buffering is applied to even to the differential logging. Since a differential log record is used for both redo and undo operations, only the differential log records can be maintained in the private log buffer.
[0099] Log Record and Page Structure
[0100]FIG. 4A shows the structure of an update log record structure in a preferred embodiment using SSN-based PROL. There may be five types of log records in SSN-based PROL: update, begin_transaction, commit_transaction, begin_checkpoint, and end_checkpoint. Among these, the abort_transaction type is not needed for SSN-based PROL because the log records of the aborted transactions on disk are not written.
[0101] The update log record consists of a log header 401 and the physical image of a slot 402. The log header 401 further consists of a slot sequence number (SSN) 407, a slot ID 406 representing the address of the slot to which the log record will be applied, a size 404 represents the size of the slot, a type 403 specifying the type of log records, and a page version number (PVN) specifying a page version.
[0102] Differently from this structure of the update record, the log records for begin_transaction and commit_transaction will have just two fields: the type and transaction ID (TID) fields. In the page header, the dirty flag indicates whether this page needs to be checkpointed or not. The bitmap field represents whether each slot is free or not.
[0103]FIG. 4B shows the organization of a page stored in main memory in SSN-based PROL. It consists of a page header 411 and multiple slots such as 412 and 413. Each page header stores a dirty flag 414, a page version number (PVN) 415, and a bit map 416 for recording the state of each slot in the page. The state is set to "occupied" if data is inserted to the corresponding slot or "free" if data is erased from the slot. Stored together with each slot is an update sequence number most recently played such as 417 or 418.
[0104] TSN-based PROL uses a slightly different log record structure and page organization from those of the SSN-based PROL. The SSN field is unnecessary in the update log record. Instead, the TSN value obtained at the commit time is attached to the begin_transaction record. In the page header, no such update sequence number field as SSN is needed.
[0105] Parallel Logging and Checkpointing
[0106]FIG. 5 shows the logging system architecture of the present invention using SSN-based PROL. The commutativity and associativity of the redo operations in PROL of the present invention enable unrestricted distribution of the log records to multiple log disks. To reduce the contention on the shared data structures, active transaction tables (ATT) such as 502 for storing the list of active transactions, private log buffers such as 503 for storing both redo and undo tranactions, and the public log buffers such as 504 for storing committed redo transactions are partitioned by the log disks.
[0107] ATT such as 502 is used to find the transactions that are reflected to the backup database, but are aborted later. This table is initialized from the active transaction list. Such list is stored in the log anchor just before the recent checkpointing completes. When reading a commit record, the corresponding entry in the ATT is removed, if exists. After the roll forward completes, the remaining transactions in the ATT are the transactions that should be undone.
[0108] The SSN-based PROL maintains a local counter on each slot. Alternatively, the TSN-based PROL maintains a global counter for counting transactions. Note that since the slot-level transactional locking is assumed, maintaining SSN does not incur any additional contention.
[0109] Algorithms 4, 5, and 6 are the update, transaction commit, and transaction abort algorithms in the SSN-based PROL. The generation of redo log records is deferred until the transaction commit time. The log buffering thread and the log flushing thread are separated from each other so that the log buffering thread committing a transaction may not remain idle until its log records are flushed. If there are any log records to flush in the public log buffer, the log flushing thread flushes them and finishes the remaining commit processing for the transactions included in the flushed log.
[0110] Algorithm 4. Update
[0111] 1. Generate the redo log record and undo log record in the private log buffer.
[0112] 2. Update the slot.
[0113] 3. Copy SSN of the slot and PVN of the page into the redo log record, and increment SSN.
[0114] Algorithm 5. Transaction Commit
[0115] 1. Generate the redo records by combining the headers and after images, and append them together with a transaction commit record.
[0116] 2. Flush the private log buffer to the corresponding public log buffer.
[0117] 3. Wait until the redo log records are written to the disk.
[0118] 4. Release all the locks that the transaction holds.
[0119] 5. Notify the user that the transaction has been committed.
[0120] 6. Remove the matching element from the ATT.
[0121] 7. Delete the private log buffer for the transaction.
[0122] Algorithm 6. Transaction Abort
[0123] 1. Apply the undo log records to the primary database.
[0124] 2. Release all the locks that the transaction holds.
[0125] 3. Notify the user that the transaction has been aborted.
[0126] 4. Remove the matching element from the ATT in a latched state.
[0127] 5. Delete the private log buffer for the transaction.
[0128] PROL can be used with parallel checkpointing, as shown in Algorithm 7. Each checkpointing thread flushes dirty pages located in its own checkpointing partition. To allow the execution of the user transactions during checkpointing, it is required to write undo log records of the ATT and active transaction list at that moment just before the transaction completes. And also, such undo logging is done for each transaction/log partition in parallel. It is assumed that such undo log records are stored on the same disk with the redo log file, but separately.
[0129] Algorithm 7. Parallel Checkpointing
[0130] 1. Create a begin_checkpoint record and append it to all the public log buffers.
[0131] 2. Choose the backup database that was the least recently checkpointed as the current backup database.
[0132] 3. For each checkpointing partition in parallel,
[0133] A. While scanning the database page by page, copy all the dirty pages into the current backup database asynchronously.
[0134] B. Wait until all the asynchronous I/Os are completed.
[0135] 4. For each transaction/log partition in parallel,
[0136] A. Hold a latch on the ATT.
[0137] B. Write the undo log of all the active transactions on the assigned disk.
[0138] C. Write the active transaction list on the assigned disk.
[0139] D. Release the latch.
[0140] 5. Append an end_checkpoint record into all the public log buffers and update log anchor with the current backup database ID and begin checkpointing the positions of the log records.
[0141] Parallel Restart
[0142] PROL of the present invention supports the fully parallel restart by allowing multiple log disks to proceed simultaneously with loading backup disk partitions. Algorithm 8 presents the steps for processing the individual log disk. Similarly, Algorithm 9 presents the steps for loading a backup disk partition.
[0143] In the case of TSN-based PROL, a global hash table is maintained to bookkeep the TSN value of each replayed slot. All slot ID and TSN pairs should be included in the hash table. But, the loaded backup slots may be excluded, because some of them will not be updated by log records and the remaining will be replayed by the log record with the same slot image.
[0144] Algorithm 8. Log Processing in Restart
[0145] 1. Read the position of begin_checkpoint record from the log anchor and mark it as the begin of the log.
[0146] 2. Initialize ATT from the active transaction list stored in the log anchor.
[0147] 3. Go backward in the log from the end until the first commit record is encountered. Mark the position as the end of the log.
[0148] 4. From the marked log begin to the marked log end, go forward in the log, doing the following:
[0149] A. For an update log record,
[0150] (i) Hold a latch on the page.
[0151] (ii) If the update record's PVN is larger than or equal to the page header's PVN, proceed to the following step. Otherwise, release the latch and ignore this record.
[0152] (iii) If the update record's SSN is larger then the slot's SSN, redo and update the slot's SSN with the record's SSN. Otherwise, ignore the current update record.
[0153] (iv) Release the latch.
[0154] B. For a commit log record,
[0155] (i) Remove the corresponding TID in the ATT, if it exists.
[0156] 5. Wait until the backup loading completes.
[0157] 6. Roll back the remaining TIDs in the ATT.
[0158] Algorithm 9. Backup Loading
[0159] 1. Find the recently checkpointed backup DB copy from the log anchor information.
[0160] 2. Read the pages of this backup DB into the backup DB buffer.
[0161] 3. For each page read in the backup DB buffer, do:
[0162] A. Hold a latch on the page.
[0163] B. If the buffered page's PVN is equal to or larger than the primary page header's PVN, proceed to the following step. Otherwise, release the latch and skip this page.
[0164] C. For each slot s in the page, do:
[0165] (i) If the slot SSN of s is larger than the SSN of the corresponding slot t in the primary database, override the image and SSN of t with the after image and SSN of t, respectively.
[0166] (ii) Otherwise, ignore the current update record.
[0167] D. Release the latch.
[0168] PROL-Based Hot Standby System Architecture
[0169]FIG. 6 shows the PROL-based hot standby system architecture of the present invention. This hot standby architecture also exploits the parallelism in log-based synchronization. Multiple log streams are established between a master server 601 and a slave server 602. Since PROL enables order-independent log replay, there is no need to merge the multiple log streams in the slave server. Instead, the slave server can replay them in parallel. Only the log records of committed transactions are propagated to the slave server.
[0170] Following the 1-safe log synchronization, the master server 601 propagates the log pages that are safely stored on the disk. The slave server 602 replays the log records in the received pages and returns the acknowledgement message to the master server. The monitor threads 603 and 604 send and receive a heartbeat message 605 periodically. If no heartbeat message arrives from the master server for a given period, the slave server 602 automatically takes over.
[0171] To deal with the inconsistency between two participating servers incurred by the 1-safe scheme, the synchronization process is done as described in Algorithms 13 and 14. The lost transactions' log records are retransmitted to the taken-over server, and the log records arrived to the taken-over server are replayed selectively. If the slot to which the lost transaction's log record is applied has been updated after taking over, the log record is ignored. As the result, the images of two servers are converged to a consistent and up-to-date state.
[0172] The simplified hot standby algorithms are described in Algorithms 10 to 14 as follows.
[0173] Algorithm 10. Takeover of the Slave Server
[0174]1. Wait until all the received log records are replayed.
[0175]2. Abort the active transactions at the moment.
[0176]3. Set the sync_position as the recently received log record ID.
[0177]4. Set the send position as the tail address of the log file.
[0178]5. Resume the transaction service.
[0179] Algorithm 11. Normal Processing of the Master Server
[0180] Iterate the following:
[0181] A. If the connection with the slave server is available,
[0182] (ii) Send a log page located from the send_position to the slave server, if it is reflected to the log disk.
[0183] (ii) Receive an acknowledgement message from the slave server, and increment the send_position by the size of the successfully sent log page.
[0184] (B) Otherwise,
[0185] (i) Wait until it is recovered.
[0186] (ii) If the recovered slave server requires the synchronization process, invoke Algorithm 14.
[0187] Algorithm 12. Normal Processing of the Slave Server
[0188] Iterate the following:
[0189] (A) If the connection with the slave server is available,
[0190] (i) Receive a log page from the master server, and replay log records in the page.
[0191] (ii) If the received log record is safely stored in the log disk, send the acknowledgement message to the master server.
[0192] (B) Otherwise,
[0193] (i) If the heartbeat message does not arrive for a period, invoke Algorithm 10.
[0194] Algorithm 13. Restart of the Failed Master Server
[0195] 1. Do the following synchronization process for lost transactions:
[0196] (i) Require the synchronization processing by asking the sync_position to the taken- over server.
[0197] (ii) Collect the transactions that are committed in the log located from the sync_position.
[0198] (iii) Send all the log records generated by the collected transactions to the taken-over server.
[0199] 2. Invoke Algorithm 12.
[0200] Algorithm 14. Synchronization Process of the Taken-over Server
[0201] 1. Send the sync position to the failed master server.
[0202] 2. For each received log record of the lost transactions, do:
[0203] (i) If the timestamp of the log record is larger than that of the corresponding slot, replay it with transactional locking.
[0204] (ii) Otherwise, ignore the log record.
[0205] D. Experimental Results
[0206] To compare the logging and recovery performance of PROL of the present invention with those of the differential logging scheme (DL) and the non-parallelized redo-only logging (ROL) scheme, a series of experiments were conducted. As previously mentioned, private log buffering is combined with PROL and ROL. For a fair comparison with PROL, differential logging was implemented with private log buffering (DL-private) as well as one with public log buffering (DL-public). Two variants of PROL were implemented: PROL with a slot sequence number (PROL-SSN) and PROL with a transaction sequence number (PROL-TSN).
[0207] A test database was used which models a simplified SMS(short message service) message table with three fields. The size of a record or a slot was 256 bytes. The primary database included 2.5 million records, and thus the size of the primary database was 620˜639 MB. Since PROL-SSN maintains the SSN for each slot, it consumes a little more space than other schemes. For the 2.5 million record database, this SSN overhead is around 19 MB.
[0208] There are two types of transactions: one inserts two records into the message table, and another deletes two records from the table. Table 1 shows the log overhead incurred by each scheme for each of these transactions: transaction begin/commit records, and the update log header. PROL-SSN and DL-public have the biggest overhead because PROL-SSN requires the update sequence number field for each update log record, and DL-public requires the previous LSN field for each update log record, where the previous LSN field is used for rolling back loser transactions without scanning all the unrelated log records. On the other hand, ROL and DL-private show the smallest overhead. Since PROL-TSN attaches an update sequence number only to a transaction begin record, its log record overhead is slightly higher than ROL or DL-private.
[0209] The experiment was conducted on a shared-memory multiprocessor PC server with four 700 MHz Xeon CPUs, 8 GB RAM, and dozen SCSI disks. The average seek time of each disk was 5.2 msec, average rotational latency was 4.17 msec, and the internal transfer rate was 20˜30 MB/sec. The server had an Adaptec 160 SCSI card, which supports up to 160 MB/sec. For a hot standby experiment, two PC servers were connected with a gigabit Ethernet switch. In each PC server, the SCSI card and a gigabit Ethernet card are linked via a 64 bit/66 Mhz PCI bus.
[0210] Logging Performance
[0211] The logging throughput was measured by varying the number of log disks, the abort ratio, and the size of a slot when the system is overloaded. The size of a group in the public log buffer was fixed at 128 KB.
[0212]FIG. 7 shows the logging throughput in TPS (transactions per second) with the
TABLE 1 The Log Overhead for a Transaction with Two Updates PROL- DL- Logging Scheme PROL-SSN TSN private DL-public ROL Log Overhead 64 B 56 B 48 B 64 B 48 B
[0213] number of log disks. The transaction abort ratio was 5%. The observation is summarized as follows.
[0214] The throughput of parallelized logging schemes increased almost linearly with the number of disks until it becomes saturated, while the non-parallelized scheme, ROL was not scalable. The best parallelized schemes outperformed ROL by about 3.5 times.
[0215] PROL-TSN, PROL-SSN, and DL-private showed almost the same throughput. The throughput of DL-public was a little lower than that of these schemes because of the contention on the public log buffer. Since a transaction generated a pair of begin/commit transaction records and two update records in this experiment, the DL-public accessed the public log buffer four times as many as other schemes with the private buffering.
[0216] The throughput of the parallelized schemes beccame saturated when the number of log disks was more than 4. The SCSI channel becomes saturated when the disks were fully used for logging.
[0217]FIG. 8A shows the logging throughput with the varying slot size. It is noted as the throughput goes down as the slot size increases, the performance of DL-public converged to that of other parallelized schemes with the private buffering because of the reduced contention on the public log buffer.
[0218]FIG. 8B shows the logging throughput with the abort ratio varying from 0% to 20% in a 4-way multiprocessor platform. Since the private log buffering does not write any log records to the disk for the aborted transactions, the required log disk I/O will slightly decrease with the increasing abort ratio while the aborted transactions contribute to the throughput. DL-public(4) shows no change with the varying abort ratio because it writes log records for the aborted transactions according to the compensation logging scheme. As the result, when the abort ratio is 20%, DL-private(4), PROL-SSN(4), and PROL-TSN(4) outperform DL-public(4) by 1.26 times, and ROL by 2.95 times.
[0219] To see the effect of the log-based synchronization on the run-time throughput, the PROL-SSN-based log shipping scheme was implemented.
[0220]FIG. 9 compares the logging throughput of the stand-alone server with that of the master server that ships PROL-SSN log records to the slave server. As the number of log disks increases, the throughput of the master server decreases to about 70% of the stand-alone server throughput. Such performance gap is due to the following two reasons.
[0221] During the log propagation to the network, the log in DRAM is transferred to both the SCSI card and the Ethernet card via PCI bus. Therefore, the logging throughput is limited by the bandwidth of the PCI bus as well as that of the SCSI channel.
[0222] When the log shipping speed is as fast as the log flushing speed, there is no need to access the log disk for the log propagation because the log buffer can hold them until it is exploded. If this is not the case, the sequential access to the log disks for log writing may be disturbed by the log shipping threads. Therefore, it is expected that the performance gap becomes larger if the log propagation rate to the network is not so high.
[0223] Recovery Performance
[0224] To gauge the practical impact of parallel recovery, the restart time during recovery was also measured. The restart time is broken down to the backup DB loading time and the log processing time.
[0225]FIG. 10 shows the log processing time in seconds with the number of log disks. 1.2 M transactions were replayed during the log processing. The log volume was 739 MB for PROL-SSN or DL-public, 650 MB for PROL-TSN, and 640 MB for DL-private or ROL.
[0226] The log processing time of the parallelized schemes decreased with the number of log disks, while the non-parallelized, ROL did not.
[0227] The log processing time of PROL-TSN was saturated when the number of disks was more than 2. The restart hash table for TSN incurred more contention with the increasing number of disks.
[0228] The log processing time of DL-public was longer than that of PROL-SSN or DL-private. This is because the log volume of DL-public was slight larger than the other two schemes due to the compensation logging for aborted transactions. For this experiment, the 5% abort ratio was assumed.
[0229] The log processing time of DL-public, DL-private, and PROL-SSN became saturated when the number of log disks approached to six, when the SCSI channel became saturated.
[0230] Two best schemes, DL-private and PROL-SSN, outperformed ROL by 4.5 times in the experimental environment.
[0231]FIG. 11A shows the log processing time with varying slot sizes in a 4-way multiprocessor platform. The log volume overhead by the update sequence number was negligible regardless of the slot size. The log processing time was measured with varying the abort ratio from 0% to 20%.
[0232]FIG. 11B shows the result when 1.2 M transactions are replayed in a 4-way multiprocessor platform. While the log volume of the private log buffering schemes decreased with the abort ratio, that of the public log buffering did not. Such difference in the log volume was reflected in the result. Note that PROL-SSN(4) and DL-private(4) outperform DL-public(4) by 1.44 times, ROL by 3.97 times.
[0233] In evaluating the overall recovery time, two types of measurement were conducted. The first one with the postfix seq measured the time for the parallel log processing separately from the time for the parallel backup database loading. The other one with the postfix para measured the recovery time when the log processing was intermixed with the backup loading.
[0234]FIG. 12A and FIG. 12B show the result when the log volume size was almost the same with the primary database size, namely 620˜639 MB. FIG. 12A show the measured restart time with two log disks and two backup disks, and FIG. 12B shows the measured restart time with four log disks and four backup disks. The following observations are made.
[0235] Intermixing the backup database loading and the log processing reduced the total recovery times of DL-public, DL-private, and PROL-SSN almost to half as shown in FIG. 12A.
[0236] The total recovery time of PROL-TSN-para was longer than DL-public-para, DL-private-para, or PROL-SSN-para in both FIG. 12A and FIG. 12B. In PROL-TSN-para, since each backup loading process also accessed the restart hash table on TSN, the contention in the hash table increased.
[0237]FIG. 12B shows that the gain in the restart time by intermixing the backup loading and the log processing was not so high as in FIG. 12A. The SCSI channel became saturated when the eight disks are fully involved in the restart.
[0238] In FIG. 12B, the difference between DL-public-para and DL-private-para came from the difference in the log volume. Since the abort ratio was fixed at 5%, the log volume of DL-public was larger than that of DL-private by 5%.
[0239] The marginal difference between PROL-SSN-para and DL-private-para came from the selective log replay of PROL. Since the backup database loading for the slot with already replayed log was skipped, the number of memory write operations was slightly smaller than that of DL variants.
[0240] PROL-SSN-para outperformed ROL-seq by 2.83 times when two backup disks and two log disks were involved, and by 3.13 times when four backup disks and four log disks were involved.
[0241] In summary, the experimental results on a 4-way multiprocessor platform show that PROL-SSN and DL-private are almost indistinguishable in terms of the logging and recovery performance
[0242] While the invention has been described with reference to preferred embodiments, it is not intended to be limited to those embodiments. It will be appreciated by those of ordinary skilled in the art that many modifications can be made to the structure and form of the described embodiments without departing from the spirit and scope of this invention.
|Cited Patent
|Filing date
|Publication date
|May 4, 1936
|Mar 28, 1939
|American Box Board Co
|Title not available
|Title not available
|Title not available
|Title not available
|Citing Patent
|Filing date
|Publication date
|Dec 19, 2003
|Sep 26, 2006
|Lenovo (Singapore) Pte. Ltd.
|Autonomic method to resume multi-threaded preload imaging process
|Mar 23, 2005
|Oct 17, 2006
|Tokyo Electron Limited
|Method and apparatus for the monitoring and control of a semiconductor manufacturing process
|Apr 30, 2003
|Feb 20, 2007
|Oracle International Corporation
|Flashback database
|May 17, 2002
|Jul 24, 2007
|Aleri, Inc.
|Database system and methods
|Aug 13, 2004
|Aug 28, 2007
|Seakr Engineering, Incorporated
|Soft error detection and recovery
|Mar 31, 2004
|Oct 2, 2007
|Microsoft Corporation
|System and method for a consistency check of a database backup
|Oct 19, 2004
|Jul 15, 2008
|International Business Machines Corporation
|Management of global counters in transactions
|Sep 9, 2005
|Nov 25, 2008
|International Business Machines Corporation
|Method to detect errors in computer systems by using state tracking
|Aug 31, 2004
|Jan 27, 2009
|International Business Machines Corporation
|Sending log records directly from log buffer
|Sep 9, 2005
|Mar 10, 2009
|International Business Machines Corporation
|Method and system to execute recovery in non-homogeneous multi processor environments
|Dec 22, 2004
|May 5, 2009
|International Business Machines Corporation
|Log shipping data replication with parallel log writing and log shipping at the primary site
|May 6, 2004
|Jul 7, 2009
|Aptare, Inc.
|System to capture, transmit and persist backup and recovery meta data
|Dec 1, 2009
|Oracle International Corporation
|Lost write detection and repair
|Jan 11, 2007
|Dec 8, 2009
|Oracle International Corporation
|Flashback database
|Jun 28, 2006
|Dec 29, 2009
|International Business Machines Corporation
|System and method for implementing a partial-blocking consistency point in a database
|Nov 3, 2006
|Jan 26, 2010
|Microsoft Corporation
|Anchor for database synchronization excluding uncommitted transaction modifications
|May 6, 2004
|Feb 2, 2010
|Aptare, Inc.
|System to manage and store backup and recovery meta data
|Apr 6, 2010
|Symantec Operating Corporation
|Method and system for incremental backup of data volumes
|Apr 13, 2010
|Fujitsu Limited
|Database system management method, database system, database device, and backup program
|Sep 9, 2008
|Jul 6, 2010
|International Business Machines Corporation
|Method and system to detect errors in computer systems by using state tracking
|Jul 20, 2010
|International Business Machines Corporation
|Apparatus, system, and method for continuously protecting data
|Jan 27, 2009
|Jul 27, 2010
|International Business Machines Corporation
|Method and system to execute recovery in non-homogenous multi processor environments
|Jul 24, 2006
|Jul 27, 2010
|Honeywell International Inc.
|Device and method to enhance availability of cluster-based processing systems
|Feb 20, 2004
|Sep 21, 2010
|Microsoft Corporation
|Systems and methods for reading only durably committed data in a system that otherwise permits lazy commit of transactions
|Dec 26, 2007
|Nov 23, 2010
|Emc (Benelux) B.V., S.A.R.L.
|Methods and apparatus for dynamic journal expansion
|Dec 26, 2007
|Nov 30, 2010
|Emc (Benelux) B.V., S.A.R.L.
|Methods and apparatus for bottleneck processing in a continuous data protection system having journaling
|Dec 12, 2006
|Dec 7, 2010
|Emc Corporation
|Methods and apparatus for multiple point in time data access
|Dec 26, 2007
|Dec 28, 2010
|Emc (Benelux) B.V., S.A.R.L.
|Method and apparatus to recover data in a continuous data protection environment using a journal
|Sep 9, 2005
|Aug 9, 2011
|International Business Machines Corporation
|Method and system for state tracking and recovery in multiprocessing computing systems
|Aug 30, 2006
|Sep 20, 2011
|Sony Corporation
|Recording apparatus, file management method, program for file management method, and recording medium having program for file management method recorded thereon
|Dec 26, 2007
|Oct 18, 2011
|Emc Corporation
|Offloading encryption processing in a storage area network
|Mar 28, 2008
|Nov 15, 2011
|Emc (Benelux) B.V., S.A.R.L.
|Consolidating snapshots in a continuous data protection system using journaling
|Jun 8, 2007
|Jan 24, 2012
|Oracle International Corporation
|Revenue management systems and methods with bill and account suppression
|Jun 28, 2006
|Feb 14, 2012
|Oracle International Corporation
|Revenue management system and method
|Jul 28, 2006
|Feb 14, 2012
|Oracle International Corporation
|Revenue management system and method utilizing database backup
|Feb 13, 2009
|Jul 3, 2012
|International Business Machines Corporation
|Disaster recovery based on journaling events prioritization in information technology environments
|May 1, 2006
|Jul 17, 2012
|Oracle International Corporation
|Revenue management systems and methods
|Jun 30, 2004
|Sep 4, 2012
|Symantec Operating Corporation
|Estimation of recovery time, validation of recoverability, and decision support using recovery metrics, targets, and objectives
|Jun 23, 2010
|Dec 11, 2012
|Emc Corporation
|Splitter used in a continuous data protection environment
|Dec 2, 2010
|Dec 18, 2012
|Emc International Company
|Replicating in a multi-copy environment
|Sep 29, 2010
|Dec 18, 2012
|Emc Corporation
|Storage array snapshots for logged access replication in a continuous data protection system
|Jun 8, 2007
|Feb 5, 2013
|Oracle International Corporation
|Revenue management systems and methods with sponsored top-up options
|Mar 30, 2010
|Mar 5, 2013
|Emc International Company
|Accessing a volume in a distributed environment
|Jun 8, 2007
|Apr 16, 2013
|Oracle International Corporation
|Revenue management systems and methods with re-rating and rebilling
|Sep 27, 2010
|Apr 30, 2013
|Emc International Company
|Virtualized consistency group using an enhanced splitter
|Jun 8, 2007
|Jun 11, 2013
|Oracle International Corporation
|Revenue management systems and methods with payment suspense management
|Sep 27, 2010
|Jul 2, 2013
|Emc International Company
|Virtualized consistency group using more than one data protection appliance
|Dec 28, 2011
|Aug 27, 2013
|Microsoft Corporation
|Database data recovery system and method
|Sep 28, 2007
|Nov 19, 2013
|Oracle International Corporation
|Techniques for automatically tracking and archiving transactional data changes
|Sep 29, 2010
|Apr 8, 2014
|Emc Corporation
|Using I/O track information for continuous push with splitter for storage device
|Jun 8, 2007
|Aug 5, 2014
|Oracle International Corporation
|Revenue management systems and methods with enhanced rollover
|Apr 1, 2013
|Sep 9, 2014
|Emc International Company
|Virtualized consistency group using an enhanced splitter
|Apr 26, 2012
|Oct 14, 2014
|Altibase Corporation
|Apparatus and method of managing databases of active node and standby node of main memory database management system
|Jan 7, 2011
|Oct 21, 2014
|Microsoft Corporation
|Transaction support for distributed data
|Sep 7, 2011
|Nov 25, 2014
|Emc Corporation
|Write signature command
|Mar 14, 2013
|Mar 31, 2015
|Emc Corporation
|Accessing an image in a continuous data protection using deduplication-based storage
|Jun 20, 2012
|Apr 7, 2015
|Microsoft Technology Licensing, Llc
|Structuring storage based on latch-free B-trees
|Jan 10, 2014
|May 5, 2015
|Emc Corporation
|Using I/O track information for continuous push with splitter for storage device
|Jun 24, 2013
|Jun 30, 2015
|Emc International Company
|Dynamic granularity in data replication
|Mar 15, 2013
|Jul 14, 2015
|Emc Corporation
|Synchronous and asymmetric asynchronous active-active-active data access
|Jun 24, 2013
|Jul 21, 2015
|Emc International Company
|Consistency across snapshot shipping and continuous replication
|Nov 7, 2011
|Jul 28, 2015
|Sap Se
|Database log parallelization
|Mar 14, 2013
|Aug 18, 2015
|Emc Corporation
|Continuous data protection using deduplication-based storage
|Mar 15, 2013
|Sep 1, 2015
|Oracle International Corporation
|Common users, common roles, and commonly granted privileges and roles in container databases
|Mar 7, 2012
|Sep 15, 2015
|Aptare, Inc.
|System to catalog and search point-in-time instances of a file system
|Jun 25, 2013
|Sep 29, 2015
|Emc Corporation
|Storage recovery from total cache loss using journal-based replication
|Jul 12, 2002
|Jun 3, 2004
|American Management Systems, Incorporated
|High performance transaction storage and retrieval system for commodity computing environments
|Jul 31, 2003
|Jul 15, 2004
|Lech Pofelski
|Backup system and method of generating a checkpoint for a database
|Dec 8, 2003
|Jun 9, 2005
|Dell Products L.P.
|Transaction transfer during a failover of a cluster controller
|Dec 19, 2003
|Jul 14, 2005
|International Business Machines Corp.
|Autonomic method to resume multi-threaded preload imaging process
|Mar 23, 2005
|Aug 25, 2005
|Tokyo Electron Limited
|Method and apparatus for the monitoring and control of a semiconductor manufacturing process
|Mar 31, 2004
|Oct 6, 2005
|Randal Paul S
|System and method for a consistency check of a database backup
|Sep 2, 2004
|Jan 5, 2006
|Jun Nagasawa
|Method and system for data processing with data replication for the same
|Aug 13, 2004
|Feb 16, 2006
|Seakr Engineering, Incorporated
|Soft error detection and recovery
|Aug 31, 2004
|Mar 2, 2006
|Cherkauer Kevin J
|Sending log records directly from log buffer
|Oct 19, 2004
|Apr 27, 2006
|International Business Machines Corporation
|Management of global counters in transactions
|Dec 22, 2004
|Jun 22, 2006
|International Business Machines Corporation
|Log shipping data replication with parallel log writing and log shipping at the primary site
|Mar 3, 2005
|Sep 7, 2006
|Hu Wei M
|Lost write detection and repair
|Mar 6, 2006
|Sep 7, 2006
|Fusionsoft Co., Ltd.
|Method of efficiently recovering database
|Mar 25, 2005
|Sep 28, 2006
|International Business Machines Corporation
|Log stream validation in log shipping data replication systems
|May 1, 2006
|Nov 2, 2006
|Portal Software, Inc.
|Revenue management systems and methods
|Aug 30, 2006
|Dec 28, 2006
|Haruo Yoshida
|Recording apparatus, file management method, program for file management method, and recording medium having program for file management method recorded thereon
|Sep 9, 2005
|Apr 12, 2007
|International Business Machines Corporation
|Method and system to recover from control block hangs in a heterogenous multiprocessor environment
|Jun 28, 2006
|Apr 26, 2007
|Alexander Rockel
|Revenue management system and method
|Dec 16, 2010
|Nov 17, 2011
|Salesforce.Com, Inc.
|Providing administrative capabilities in a multi-tenant database environment
|Apr 26, 2012
|Microsoft Corporation
|Database data recovery system and method
|Jan 7, 2011
|Jul 12, 2012
|Microsoft Corporation
|Transaction support for distributed data
|Dec 19, 2012
|Jun 19, 2014
|Microsoft Corporation
|Main-memory database checkpointing
|Apr 3, 2014
|May 28, 2015
|Juchang Lee
|Transaction Private Log Buffering for High Performance of Transaction Processing
|Nov 12, 2007
|Mar 18, 2009
|Synchronization process of distributed control system operator stations EMS memory database structure storage
|Aug 30, 2012
|May 8, 2013
|Sap Ag
|Database log parallelization
|May 6, 2004
|Nov 18, 2004
|Aptare Inc
|A system to manage and store backup and recovery meta data
|Mar 31, 2004
|Nov 18, 2004
|Oracle Int Corp
|Flashback database
|Nov 21, 2006
|Jun 14, 2007
|Apparatus, system, and method for continuously protecting data
|Jun 16, 2010
|Jan 6, 2011
|International Business Machines Corporation
|Disaster recovery for databases
|U.S. Classification
|International Classification
|Cooperative Classification
|European Classification
|Aug 19, 2002
Owner name: TRANSACT IN MEMORY, INC., KOREA, REPUBLIC OF
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:CHA, SANG K.;LEE, JUCHANG;KIM,KIHONG;REEL/FRAME:013212/0077
Effective date: 20020813
|Feb 15, 2007
Owner name: SAP AG, GERMANY
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:TIM SYSTEM INC.;REEL/FRAME:018907/0825
Effective date: 20060613
|Aug 19, 2008
|Certificate of correction
|May 27, 2011
|Fee payment
Year of fee payment: 4
|Aug 26, 2014
Owner name: SAP SE, GERMANY
Free format text: CHANGE OF NAME;ASSIGNOR:SAP AG;REEL/FRAME:033625/0334
Effective date: 20140707
|May 29, 2015
|Fee payment
Year of fee payment: 8


text:
The new Yardbarker app is here! Install on your iPhone, iPad or Android device. You can also find us in Google Newsstand.
Nick Watney hopeful he is turning the corner
LOS ANGELES (AP) One month into the new year, Nick Watney already is putting last year out of his mind. Watney was hampered early by a back injury that forced him out of the final round at Doral, and the birth of his first child the following week. But those should have been merely speed bumps. He didn't need to elaborate on a season that produced only two top 10s and knocked him...
Via AP on Fox
February 17, 2015 | Discuss
PGA Tour: Weekend Watch at Pebble Beach
Brandt Snedeker This weekend at the AT&T Pebble Beach National Pro-Am is looking promising for a lot of golfers. Brandt Snedeker, my dark horse pick is in the lead after two rounds, he is sharing that lead with Matt Jones at -12. There are ten players at -1o close behind the two golfers, chasing for the title. Among those ten golfers are the likes of Billy Horschel and Daniel...
Via isportsweb.com
February 14, 2015 | Discuss
PGA Tour: Watney takes 1-stroke lead at Wyndham
Not many players birdied the tricky final hole at the Wyndham Championship on Saturday. Nick Watney did to top leaderboard. Watney made a 20-foot birdie putt from the right edge of the green on the par-4 18th for a 5-undee 65 and a one-stroke lead. The five-time PGA Tour winner had a 14-under 196 total with only one bogey through three trips around Sedgefield Country Club. "I...
Via AP on Fox
August 16, 2014 | Discuss
Sergio Garcia Surprises Golfers, Hands Out Free TaylorMade Drivers
It's a special day when any amateur golfer tees it up at Bethpage Black on Long Island, N.Y., home of the 2002 U.S. Open (won by Tiger Woods), '09 U.S. Open (Lucas Glover), '12 Barclays (Nick Watney), and future home of the 2019 PGA Championship and 2024 Ryder Cup. It ranks No. 8 on Golf Digest's list of America's 100 Greatest Public Courses.Might the day be even better...
Via Back9Network
June 30, 2014 | Discuss
Check Out Nick Watney's MGM Grand Golf Bag
You may have noticed something unique about Nick Watney during the first round of the 2014 WGC – Cadillac Championship at Doral. No, it wasn't a flashy new flat-brim hat or even a new pair of shoes.Take a look at this sweet-ass MGM Grand Las Vegas tour staff bag!If the logo of the massively popular MGM Grand casino wasn't enough to impress you, the Masters-inspired color palate...
Via Back9Network
March 08, 2014 | Discuss
PGA Tour 2013: And the Winner of the BMW Championship is…..
Nick Watney Hitting a Shot From the Fairway During Sunday's Round at the BMW Championship. Zach Johnson Holding the Trophy After a Fantastic Display of Golf on Sunday at the BMW Championship. Accurate golfers have been the hot players at this year's BMW Championship as we've seen Jim Furyk, Steve Stricker and Zach Johnson dominate the field but only one can come out on top...
Via The GP Golf Report
September 16, 2013 | Discuss
Golf News
Delivered to your inbox
You'll also receive Yardbarker's daily Top 10, featuring the best sports stories from around the web. Customize your newsletter to get articles on your favorite sports and teams. And the best part? It's free!
Follow us on Twitter

subset: commoncrawl


text:
DEFINITION of 'Shared National Credit Program'
A program formed in 1977 to provide an efficient and consistent review and classification of any large syndicated loan. The program was established by the board of governors of the Federal Reserve System, the Federal Deposit Insurance Corporation, and the Office of the Comptroller of the Currency.
BREAKING DOWN 'Shared National Credit Program'
The program currently covers any loan or loan commitment of at least $20 million that is shared by three or more supervised institutions. The agencies' review is conducted annually, usually in mid-year. In its 2009 review, the Federal Reserve said that "criticized assets," which included large syndicated loans classified in categories including doubtful or loss, reached $642 billion, up from $373 billion in the preceding year.
Office Of The Comptroller Of The ...A U.S. federal agency that serves to charter, regulate and supervise ...
Credit FacilityA type of loan made in a business or corporate finance context. ...
Federal Deposit Insurance Corporation ...The U.S. corporation insuring deposits in the U.S. against bank ...
Syndicated LoanA loan offered by a group of lenders (called a syndicate) who ...
Federal Reserve System - FRSThe central bank of the United States. The Fed, as it is commonly ...
Bank DepositsMoney placed into a banking institution for safekeeping. Bank ...
Credit Default Swaps: What Happens In A Credit Event?The credit crisis of 2008 prompted important changes to the settlement of credit default swaps.
When Financial Crisis Strikes The Bank Of Mom And DadIf you really want your kids to learn to be financially responsible adults, it's time to stop giving them money.
Debt ReckoningLearn about debt ratios and how to use them to assess a company's financial health. You could save a lot of money!
Investing Basics
How to Use Boring CDs to DiversifyMarkets are volatile and are in for more punishment. CDs can help investors earn some interest while they're waiting out the storm.
Breaking Down the Federal Reserve's Dual MandateThe Fed has been tasked with a dual mandate by Congress to achieve monetary stability. We explain what the dual mandate is and what it means.
Investing News
Are Stocks Cheap Now? Nope. And Here's WhyAre stocks cheap right now? Be wary of those who are telling you what you want to hear. Here's why.
Should the Fed Be More Worried About Asset Bubbles?While the Fed should be concerned that assets bubbles might impact economic stability, monetary policy is not the best tool to mitigate this threat.
Mutual Funds & ETFs
Mutual Funds Are Not FDIC Insured: Here Is WhyFind out why mutual funds are not insured by the FDIC, including why the FDIC was created and how to minimize your risk with educated mutual fund investments.
What's the 1913 Federal Reserve Act?The 1913 Federal Reserve Act was a pivotal congressional act that helped establish the Federal Reserve System as it exists today. It is one of the United States financial system's most influential ...
Investing News
Could a Rate Hike Send Stocks Higher?A rate hike would certainly alter the investment scene, but would it be for the better or worse?
Who bears the risk of bad debts in securitization?Bad debts arise when borrowers default on their loans. This is one of the primary risks associated with securitized assets, ... Read Full Answer >>
Why are mutual funds not FDIC-insured?Mutual funds are not Federal Deposit Insurance Corporation (FDIC)-insured because money invested in funds are not considered ... Read Full Answer >>
How is the Federal Reserve audited?Contrary to conventional wisdom, the Federal Reserve is extensively audited. Politicians on the left and right of a populist ... Read Full Answer >>
Who decides when to print money in the US?The U.S. Treasury decides to print money in the United States as it owns and operates printing presses. However, the Federal ... Read Full Answer >>
Why do some people claim the Federal Reserve is unconstitutional?The U.S. Constitution does not mention the need for a central bank, nor does it explicitly grant the government the power ... Read Full Answer >>
How can the federal reserve increase aggregate demand?The Federal Reserve can increase aggregate demand in indirect ways by lowering interest rates. Aggregate demand is a measure ... Read Full Answer >>


text:
TerraTrike will be attending the 2 largest recumbent shows this year Recumbent Cycle Convention will be held onSeptember 25-27at the Sharonville Convention Center, Cincinnati, Ohio. Since 2011, the annual Recumbent Cycle-Con has become the world's first and largest annual gathering of all the major recumbent bike, trike and tandem manufactu...Read More
The Climate Ride produces charitable multi-day rides. Participants make an extraordinary contribution to a cause they care about. They help to provide much-needed financial support, raising awareness, engaging other humans, and helping to build a national network of supporters. The reward for this commitment is the experience of a lifetime throu...Read More


text:
Don't forget to enter the #FPBFF Contest for your chance to explore the Big Apple! (…) Read more
Tagged: nyc
Hidden Gem Of New York: Liz Christy Community GardenAug 18, 2015
A hidden gem within NYC that might just inspire you to explore your own backyard… (…) Read more
Suite Caroline Salon X Free People: Learn How To Style The Ear Cuff To Hair ChainAug 14, 2015
We partnered up with one of our favorite NYC salons to bring you something exciting all month long! (…) Read more
In The Kitchen With Lael CakesMay 19, 2015
As I walked into Emily's large Brooklyn loft a few weeks back, my initial reaction was, "Man, I'm doing it all wrong." (…) Read more
Connecting With Nature In A Jungle Made Of ConcreteApr 21, 2015
As creatures of the earth, we all have a profound connection to Mother Nature. (…) Read more
In Her Closet: Just AnotherApr 19, 2015
Lisa Dengler is doing a lot of things right. (…) Read more
How Do You Unplug?Apr 14, 2015
The past few times I've been in NYC, it's been primarily for work. (…) Read more
8 Ways To Make A Rented Apartment Feel Like HomeApr 02, 2015
For lots of us, it feels as if we're constantly moving to a new place to call home. In the past five years, I've moved eight times. (…) Read more
Designer Interview: Phyllis+Rosie JewelryMar 26, 2015
A pretty special jewelry collaboration just hit our site, and I'm here to introduce you all to the talent behind Phyllis+ Rosie Jewelry. (…) Read more
Embracing Small-Space LivingMar 19, 2015
A year ago, I was living in Philly. (…) Read more
Commuter ProblemsMar 13, 2015
So this morning, I'll be heading up to NYC. (…) Read more
A Weekend in NYC With Claire GeistMar 10, 2015
When it comes to tackling NYC and its neighboring boroughs, there's no one better to turn to than a local. (…) Read more


text:
Església de la Puríssima Concepció I Assumpció de Nostra Senyora
Platja de l'Eixample
Silom Spa
A touch of the Orient is the promise in this city spa, where you can combine a series of Thai, aromatic and other massages with aroma...
La Casa Elizalde
Cultural organisation La Casa Elizalde runs several Barcelona walks (which generally occupy a Saturday morning and cost up to €9.25 per...
Les Gens Que J'Aime
This intimate basement relic of the 1960s follows a deceptively simple formula: chilled jazz music in the background, minimal lighting...
Ca la Nuri
With its white and grey tones, and schools of silver fish shapes seemingly skudding like wind-driven clouds along one wall, this is a...
Carrer de Roger de Llúria 70 · interesting places nearby
Església de la Puríssima Concepció I Assumpció de Nostra Senyora information
One hardly expects to run into a medieval church on the grid-pattern streets of the late-19th-century city extension, yet that is just what this is. Transferred stone by stone from the old centre in 1871–88, this 14th-century church has a pretty 16th-century cloister with a peaceful garden.
Behind is a Romanesque-Gothic bell tower (11th to 16th century), moved from another old town church that didn't survive, Església de Sant Miquel. This is one of a handful of such old churches shifted willy-nilly from their original locations to L'Eixample.


text:
Thursday, 29 January 2009
Is Carla Connor the new Elsie Tanner?
Has Coronation Street finally found the new Elsie Tanner in Carla Connor? I'm starting to think that perhaps they might have.
Missed Corrie this week?
Catch up with the Coronation Street Weekly Updates.
5 comments:
I know what you mean but I won't be able to agree until we see Carla in her slip putting on her lippy.
- 30 January 2009 at 10:06
I agree full heartedly. Carla Connor (refuse to call her Gordon) is my favourite Corrie character and does echo the legend that is Elsie Tanner. Long may Carla be in the soap. Going to miss her whilst Alison is on maternity leave, she's an asset to the show.
- 30 January 2009 at 11:57
- Tvor said...
Possibly. Sunny Jim....nobody wears slips much these days, you'll have to be content with a basque ;) Gotta move with the times and after all, she *does* sell underwear!
- 30 January 2009 at 12:07
- Flaming Nora said...
And her nighty was used as an almost murder weapon!
- 30 January 2009 at 13:48
- Reginald Fah-Fah said...
The Earl, Reginald thinks Carla Connor is absolutely marvellous and a shining star of Corrie!!!
I say...comparing Carla with Elsie Tanner is a big mistake!Carla has her own shine and is a star in her own right!
Myself and James McTaggart were out shooting and fox hunting at the weekend, as well as discussing our frivolous pastimes, James mentioned to me that he had seen a classic television soap called 'Coronation Street' and found himself 'a huge fan of Carla Connor!'
We agreed to disagree while pigeon shooting! Reginald's favourite Corrie character is Michelle (Kym Marsh)because the Rovers Return wouldn't be the same without her!
They are both 'Strong, dark, mouthy, pushy, full of herself and nobody's fool, ready to put the world to rights with another lashing of lipstick.' Every Tories'perfect wife!
- 31 August 2010 at 01:05
Our Top 10 Blog Posts in last 7 days (automatically updated)
Coronation Street has an exciting opportunity for a qualified Tutor to work on an ad-hoc, freelance basis with children aged 5-16. The Tu...
Oh, it's a cutie. Wet nose, shiny coat, eager to please and lovable. But enough about Steve McDonald, what about his new little dog th...
It's all over the news today, and ITV News has also carried the story that Simon Gregson will be taking several months off due to ill...
If you're wondering where you've seen Callum's mum Marion in Coronation Street before, then wonder no more. Marion is played...
ITV has confirmed that Ryan Thomas is to leave Coronation Street next year. Having been on the soap for 15 years, playing builder Jason ...
In this week's TV Times magazine there's a good interview with Jennie McAlpine, who plays Fiz Stape in Coronation Street. Jennie...
We're often asked where you can find the clothes fans' have seen characters wearing on Coronation Street. And with the help of t...
Here in one handy little blog post is everything you need to know about Coronation Street's upcoming live episode. When is it ...
I seem to be swimming against the tide of popular opinion on this one. Most people I've encountered on Twitter seem to love the blos...
Sunday 13th September at 7pm for one hour JASON HAS ONE THING ON HIS MIND: REVENGE Jason gets Callum's address from David and heads out ...


text: I was hoping it will be easy to get a churascaria around here, as the Gauches are known for great BBQ but happens to be the opposite......found more of Lanches around. This restaurant is well know around here in NH, so any taxi you flag should be able to get you there. The reception at the... read more


text:
Saturday, October 10
By 92Q, on Mar 14, 2012 at 9:06am
Full Story »
By Kelson, on Mar 14, 2012 at 9:05am
By Dre Johnson, on Mar 14, 2012 at 8:13am
Copyright ©2015,
Interactive One, LLC . All Rights Reserved
Powered by WordPress.com VIP
Get every new post delivered to your Inbox.
Join 15,043 other followers


text:
2040 Vision
Strategic Framework
For the last 25 years, we have been successfully helping people in the West build the communities they want to live in while preserving the West that brought them here. Discover our vision for our next 25 years.
Arizona Public Media
I-11 Corridor Initiative
In a recent PBS piece, our Sun Corridor Program Director talks about Sonoran's involvement in the project's development. He goes on to discuss the Environmental Impact Statement process and the critical concept of integrated infrastructure. As always, it is vitally important that stakeholders work together from start to finish.
Resources - State Trust Lands
Western Lands and Communities State Trust Lands page
State Trust Land Conservation Profiles
The Sonoran Institute has long been involved in the area of state trust land conservation and efforts to effectively manage the assets on behalf of the beneficiaries, primarily k-12 public education. With funding from the Nina Mason Pulliam Charitable Trust and in collaboration with conservation groups, municipalities, and counties across Arizona, the Sonoran Institute has assembled state trust lands suitable for conservation into a single database. The resulting profiles focus on conservation values. Political values are left for another day.
The conservation profiles included here have been identified by stakeholders around the state as suitable for conservation for various reasons: preservation of wildlife corridors; archeological and historical gems; preservation of ancient cultures; and large landscape conservation to name a few.
The four-page conservation profiles are available to inform policy makers, conservation advocates, land managers, and neighbors who love their special corner of Arizona and want to make the case for inclusion in conservation efforts. These profiles explain the importance and location of the land and the reason it deserves protection. Thanks to the local organizations that care deeply about these parcels and provided information, photos, and their support.
The maps on the following pages show the conservation areas by section within the state – Northern Arizona, Central Arizona, and Southern Arizona.


text:
Verified ID
Email AddressVerified
Phone Number(▒▒▒) ▒▒▒- ▒▒ 98
Reviewed24 Reviews
Offline IDPersonal Info
About Me
- School
- Baruch college
- Work
- teacher
- Languages
- English
- David's Airbnb Symbol
Hey, I'm David!
New York, New York, United States · Member since March 2014
You have reported this user. Undo?
A fun loving couple with a passion to help out others in times of need. We try to make the most of our guests' time along with ours by being accommodating at all times within our capacity. We are gay and prefer open minded, fun, clean and trust worthy people since it is our home. We like kids as long as they are well behaved and if you have a small dog that is house trained, and you have a bed for him/her, you can bring it along too.


text:
Fixed PriceDelivery by December 20, 2013
I am looking for a talented developer/ team of developers who can help us make some upgrades to our analytics application that is currently up and running on dekeanalytics.com.
This is a cloud based analytics app written in C# that is plugged into the Google Analytics API.
It gives users the ability to see certain analytic website data about their website.
We need to make some upgrades to the app.
Please only apply if you have experience using the Google Analytics API and have built cloud based applications before.
When you apply please tell me exactly what applications you have built in the past that are similar to this one.
After you apply I will give you access to the app so you can take a look around.
Price is negotiable. Will have more work for you if this project works out.
About the Client
United States
Wheat Ridge 07:38 AM
Jobs Posted
83% Hire Rate, 1 Open Job
Total Spent
40 Hires, 3 Active
Avg Hourly Rate Paid
10 Hours


text:
Launched in May 2011 the Bristol Music Trust is a new and exciting independent charity with two distinct aims.
The first is to manage the operations of Colston Hall, Bristol's largest concert venue, where people have been enjoying music since 1867; developing a diverse musical programme, leading a major capital refurbishment programme and securing its commercial success.The Hall presents a wonderful opportunity to build a diverse Music & Arts Centre; comprising a traditional concert hall, a smaller intimate hall and an inspiring newly built foyer.
The second aim is to drive forward music across Bristol; working in partnership with city wide organisations; leading on musical education, commissioning new works, supporting local musicians, and promoting music across all communities to build on the vibrant talent already in the city.
Much more, we want the Bristol music scene to stand out with its own unique culture and internationally renowned reputation.
Bristol Music Trust's vision is to put music at the heart of Bristol life, and to lead a strategic, holistic, citywide approach to promoting music so that it flourishes through performance, composition, participation and education.
Work experience/student placements/Internships – please note that we don't currently have the capacity for a work experience/student placement programme, and whilst this may change in the future, we are not currently taking enquiries around work experience or student placements. We also don't currently have any internship opportunities available.
Speculative job enquiries – unfortunately, we are unable to accept CVs or speculative applications/enquiries for roles, all available opportunities are listed here on our jobs page, we regret that we are also unable to keep enquiries on file, but you can follow us on Facebook and twitter where such updates are announced . For any other enquiries not answered here, please email email@example.com
Bristol Music Trust/Colston Hall – Box Office Assistants x 2
£7.86 per hour / zero hours
Bristol Music Trust is recruiting for Box Office Assistants, to join our busy Customer Service team at Colston Hall, the South-West's largest concert hall and Bristol's major music venue. You will be required to provide an efficient and effective box office service to hirers, promoters and the general public, providing high standards of customer care at all times. The successful candidate will be proactive, enthusiastic, and well-organised, ideally with previous experience of working in a box office, and an intuitive understanding of what makes for great customer service. You will be expected to work flexibly, including evenings and weekends, to meet the needs of the venue.
To apply for this role please read the supporting documents and complete and return the Bristol Music Trust application form by 9.00am on Monday 12th October 2015 to email@example.com.
Box office Assistant JD with TCs – Sep 2015
Equal Opportunities Monitoring Form


text:
We provide platforms for independent artists to master and promote their craft; Learn more about the event independent artists call home!
We bring you behind the scenes and into the minds of the independent artists starting to make noise.
We provide affordable recording solutions to independent artists who are looking to get a quick start.
Welcome to WordPress. This is your first post. Edit or delete it, then start blogging!Read More


text:
G. Gleason 1G., and W. H. Porter '98, will speak at the annual convention of the Y. M. C. A. of the colleges of Massachusetts and Rhode Island, to be held at Worcester tomorrow afternoon. The other Harvard delegates to the convention are: E. B. Barton '98, J. P. Sheffield 3L., L. E. Hubbard '98, E. F. Phillips '98, H. J. Bennett '98, and H. C. Ward '99.
Read more in NewsB. A. A. Indoor Meeting.
Want to keep up with breaking news? Get the latest, straight to your inbox.


text:
GMP sewer work planned on Merchants RowBy Gordon Dritschilo
Staff Writer | July 13,2013A section of Merchants Row will become one lane by night over the next couple of weeks.
The closure will accommodate the installation of a new water and sewer connection at Green Mountain Powers Energy Innovation Center in the former Eastman building. GMP officials stressed that one lane will remain open at all times and the closure will only happen at night.
Theyll fill (the hole) back in every night or day, I guess at the end of the work shift, said Steve Costello, GMPs vice president of generation and energy innovation. Itll be gravel it wont be paved but people will be able to travel up and down the street as normal during the day.
The work is scheduled to begin Monday and expected to take up to two weeks. Work will run from 9 p.m. to 6 a.m., except for Tuesday, when it will begin at 6 p.m.
Costello said the existing water and sewer connection is roughly 100 years old.
The current sewer pipe is an old clay pipe we dont really want to rely on, Costello said.
The hookup will also get equipment designed to prevent sewage backflow, thus easing fears about flooding in the building.
Costello said the existing water connection is a 1-inch pipe which will not be able to meet the buildings water needs.
GMP bought the long-vacant art supply store which some city officials described as one of the most blighted locations in Rutland last year. When finished, the Energy Innovation Center will house several of the utilitys employees along with representatives of Efficiency Vermont and NeighborWorks of Western Vermont.
A museum component and a college lecture series is planned for the space, which is slated to open in the fall.
email@example.com IN This Just In
Federal agents
arrest residents
Two Rutland residents pleaded innocent in... Full Story
KILLINGTON — Killington Resort says it will be hosting a women's World Cup giant... Full Story
BARRE — A Granite City roofer with a lengthy criminal history has been sentenced to up ... Full Story
- Most Popular
- Most Emailed


text:
Top Definition
When he bent over you and his shorts slid down you could see he had grass in the gutter.
by Cracker Squirrel April 14, 2014
Type your email address below to get our free Urban Word of the Day every morning!
Emails are sent from firstname.lastname@example.com. We'll never spam you.×


text:
$1,350—$1,650 1—2 bed 1170 Galapago Street, Denver CO, 80204
from $1,350 1—2 bed
1170 Galapago Street, Lincoln Park, Denver, 80204
Commute to Downtown Englewood
The rent at Marquis at Parkway Apartments ranges from $1,350 for a one bedroom to a $1,650 two bedroom. Apartments in this building are about 41% less expensive than the Lincoln Park average. This is a pet-friendly apartment.
Marquis at Parkway Apartments is a 11 minute walk from the D-Line Light Rail D-Line, the F-Line Light Rail F-Line and the H-Line Light Rail H-Line at the Colfax at Auraria Station stop.
This apartment is in the Lincoln Park neighborhood in Denver. Nearby parks include Sunken Gardens Park, Lincoln Park and Civic Center Historic District.
Marquis at Parkway Apartments has a Walk Score of 90 out of 100. This location is a Walker's Paradise so daily errands do not require a car.
1 & 2 Bedroom Luxury Apartment Homes, Free Tanning Beds, Free Shuttle Service Downtown & to Auraria Campus, Oversized Balconies with Fantastic Views, Huge Fitness Center - FREE FITNESS CLASSES, Billiards and Lounge Area, Enclosed Dog Park with Agility Course, Movie Theater, Walk, Bike, Ride Everywhere, Close to I-25, 6th Ave. West , Busline & Lt. Rail, Business Center, Resort Style Pool and Hot Tub
Pets allowed:
Lincoln Park, Denver, 80204
Daily errands do not require a car.
Transit is convenient for most trips.
Flat as a pancake, excellent bike lanes.
About your score
Explore how far you can travel by car, bus, bike and foot from Marquis at Parkway Apartments.
Marquis at Parkway Apartments has excellent transit which means transit is convenient for most trips. Car sharing is available from RelayRides and Hertz On Demand.
Rail lines:
D-Line Light Rail D-Line
0.4 mi
H-Line Light Rail H-Line
0.4 mi
F-Line Light Rail F-Line
0.4 mi
E-Line Light Rail E-Line
0.6 mi
C-Line Light Rail C-Line
0.6 mi
W-Line Light Rail West Line W-Line
0.9 mi
Bus lines:
9 West 10th Avenue
0.1 mi
1 West 1st Avenue
0.2 mi
52 W 52nd Avenue / South Bannock
0.3 mi
16 West Colfax Avenue
0.3 mi
16L West Colfax Limited
0.3 mi
Car shares:
Bike shares:
1170 Galapago Street compared to all of Denver.
Average crime area.
Average crime area.
How do you feel walking near 1170 Galapago Street?
Thanks for your feedback!
View all Lincoln Park apartments on a map.
Search for homes in Denver or find a Denver real estate agent who knows walkable neighborhoods. Links will open on our partner site Redfin.com.
Marquis at Parkway Apartments is in the Lincoln Park neighborhood. Lincoln Park is the 5th most walkable neighborhood in Denver with a neighborhood Walk Score of 84.
Moving to Denver from another city? View our moving to Denver guide.


text:
Real Canadian Superstore – No Tax Event(April 11th)
Real Canadian Superstore NO TAX EVENT
Save the taxes on almost everything at your local RCSS location.
Event takes place on April 11th, 2015 only.
What you will find at your local Real Canadian Superstore
President's Choice Products: Find a wide range of PC products that will meet your high standards as well as your pocketbook.
Joe Fresh: Apparel for men, ladies, and kids. Find the latest styles and trends at great prices. Fantastic money-saving coupons are offered from time to time as well.
Photolab: Find prints, photo books and greeting cards. Ongoing promotions to help you save on your photo needs. Capture those memories here.
Gift Of Choice: Gift Cards are the way to go. Give the gift of choice. No more re-gifting when they get one of these Bon Apetite cards. Gift cards can be used at the following locations: Kelseys, Milestones, Swiss Chalet, Montana's and Harveys.
President's Choice Cooking School: Offers kids and adult cooking classes. If you would like to learn a little more about cooking ideas, get tips and great recipes, this program is for you. Enjoy a day or night out with great company and great food.
View upcoming classes or register here.
Latest coupons from Real Canadian Superstore
The newest coupon that is being offered right now is;
Save $1.00 on any Egg Creations or Egg Creations French Toast Blend product(500g)
Love the Superstore on Portage Avenue.
Love the Canadian Super store of Lethbridge


text:
you are viewing a single comment's thread.
Why do you keep saying that it will head much lower short term?
because the shorts are definitely now in control and the company just sits back and says nothing to respond. they just don't know what they are doing when it comes to IR.
Management doesn't have a clue how to deal with the investment community and their head of IR is a very young man who is very green and has no experience to be able to help.
The flip side though is that the company could come out with a major announcement at any time that will probably send shares to between $8 and $10 instantly.
I wouldn't want to be short Ampio because there is too little to gain and way to much to lose.
On the other hand, the investors who are long have very little to risk and huge returns to gain in a very short timeframe.
Ampio is a screaming buy but it would be a much better buy if management knew how to handle IR like the successfully traded companies.
Sentiment: Strong Buy
Based on your lack of faith in management's competency their deteriorating cash position should be of increasing concern to you
AMPE for sure should have hired you to run IR. Your SA article that pumped this to 6 was great, plus, as you say, you know more about the company than the analysts who follow it.
Strange, though that by your own admission you got this: "My understanding is that the final phase III trial has already been completed and that they are only waiting for the FDA's response. I believe these trials took place in the U.S. recently." wrong.....
Maybe AMPE should fly you back out so you can write another pump piece......


text:
Click any word in a definition or example to find the entry for that word
90% of the time, speakers of English use just 7,500 words in speech and writing. These words appear in red, and are graded with stars. One-star words are frequent, two-star words are more frequent, and three-star words are the most frequent.
The thesaurus of synonyms and related words is fully integrated into the dictionary. Click on the thesaurus category heading under the button in an entry to see the synonyms and related words for that meaning.more
Could you just give us a general description of the work you do?
Her predictions are so general that they could apply to anybody.
We are prepared to discuss the deal in general terms, but the details are confidential.
As a general rule, shorter sentences are easier to understand.
I get the general impression that they aren't very happy.
He has this general attitude of superiority toward his colleagues.
Your general physical condition seems remarkably good.
a good general introduction to linguistics
science programmes that are accessible to the general viewer
The building and plumbing work will be done by a general contractor.
This is the British English definition of general. View American English definition of general.
Change your default dictionary to American English.
in an online conversation, repeatedly asking a person questions which suggest that you are interested in what they are talking about, but are actually intended to annoy themBuzzWord Article
Open Dictionary
an electric skateboardadd a word
More submissionsview entries
A must for anyone with an interest in the changing face of language. The Macmillan Dictionary blog explores English as it is spoken around the world today.global English and language change from our blog