text:
|Publication number
|Publication type
|Application number
|Publication date
|Jan 2, 2007
|Filing date
|Aug 28, 2003
|Priority date
|Aug 29, 2002
|Fee status
|Also published as
|Publication number
|Abhinav Jalan, Retesh Chadha
|Original Assignee
|Indian Institute Of Information Technology
|Export Citation
|BiBTeX, EndNote, RefMan
|Patent Citations (17), Non-Patent Citations (19), Referenced by (44), Classifications (21), Legal Events (4)
|External Links: USPTO, USPTO Assignment, Espacenet
The present invention relates to the field of parallel and distributed computing and to the field of object oriented software design. More specifically, it relates to a method and system for executing a sequential program in parallel with automatic fault tolerance.
Parallel computing is the use of two or more processors (computers) in combination to solve a single problem. Parallel computing involves writing concurrent programs. In writing a concurrent program, the programmer has to figure out how to break the problem into pieces, and has to figure out how the pieces relate to each other.
There are different ways of running a concurrent program on an execution platform. The program may be executed on a uni-processor machine, for example, using a threading system, or on a parallel computer comprising plurality of processors. While concurrency is a semantic property of a program, parallelism pertains to its implementation as determined by the compiler, libraries and other systems software.
Distributed computing is a specialized form of parallel computing in which the processing nodes (computers) are physically distributed and are interconnected. These interconnections may or may not be reliable. However, the computers must cooperate in order to maintain some shared state to work on the given problem. Distributed Computing harnesses the idle processing cycles of the workstations on the network and thus makes them available for working on computationally intensive problems that would otherwise require a supercomputer or workstation/server cluster to solve.
The development of parallel programs is a tedious task and involves numerous skills other than the general programming skills on the part of the programmer. The development involves division of the problem into parallel executable fragments and synchronizing the parallel executing processes with each other in order to produce a proper result. The programmer must also handle transfer of data from one process to another. Furthermore, distributed systems used to run the parallel programs are unreliable and prone to system shutdowns and network failures. In order to make a system fault tolerant, a programmer has to encode the necessary complex instructions in the system to recover from a failure, which takes a lot of extra effort.
Debugging of a concurrent program is even more tedious than building it. In a method of debugging a concurrent program concurrent program is serialized and the programmer is provided with the tools to debug it as a sequential program [U.S. Pat. No. 5,860,009, Naoshi Uchihira, Shinichi Honiden, Toshibumi Seki, "Hypersequential Programming: A New Way to Develop Concurrent Programs"]. After debugging of the program, its concurrency is restored back and is executed as parallel processes.
There are many approaches for achieving parallelism. In one approach called as data parallelism, in order to add parallelism to a programming language, the language is extended, that is, the compiler is extended to recognize the new language constructs. While such newer extended languages provide enhanced performance they are limited by a lack of portability between operating systems. Moreover, the programmer needs to learn the new language constructs. Parallel compilers are usually based on data parallel programming model. High Performance FORTRAN (HPF) and Data Parallel C Extensions (DPCE) support data programming. In this model, distribution of data at a very high level is specified using parallel variables. This approach is also limited by the type of tasks that can be parallelized and cannot be used for general purpose parallel computation.
The other approach to design and implement a parallel program, rather than using a new extended compiler, is to use Message Passing Libraries (MPL). In this model, processes communicate by sending and receiving messages. Data transfer requires cooperative operations to be performed by each process (a send operation must have a matching receive). Programming with message passing is done by linking with and making calls to libraries which manage the data exchange between processors. MPI (Message Passing Interface) and PVM (Parallel Virtual Machine) are standard message passing libraries providing concurrency among processes [Message Passing Interface Forum, "MPI: A message-passing interface standard"]. In these libraries, it is the programmer's responsibility to resolve data dependencies and avoid deadlocks and race conditions.
In other approach, called control parallelism or task parallelism or functional parallelism, work is divided into multiple threads. In this model different tasks are executed at the same time. It requires all subroutines to be thread-safe. OpenMP is based on this model [Leonardo Dagum, Ramesh Menon, "OpenMP: An Industry-Standard API for Shared-Memory Programming"]. OpenMP uses the fork-join approach of parallel execution with threads. Routines for locking the data are to be used by the programmer for handling synchronization. OpenMP FORTRAN implementations are not required to check for dependencies, conflicts, deadlocks, race conditions or other problems that result from incorrect program execution. TOPC (Task Oriented Parallel C\C++) is a software library built on master slave model [G. Cooperman, "TOP-C: a task-oriented parallel C interface"].
It is now well accepted that the object paradigm provides good foundations for the new challenges in concurrent and distributed computing. Object notions, rooted in the data-abstraction principle and the message-passing metaphor, are strong enough to structure and encapsulate modules of computation and flexible enough to match various granularities of software and hardware architectures. Programs structured around objects are modular, and easier to understand and modify. However, in addition to these advantages, integrating concurrency and synchronization with data abstraction offers benefits that are particular to parallel programming. As a result, many object-based concurrent, parallel, or distributed models, languages, or system architectures have been proposed like Abcl, Actel, Actor, Argus, Concurrent Smalltalk, COOL, Eiffel, Emerald, Hybrid, Nexus, Parmars, POOL-T, Presto [Jean-Pierre Briot, Rachid Guerraoui, Klaus-Peter Lohr, "Concurrency and distribution in object-oriented programming"].
Several object oriented implementations for supporting basic concurrency exist. Various encapsulations for providing an object oriented interface over the basic operating system services for process management have evolved. Synchronization has been simplified through the use of synchronized procedures associated with each object. Library provided as part of software development kit of JAVA is a perfect example. Some modern implementations have made introducing concurrency in the program much easier through active object. Active objects provide a view of object as a process. Concurrency in active objects can then be viewed as the parallel execution resulting from the creation of these active objects and their interactions with one another. Calls to active objects act like message exchange between two processes. Similar to the active objects are actor-based languages. Actors are self-executing objects, each having a unique address and a mailbox. Actors communicate by sending messages asynchronously and executing concurrently.
All the above methods of achieving concurrency involve organization of the program along interacting parallel executing processes. They involve division of either data or tasks into parallel executing fragments. Processes-have to communicate and are required to be synchronized with each other in order to get right results.
Work has also been done in the direction of conversion a sequential program to parallel executable code [U.S. Pat. Nos. 5,088,034 and 5,452,461, W. Blume, R. Doallo, R. Eigenmann, J. Grout, J. Hoeflinger, T. Lawrence, J. Lee, D. Padua, Y. Paek, B. Pottenger, L. Rauchwerger, and Peng Tu, "Parallel Programming with Polaris", M. W. Hall, J. M. Anderson, S. P. Amarasinghe, B. R. Murphy, S. W. Liao, E. Bugnion, and M. S. Lam, "Maximizing multiprocessor performance with the SUIF compiler", C. Polychronopoulos, M. Girkar, M. R. Haghighat, C.-L. Lee, B. Leung, and D. Schouten, "Parafrase-2: a new generation parallelizing compiler" and P. Banerjee, J. A. Chandy, M. Gupta, E. W. Hodges IV, J. G. Holm, A. Lain, D. J. Palermo, S. Ramaswamy, and E. Su., "The PARADIGM Compiler for Distributed-Memory Multicomputers"]. A compiler is extended to find data dependencies between parts of the program and independent parts are made to execute in parallel. Other methods of parallelization include 'Inter-procedural analyses and ' Symbolic analyses. A suitable parallel code is built and put in place of the sequential code. However, these compilers are faced with an inherent limitation in their capability to find parallel part in a program containing operation directly on memory addresses. Dependence or non-dependence between any two parts is sometimes known only at runtime or depends on program input; these compilers, which rely on prior division of the program, cannot resolve them. Most of these implementations are long way to come into actual practical use. Some other implementations require the programmer to select and link components in a dataflow graph to specify dependency between them [U.S. Pat. No. 5,999,729], while some others resolve the dependency by having questions and answers between a system and the user [U.S. Pat. No. 6,253,371]. Another method to bring parallelism to a sequential program is by allocating instructions to the processors depending on the memory address contained in the operand [U.S. Pat. No. 5,619,680]. However, the parallelism achieved is instruction level and its scalability is highly limited.
In one method of execution of sequential program in parallel is to serially label the steps to access or modify data variables in accordance with the intended sequence of the whole sequential program and then executing parts in parallel under a control system such that all memory accessing and modifying operations are executed only in the sequential order [U.S. Pat. No. 5,832,272]. The system however got huge overheads and requires special hardware is not applicable on programs with complex memory accesses.
In some object oriented systems, future objects are used for parallelization of a sequential program [U.S. Pat. No. 5,404,521, Neelakantan Sundaresan, "Extending the Standard Template Library for Parallelism in Coir<Futures>", Rohit Chandra, Anoop Gupta, John L. Hennessy, "COOL: An Object-Based Language for Parallel Programming"]. In such systems, computation intensive subroutines are called asynchronously. Asynchronous call means that the called procedure executes in parallel with rest of the program. The result of the subroutine is made to be stored in a 'future' object after it returns. The main program continues to run with the subroutine concurrently. If the main program accesses the future object before subroutine returns, the program simply blocks to await the desired result. The mechanism does not allow access to future object holding return value of a remote procedure in the main program or passing it as an argument to another procedure. Thus by using futures, parts of a sequential program are executed in parallel, with implicit synchronization. The programmer does not handle the synchronization in the future object; it is inbuilt. Communication is also implicit in form of arguments (from main program to subroutine) and futures (from subroutine to main program).
However, many limitations remain in the usage of future objects and to the level of parallelism achievable through them.
Also, futures are incompatible with references, especially on a distributed memory system. References play an important role in any programming language system. Nonconformity of any architecture to references severely limits its capability to be used in various complex systems. In distributed systems, support for references involves the complex task of not only the synchronization over referred data, but also of maintaining the linkage structure (how data are connected), which is subject to changes during program execution, together with providing parallelism.
In all, use of futures does not bring true parallelism to a sequential program. Futures can be used to execute in parallel simple procedures only, which take only [in] arguments ([inout] not supported, i.e. C++ pointer or reference arguments are unsupported) and in which arguments themselves do not contain references.
Another popular mechanism for distributed computing is through RPC. The semantics of RPC are identical to the semantics of the traditional procedure call, except that while a normal procedure call takes place between procedures of a single process in the same memory space, RPC takes place between a client and a server process on different systems connected through a network. Like a normal procedure call, RPC is a synchronous operation, i.e., the client process is blocked until processing by the server is complete. To gain parallelism RPC has been extended to asynchronous calls also. Futures can be employed for synchronization in a limited manner [U.S. Pat. No. 5,999,987, Murat Karaorman, John Bruno, "Introducing concurrency to a sequential language"].
The object of this invention is to obviate the above mentioned drawbacks and provide a method and system for executing a sequential program in parallel with automatic fault tolerance.
The second object of the invention is to simplify the development of parallel and distributed programs that can be run on various platforms. The platform can be a multi processor computer, a high speed cluster dedicated to parallel computing, a network of computers, where people do their day to day work, or even the Internet.
Yet another object of the invention is to enable users to exploit their existing computer hardware to solve much larger problems at minimal additional cost. It is easy to observe that any computer on a network is not fully utilized at all times except during short processing bursts that require more resources for execution. The invention allows idle processing cycles of such computers to be used to execute a large application distributed over the network.
These and above objects are achieved by providing a system and method to allow a programmer to write a sequential program containing parallel procedures without any extra effort required to write a parallel program using traditional programming platforms. The programmer can specify procedures in a sequential to be executed in parallel using special language constructs, and the system automatically executes them in parallel with the rest of the program. The system does so by dividing the program into jobs that can be executed in parallel to each other. Data types of the arguments to the parallel procedure are changed to synchronous objects that are special wrapper classes around the regular data types provided by the language platform. The system implicitly takes care of inter-process communication and synchronization over the argument values required between various executing threads and processes of the program and therefore the programmer also does not require taking care of any deadlock in the program. For the purpose of debugging, the system has an implicit feature that program can be debugged in sequential mode.
The system supports references in the argument variables. It provides synchronization over the referred data also. The system also supports alterations in the linkage structure of the data (how data are connected with each other through references) and maintains the synchronization together with linkage structure modifications.
The system resolves the dependencies (including those generated on account of references) at run-time and parallelization is done to the level where actual dependency is encountered. Therefore the system is much more efficient in parallelizing programs.
The system may be built by extending the compiler, making the task of specifying the parallel procedures and its arguments simpler. However, the system is easily practicable on a language system without any compiler extensions through the library mechanism and therefore can be easily ported to any platform.
The invention utilizes the untapped computing power of the nodes connected to the network by transferring the execution of the jobs to these nodes. Various threads and processes of the program can be executed on any or all the nodes connected to the network. The system can automatically detect resources (including new processing nodes) on the network and add them to its resource pool as and when any new resource is added to the network. The system can automatically select any node on the network having sufficient free resources to execute the job or the programmer can use special language constructs to specify a particular node for job execution. The program can be assigned a low priority so that it runs unobtrusively in the background not affecting other tasks being carried out on the host machine.
In addition, the invention provides automatic fault tolerance, i.e., no extra effort is required on the part of the programmer to write any code to recover from any fault that may occur on the network; the system does that automatically. The fault may be a process failure, a processing node failure or communication link fault. If a node fails during execution of a program all the jobs given to it are transferred to some other machine unobtrusively. Thus, a program can be executed on network are used for other activities, where computers go on and off regularly.
The invention will now be described with reference to the following figures:
The invention pertains to the area of parallel and distributed computing. It simplifies the way parallel and distributed applications are programmed. Simplification is achieved by obviating the need to write a parallel program in form of parallel executing processes. The system executes in parallel parts of a sequential program. Writing a parallel application involves great effort and skills on the side of programmer. Parallel programming involves extra programming load of data communication and inter-process synchronization. Program is to be divided into a set of collaborating processes or threads. Data is to be divided and then transferred from one process to another in an efficient manner, so as to maximize the performance. Synchronization between the processes is also to be taken care of. In case, the parallel execution is to be distributed over to other computers on the network, these computers must also be configured.
The invention provides a framework for executing a sequential program in parallel. To write a parallel application using the framework, parts to be executed in parallel are specified in the form of separate procedures marked as 'parallel'. Therefore, the granularity, i.e. the smallest part of the program that can be divided into parallel running fragments, is the part of the program written in parallel procedures. During execution, these parallel procedures are executed in parallel to rest of the program.
For achieving high performance, a parallel application is often distributed to various processing nodes in a cluster or on a network. For the parallel application to run successfully, these nodes need to be kept in isolation and have to be fully dedicated only for the parallel applications. A failure in any of the nodes may lead to the failure of the complete application. If the programmer has to incorporate fault tolerance, he has to further program by setting up the check points and writing code for the recovery from each checkpoint on a failure between the checkpoints.
The invention in addition to providing a framework for executing a sequential program in parallel also provides for automatic fault tolerance. The onus of writing code for fault recovery is not on the programmer but is taken care of by the framework itself. Automatic recovery from the fault without any extra effort on the part of the programmer is supported. (Many systems to detect faults already exist).
The framework supports dynamic resource (mostly processing units) addition. Applications made over this framework can also be executed over a network in use for other activities. Processing nodes that are used for other activities can also be taken in. As the system is fault tolerant, these nodes can go up and down not resulting in complete failure. On each failure system performance degrades gracefully. The parts of the parallel application can be executed as low priority processes running only when a node is idle and thus not interfering with the other activities on the node. These features make the invention one of the most cost-effective supercomputer that can be put on already built network of corporate offices, research and educational institutes.
Parallel procedure is specified at calling point by specifying parallel procedure ID and its arguments to the system. ID of the parallel procedure may be in the form of address of the procedure, or it can be any other ID that can be resolved to the address of the parallel procedure. It can be the name of the parallel procedure in a programming language system that allows resolving of procedures through their names. It can be any other symbol that the programming language system is able to resolve with the procedure. It can also be any other symbol explicitly bound in the program to the procedure. A symbol can be bound with the parallel procedure using a symbol table. In systems with extended compiler parallel procedure can be specified by declaring it as 'parallel'.
Data communication between the parallel executing processes is through the arguments to the procedure. Argument variables are replaced by objects of special wrapper synchronous classes to the argument data. These wrapper classes provide all the functionality and properties of argument data type and in addition take care of synchronization involved in transfer of argument variables' value. On running the program, the system executes the specified procedures in parallel to the rest of the program, giving the same result it would have given had the program executed sequentially.
Parallel procedures can be specified in the program to the framework in the following ways:
In the code, job object is created by 'CreateJob' function, in which the ID of the parallel procedure is specified as 'ProcedureID'. Then the arguments 'Argument1', 'Argument2' and 'Argument3' are specified by passing them to job through 'AddArgument'. The second parameter to 'AddArgument' is the type of the argument to parallel procedure. Type of the argument is discussed later in the specification. The job is then set to execute in parallel with the call to 'ExecuteParallel'.
The job object is not required to be created by the programmer explicitly, the function 'ExecuteParallel' automatically creates it. The function is called by specifying the ID of the parallel procedure, that is, ProcedureID and its arguments 'Argument1', 'Argument2' and 'Argument3'. Second argument to 'ExecuteParallel' is number of arguments to parallel procedure.
class ParallelProcedureclass1 : ParallelProcedureClass
SynchronousInt * Argument1;
SynchronousFloat * Argument2;
SynchronousInt * Argument3;
void ParallelProcedure( );
/* in the main procedure */
ParallelProcedureClass1 ParallelProcedureObject;
ParallelProcedureObject.Argument1 = new SynchronousInt;
ParallelProcedureObject.Argument2 = new SynchronousFloat;
ParallelProcedureObject.Argument3 = new SynchronousInt;
A new class 'ParallelProcedureClass1' is derived from the predefined class 'ParallelProcedureClass'. The code of the parallel procedure is written in the class's overridden member function 'ParallelProcedure' and the arguments to the parallel procedure 'Argument1', 'Argument2' and 'Agument3' are specified as the class's member variables. The parallel procedure is executed by creating the class's object and calling the 'ExecuteParallel' function of the parallel procedure object.
In the given code, the parallel procedure 'ParallelProcedure1' is specified by declaring it with the keyword 'parallel'. 'ParallelProcedure1' is then called like any other procedure in the program and it gets executed in parallel.
A parallel procedure executes as a slave process to the master process from which it is called. From within parallel procedures, any number of parallel procedures can be called. The calling process then acts as master to all the new processes formed. This is shown in
Within one parallel procedure, another parallel procedure can also be called with the newly spawned slave process also executing in parallel to rest of the program. Thus, a slave process of a process may be a master of others. This is shown in
Data transfer between the master and slave processes is done through arguments passed to the parallel procedure. Data needed in the parallel procedure is put on the argument variables in the master process and any data needed from the parallel procedure is written back on the argument variables in the parallel procedure. Arguments can be passed as follows:
Type of the arguments can also be determined at compile time. The compiler may determine whether or not arguments can be modified in the parallel procedure through a simple analysis of the control flow graph of the parallel procedure. If any of the argument cannot be modified, the argument type may be set as non-returnable else returnable.
For the program to give correct results, values of variables at any instance of accessing them must be consistent with values they would have, had the program been executed sequentially with no parallel procedure executing in parallel. In the parallel model, synchronization of data between master and slave processes is also to be taken care of. Variables sent as arguments in the parallel procedure may not have 'right values' in them at all instances during the execution. The 'right value' is what it would have got on sequential run of the program. For example, a parallel procedure is called and its 'returnable' argument is modified in it. As the parallel procedures execute in parallel to rest of the program, the argument variable may be accessed outside the parallel procedure before its completion. It may be accessed in the master process or it may be sent to some other parallel procedure also. Now, the value of the variable is inconsistent with the value it should have been, had the program been executed sequentially. Its value must be first updated back from the modifying parallel procedure and then only it must be accessed in other threads.
Following are the cases to be dealt with to achieve synchronization in the aforementioned model of parallel execution of a sequential program:
The framework provides automatic synchronization without requiring any extra input in the program. No effort regarding data dependencies or synchronization between different processes is required to be made by the programmer; only the way arguments are passed to the parallel procedure is changed. Arguments are passed in the form of special synchronous objects, which encapsulate the data carried by the arguments. These synchronous objects act as wrapper objects around argument data.
Synchronous objects are instances of 'SynchronousObject' classes. If data of type 'signed integer' is to be sent as an argument to a parallel procedure, an object of 'SynchronousInt' class is instantiated and its reference is passed as argument to the parallel procedure. 'SynchronousInt' is wrapper class of 'signed integer'. New 'SynchronousObject' classes are made by deriving new classes from the 'SynchronousObjectBase' class provided by the framework. In-built wrapper classes of some basic data types are derived from 'SynchronousObjectBase' class only and custom 'SynchronousObject' classes can be similarly constructed. For example, if a complex number is to be sent to the parallel procedure, a new class is made by deriving it from the 'SynchronousObjectBase' class containing a real and an imaginary part.
The 'SynchronousObject' classes also support object creation on fly. In the slave process, the same copy of the object is not operated upon; rather a 'child synchronous object' of each synchronous object, which is passed as argument, is created. Child object in the slave process is a copy of the parent object passed as argument in the master process. All operations on the argument objects are done on slave process's local copy. If the object is passed as 'returnable' then after completion of the parallel procedure, object in the master process is updated back.
These wrapper classes provide all the properties and functionality of data-types, and in addition also provide for synchronization of data contained therein. Objects of these classes are built with the ability to block the thread accessing them, if they do not have the 'right values'. Objects release the blocked thread after they are updated with 'right values'. Thus, on accessing an argument in master or slave process, it is ensured that accessing process gets the 'right value'.
If a synchronous object does not contain 'right value', it is 'unavailable' to the process. The process gets blocked on accessing an 'unavailable' object. An object becomes 'available' to the process only after it is updated with the 'right value'. Following are the rules adhered to by the system to determine the availability and value transfer of a synchronous object:
Child synchronous object value may be transferred to the master process before the completion of the parallel procedure also; it can be transferred right after the execution of last modifying instruction on the object in the parallel procedure. The last modifying instruction may be determined by specifying it by the programmer by setting the object as 'over'; it may be determined by the compiler or it may be taken as the point of completion of the parallel procedure.
Since the accessing thread is to be blocked, if the object is 'unavailable', direct access to the data contained in a synchronous object is not given. Every object access is accompanied by a lock and a subsequent unlock of an 'accession lock' associated with each synchronous object. This 'accession lock' can be any synchronization object, like a semaphore, provided by the operating system. When the object is unavailable, its 'accession lock' is not free and process gets blocked when trying to lock it; resumed only after the object becomes 'available'. Thus, it is ensured that on accessing a synchronous object, the process gets the 'right value' only.
To implement the 'availability' of object as mentioned, a concept of 'ownership' of object is employed. In the process in which the object is created, the ownership of the object is with the process itself or with any of the jobs to which the object has been passed as argument or rather added to the job. Object ownership signifies who can modify the object in the process. Ownership with the process signifies that object is available to the process for modification. If object's ownership is with some job, then the object value is to be updated from the corresponding slave process and the object is 'unavailable' to the master process. In the slave process, ownership of an object can be with the 'source' corresponding to when object value is not received from master process.
Rules for ownership transfer in master process are as follows:
In slave process, following rules are followed:
In the ownership transfers involving job in which object has been added as non-returnable, an optimization is done. These jobs are bypassed during the ownership transfer and object values are transferred to the corresponding slave processes.
Implementation of ownership involves that every object maintains a list of an 'object info' structure. The structure contains a reference to the object; it also contains the status of the object with respect to the job, that is, whether the object values have been 'sent' to the slave process or not. If object is passed as 'returnable', the structure also stores if object value have been 'received' back from the slave process or not. Separate 'object info' lists are maintained for 'returnable' and 'non-returnable' objects. A single list can also be used with additional information about each object's type 'returnable' or 'non-returnable' in each element of the list. 'Object info' also has a pointer to the object's object info in the 'next owner' of the object in the ownership queue. If there is no next owner, the pointer is null. Object itself contains the pointer to the object's 'object info' in the current owner. Ownership queue of an object is nothing but this linking of the 'object info' corresponding to the 'current owner' with the 'object info' corresponding to 'next owner' and so on. In the queue, jobs get inserted and deleted by just changing the linkage of 'object info'.
#define DYNAMIC_CREATEClassName
static ClassInfo ClassInfoClassName::CreateObject;
virtual ClassInfo& GetClassInfo
return ClassInfo;
static ClassName * CreateObject
return new ClassName;
In order to transfer object value from one process to another, every synchronous object need to have procedures for serialization and deserialization. Serialization is the process of copying objects on a data stream. Deserialization is the process of updating the object by copying data from the data stream to the object variables. In the 'serialize' procedure, data in each variable in the object is copied to the 'argument stream' one by one. In the 'deserialize' procedure object is updated back by copying data from the 'argument stream' to the object variables in the same order. 'Argument stream' acts as an abstraction to the data stream. 'Serialize' is called when sending the value of a synchronous object and 'deserialize' is called when receiving and updating the value of the synchronous object. A sample code of serialize and deserialize procedure is given next.
Class NewSynchronousObject : BaseSynchronousObject
int Var1;
float Var2;
int Var3;
void serialize (stream ArgumentStream)
void deserialize (Stream ArgumentStream)
On a call to a parallel procedure, a job object is constructed as shown in detail in
After the creation of job object, a slave process is created and ID and arguments' 'class info' are transferred. Subsequently, child synchronous objects are created, their references are put on the process stack and parallel procedure is called. Parallel procedure takes its own course to complete.
After each synchronous object's class info has been transferred to the slave process and child synchronous objects are created, values of the objects owned by the job are sent. Object values of the rest of the objects are transferred to the slave process as the job in the master process gets their ownership. After the completion of the parallel procedure, value of data in 'returnable' objects is sent back to the master process to update back the parent objects.
Entire communication between any two nodes takes place though separate 'communication threads', and the job executing processes including master process are never blocked over I/O requests for master—slave communication. Communication threads support parallel transfer of data to more than one node, drastically improving distribution efficiency.
In many cases, a process may get terminated before it transfers object values to other processes slave or master. Following are the possibilities:
It is therefore the requirement, that object transferred must be maintained even after the process termination or object going out of scope. A synchronous object for the above reasons is not deleted until all object transfers have taken place through the process. Memory management of the synchronous objects is therefore not handled by the programmer. Synchronous object are not created on stack as they are required to be kept even after termination of a process or ending of scope of a program block. One way to accomplish it is that after a synchronous object is marked as 'over' explicitly by the programmer, when it is no longer required in the program; the system deletes the object after all the transfer processes have taken place involving the object.
The same result can also be achieved though automatic garbage collection. When an object becomes inaccessible in the program any references to the object ceases to remain, the object is automatically garbage collected. Since the framework also stores the object reference if the object is to be required in any object transfer, the object is deleted only after object transfers. In conventional systems, not supporting automatic garbage collection, automatic garbage collection can be achieved through a well known technique of reference counting using 'smart pointers'. Garbage collection through 'smart pointers' can be easily accomplished though software libraries. Each object maintains its reference count, which is adjusted for any reference modification. As the count becomes zero, the object is deleted
Similar to arguments to a parallel procedure, the procedure's return value can also be synchronized. Synchronization on return value can be implemented on similar lines as that on 'returnable' objects, wherein a parallel procedure returns a reference to a synchronous object only. Container synchronous object of the return value can be specified to the job and the object can be added to the job in the same manner as a 'returnable' object is added. On completion of the parallel procedure, value of the return object is sent to the master process.
However, the above model is not capable of handling sequential programs wherein arguments may contain references or pointers to some other objects. If an object containing references to the data outside the object is sent to another process, then the data must also be transferred along as the data may be accessed in the slave process through the object. This transfer of data poses the problem of synchronization of this data also. This data can be referred through the references that are not in the object. After the data has been transferred to the slave process, one of its references remains in the first process. Two copies of the data, one in the master process and other in the slave process, can be modified independent of each other. Resulting in inconsistencies in data in one of the processes involved. However, if this data is also in the form of synchronous object, the synchronization problem gets solved automatically.
Synchronous objects may therefore contain references to other synchronous objects only. An object is reachable from another object, if the reference to the former object is contained in the latter or any other object reachable from the later object. In simple terms, a synchronous object A is reachable from another synchronous object B, if object A can be accessed in the program through object B. For instance, in
When an object is added to a job all the objects reachable from it are also added to the job. Similarly, if an object is transferred from one job to another, all the objects reachable from it are also to be transferred. As and when a synchronous object becomes owned by the job, its value is transferred to the slave process. An object, reachable from a 'returnable' object is treated as returnable, and an object, reachable from a 'non-returnable' object is treated as a 'non-returnable' in the job. Object values of all the reachable objects, which have been added as 'returnable', are also sent back to the master process on completion of the parallel procedure, corresponding objects in the master process are updated, and their ownership is released by the job. Ownership transfer of objects reachable from 'non-returnable' objects is also dealt with as that of 'non-returnable' objects.
When adding an object to a job, the framework does not have prior information about the references contained in the object, therefore it uses 'serialize' and 'deserialize' procedures of the object to gain this information. In 'serialize' and 'deserialize' procedure of a synchronous object, the references to synchronous objects contained in it are also passed to the argument stream in the same order. When the objects' 'serialize' procedure is called, references to synchronous objects, having their references in the object are added to the argument stream. Argument stream does not copy the reference value possibly the address of the object in the memory to any data stream, rather it passes the reference to the job. The object is added to the job in form of a new entry in the 'object info' list and object's serialize procedure is called again to gain information about the references further contained in this object. Thus, all objects that are reachable from an argument object get serialized recursively. After all objects reachable from any one argument object have been added, the same process is repeated for remaining objects. The following sample code illustrates a new synchronous object class that contains references to other synchronous objects.
Class NewSynchronousObject : BaseSynchronousObject
SynchronousInt * Var1;
SynchronousFloat * Var2;
SynchronousInt * Var3;
void serialize (Stream ArgumentStream)
void deserialize (Stream ArgumentStream)
ArgumentStream.deserialize Var1;
ArgumentStream.deseriaiize Var2;
ArgumentStream.deserialize Var3;
In the code given above, the class NewSynchronousObjectClass is derived from BaseSynchronousObjectClass and contains other synchronous objects 'Var1', 'Var2' and 'Var3'. In the 'serialize' function, all the synchronous objects are passed to the argument stream and in 'deserialize' function, all the objects references are updated back from the argument stream.
To those in the ordinary skill in the art, it would be apparent that instead of 'serialize' and 'deserialize' a different set of procedure could also have been used. The functionality of fetching the linkage information of the object at run-time may also be added by the compiler.
'Class info' of each object in the 'object info' list is transferred to the slave process along with parallel procedure ID.
References in the synchronous object are initialized to newly created objects. This is also done in the same recursive way, but this time through 'deserialize' procedure. In 'deserialize' procedure, argument stream updates the object references to the newly created objects. 'Deserialize' is called in the same order as serialize. Argument stream takes object reference from the 'received object list' and updates the references contained in the synchronous objects. As the order of deserialization is same as that of serialization and order of object in both 'received object list' in the slave process and 'object info' list in the corresponding job object in the master process is same, same objects similar to parent objects are linked together.
Object values are transferred from master process to the slave process as job gets their ownership. Value of all the objects available to the master process at the point of calling the parallel procedure is sent right away as job gets objects' ownership as they are added to the job. Along with each object's value its position in the 'object info' list in the job is also transferred. In the slave process, after getting the position of object same in the 'received object list', object value is deserialized to the corresponding object. Similarly, after the completion of parallel procedure, each 'returnable' object's value along with the object position in the 'received object list' is transferred back to the master process, as object becomes available to the slave process.
A reference to a synchronous object is link between the two objects. Link is directional and directed towards the referred object. When an object gets transferred from one process to another along with objects reachable from it, the linkage structure of the synchronous object is maintained, i.e. all the paths through which one object was reachable from another is maintained. To maintain this linkage structure, cross-references and circular references in the objects have to be specially taken care of. If the objects are assumed to be nodes of a graph and links connecting these nodes to be directional edges, then the cross reference corresponds to two paths to reach from one to another. Circular reference corresponds to a loop in the graph. A special consideration while handling cross-references and circular references in linkage structure is to avoid sending an object, reachable from other objects through more than one path, more than once.
Cross-reference can be illustrated in
Circular reference linkage structures are illustrated in
Adding objects to a job with cross-references or circular references is similar to adding an object as argument to the job more than once. If an object is added twice to a job with same type either both as 'returnable' or both as non-returnable, only one copy of object must be formed in the slave process. All the references of the object in the slave process must point to the same copy of the object. In case of cross-reference and circular reference, the object is added to job twice. In order to deal with multiple additions of objects, 'object info' also stores the first occurrence of same type of the object in the job object info list. When transferring object info list, first occurrence of the object is also transferred along with their class info. When creating child objects in the slave process, if the first occurrence position is not position of the object entry in the list, new instance of the object is not made corresponding to this entry and in the 'received object list', reference in the first occurrence position is copied at this position. Thus during deserialization, same object gets linked and linkage structure is maintained. If object is added twice with different types then two different copies are made in the slave process, as the two types have very different functionality and cannot be mixed by the framework.
Another problem that needs to be addressed is when linkage structure changes are made in the program. The above model of recursively adding the objects to the job fails when the references contained in the synchronous objects are altered. The model adds all the objects reachable from an added object irrespective of whether the object is available to be owned by the job at that point or not. References in the object may be modified before the object ownership is transferred to the job making the linkage structure in the process corresponding to the job inconsistent with what it should have been. The model even does not reflect linkage structure changes in one process to other. No information regarding any change of linkage structure is passed back to the master process, if some changes have been made in the slave process and therefore the changes do not get prorogated to the jobs next in the ownership queue. If a new object is created in the slave process and gets included in the linkage structure of objects, which are to be updated back to master process, neither the new object's class info nor object's value or any info regarding linkage structure alterations is sent back. Modifications in the references contained in a synchronous object received from master process may include:
Even if linkage information is sent back and corresponding new objects are created and linked in the master process, changes in the linkage structure may still not get reflected in the jobs ahead in the 'ownership queue'. For instance, a synchronous object A containing a reference to another object B is added to two jobs J1 and J2 as 'returnable'. A is now unavailable to J2 till J1 releases its ownership. In J1, reference to B may be modified to a new object C, created in J1 only. However in J2, along with A, B also got added, resulting in inconsistency in the linkage structure of A.
To deal with, referred objects are not added to a job till the job gets the ownership of the referring object. Delaying addition of referred objects precludes their premature addition, when the linkage structure is subject to changes, till the ownership of the referring object is transferred to the job.
This process of transferring ownership of objects and adding referred objects is termed as 'update' of the job further in the specification. 'Update' process is done when any of the added objects becomes available to be owned by the job.
Master process maintains a 'jobs to be updated list'. 'Jobs to be updated list' contains jobs that have at least one object in their 'new objects available list'. i.e. those that are required to be updated. The system updates all the jobs one by one, removing their entry from the list as they are processed. Only when a job's 'new objects available list' is empty and an object gets added to it, the job gets added to the 'jobs to be updated list'. Thus, the job can get added to the list only when it is created and any of new objects added to the job is available to be owned by the job, or when an object's value update is received and object becomes available to be owned by the job.
The new approach of delayed addition of objects poses a new problem of keeping the order of ownership transfer of the objects consistent with the order of call to parallel procedure. Ownership of an object may get transferred to a job and subsequently its value may also be transferred before it is transferred to another job whose parallel procedure is called before. The ownership must actually be transferred to the jobs in the calling order of parallel procedures. To give an example, ownership of a synchronous object A containing a reference to another synchronous object B is released by job J1. If object B is already added to job J3 and object A to job J2, with J2 being called before J3, then B gets added to the 'new objects available list' of J3 and A gets added to the 'new objects available list' of J2, and both J2 and J3 are added to the 'jobs to be updated list'. Now, if J3 gets updated before J2, the ownership of B gets transferred to J3, and in J2, ownership of A gets transferred and object B gets added to the job, leading to inconsistency in ownership transfer and consequently in program flow also. However, if J2 would have been updated before J3, then after ownership of object A would have been transferred to J2, object B would have been added to J2, as well to the 'new object available list' of J2 and its corresponding entry in the 'new object available list' of J3 would have been removed, making B unavailable to J3 to own. Only after ownership of object B would have been released by J2, B's ownership would have transferred to J3. Thus by ensuring the update of jobs in the order of call to parallel procedures and by removing an object from the 'new object available list' of a job if the object gets added to another job whose parallel procedure is called before, the order of ownership transfer is kept consistent with the order of call to parallel procedures.
To keep the order of update consistent with the order of call to parallel procedure, jobs are numbered by 'call number' in the order in which their parallel procedures are called. Numbering is done during the creation of job object. And updating of the jobs is done in the order in which their parallel procedures are called, that is, in the order of their 'call number'.
In the case demonstrated above, it has been assumed that a job does not release ownership of an object till it releases the ownership of all the objects from which it is reachable. The setup would fail if ownership of an object were released before the release of ownership of the object from which it is reachable. Those with ordinary skill in the art would appreciate that the model already takes care of the condition, given the last modifying instruction is taken to be end of the parallel procedure. After the completion of parallel procedure, the slave process transfers object value as it becomes available to the process. Those with ordinary skill in the art would also be able to recognize the recursive relationship of the sequence of transfer of value of two objects when one is reachable from other and would be able to easily see that value of the object reachable from other cannot be transferred before the value of other. Similarly from master process also, value of the object reachable from other object cannot be transferred before the value of other.
Master process also maintains an 'objects to be freed list'. When value update of an object is received in the master process, the object is not freed, i.e., object does not become accessible to the process even if no next owner exists. Instead, it is added to the 'objects to be freed list'. After receiving an object value and release of ownership of the object by the corresponding job, all the jobs in 'jobs to be updated list' are updated. When no job is left to be updated, all the objects in 'objects to be freed list' are made 'available' to the master process. Reason for not freeing the object is that in the process of updating jobs, the object may be added to another job and therefore must not be made 'available' to the master process. When the job gets added to another job, it gets removed from 'objects to be freed list'.
This process is illustrated in the activity diagram in
To further elaborate the process, an example is provided as follows: Suppose, synchronous objects A and B are sent to job J1 as returnable, and object A is also added to job J2. In job J1, B becomes reachable from A, that is, its reference is put in A. After job J1 releases the ownership of A and B. B is added to the 'objects to be freed list' and is not made available to the master process. However, object A is added to the 'new objects available list' of J2. So, J2 gets added to the 'Jobs to be updated list'. When J2 is updated, object B also gets added to job J2 resulting in it being removed from the 'objects to be freed list'. Thus, B is not made available to the master process.
The process of insertion of job in the 'ownership queue' of the object after addition of the object in the job is illustrated in
As the objects referred from an unavailable object are not added to the job till the object becomes available, they cannot be added to the job recursively depth first. First, all the objects referred from an object in the 'new object available list' are added to the job and then new objects added to the 'new objects available list' are opened further. This is effectively adding of the objects in breadth-first manner.
Order in which objects are added to the job is not fixed, as it depends on whether the referring object is available or not at the point of its addition to the job. It is also subject to changes in the linkage structure of the object. Since the order of addition of objects is not fixed, an extra data structure containing the linkage information of objects is stored in the 'object info'. This data structure is a 'link info list'. An element in the 'link info list' in object's 'object info' contains position of referred objects in the 'object info' list.
As a new object is added to the job, an entry of the object's position is made in the 'link info' list in the 'object info' of the referring object. If an object has already been added to the job and is again passed to the argument stream case when object is referred from more than one object, then only the entry corresponding to the new link is made in the 'link info list' of the referring object's 'object info' in the job. For example, if an object A contains a reference to object B. Both A and B are passed as arguments to job J1. If B gets added to the job before A and A becomes available to the job later, then rather than adding B again A refers to B, only an entry corresponding to the position of B in the job is made in the 'link info list' in A's 'object info'.
While returning objects back to master process from the slave process, a similar approach to that employed in master process is applied. In the slave process, a 'return object list' of objects is built on the similar lines as 'object info' list in each job in the master process; each element of the list is similar to 'object info'. After completion of the parallel procedure or after the last modifying instruction on the object is executed, all the 'returnable' objects received from the master process their references are present in received object list are 'added' to the list. A job object corresponding to master process 'source' is made as object values are now to be transferred to master process. 'Return object list' is actually an 'object info' list in 'source' job object, which is always the last owner of the object in the slave process. As the ownership of an object is transferred to 'source' job, object value is transferred to the master process. Similar to the jobs in master process, a 'new objects available list' is also maintained in 'source' object. 'Source' object also gets added to 'jobs to be updated list' and is always updated in the last as other jobs.
In the new model proposed, not only object update transfer but 'class info' transfer or object list update and object value transfer also includes link info transfer may take place multiple times. As ownership of any object gets transferred to a job, which may get transferred when constructing the job or when ownership of an object is released by a job just ahead in the 'ownership queue', new objects may get added to the job. Object value of the newly owned object and list update corresponding to newly added objects are transferred. Always, object list update transfer precedes object value transfer. This ensures that position contained in the 'link-info list' of each object is a valid object position in the receiving process, that is, the referred objects are created in the receiving process before references in the referring object are initialized.
Object list updates in master and slave process is explained as follows. The process of transferring object list updates from master process is shown in
Object value updates in master and slave process is explained as follows. After an object is removed from the 'new objects available list', object is added to 'objects to be transferred list'. 'Objects to be transferred list' is the list of objects that have become available to the job and their value is to be transferred to the slave process. Transferring of values of all the objects from the master process is illustrated in
Distributed systems are prone to network failures and computer shutdowns and crashes. The invention provides automatic fault tolerance, wherein the programmer is relieved from writing code for fault detection and recovery. Fault detection is trivial and can be easily done by pinging nodes at regular intervals; the master process can also verify if the slave process to which it transferred parallel procedure execution is running as normal or not.
There can be three types of faults: process failure faults, node failure fault or a network failure fault. 'Process failure' is abnormal termination of any of the executing processes. If a process gets abnormally terminated, the master process can throw the job again. Thus, a recovery from a 'process failure' is possible by simply rescheduling the process again on the same or any other node. A 'node failure' can be recovered from, by rescheduling each of the processes executing on it to other nodes.
Since 'returnable' objects are not 'available' to the master process till the completion of parallel procedure, their data content remains intact. However, 'non-returnable' objects can be modified. Therefore, to support re-execution of parallel procedure, value of objects added as 'non-returnable' need to be stored linkage information is already stored in the object info. Whenever a value of an object, which is added as non-returnable to the job, is transferred to the slave process, it is serialized to a buffer also. During rescheduling, object's value is transferred directly from the buffer.
A fault may also occur when updating a 'returnable' object on master process while the object is updated only partially, destroying the data content of the returnable object to be used for rescheduling. Therefore, 'returnable' objects also need to be serialized to the buffer. Another scenario where 'returnable' objects need to be serialized is when a fault occurs between two object updates on the master process. Few objects may get completely updated and their ownership may be released by the job, while the rest of the objects may still be owned by the job, waiting to be updated. After ownership is released by the job, the objects in the master process may be modified. However, for rescheduling the slave processes in the event of some failure, the object's value that needs to be transferred is the one that it had at the point of calling the parallel procedure initially. Therefore, objects added as 'returnable' are also serialized in the buffer.
Rescheduling processes may lead to the problem of double updates of 'returnable' objects in the master process. One update can be from the original process and the other from the rescheduled process. To handle this problem, the job object maintains information about the status 'sent' or 'received' of the object; if the object has been 'received', new object updates are discarded.
Storing each object in the buffer may lead to unacceptably high memory usage. If an object is not modified between two calls to parallel procedures and has been added as non-returnable to the first job, only a single copy of the buffer may be used by both the jobs, to reduce the memory usage, since the object value does not get modified between the calls to the parallel procedures. As a further extension, an 'object value buffer' may be shared between the jobs in which the object has been added as 'non-returnable' consecutively and the first job in which object is added as 'returnable' thereafter, given the object is not modified in the process between these calls.
To implement common 'object value buffer' when object has same value in more than one call to parallel procedures, buffer of each object is kept separate in the job, with each 'object info' holding a pointer to the 'object value buffer' corresponding to the job. In case of a common buffer, they point to the same buffer. Beside the 'object info', object itself also contains pointer to the 'object value buffer' made during last serialization operation on the object. In addition, object also contains a flag, which is 'true' only when 'object value buffer' pointed by the object contains object's current value, that is, object has not been modified after the last serialization operation. When any modification is made to the object, the flag is set to 'false' and when object is serialized, when added as 'non-returnable', the flag is set as 'true', and when added as 'returnable', the flag is set 'false'. If object value is sent with the flag being set as 'true', new buffer is not made and the same buffer is reused by making the 'object info' in the new job to refer to the same buffer; buffer pointer is simply copied from the object. If the flag is set as 'false', a new copy of the buffer is made and subsequently, the buffer pointers, both in object and 'object info', are changed to the new buffer.
In the depicted example, a node executing the master process 28.2 is connected to the network 28.1. In addition, other nodes 28.3, 28.4 and 28.5 are also connected to network 28.1. These nodes 28.3, 28.4 and 28.5 may be, for example, personal computers, network computers or any other type of processing unit. In the depicted example, the node executing the master process 28.2 creates the jobs and sends them to the slaves 28.3 and 28.4 for execution. The slave nodes 28.3 and 28.4 may further transfer the execution of new jobs to some other slave nodes 28.5 or among themselves.
Network data processing system may include additional computing systems and other devices not shown in the depicted example. In the depicted example, network data processing system is built over a typical Local Area Network with network 28.1 representing a collection of networks and gateways that use the TCP/IP suite of protocols to communicate with one another. Of course, network data processing system may also be implemented over a number of different types of networks, such as for example, a high-speed network connecting coupled memory—processor sets, a wide area network WAN, or even may be extended to the Internet.
Those of ordinary skill in the art will appreciate that the means to execute the program are instructions for operating on the nodes of the system. The means are capable of existing in an embedded form within the hardware of the node or may be embodied on various computer readable media. The computer readable media may take the form of coded formats that are decoded for actual use in a particular information processing system. Computer program means or a computer program in the present context mean any expression, in any language, code, or notation, of a set of instructions intended to cause a system having information processing capability to perform the particular function either directly or after performing either or both of the following:
At the computer system executing the master process, the keyboard 29.11, mouse 29.12 and other input devices connected to the computer system through the I/O interface 29.10 are used to input the sequential program wherein the parts of the program to be executed in parallel as 'parallel procedures' are specified. Following this the program is executed and the instructions encoded in the means to execute the program in parallel are transferred from the storage device 29.5 to the memory 29.4, which holds the current instructions to be executed by the processor 30.3 along with their results, through the internal communication bus 29.2. The processor 29.3 executes the instructions by fetching them from the memory 29.4 to create jobs for each call to a parallel procedure. The computer system either executes the job by itself or uses the networking interface 29.8 to send the jobs to the target nodes over a network such as LAN 29.13. At the slave node, the job is received over the LAN 29.13 through the networking interface 29.8 and the processor 29.3 executes the job.
Those of ordinary skill in the art will appreciate that the hardware depicted in
The computing environment also provides for interface to its administrator through a Process viewer. The Process viewer is an application started by the registry service at startup and is responsible for showing the state of execution of the programmer's application at any moment of time. It shows all the information about the nodes involved in the distributed processing. The information may include its hardware information or resource availability on the each computing node. The process viewer can also be used for getting the information about each process running on the system along with its ID, its priority, state ready or suspended, its execution time etc. The process viewer can also be used to control the distribution of processes and to suspend processes or forcing process distribution to a particular node or set of nodes.
It is important to note that while the present invention has been described in the context of a fully functioning data processing system, those of ordinary skill in the art will appreciate that the present invention is capable of being distributed in the form of computer instructions forming a computer program in which these instructions may be embodied on various computer readable media. The present invention applies equally regardless of the particular type of signal bearing media actually used to carry out the distribution. Examples of computer readable media include recordable-type media, such as a floppy disk, a hard disk drive, a RAM, CD-ROMs, DVD-ROMs, and transmission-type media, such as digital and analog communications links, wired or wireless communications links using transmission forms, such as, for example, radio frequency and light wave transmissions. The computer readable media may take the form of coded formats that are decoded for actual use in a particular data processing system.
|Cited Patent
|Filing date
|Publication date
|Jun 26, 1989
|Jun 4, 1991
|Mcc Development, Ltd.
|Parallel processor system for processing natural concurrencies and method therefor
|Jan 25, 1989
|Feb 11, 1992
|Hitachi, Ltd.
|Compiling method for determining programs to be executed parallelly by respective processors in a parallel computer which transfer data with a data identifier to other processors
|Oct 5, 1993
|Apr 4, 1995
|Top Level Inc.
|Opportunistic task threading in a shared-memory, multi-processor computer system
|Dec 3, 1993
|Sep 19, 1995
|Hitachi, Ltd.
|Program parallelizing apparatus capable of optimizing processing time
|Mar 15, 1994
|Nov 3, 1998
|University Of Westminister
|Apparatus and method for parallel computation
|Apr 27, 1995
|Jan 12, 1999
|Kabushiki Kaisha Toshiba
|Programming method for concurrent programs and program supporting apparatus thereof
|Mar 6, 1997
|Dec 7, 1999
|Continuum Software, Inc.
|System and method for developing computer programs for execution on parallel processing systems
|Feb 9, 1995
|Dec 7, 1999
|International Business Machines Corporation
|Concurrent processing in object oriented parallel and near parallel
|Jul 22, 1998
|Mar 20, 2001
|Cisco Technology, Inc.
|Component extensible parallel execution of multiple threads assembled from program components specified with partial inter-component sequence information
|Feb 16, 1993
|Jun 26, 2001
|Hitachi, Ltd.
|Method for supporting parallelization of source program
|Apr 4, 1997
|Oct 2, 2001
|Texas Instruments Incorporated
|Computer operating process allocating tasks between first and second processors at run time based upon current processor load
|Mar 25, 1996
|Oct 30, 2001
|Torrent Systems, Inc.
|Apparatuses and methods for programming parallel computers
|Jul 22, 1998
|Jan 7, 2003
|Cisco Technology, Inc.
|Dynamic determination of execution sequence
|Aug 29, 1995
|Jan 27, 2004
|Object Technology Licensing Corporation
|Object-oriented operating system
|Jul 22, 1998
|Jul 27, 2004
|Cisco Technology, Inc.
|Persistent storage of information objects
|Feb 18, 2003
|Oct 25, 2005
|Cogent Chipware Inc.
|Processor cluster architecture and associated parallel processing methods
|Oct 18, 2002
|Feb 28, 2006
|Panasas, Inc.
|Hashing objects into multiple directories for better concurrency and manageability
|B.L. Massingill, Experiments with Program Parallelization Using Archetypes and Stepwise Refinement; Parallel Processing Letters, vol. 9, No. 4, 1999, pp. 487-498.
|B.R. Bryant et al., Automatic Parallelization of Object-Oriented Programming Languages Using Tuple Space; 1995 ACM, pp. 89-96.
|C. Chu et al., Parallelizing Subroutines in Sequential Programs; 1994, IEEE Software, pp. 77-85.
|C.T.H. Everaars, et al., Restructuring Sequential Fortran Code into a Parallel/Distributed Application; 1996 IEEE, pp. 13-22.
|C.T.H. Everaars, et al., Using Coordination to Restructure Sequential Source Code into Concurrent Program; ICSM 2001: 342-351.
|H.E. Bal et al., Programming Languages for Distributed Computing Systems; ACM Computing Surveys, vol. 21, No. 3, Sep. 1989, pp. 262-322.
|J. Briot et al., Concurrency and Distribution in Object Oriented Programming; ACM Computing Surveys, vol. 30, No. 3, Sep. 1998, pp. 291-329.
|J.M. Jezequel et al., A Parallel Execution Environment for a Sequential Object Oriented Language; 1992 ACM, pp. 368-376.
|M. Karaorman, et al., Introducing Concurrency to a Sequential Language, Communications of the ACM, Sep. 1993, vol. 36, No. 9, pp. 103-116.
|MPI: A Message-Passing Interface Standard; 1994.
|N. Sundaresan, Extending the Standard Temple Library for Parallelism in Coir<Futures>; Third USENIX Conference on Object-Oriented Technologies (COOTS), 1997, pp. 159-174.
|N. Uchihira et al., Hypersequential Prgramming; 1997 IEEE, pp. 44-54.
|R. Choudra et al., COOL: An Object-Based Language for Parallel Programming; 1994 IEEE, pp. 13-26.
|The MPI Forum, MPI: A Message Passing Interface; 1993 ACM, pp. 878-883.
|TITLE: ADA for closely coupled multiprocessor targets, author: Cholerton, ACM, 1989.
|TITLE: Low-Cost Task Scheduling for Distributed -Memory Machines, author: Radulescu et al, IEEE, Jun. 2002.
|TITLE: Object Distribution in Orca using Compile-Time and Run-Time Techniques, author: Bal et al, ACM, 1993.
|TITLE: Task concurrency management methodology to schedule the MPEG4 IM1 player on a highly parallel processor platform, author: Wong et al, ACM, 2001.
|Z. Abdelouahab et al., Concurrency in Object Oriented Language Actel; 1997 IEEE, pp. 291-296.
|Citing Patent
|Filing date
|Publication date
|Jun 4, 2004
|May 20, 2008
|Sun Microsystems, Inc.
|Thread suspension and method in a multi-threaded environment
|Dec 23, 2003
|Jul 8, 2008
|Kabushiki Kaisha Toshiba
|Method and program for linking dynamics simulation and kinematic simulation
|Jul 9, 2004
|Mar 9, 2010
|Microsoft Corporation
|Implementation of concurrent programs in object-oriented languages
|Nov 22, 2005
|Apr 27, 2010
|Siemens Aktiengesellschaft
|Architecture for a computer-based development environment with self-contained components and a threading model
|Feb 20, 2007
|Jan 4, 2011
|International Business Machines Corporation
|Method and system for detecting synchronization errors in programs
|May 5, 2009
|Jul 12, 2011
|Edss, Inc.
|TICC-paradigm to build formally verified parallel software for multi-core chips
|Aug 12, 2009
|Aug 7, 2012
|Apple Inc.
|Managing extrinsic processes
|Dec 1, 2010
|Jul 1, 2014
|Fujitsu Limited
|Node computation initialization technique for efficient parallelization of software analysis in a distributed computing environment
|Mar 14, 2013
|Nov 18, 2014
|Microsoft Corporation
|Providing distributed array containers for programming objects
|Jun 29, 2012
|Dec 30, 2014
|Microsoft Corporation
|Implementation of distributed methods that support generic functions
|Dec 1, 2009
|Feb 3, 2015
|Kpit Technologies Limited
|Method and system for parallelization of sequential computer program codes
|Sep 30, 2013
|Jun 30, 2015
|The Research Foundation For The State University Of New York
|System and method for security and privacy aware virtual machine checkpointing
|Feb 9, 2009
|Aug 18, 2015
|Microsoft Technology Licensing, Llc
|General purpose distributed data parallel computing using a high level language
|Jun 29, 2012
|Nov 3, 2015
|Microsoft Technology Licensing, Llc
|Partitioned array objects in a distributed runtime
|Oct 25, 2013
|May 17, 2016
|Apple Inc.
|Efficient service discovery for peer-to-peer networking devices
|Oct 22, 2014
|May 31, 2016
|Microsoft Technology Licensing, Llc
|Providing distributed array containers for programming objects
|Apr 27, 2016
|Jan 3, 2017
|Microsoft Technology Licensing, Llc
|Providing distributed array containers for programming objects
|Jun 5, 2009
|Jan 17, 2017
|Microsoft Technology Licensing, Llc
|Language-based model for asynchronous operations
|Jun 29, 2015
|Jan 24, 2017
|The Research Foundation For The State University Of New York
|System and method for security and privacy aware virtual machine checkpointing
|Feb 13, 2015
|Feb 21, 2017
|International Business Machines Corporation
|Determining an ordering to use to open and close programs that call other programs
|Dec 30, 2013
|Feb 28, 2017
|Sas Institute Inc.
|Automated cooperative concurrency with minimal syntax
|May 23, 2014
|Jun 13, 2017
|Microsoft Technology Licensing, Llc
|Framework for authoring data loaders and data savers
|Jan 17, 2017
|Jun 27, 2017
|Microsoft Technology Licensing, Llc
|Language-based model for asynchronous operations
|Jul 23, 2015
|Aug 1, 2017
|Microsoft Technology Licensing, Llc
|General purpose distributed data parallel computing using a high level language
|Dec 21, 2012
|Aug 29, 2017
|Nvidia Corporation
|System and method for compiling or runtime executing a fork-join data parallel program with function calls on a single-instruction-multiple-thread processor
|Dec 28, 2015
|Sep 19, 2017
|The Research Foundation For The State University Of New York
|System and method for validating program execution at run-time
|Jun 18, 2015
|Sep 19, 2017
|The Research Foundation For The State University Of New York
|Continuous run-time validation of program execution: a practical approach
|Jun 2, 2008
|Sep 26, 2017
|Microsoft Technology Learning, LLC
|Composable and cancelable dataflow continuation passing
|Dec 23, 2003
|Sep 23, 2004
|Koichi Kondo
|Mechanism simulation method and mechanism simulation program
|Jul 9, 2004
|Jan 26, 2006
|Microsoft Corporation
|Implementation of concurrent programs in object-oriented languages
|Jun 30, 2005
|Jun 15, 2006
|Park Yong-Jun
|Method and apparatus to upgrade software of an apparatus having a plurality of microprocessors
|Nov 22, 2005
|Jun 22, 2006
|Detlef Becker
|Architecture for a computer-based development environment with self-contained components and a threading model
|Aug 16, 2005
|Feb 22, 2007
|Chan Wilson W S
|Affinity-based recovery/failover in a cluster environment
|Feb 20, 2007
|Aug 21, 2008
|International Business Machines Corporation
|Method and system for detecting synchronization errors in programs
|Jun 2, 2008
|Dec 3, 2009
|Microsoft Corporation
|Composable and cancelable dataflow continuation passing
|May 5, 2009
|Dec 10, 2009
|Edss, Inc.
|Ticc-paradigm to build verified parallel software for multi-core chips
|Feb 9, 2009
|Aug 12, 2010
|Microsoft Corporation
|General purpose distributed data parallel computing using a high level language
|Mar 18, 2010
|Nov 4, 2010
|Nec Laboratories America, Inc.
|Symbolic predictive analysis for concurrent programs
|Jun 5, 2009
|Dec 9, 2010
|Microsoft Corporation
|Language-based model for asynchronous operations
|Aug 12, 2009
|Feb 17, 2011
|Erwin Hom
|Managing extrinsic processes
|Dec 1, 2010
|May 3, 2012
|Indradeep Ghosh
|Node computation initialization technique for efficient parallelization of software analysis in a distributed computing environment
|Dec 21, 2012
|May 8, 2014
|Nvidia Corporation
|System and method for compiling or runtime executing a fork-join data parallel program with function calls on a single-instruction-multiple-thread processor
|Dec 30, 2013
|Oct 2, 2014
|Sas Institute Inc.
|Automated cooperative concurrency with minimal syntax
|Dec 1, 2009
|Jun 10, 2010
|Kpit Cummins Infosystems Ltd.
|Method and system for parallelization of sequencial computer program codes
|U.S. Classification
|717/149, 718/106, 718/102, 717/126, 712/32, 719/316, 714/25, 719/330, 712/31, 714/34, 717/124
|International Classification
|Cooperative Classification
|European Classification
|May 4, 2004
Owner name: INDIAN INSTITUTE OF INFORMATION TECHNOLOGY, INDIA
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:JALAN, ABHINAV;CHADHA, RETESH;REEL/FRAME:015299/0972;SIGNING DATES FROM 20040428 TO 20040429
|Aug 9, 2010
|Maintenance fee reminder mailed
|Jan 2, 2011
|Lapse for failure to pay maintenance fees
|Feb 22, 2011
|Expired due to failure to pay maintenance fee
Effective date: 20110102


text:
FREE SHIPPING on orders of $85 or more
An excerpt from www.HouseOfNames.com archives copyright © 2000 - 2017
Anglo-Saxon culture. It comes from when the family lived in the county of Cumberland. As a general rule, the greater the distance between individuals and their homeland, the larger the territory they were named after. For example, people who only moved to another parish would be known by the name of their original village, while people who migrated to a different country were often known by the name of a region or country from which they came.
Comerland Early Origins
Cumberland, a historic county of North West England that existed from the 12th century until 1974. The earliest record of the place was when it was listed as Cumbraland in 945 in the Anglo Saxon Chronicle and literally meant "region of the Cymry or Cumbrian Britons" from the Old English words Cumbre + land. [1]CITATION[CLOSE]
Mills, A.D., Dictionary of English Place-Names. Oxford: Oxford University Press, 1991. Print. (ISBN 0-19-869156-4)
Comerland Spelling Variations
Comerland Spelling Variations
Before the last few hundred years, the English language had no fast system of spelling rules. For that reason, spelling variations are commonly found in early Anglo-Saxon surnames. Over the years, many variations of the name Comerland were recorded, including Cumberland, Cumbeland, Cumberlande, Comerland and many more.
Comerland Early History
Comerland Early History
This web page shows only a small excerpt of our Comerland research. Another 113 words (8 lines of text) covering the years 1191, 1301, 1524, 1670, 1703, 1631 and 1718 are included under the topic Early Comerland History in all our PDF Extended History products and printed products wherever possible.
Comerland Early Notables (pre 1700)
Comerland Early Notables (pre 1700)
Another 24 words (2 lines of text) are included under the topic Early Comerland Notables in all our PDF Extended History products and printed products wherever possible.
The Great Migration
The Great Migration
To escape oppression and starvation at that time, many English families left for the "open frontiers" of the New World with all its perceived opportunities. In droves people migrated to the many British colonies, those in North America in particular, paying high rates for passages in cramped, unsafe ships. Although many of the settlers did not make the long passage alive, those who did see the shores of North America perceived great opportunities before them. Many of the families that came from England went on to make essential contributions to the emerging nations of Canada and the United States. Research into various historical records revealed some of first members of the Comerland family emigrate to North America: William Cumberland who arrived in Philadelphia in 1873.
Comerland Family Crest Products
Comerland Family Crest Products
See Also
See Also
Other References
The Comerland Family Crest was acquired from the Houseofnames.com archives. The Comerland Family Crest was drawn according to heraldic standards based on published blazons. We generally include the oldest published family crest once associated with each surname.
This page was last modified on 29 January 2014 at 12:38.
Sign Up
FREE SHIPPING on orders of $85 or more

subset: commoncrawl


text:
SEEC Co-Chairs Statement on the Nomination of Scott Pruitt to head the Environmental Protection Agency
Today, President-Elect Trump nominated Oklahoma Attorney General Scott Pruitt to head the Environmental Protection Agency. SEEC Co-Chairs Gerry Connolly (VA), Doris Matsui (CA), and Paul Tonko (NY) released the following joint statement:
"We are extremely concerned by President-Elect Trump's nomination of Oklahoma Attorney General Scott Pruitt to head the Environmental Protection Agency (EPA). As Oklahoma Attorney General, Scott Pruitt has demonstrated time and again that he values his ties to the fossil fuels industry over any efforts to protect public health, safety, and the environment. Beyond his record of repeatedly trying to stymie and rollback important environmental protections, Pruitt is a climate change denier. Any individual that rejects the scientific consensus supporting climate action should have no place at the EPA–an agency charged with developing science-based protections for our health and environment. Congress has charged the EPA with the task of protecting our nation's clean air and water for the benefit of our children and families. Based on Scott Pruitt's record we have little faith in his ability to fulfill the EPA's lifesaving mission, and his nomination must be opposed."


text:
Reviewing Stoneridge (SRI) and Its Peers
Stoneridge (NYSE: SRI) is one of 32 publicly-traded companies in the "Auto, Truck & Motorcycle Parts" industry, but how does it contrast to its rivals? We will compare Stoneridge to related companies based on the strength of its institutional ownership, earnings, dividends, risk, valuation, profitability and analyst recommendations.
This table compares Stoneridge and its rivals' net margins, return on equity and return on assets.
|Net Margins
|Return on Equity
|Return on Assets
|Stoneridge Competitors
Valuation & Earnings
This table compares Stoneridge and its rivals gross revenue, earnings per share (EPS) and valuation.
|Gross Revenue
|Price/Earnings Ratio
|$759.88 million
|$85.94 million
|Stoneridge Competitors
|$5.85 billion
|$698.05 million
Stoneridge's rivals have higher revenue and earnings than Stoneridge. Stoneridge is trading at a lower price-to-earnings ratio than its rivals, indicating that it is currently more affordable than other companies in its industry.
Analyst Ratings
This is a summary of recent recommendations for Stoneridge and its rivals, as reported by MarketBeat.
|Sell Ratings
|Hold Ratings
|Buy Ratings
|Strong Buy Ratings
|Rating Score
|Stoneridge Competitors
Stoneridge currently has a consensus target price of $19.50, suggesting a potential downside of 14.10%. As a group, "Auto, Truck & Motorcycle Parts" companies have a potential upside of 3.53%. Given Stoneridge's rivals stronger consensus rating and higher possible upside, analysts clearly believe Stoneridge has less favorable growth aspects than its rivals.
Volatility & Risk
Stoneridge has a beta of 1.44, meaning that its stock price is 44% more volatile than the S&P 500. Comparatively, Stoneridge's rivals have a beta of 1.40, meaning that their average stock price is 40% more volatile than the S&P 500.
Insider & Institutional Ownership
87.8% of Stoneridge shares are owned by institutional investors. Comparatively, 71.1% of shares of all "Auto, Truck & Motorcycle Parts" companies are owned by institutional investors. 5.3% of Stoneridge shares are owned by company insiders. Comparatively, 12.3% of shares of all "Auto, Truck & Motorcycle Parts" companies are owned by company insiders. Strong institutional ownership is an indication that hedge funds, endowments and large money managers believe a company will outperform the market over the long term.
Stoneridge rivals beat Stoneridge on 8 of the 13 factors compared.
About Stoneridge
Stoneridge, Inc. is a designer and manufacturer of engineered electrical and electronic components, modules and systems for the automotive, commercial, motorcycle, off-highway and agricultural vehicle markets. The Company operates through three segments: Control Devices, Electronics and PST. The Company's Control Devices segment designs and manufactures products that monitor, measure or activate specific functions within a vehicle and includes product lines, such as sensors, switches, valves, and actuators. The Company's Electronics segment designs and manufactures electronic instrument clusters, electronic control units and driver information systems. The PST segment is engaged in the design, manufacture and sale of in-vehicle audio and video devices, electronic vehicle security alarms, convenience accessories, vehicle tracking devices and monitoring services primarily for the automotive and motorcycle industry. It operated in 25 locations in 12 countries, as of December 31, 2016.
Receive News & Ratings for Stoneridge Inc. Daily - Enter your email address below to receive a concise daily summary of the latest news and analysts' ratings for Stoneridge Inc. and related companies with MarketBeat.com's FREE daily email newsletter.


text:
- 2017 Family Cookie Manual (online flipbook)
- 2017 Spanish Family Cookie Manual (online flipbook)
- Generic Goal Getter Order Card - Use this card to sell extra boxes to make even cases.
- Gift of Caring Donation Receipt
- Girl Rewards
- Girl Scout Internet Safety Pledge
- Online Product Sales
Permission Form
- Product Sales Permission Form (pdf)
- Safety First! Guidelines and tips for staying safe while selling cookies.
Tools for Cookie Program Success
Little Brownie Bakers
Want to make this cookie season the best one yet? Check out some of the fun activities and materials for girls from our partners at Little Brownie Bakers.
2018 Girl Scout Cookie Program Resources
NEW! Setting Troop Goals Guide: Guide to get girls engaged in troop goal setting, from activities to the Girl Scout Cookie Sale. View as an online flipbook or download pdf.
2017 Troop Manual & Tip Sheets.
- 2017 Family Cookie Manual (online flipbook)
- 2017 Spanish Family Cookie Manual (online flipbook)
2017 Troop Cookie Manual (online flipbook)
- 2017 Troop Cookie Manual (pdf)
- Online Marketing Quick Sheet
- Girl Rewards Check List
- How to Conduct a Booth Sale
- Troop Booth Sale Worksheet
- Gift of Caring Quick Sheet
Cookie Program Newsletters
Talking Points: Sensitive Issues
Cookie Resources for Girls & Parents
Cookie Resources for Troops
Cookie Resources for Service Units and Regions
Want to know about all the benefits of selling Girl Scout Cookies? Looking for a one-stop shop with the latest safety guidelines, information on earning the Girl Scout Cookie Activity pin, and tips for parents and volunteers working with Girl Scout Cookie professionals?
Girl Scouts of the USA's Cookie Resources
Here you'll find handouts, posters, videos, and other materials to help you have fun and stay safe as you build your cookie business.


text:
Phase 1 Study of MGD007 in Relapsed/Refractory Metastatic Colorectal Carcinoma
- Full Text View
- Tabular View
- No Study Results Posted
- Disclaimer
- How to Read a Study Record
|Colorectal Carcinoma
|Drug: MGD007
|Phase 1
|Study Type:
|Study Design:
|Allocation: Non-Randomized
Intervention Model: Parallel Assignment
Masking: None (Open Label)
Primary Purpose: Treatment
|Official Title:
|A Phase 1, First-in-Human, Open Label, Dose Escalation Study of MGD007, A Humanized gpA33 x CD3 Dual-Affinity Re-Targeting (DART®) Protein in Patients With Relapsed/Refractory Metastatic Colorectal Carcinoma
- Characterize dose limiting toxicity and establish a maximum tolerated dose and schedule [ Time Frame: Cycle 1 of a 6 week cycle ]Safety is based on evaluation of adverse events (AEs) and serious adverse events (SAEs) from the time of study drug administration through the End of Study visit. The MTD will be defined separately for both the weekly and every three week schedules of MGD007 administration, and will be determined as the highest dose level at which the incidence of DLT is < 33% during the first cycle of MGD007 treatment.
- Characterize the PK and Immunogenicity of MGD007 [ Time Frame: Beginning of treatment through end of treatment, an expected duration of less than 12 months ]Serum concentrations of MGD007 will be monitored. PK modeling will be performed and an appropriate model will be selected to describe the data.
- Describe any evidence of anti-tumor activity [ Time Frame: Every 6 weeks until End of Study, an expected duration of less than 12 months ]Obtain regular radiographic and clinical evaluations to assess treatment response.
|Estimated Enrollment:
|Study Start Date:
|September 2014
|Estimated Study Completion Date:
|March 2019
|Estimated Primary Completion Date:
|June 2018 (Final data collection date for primary outcome measure)
|Assigned Interventions
Experimental: Does Escalation Arm A
MGD007 treatment once weekly
Drug: MGD007
MGD007 is a gpA33 x CD3 bi-specific antibody-based molecular construct referred to as a DART molecule. MGD007 will be administered as a single agent.
Experimental: Dose Escalation Arm B
MGD007 treatment once every 3 weeks
Drug: MGD007
MGD007 is a gpA33 x CD3 bi-specific antibody-based molecular construct referred to as a DART molecule. MGD007 will be administered as a single agent.
Experimental: Dose Expansion Arm C
MGD007 once every 3 weeks for K-ras wild-type and mutant metastatic CRC
Drug: MGD007
MGD007 is a gpA33 x CD3 bi-specific antibody-based molecular construct referred to as a DART molecule. MGD007 will be administered as a single agent.
Experimental: Dose Expansion Arms
MGD007 2, 3, 6, or 12 doses/cycle
Drug: MGD007
MGD007 is a gpA33 x CD3 bi-specific antibody-based molecular construct referred to as a DART molecule. MGD007 will be administered as a single agent.
Detailed Description:
This is an open-label, multi-center, Phase 1 dose-escalation study to define a MTD, describe preliminarily safety, and to assess PK, immunogenicity, and potential anti-tumor activity of MGD007 in patients with relapsed or refractory metastatic colorectal cancer.
In the initial phase of the study, two dose schedules will be assessed in dose escalation, once weekly and once every three weeks administration of single agent MGD007. Following the establishment of an MTD, additional patients will enroll in four separate dose expansion cohorts to further optimize the dose and schedule of MGD007 administration.
In all segments of the study, patients who benefit from MGD007 treatment and continue to meet eligibility may continue treatment in Cycles 2 and beyond.
Choosing to participate in a study is an important personal decision. Talk with your doctor and family members or friends about deciding to join a study. To learn more about this study, you or your doctor may contact the study research staff using the contacts provided below. For general information, Learn About Clinical Studies.
|Ages Eligible for Study:
|18 Years and older (Adult, Senior)
|Sexes Eligible for Study:
|Accepts Healthy Volunteers:
Inclusion Criteria:
- For the dose escalation cohorts, histologically-proven metastatic colorectal adenocarcinoma that is refractory to 2 prior standard treatment regimens or standard treatment was declined.
- For the dose expansion cohorts, histologically-proven metastatic colorectal adenocarcinoma that is refractory to 1 prior standard treatment regimen or standard therapy was declined.
- Eastern Cooperative Oncology Group (ECOG) performance status of 0 or 1
- Life expectancy of at least 12 weeks
- Measurable disease
- Intolerance to at least 2 prior standard therapy regimens
- Acceptable laboratory parameters
- Adult (≥18 years old)
Exclusion Criteria:
- Known brain metastasis
- Any prior history of or suspected current autoimmune disorders (with the exception of vitiligo, resolved childhood atopic dermatitis, prior Grave's disease)
- Prior history of allogeneic bone marrow, stem-cell, or solid organ transplantation
- Prior treatment with checkpoint inhibitors and other immunotherapy treatments, including anti-LAG-3, anti-PD-1, anti-PD-L1 or anti-CTLA-4 antibodies, if less than 5 half lives before study drug administration
- Prior history of Grade 3 or greater drug-related diarrhea/colitis during treatment with checkpoint inhibitors or other immunotherapy treatments.
- Treatment with any local or systemic anti-neoplastic therapy or any other investigational agent in the 4 weeks prior to study drug administration
- Require, at the time of study entry, treatment with steroids > 10 mg/day of oral prednisone (or equivalent), except topical use, steroid inhaler, nasal spray or ophthalmic solution
- History of clinically significant cardiovascular disease, gastrointestinal disorder, or significant pulmonary compromise.
- Second primary malignancy that has not been in remission for greater than 3 years, with the exception of non-melanoma skin cancer, cervical carcinoma in situ,or squamous intraepithelial lesion on PAP smear, localized prostate cancer (Gleason score <6), or resected melanoma in situ.
To learn more about this study, you or your doctor may contact the study research staff using the contact information provided by the sponsor.
Please refer to this study by its ClinicalTrials.gov identifier (NCT number): NCT02248805
Please refer to this study by its ClinicalTrials.gov identifier (NCT number): NCT02248805
|Contact: Joanna Lohr
|Contact: Susan Brann
|United States, Florida
|H. Lee Moffitt Cancer Center & Research Institute, Inc
|Tampa, Florida, United States, 33612
|Contact: Tiffany Romershausen, MPH 813-745-2146 firstname.lastname@example.com
|Principal Investigator: Richard Kim, MD
|United States, Maryland
|Johns Hopkins Sidney Kimmel Comprehensive Cancer Center
|Baltimore, Maryland, United States, 21205
|Contact: Ellen Lilly-Foreman, RN, OCN 443-287-4961 email@example.com
|Contact: Thomas Brown 410-502-5328 email@example.com
|Principal Investigator: Daniel Laheru, MD
|United States, Massachusetts
|Massachusetts General Hospital
|Boston, Massachusetts, United States, 02114
|Contact: Susan Sheehan, RN 617-724-4000 email@example.com
|Principal Investigator: David P Ryan, MD
|Dana Farber Cancer Institute
|Boston, Massachusetts, United States, 02215
|Contact: Margaret Carey 617-632-5575 email@example.com
|Principal Investigator: David P Ryan, MD
|United States, North Carolina
|Duke University Medical Center
|Durham, North Carolina, United States, 27710
|Contact: Anthony Amara 919-668-1861 firstname.lastname@example.com
|Principal Investigator: Herbert Hurwitz, MD
|Carolina Biooncology Institute
|Huntersville, North Carolina, United States, 28078
|Contact: Ashley McClain 704-947-6599 ext 117 firstname.lastname@example.com
|Principal Investigator: John Powderly, MD
|United States, Oregon
|Providence Portland Medical Center
|Portland, Oregon, United States, 97213
|Contact: Yue-yun To, RN 503-215-2855 firstname.lastname@example.com
|Principal Investigator: Todd Crocenzi, MD
|Study Director:
|Jan Davidson-Moncada, MD
|Responsible Party:
|ClinicalTrials.gov Identifier:
|NCT02248805 History of Changes
|Other Study ID Numbers:
|First Submitted:
|September 18, 2014
|First Posted:
|September 25, 2014
|Last Update Posted:
|October 30, 2017
|Last Verified:
|October 2017
Keywords provided by MacroGenics:
colon cancer
colorectal carcinoma
stage IV colorectal cancer
Additional relevant MeSH terms:
Colorectal Neoplasms
Neoplasms, Glandular and Epithelial
Neoplasms by Histologic Type
Intestinal Neoplasms
Gastrointestinal Neoplasms
Digestive System Neoplasms
Neoplasms by Site
Digestive System Diseases
Gastrointestinal Diseases
Colonic Diseases
Intestinal Diseases
Rectal Diseases