1. A system for checkpointing data in a parallel computing system having a plurality of computing nodes, each node having one or more processors and network interface devices for communicating over a network, said checkpointing system comprising: one or more network elements interconnecting said network interface devices of computing nodes via links to form a network; a control device to communicate control signals to each said computing node of said network for stopping receiving and sending message packets at a node, and to communicate further control signals to each said one or more network elements for stopping flow of message packets within said formed network; and, a control unit, at each computing node and at one or more said network elements, responsive to a first control signal to stop each of said network interface devices involved with processing of packets in said formed network, and, to stop a flow of packets communicated on links between nodes of said network; and, said control unit, at each node and said one or more network elements, responsive to a second control signal to obtain, from each said plurality of network interface devices, network packet data included in said packets currently being processed including information of packets in transit between said links, and to obtain from said one or more network elements, current network state information, said control unit at each node responsive to a further control signal to obtain data associated with a user application, said obtained data including a user application program and associated program data, and a memory storage device adapted to temporarily store said obtained network packet data and said obtained network state information, wherein, each said control unit at each computing node and at one or more said network elements are responsive to further control signals to subsequently restore to said network said obtained network packet data and said obtained network state information; and, restart said network, and wherein said one or more network elements interconnecting said network reside on separate physical chips interconnected via communication links, said system further comprising a separate control network operatively connected with said nodes and said network elements, said control unit determining whether selective restarting of system-only packets is permitted, and in response to determining selective restarting is not permitted, said control unit initiating storing over the control network of both said network packet data and said obtained network state information, and resetting said network prior to checkpointing said obtained user application program and data.