Dynamic simulation for transient stability assessment is one of the most important, but intensive, computational tasks for power system planning and operation. Several commercial software tools provide functionality for performing multiple dynamic simulations such as those in contingency analysis simultaneously on parallel computers. Nevertheless, a single dynamic simulation is still a time consuming process performed sequentially on one single computing core as the tools were originally designed. Modern high performance computing (HPC) holds the promise to accelerate a single dynamic simulation by parallelizing its kernel algorithms without compromising computational accuracy. Parallelizing a single dynamic simulation is a much more challenging problem than the contingency-type parallel computing. It requires a good match between simulation algorithms and computing hardware. This paper provides guidance for such a match so as to design and implement parallel dynamic simulation to maximize the utilization of computing hardware and the performance of the simulation. The guidance is derived through comparative implementation of four parallel dynamic simulation schemes in two state-of-the-art HPC environments: 1) message passing interface and 2) open multi-processing. The scalability and speedup performance of parallelized dynamic simulation are thoroughly studied to determine the impact of simulation algorithms and computing hardware configurations. Several testing cases are presented to illustrate the derived guidance.