1. An apparatus for parallel processing comprising: a control processor for fetching and distributing single instruction multiple data (SIMD) instructions to a plurality of processing elements (PEs), wherein one of the SIMD instructions is a thread start (Tstart) instruction which causes the control processor to pause its instruction fetching; a local PE instruction memory (PE Imem) associated with each PE and containing local PE instructions for execution on the local PE; local PE Imem fetch, decode, and execute logic associated with each PE; instruction path selection logic in each PE is used to select between control processor distributed instructions and local PE instructions fetched from the local PE Imem, wherein each PE is initialized to receive control processor distributed instructions; and local hold generation logic associated with each PE, wherein a PE receiving a Tstart instruction causes the instruction path selection logic to switch to fetch local PE Imem instructions.