1. A computer program product comprising: a non-transitory computer readable storage medium that stores program code for reducing latency in a data processing system having a multi-level cache hierarchy including a higher level cache having a lower access latency and a lower level cache having a higher access latency, wherein the program code causes the data processing to perform: generating a main thread and an assist thread from a source code object, wherein the assist thread includes a smaller subset of the instructions in the main thread; scheduling the assist thread to execute in conjunction with the main thread; constraining execution of the main thread relative to the assist thread, such that each instruction in the main thread having a corresponding instruction in the assist thread executes after the corresponding instruction in the assist thread; making a memory access requested by an instruction of the assist thread directly to the lower level cache without first accessing the higher level cache and making a corresponding memory access requested by an instruction in the main thread to the higher level cache; and responsive to a memory access indicated by a selected memory reference instruction in the assist thread missing in the lower level cache, determining whether a subsequent assist thread instruction depends on the selected memory reference instruction and, if so, taking action based on an assist thread policy, wherein the assist thread policy is selected from a set of policies consisting of awaiting completion of the selected memory reference instruction, predicting an outcome of the selected memory reference instruction, and predicting the outcome of the selected memory reference instruction and verifying the outcome upon completion.