1. A computer-implemented method performed by at least one processor when executing an accelerator work allocation mechanism for allocating work to a hardware accelerator in a first runtime of a computer program in a computer system that includes the hardware accelerator and at least one processor executing the computer program in the first runtime, the method comprising the steps of: reading an accelerator policy included within the accelerator work allocation mechanism that specifies criteria that determines when a function is allocated to the hardware accelerator; analyzing the computer program; allocating during a first runtime of the computer program the function in the computer program to the hardware accelerator when the criteria in the accelerator policy is satisfied based on processor estimates of performance for the function and hardware accelerator estimates of performance for the function; and executing by the hardware accelerator the function during the first runtime of the computer program.