1. An arithmetic processing device comprising: an instruction control unit that outputs an instruction fetch request; a primary cache that has a request port that enters the instruction fetch request output from the instruction control unit and a cache processing unit that outputs instruction data corresponding to the instruction fetch request in response to an input of the instruction fetch request; and a secondary cache that returns the instruction data to the primary cache in response to a data request output from the primary cache when a cache error is caused in the primary cache, wherein the request port has a primary port that is provided in common to a plurality of threads and stores information on the instruction fetch requests of the respective threads, a plurality of secondary ports that are provided corresponding to the plurality of threads, respectively, and each of which stores entry numbers of the primary port, and a request determination unit that determines, from among the entered instruction fetch requests input in the request port, an instruction fetch request to be input to the primary cache, by placing priority on an instruction fetch request waiting for being input to the primary cache, in an order in which the instruction fetch requests of the respective threads are output from the instruction control unit as for the instruction fetch requests of the respective threads.