1. An arithmetic processing device comprising: an instruction control unit that outputs an instruction fetch request; a primary cache that has, a request port that enters the instruction fetch request output from the instruction control unit, an instruction cache memory that stores an instruction data, and a cache processing unit that outputs instruction data corresponding to the instruction fetch request in response to an input of the instruction fetch request from the request port; and a secondary cache that returns the instruction data to the primary cache in response to a data request output from the primary cache when a cache error is caused in the primary cache, the request port has a primary port that is provided in common to a plurality of threads and has a plurality of entries each of which stores information on the instruction fetch requests of each of the plurality of threads, a plurality of secondary ports that are provided corresponding to the plurality of threads, respectively, and each of which stores entry numbers of the plurality of entries in the primary port, the primary port has entries fewer in number than a total number of the entries of the plurality of secondary ports, and a request determination unit that selects an instruction fetch request waiting for being input to the primary cache, respectively from among the instruction fetch requests that are stored in each of the plurality of secondary ports, in an order in which the instruction fetch requests are output from the instruction control unit, and determines an instruction fetch request to be input to the primary cache, from among the instruction fetch requests that are selected from the plurality of secondary ports respectively, in a predetermined order of the plurality of threads.