1. An arithmetic processing device, comprising: a branch history storage unit configured to store past branch history information of conditional branch instructions; first prediction units configured to output branch prediction information of a fetched conditional branch instruction based on the branch history information stored by the branch history storage unit; a second prediction unit configured to store a branch taken consecutive number of times being the number of times in which branches of the conditional branch instructions are consecutively taken and a branch not-taken consecutive number of times being the number of times in which the branches of the conditional branch instructions are consecutively not taken to a pattern information storage unit, and output branch prediction information of the fetched conditional branch instruction based on the past branch taken consecutive number of times or branch not-taken consecutive number of times stored at the pattern information storage unit; selecting units configured to selectively output the branch prediction information output from the first prediction units or the branch prediction information output from the second prediction unit; and a selector configured to output a next instruction address of the conditional branch instruction or a branch target address of the conditional branch instruction to an instruction fetch unit in accordance with the branch prediction information output from the selecting units.