1. An arithmetic processing device, comprising: a branch history storage unit configured to store past branch history information of conditional branch instructions; first prediction units configured to output branch prediction information of a fetched conditional branch instruction based on the branch history information stored by the branch history storage unit; a second prediction unit configured to store a branch taken consecutive number of times, being a number of times in which branches of the conditional branch instructions are consecutively taken, and a branch not-taken consecutive number of times, being a number of times in which the branches of the conditional branch instructions are consecutively not taken, to a pattern information storage unit, and output branch prediction information of the fetched conditional branch instruction based on the branch taken consecutive number of times or branch not-taken consecutive number of times in the past stored at the pattern information storage unit; selecting units configured to output the branch prediction information output from the second prediction unit when a pattern having the branch taken consecutive number of times and the branch not-taken consecutive number of times appears twice or more in the past in the branch history information, and output the branch prediction information output from the first prediction units when a pattern having the branch taken consecutive number of times and the branch not-taken consecutive number of times appears once or less in the past; and a selector configured to output a next instruction address of the conditional branch instruction or a branch target address of the conditional branch instruction to an instruction fetch unit in accordance with the branch prediction information output from the selecting units.