1. An arithmetic processing apparatus performing arithmetic by a neural network in which a plurality of processing layers are hierarchically connected, the arithmetic processing apparatus corresponding to one of the plurality of processing layers comprising: a convolution arithmetic portion receiving an input data that is input from an other of the plurality of processing layers, performing convolution arithmetic to the input data, and, in each arithmetic cycle, outputting a part of all convolution arithmetic result data required for single pooling processing; and a pooling processing portion performing the single pooling processing to the all convolution arithmetic result data obtained by the convolution arithmetic portion, wherein the pooling processing portion performs the single pooling processing before executing activation processing.