1. A parallel processing device that computes a hierarchical neural network, the parallel processing device comprising: a plurality of units; a distribution control section that outputs control data to the plurality of units, the control data including a input value and a selection unit number; a common storage section that stores in advance a plurality of coupling weights, each of a plurality of layers of the hierarchical neural network having at least one of the plurality of coupling weight, the plurality of coupling weights being shared by plural ones of the plurality of units, a unit input bus; and a unit output bus, each of the plurality of units being identified by a characteristic unit number that is predetermined identification number, respectively, the each of the plurality of units including: a data input section that received the control data from the distribution control section through the unit input bus; a unit number match judgment section that judges as to whether the selection unit number included in the control data matches the characteristic unit number; a unit processing section computes a output value and a new coupling weight based on the input value and one of the plurality of coupling weights by a computing method predetermined for the each of the plurality of units; and a data output section that outputs the output value through the unit output bus when the unit number match judgment section judges that the selection unit number matches the characteristic unit number, the unit processing section executing computation of the output value in a direction from an input layer to an output layer in the hierarchical neural network, the input layer being defined as a first layer, and layers from the input layer to the output layer being sequentially numbered, the unit processing section executing computation of the new coupling weight in a direction from the output layer to input layer in the hierarchical neural network, and the one of the plurality of coupling weights updating to the new coupling weight.