How much can bias decrease performance of the network at the beginnng of the training?
Bias takes care of variables which are latent for i.e you did not include it in the training set.

So you are trying to overfitt on training set, but now you introduce pertrubance. Ofcourse it will have worse performance.