USB switching power supply filter
Using a switch-mode regulator (in the Duracell USB PSU) for "a high sensitivity sensor board" is your problem.  Sensitive analog electronics quickly succumbs to the noise inherent with such regulators.  You can go to town with filtering (and simply throwing a cap of every type is not the ideal way to go about it), but some still gets through - whether such efforts are enough for your needs can't be gleaned from what you've told us.

It's traditional to add a linear regulator stage between the raw switch-mode PSU and analog circuitry.  If your analog circuitry will happily operate at less than 5V, then a LDO (low drop-out voltage) linear reg can be added, but this assumes a lot about the analog circuitry (head-room, voltage reference in an ADC, etc).