Does a battery powered resistor have Johnson Noise?
yes. Even a resistor with no power source will generate Johnson noise. This is thermal noise due to random movement of electrons in the resistor itself.
f in the formula is the bandwidth across which you wish to calculate the noise, it has nothing to do with the signal applied (or not) to the resistor.