How can you check battery cell noise using oscilloscope
You won't likely be able to see any noise from the battery on an oscilloscope because the noise floor of the oscilloscope is too large.
To actually make a useful measurement, you'll likely need to use (eg. SR560) or make a low-noise AC-coupled amplifier that covers the bandwidth of interest.
Provided you stay within the limits the oscilloscope is rated for (typically marked on the oscilloscope front or back panel) you won't damage the oscilloscope.