Oscilloscope probe measure squarewave
Yep, this is typical when your ground connection is not suitable for the frequencies you're working with. You may want to use a ground spring to reduce both the inductance-related peak and the noise: