What does a gear 'spinning out' mean?
"Spinning" your legs as fast as they can go (i.e. pedalling at the highest cadence you are able to sustain), so going as fast as physically possible in the gear you are in. 

Usually at that point you'd change up a gear to bring the cadence back down to a manageable level, but if you are already in your highest gear that's not possible so you're at the limit of what the combination of you and your bike's gearing can physically achieve.

When you reach this point you will often feel like you're bouncing in the saddle as you sacrifice pedal stroke quality in order to try and keep you legs spinning fast enough. It can also feel like you have less of a connection with the bike, and by extension the road, as you can't spin your legs fast enough to have any resistance / anything to push against - these two things combined can quickly lead to a lack of stability on the bike. You can see this in action in videos of "roller races" (literally racing on static rollers), where the resistance provided by the rollers is very low.

As an aside, the "Velominati" refer to the speed at which you "spin out" as Escape Velocity.