What's the difference between USB and non USB oscilloscope?
Although this is a very opinion-based matter, there are two things to keep in mind. I presume here, with "USB scope" you refer to an oscilloscope that has no display or controls, and requires a computer to operate. I presume you do not mean an oscilloscope that can be connected to a computer, since any modern scope is capable of doing this.

USB scopes are often made to a price - they are aimed at entry-level buyers, who consider 500 USD/EUR/Whatever too much. Hence, they will, in my experience, be of lower quality. Exceptions to this exist - I've heard many good things about the picoscope, and of course there are devices such as the NI virtualbench that are excellent products (however I think this is not  the price range you had in mind when asking this question).

The main difference is in the interface. In the end, most oscilloscopes are the same (to the untrained eye): Apart from a bandwidth difference between models, digital scopes are just fancy analog-to-digital converters with trigger systems and a screen. But this latter is key: The triggering system, its capabilities, and the software to control the entire thing is what makes-or-breaks an oscilloscope - esp. to someone who is at the knowledge level to ask this question. For every day, hobby use, I much prefer the physical nature of oscilloscope control knobs. Fiddling about with mouse sliders and such just never seems to work fast for quick-and-dirty measurements. 

And finally as a sidenote: Consider the lifespan of a product and its dependence on a second system. If you purchase a USB oscilloscope, there is a good chance you won't be able to use it in 10-15 years. No more driver updates, computers (in whatever form they are then) no longer have USB interfaces, the software doesn't support your device, etc.. A "classic" oscilloscope is fully integrated and stand-alone. As long as the thing powers on, you know it will be able to take measurements. You need not rely on the support of the device on a second system.