Didn't Apple use to provide USB stick with OS X on it?
Yes, The USB recovery drive shipped first with a MacBook Air and Lion was also sold as a full retail OS on USB write only format instead of the retail DVD/CD that used to be available for purchase.

Apple discontinued this practice a few years ago and has started distributing OS X via the App Store exclusively as a digital download. You can make a bootable version of the installer from the App Store version if you have a USB stick handy.

There's a free, AppleScript-based program called DiskMaker X that will help you build a bootable USB stick for Mavericks. You'll have to download the installer from the App Store first. Just cancel the installation after the download and Mavericks will remain in your Applications folder for DiskMaker X to find and use.