Why do I need internet to listen to my music library?
Yipee! I figured it out! My problem is, as I forgot to mention in the question, I just got a new iPod. Aaaaaaand, I never realized it, but apparently when you get a new device, you have to download all your music into your library. Otherwise, you can still listen to it, but only over internet. Probably to save space. Downloading your songs can be done by tapping on the song or album, and then tapping "Make available offline".

Thank you everyone for your help!