I’m afraid I might have given everything away in the headline here. Album Art Grabber is a free Android app that does one thing, and it does it well. It grabs missing album art for your music collection.
Basically, you fire up the app and it will show you a list of albums stored on your device. Click the Grab Missing Artwork button and it will go online, look for images matching those albums, and download them to your device. Next time you open the Android music app, you should see the correct artwork for each album.
The images come from Last.fm, so there’s a pretty good library of artwork — but it’s not perfect. Don’t expect your best friend’s garage band to show up automatically.