Impossible to invoke a method ("limit") on an array... but it's not an array?
I'm not calling .all(), so this is still an object, right?


Nope.  When you execute the for loop, it will explicitly run the database query and return the array of results.  You're probably getting a deprecation error logged using that syntax as well since Craft wants you to use {% for album in albums.all() %}

See here in the docs:  https://docs.craftcms.com/v3/changes-in-craft-3.html#treating-queries-as-arrays

If you want the first results for the album cover, you can use:

{% set albumCover = album.townPhotoAlbum[0] %}