How do I find the number of our association's members at the end of each calendar year?
The CiviCRM Membership records themselves don't provide a complete or reliable history of memberships, which is why reports based on them (i.e. the Membership reports) aren't useful to you for this purpose. 

They do contain a "join" date separately from the "start" date which could be used to make a reasonable guess about membership totals in the past, but this isn't reliable (e.g. if a membership is deleted, or if this field is not used as expected.)

The churn extension, as well as the membership dashboard, make use of the CiviCRM Activity records to calculate when a new membership is created or renewed or expires (because each of those leave behind an activity record). This is probably the best tool for what you want, though it can be misleading as well (especially if you've imported your membership records into CiviCRM from a different sytem).

I haven't seen any off-the-shelf Civi Reports that make use of these records.

My general advice to anyone who wants to have historical comparisons of membership data is to download the membership data at defined intervals.