How do I get the Calendar app to stop asking me for my gmail password?
It turns out the issue for me is that OSX has created 2 entries in the Calendar app for my Gmail account (even though I only have the account entered once under the Internet Accounts system pref).

The Calendar app won't let me delete the duplicate since it thinks the duplicate is managed under the Internet Accounts pref. Deleting and re-adding the gmail account in Internet Accounts didn't help either.  After deleting the Internet Account, the duplicate still sticks around in the Calendar app is not deletable since the Calendar app can't delete it, and there is no corresponding Internet Account for that calendar.

I got around the issue by un-checking the "Enable this account" checkbox for the duplicate in the Calendar app (since it won't let me delete it).  That at least makes it stop checking for calendar updates so I don't get pestered for passwords.