Contextual links only appear the first time
I figured out that someone else has hit this error, and has contributed a patch for it:

https://www.drupal.org/node/2834346

The patch stops the Contextual Links module from adding a contextual link placeholder to the array if it's already in there.  I'm still not quite sure why there's a cookie for the contextual link (as I said in the comment above) if it's going through them and adding them anyway.  If anyone else knows more about this issue, feel free to answer.