MySQL Replication not proceeding
This turned out to not be a mysql issue at all.  The network team recently installed a new security device that will block packets on certain rules.  A legitimate database write contained a sequence of characters the device deemed nefarious.

The overall connection handshake for replication was able to make it fine but then it just sat there asking the master for the next log entry whose packets were never making it back.

As far as the broken slave was concerned, it was up to date b/c it had executed the most recent event in the relay logs it had received.