what will cause a sql server AG readonly replica to stop serving readonly clients?
But suppose the primary node goes down? Or a database is removed from the AG on the primary? Will the secondary replicas keep doing their thing in those scenarios?

I believe you are taking about unexpected shutdown or similar scenario where primary server not available in this case first WSFC will see if it is in position to update cluster registry to set the status like not synchronizing if the server shutdown is immediate Registry may not get updated with the state. When primary is down it would not send log blocks to secondary and if secondary is also done redoing and undoing all the log records it will also not do any work but wait for log records to be pulled when primary comes online. But let us say when the primary went down secondary was behind it, primary it will continue its process of replaying the log records to come alike primary.

Or a database is removed from the AG on the primary

Database will NOT be removed from AG unless user does that, so my above statement does not apply to this.