Small MySQL table swap/replace taking incredibly long time
Rename table requires a LOCK on the table to complete.

In your 0sec case there were no connections using the table and it works immediately.

In you 8 min case, The RENAME table waits in a stage something like "waiting for table metadata lock" until the existing SELECTS are done. Also the SELECTs that are issued after the RENAME waits on the lock. Hence the hang you notice.

You could run show full processlist when this occurs to confirm.

To avoid this, you will have to time the rename in the same app that is reading or kill the existing connections that are holding the locks. 

Perhaps the SELECT queries need to be tuned as well. 40K row table shouldn't take that long to return. Since you mentioned all the queries are by ID, is there an INDEX on ID?