Slow access to JSONB column
The JSONB object is probably large enough that it is compressed and stored out of line, in a TOAST table.  Having to retrieve and decompress the TOASTed value for every row is going to slow things down by a lot.

I just wouldn't use JSON or JSONB for this purpose if I wanted speed.  If you need to access a small number of keys, pulling them out into the table proper could help.