PostgreSQL on Amazon RDS: out of memory error on bytea field > 64MB
PostgreSQL's handling of large binary fields is very memory-inefficient. It's quite normal for it to keep three or more copies of a given field in memory when procesing a row, albeit briefly. One TOASTed Datum, one deTOASTed Datum, one stringified hex-formatted Datum. I wouldn't be surprised if SSL added another copy.

That said, this doesn't really look like a memory issue to me. Lets say somehow PostgreSQL insanely stored TEN copies of the field. It's only 63MB. You'd need 640MB of RAM. That's nothing for your server. So unless RDS's postgres is patched to do things normal PostgreSQL doesn't do, or runs with some weird control groups configuration or something, it's unlikely to be a true memory problem.

I suspect the real issue is something else.

work_mem isn't relevant here, as it's only considered for things like sorts. It's not a memory quota for a backend.

"Out of memory" can be reported in error cases where something fails that should not usually fail, under conditions that are usually caused by not being able to allocate memory. Given that, I'm very suspicious of the "bad write retry" report from OpenSSL.

You've excluded issues with renegotiations, but in an opaque system like RDS it's hard to tell what else it might be.

Can you show more of the logs? That may be informative. Especially if you retry after setting log_error_verbosity = verbose and log_min_messages = debug1 in your configuration.

Also, what's the output of SELECT pg_column_size(x), octet_length(x) FROM mytable WHERE ..., where x is the problem row?

(Edit your question to add more info, then comment here to notify me)

If this were normal unpatched PostgreSQL on a real host I'd get you to attach gdb and capture the error so we could see exactly what's happening. But in this case your best bet will be to contact Amazon's support, since they're the only people who can actually see what's happening on the host.