BIG Binary column - PostgreSQL 9.2
PostgreSQL's MVCC is used over-writing mechanism,  which is roughly mentioned to this document , so all updated rows are inserted into the table. Thus, you cannot get free space.  On the contrary, the table size is temporarily increased.

(Even though autovacuum feature works  very efficiently,) I think you have to do VACUUM FULL in order to certainly remove dead rows(tuples) and dead pages after updating. That way (do UPDATE SET NULL and VACUUM FULL), the size of the table will be small.