What happens when the lock limit is reached?
max_locks_per_transaction does not apply to rows, but to objects.
The fact that rows do not count is made explicit in the documentation:

From https://www.postgresql.org/docs/current/static/runtime-config-locks.html


  This parameter controls the average number of object locks allocated
  for each transaction; individual transactions can lock more objects as
  long as the locks of all transactions fit in the lock table. This is
  not the number of rows that can be locked; that value is unlimited.


The information necessary to row locking is written in the row itself.