What SQL execution time should I expect when querying 4k records in a 150M records table?
It looks like you were measuring the query right after importing a lot of data into the table, so your query has to set hint bits, or like shared_buffers is so small that the query finds no free buffers and has to write out some (written=1118).
Try an index that includes id so you can get an index only scan and VACUUM the table:
CREATE INDEX ON answer_submission (student_id) INCLUDE (id);
VACUUM answer_submission;

That should make the query fast.