Why are the initial queries up to minute slow on a 400M record PostgreSQL table?
That are caching effects. The first time, data are read from disk, the other executions already find the data in RAM.
There are several options:

Have enough RAM in the machine to cache the whole table and the index. Then use pg_prewarm to load it into the cache before you begin.

Get faster storage.

Use a covering index (which may get large):
CREATE INDEX ON result (p_id) INCLUDE (id, text, answer) WHERE (dirty IS NULL);

See that the index is cached in RAM.