How does the optimization process in hindsight experience replay exactly work?
Yes, you need to add goal state as an input otherwise you won't know what goal you are trying to pursue. It might take a lot of memory to run HER on ATARI games though, also defining final goal is not straightforward for such environments