Calculating phase response of filter with latency
Thanks to everyone who provided helpful comments. As some suggested shifting the signal might be the way to go, but this results in the blue line where the green line is the phase I would expect.

It turns out to correct the phase after taking the DFT of the impulse response of the filter when the filter has a delay each bin needs the following angle adding to it:

$$
\frac{ 2\pi i L} {N}
$$

Where $L$ is the latency in samples, $N$ is the FFT size and $i$ is the $i$th FFT bin. So I was missing this as step 5.

For anyone else who stumbles upon this, the phase will end up being neither wrapped or unwrapped. So you will need to take the remainder of the final phase modulus $\pi$. This will give you the normal wrapped phase (as a correctly quadranted arctan would produce by just taking the phase of the complex valued FFT bin) and then you can unwrap as normal.