Typical vs Maximum External Flash Write Times
The way I understand the flash write process (others please chime in) is that during the write operation,

The flash monitors the charge on the cell (the floating gate),

Detects when that charge reaches a specified level

Continues the write process for some period of time to ensure some margin


Therefore, to some extent, the write time is a statistical process, with the absolute max being the 4 ms specified in the data sheet.
So if you want a robust design, you need to design using the 4 ms worst case number.