I2C Level translator 5 V to 1.8 V using MOSFET, output look weird
I2C output is an open drain output. It means it can not provide 5V at output. It can only switch between 0 (low) and High-Z  (high - in this case pullup resistor provides 5V at the line, and not i2c Output).
Your simulation should look like:

LT Spice uses some parameters when you don't give it right model. As you see it looks good.
But as you can see at Page 3 of AppNote (Chapter 2), this solution is not good for high speed mode.
The bidirectional level shifter can be used for both Standard-mode (up to100 kbit/s) or in Fast-mode (up to 400 kbit/s) I2C-bus systems. It is not intended for Hs-mode systems, which may have a bridge with a level shifting possibility.
If you try with BSS123 FET in your simulation you don't get perfect result. FET can't turn off fast, because of gate-drain and gate-source capacitance.
You could maybe try with other FETS and different values of pullup resistors.