Estimating Path Loss from Measured 2.4Ghz/Wlan Signal Strength to get Distance
I believe the router's receiver LNA has AGC (Automatic Gain Control), which attenuates the signal level if the received power is stronger than the LNA power limit. This is probably why you get -30 dBm received power level even when Tx and Rx are next to each other.

Assuming the formula you have included is a good approximation, I believe you can get a good estimate when RSS is in the mid-range of your RSS measurements, since that is where AGC gain is relatively stable.