We're assuming that the adversary knows e, so if there's only a small possible set of m values, the adversary can just try them all and see which one maps, so it's very dangerous to use a small m as the input message for RSA. 