Why objects usually don't exibit fluorescence/phosphorescence?
You are correct in most of what you write. Absorption in the visible (and uv) parts of the spectrum are of far larger energy that IR and microwave and the lowest electronic transition involves promoting an electron from the highest populated orbital into the lowest unoccupied one. Once the excited state is formed it can emit via fluorescence and phosphorescence, which are both at a slightly lower energy that the light absorbed and this is because higher energy vibrational levels can be populated in the ground state. (Look for picture of a Jablonski diagram). 

However, fluorescence is emitted into all directions ($4\pi$ radians due to molecular motion between absorption and emission) and only a small part of this could enter the eyes but also and most importantly there are non-radiative processes that competitively remove the energy from the excited state ultimately populating high vibrational levels in the ground state from which energy is rapidly lost into the surroundings. These non-radiative events mean that for most types of molecules the fluorescence yield is small, which means that only a few % of the excited state's energy ends up as photons. The non radiative processes are called intersystem crossing between singlet and triplet excited state and internal conversion between states of the same spin state. The processes causing these (besides and chemical reaction) are anharmonicity in molecule vibrations and particularly heavy atoms or unpaired electrons as part of the molecule itself or in nearby molecules such as in a solvent. Some molecules may also have symmetry forbidden optical transitions in emission but not in absorption: carotenes are of this type.