Can mountain bike dirt remaining on different parts for long time damage the bike?
The main concern is water, and any other substance that might promote corrosion, such as road salt.

Leaving mud or dirt on the frame provides material for water to soak into. Even if you let it dry out it would held water for longer than it would take to evaporate otherwise, and can soak up more water later.

You obviously want to get grit and dirt off of your chain and sprockets as it sticks to the lubricant and promotes wear.

Bearings do have seals, so if you are careful with degreaser there should not be a problem  getting it into bearings and affecting the grease or other lubricant.

Update:

As brought up in comments there are areas that are more prone to damage through corrosion than others. Fork stanchions in particular.