What does "Baking" mean?
There are a couple different kinds of baking.
Texture baking
Baking could mean that you calculate specified data, and save it to a texture, which then you use in your material.
The purpose of baking is to save in render times, because Blender recalculates all shadows, lighting and what not, every single frame, which results in a lot of time wasted time for long animations.
However, render baking should only be used on objects which don't move, or aren't affected by other moving objects (except for the camera), otherwise the viewer will notice that shadows are moving when they shouldn't be, or are not moving when they should, breaking the illusion of realism.
Physics baking
Baking can also mean precalculating physics simulations such as Cloth or Rigid Body. For this Blender saves them to a file, which then loads when rendering or playing back the animation so that the physics don't have to be recalculated every time.