Is the error in a "5%" resistor consistent across measurements?
The answer to your questions is mostly covered in the data sheets. A 5% tolerance resistor will also have a specification for temperature drift, "load life" (drift with time under certain environmental conditions) and so on. It's possible to make a 1% resistor that is just as crappy as a 5% resistor in stability, it's just trimmed closer to begin with (and at a certain temperature). 

Calibration can reduce the initial inaccuracy, but it won't reduce the other kinds of drift. The drift will determine whether you can make a 0.1% circuit with 1% resistors or a 0.5% circuit with 5% resistors. 

Here's a typical temperature characteristic for a cheap resistor. 


There are many other types of drift described on page 6 of the above linked PDF file. 

For example, 



It's up to the designer to come up with an error budget and interpret the data sheet numbers to see if the part will fit the job, given the environment it will live in and the desired product life and required reliability. 

Generally if a part has a close tolerance and a low temperature coefficient of resistance (TCR) the other drift specs will be good as well, but you do have to check the data sheet if it matters for your application. For a pullup resistor, a change of 20% makes little difference, but if it's a voltage divider on a measuring circuit, a 0.1% change might be very important.