How do test pilots determine an aircraft's capabilities?
Short:  The pilot has a well-defined range of discretion within progressively increasing bounds very tightly set as part of a structured development program aimed at eliciting maximum information regarding actual versus theoretical flight envelope in all conceivable modes and situations. 

Longer: A test pilot's main task is to carry out a progressive program of tests aimed at investigating the correspondence between designed and actual capabilities, and identification of any unexpected variations from nominal which occur along the way. The designers know everything they consider the aircraft should do and shouldn't do and should be able to do long before it first flies. The job of the test pilots is to find out the places where the designers are wrong by carrying out an extremely well planned program while applying her or his skill to come back alive when the designers and/or manufacturers get it wrong. 

The pilot will usually have input into the process of deciding the rate which the program is carried out, how large progressive steps are and attempting to anticipate in advance what is likely to be viable or problematic. Their feedback along the way will help inform decisions about modifying program objectives or rate of implementation or lead to modification to the aircraft. A highly skilled test pilot is a member of a synergistic team and while program decisions are the responsibility of the 'project manager' (or equivalent) it is likely that what the pilot says and wants will be close to indistinguishable from what actually is done. 

However, the actual program on a given occasion is not the pilot's responsibility and limits on what are attempted or implemented are well-defined in advance. The pilot will be given discretion in advance regarding ranges of parameters which may be investigated, e.g., closeness of initial approach to Mach 1, rate of ascent or g forces applicable in various modes*. And the safety of the aircraft first and the pilot second (in many cases) is left to the pilot's discretion BUT subject to substantial guidance, e.g., 'bringing the aircraft back intact' is often going to provide far more information when a problem occurs so a pilot may make every effort to do so in situations where an ejection may otherwise be preferred. A  pilot who ejects more than most may or may not be criticized for any given decision but is less likely to be given as many chances in the future to risk their life even more than before. However, a pilot who crashes fatally while trying to save an out of control situation will in the future only take part as the subject in training videos and in establishing "see if you could have handled this one" simulation scenarios. 

Striking the right balance between these two 'modes' is what makes a great test pilot. But they still don't just get to go out and "see what it can do". 



A well judged ejection occasion - genuine.  

From here

English Electric Lightning early testing.  Prior ejection would very likely have been justifiable but is unlikely to have been considered. The pilot (George Aird) survived with relatively minor fractures - in part caused by descending through the roof of a glasshouse :-). Pilot was on final for an emergency landing as a fire was attacking the rudder actuator. At 100 feet altitude and 10 seconds from touchdown, Murphy won, the actuator support burned through and the plane pitched nose down. Go!



Glasshouse with pilot and ejector seat entry points (side by side) ringed in red. Canopy caused damage at far end. 



Photographers are lucky to get one opportunity like this in a lifetime.
I'm still waiting :-)



What do I know?: Layman. Engineer. Non pilot. Read too many test-pilot autobiographies. The ones you read tend to be from those who are good and didn't die. 




I was recently rererereading Roland Beaumont's "Phoenix into Ashes" - an  account from the leading test-pilot of many (most) of the early British jet aircraft. He recounted the loss of a test aircraft while pulling 9g (as I recall) in a situation which should have been inside the design envelope. The pilot died. The observer lived to provide 'human black box' functionality. It was established that fuel transfer protocols were not correctly followed, leading to over-stressing part of the airframe and thus fatal loss of control, i.e., "just" mishandling fuel tank sequencing may kill you. Nowadays computers will tend to stop this sort of error. But Murphy has many alternatives.