What are the must-checks for purchasing an airplane?
There are plenty of great $20,000 aircraft out there - the hard part is finding them.

"What to look for" is an incredibly broad question, and varies substantially between different types of aircraft: You would not look for the same things on a Piper Comanche as you would on a Cessna 182.
Accordingly the first thing you want to do is find a mechanic who is an expert in the particular type of aircraft to help you check it out and do a "pre-purchase inspection".  

Finding those mechanics is actually pretty easy: Look for a "type club" that supports the aircraft you're considering and ask them for recommendations (e.g. for a Cessna I'd talk to the Cessna Pilot's Association, for a Cirrus COPA, for most Piper aircraft the Piper forum, for Beechcraft Beech Talk, etc.

Talk to folks on those forums (and to your mechanic) about what you, as a buyer, should be looking at before you get the mechanic involved for the more detailed inspection. There are lots of things you as a pilot (even a student pilot) can determine just by looking at a plane (and/or the logbooks).

Some key items to look at before you go see the plane:


NTSB Records
Like Larian mentioned go to the NTSB site (or the equivalent if you're not in the USA) and search for accident records. Not every accident will show up in the database, but sometimes you'll find out that the cream puff aircraft being advertised has actually been "substantially damaged" in an accident. The NTSB database search is free and there's no reason not to do it.
Logbooks
All aircraft have logs, so if possible get a copy of them before you even go to look at the plane.
Read through the logs (yes, all of them): You'll find all sorts of interesting things in there, but what you're mainly looking for is good, consistent maintenance and regular use of the aircraft. As a secondary concern you're looking for major repairs (or significant minor ones) that you might want to ask your mechanic about.
FAA Records
You can request a copy of the FAA's records for an aircraft online for a small fee.  This includes registration information and any records of major repairs/alterations ("Form 337") filed for the aircraft.  It is generally an "interesting subset" of the logs.
This is something you'll want to have handy when your mechanic looks at the plane, though it's not something you need to have before you go look at it yourself. Theoretically the owner should have all the documents that this record contains as part of their records/logbooks, but in practice paperwork sometimes gets "lost".


If all that looks good it's time to look at the plane:


How is the overall condition of the aircraft?
Cosmetics can lie, so don't be dazzled by a shiny coat of paint and a nice interior:


Is the on-board paperwork (ARROW) all in order?
Are the tires, hoses, and other rubber components in good condition?
Is the engine bay (relatively) clean? Are there any leaks, drips, or other items of concern?
Are there obvious signs of damage (skin patches, dents, fabric tears, etc.)?
Are there obvious signs of corrosion (bubbles under the paint, etc.)?
Is the paint in overall good condition (no major chips, no cracks, no flaking)?
Is the aircraft itself clean?
(As much as cosmetics lie, a plane that looks like it's been neglected probably has been...)

Does the aircraft pass a very thorough preflight?
Go through the preflight checklist with an eye for anything abnormal -- ask yourself "Would I fly this aircraft today?".
If the answer is no, think carefully about why.
If the answer is yes it may be time to test-fly with the current owner.
Does the aircraft operate correctly on the test flight?


Do the engine and propeller operate properly?
Do all the instruments and avionics function correctly?
Does the aircraft fly well? Can you trim for hands-off straight and level flight?



If the aircraft passes all your checks it's time to turn your mechanic loose on it - they will inspect the aircraft for problems (major and minor) that you as a pilot might not notice which can wind up costing you a lot of money. This pre-purchase inspection (and the ensuing negotiation based on what the mechanic finds) is usually the last stop before handing over your money and taking the keys.