How can I determine whether a student has written an excellent paper themselves, or hired someone else to do it?
Are there any techniques I could use to determine whether an essay that I suspect was written by someone else, but not published elsewhere, is a student's own work? 


Ask the student to come to your office hours or a private meeting. 

Then, say: "I was really impressed by your recent paper, it was an excellent piece of work! Let's discuss it some more."

Student who did the work will be able to discuss said work intelligently.

Student who didn't do the work is unlikely to be able to carry on an intelligent conversation about said work. 

(This still isn't proof of any kind, and you can't accuse student of cheating without some evidence - but if you're lucky, student will at least be spooked enough to never do it again.)


  Are there any steps I can take to ensure I don't receive questionable essays in the first place?


Assign essays that are a few steps beyond, but still closely tied to, what's been discussed in class, such that somebody who wasn't in the class will not be able to produce the kind of essay you're expecting (the effectiveness of this depends very much on the subject matter).