What fabric can I use for these clothes?
I'm not really going to address your question, but the underlying problem.

The fact that costumes usually look like costumes and not like "legit" outfits, is because they are produced for temporary use: their cheap materials (and often cheap manufacture) keep prices at a minimum. 

You can indeed improve on the fabric, but, combined with the working hours you'll have to put in it, you will have to see if this option is preferable to buying a higher quality costume. 

My advice, depending on where you live, would be to visit a couple of thrift stores. Find pants, a waistcoat, a hat, a long coat, and any additional accessoires (since you're assembling your own outfit, you might even be able to wear parts of that outfit in daily life, making the investments even more worthwhile).
Whatever you still need you can now make from higher quality fabric, since you'll need less. You might even be able to find proper types of textile in those very same stores.

As a last tip: I believe you can get convincingly good-looking steampunk glasses and pocket watches for relatively low prices online.