How can I use an ATX power supply for my own experiments?
I don't want to whine too much about safety and I don't like kilometers long safety notices which don't say anything, but I feel that my behavior is justified in this case, so here it goes: If you're uncomfortable with opening the supply, DON'T USE IT AT ALL!!! IT COULD BE VERY DANGEROUS!!! I'll try to explain a bit later why.

I'm actually planing to modify a PC power supply for my project too, so I have several useful links to share.

Take a look at following links:
http://web2.murraystate.edu/andy.batts/ps/powersupply.htm
http://www.wikihow.com/Convert-a-Computer-ATX-Power-Supply-to-a-Lab-Power-Supply
http://www.instructables.com/id/Convert-an-ATX-Power-Supply-Into-a-Regular-DC-Powe/
http://www.instructables.com/id/Yet-Another-ATX--Lab-Bench-Power-Supply-Conversion/
http://www.mbeckler.org/powersupply/
http://www.marcee.org/Articles/PCPowerSupply.htm

They show several projects where people used ATX power supplies for home experimenting. 

I'd also recommend that you read some basic theory about switching power supplies. It will be clear to you then why yours isn't starting. I don't have any interesting links at the moment, but here's a start.

Basically, for SMPS to work, they need a minimum load. That's why yours isn't starting. When power supply is correctly connected to a computer, the computer will provide minimum load to start the supply. As you can see, if you follow the links above, people get around that by providing a low resistance resistor with high power rating and connecting it to the supply. The resistor will provide load for the supply to start. Some articles say that one on the +5 V line is needed, but on some supplies a resistor may be needed on each of the power lines. That's one of the reasons why you should open the supply and take a look how it's working inside. You should be able to identify which cable is connected to which power line. If you can't then I'd recommended that you do some more researching before doing anything. Some articles mention that they use 10 Ω 2 W resistors on + 5 V. I'd use a bit stronger resistor, so its power rating wouldn't be exceeded (but articles also mention that resistors are cool).

About sensing wires: They are there to allow supply to regulate itself and some supplies may have as much as one for each power line. Articles I linked talk about them and on how to identify them. If I'm correct, they should already be connected to the 24 pin connector and ready to work, but as you mentioned, you didn't connect anything to the connector, so it's not surprising that they are making problems. You'd have to modify PSU cable if you want to solve the problem with them, but if you're still uncomfortable with opening the supply, I'd recommend that you don't try modifying the cable.

As for detecting if the supply is working correctly, use the PWR_OK pin and a voltmeter. You should get +5 V once the supply has stabilized itself.