Computer controlled electronics
Your question is probably too broad to remain open, so I'll make this brief.  There are many development boards out there that interface to a computer in some form, like serial, USB, or over the network.

Serial is getting really old now and less used, and you don't have to do anything special to talk to a device over the network.  If you want to get into the low levels of talking to a device more directly, USB is probably a good thing to experiment with.  The state of USB drivers is far from ideal.  You can't make your own drivers anymore on 64 bit Windows systems without going thru a bunch of certification hassle and expense.  There is some generic USB driver that surfaces most things to the app level, which may allow ordinary people to still write app code for their own USB devices.  I haven't looked at this in detail yet, so could be off a bit.  There is also LibUsb on Linux and Windows.  I think that's a separate open source effort, but don't know what exactly its capabilities are.

So get a development board with some existing capabilities and documented USB interface, and give it a try.  There are many out there, including our own ReadyBoard-02.