Single-board computer
It depends a lot on which embedded system are you planning to use.
A good alternative to desktop replacement (with all your requirements) is Intel DG45FC mini-atx. With a core 2 duo socket and an embedded graphic processor you can make any application run as in a desktop.



A cheaper and smaller alternative is D410PT (or D510MO for dual core processor) with intel atom processor. Its also an x86 processor which is pretty good if you're not planning to go on more embedded kind of operational systems, but still powerfull enought for the majority of the tasks.



Another alternative is beagle board. The built in processor is an core ARM. In this case you'll need to build your program based in Android, Win CE or some Linux flavors. I think that maybe this is the alternative that may suit you better.