Linux (OS) on Custom Hardware (Custom Motherboard)
This is too broad a question to answer, but in general software will be a killer. You only need to look at the low take up of the higher powered Raspberry Pi clones to see that the software and community is really what makes the Raspberry Pi so successful. 

Realistically, you will need to a hire a professional developer if you want to make this design from scratch. If you don't want to or can't do that, an off the shelf solution (for example the RasPi Compute Module) will be the best way to go unless you're willing to learn a lot about the internals of the kernel.