How can I hold Bitcoins safely in the long term? Say I wished to hold a lot of bitcoins for many years as an investment. My worry would be about the long term safety of the different wallet mechanisms. If it's a website - what if it goes bust or goes down? If it's on my phone - then what if I lose my phone?... 

Correct me if I'm wrong but presumably a bitcoin is essentially a long string of numbers, so presumably I could physically write this down on paper, make a few copies as backup and bury them in the garden (or whatever), then I wouldn't have to worry about the website going down.

I guess that my ultimate question is, how can I store some bitcoins for many years without having to worry about the security of my wallet?
You can store the bitcoins in wallet on desktop or mobile. As it's software, you can create multiple copies of the keys and keep it in different devices, say USB, email, desktop, mobile, drop box etc

Yes you can also store it as paper, but it would be humongous to do on your own.