How do I setup a server in the cloud for machine learning? As datasets and the number of parameters get larger, it becomes increasingly difficult to run validation locally because of limited disk size and computing power.  As a result, people might use an old laptop or a server in the cloud.  In particular, I'm interested in learning more about the second option.

For the setup, it sounds like I will need a storage system and a computing system.  Is this S3 and EC2?  Redshift and Amazon ML?  Combinations from some other vendor?  At the moment, I'm asking as an amateur participant for Kaggle, but I'd also be curious to know what the pros use.

On the machine itself, is there any way to interact with a GUI rather than a command line to set this up? Would I have to install Anaconda or some other Python distribution before getting started?  Is there a specific file structure to use? What are the main pitfalls to watch out for?

Ultimately, I am looking for the practical advice to set everything up.  If you're able to just offer links to documentation, that would be immensely helpful as well.  Thanks!
If you are looking for an external storage, then I would suggest you Redshift.

Redshift is a central warehouse and a columnar data store. It allows complex and huge data aggregations and joins; so it is a nice bet for serious Kaggle participants. (I personally use Redshift as data science architecture for kaggle.)

Combinations from some other vendor?

No, a combination of Redshift and S3 is enough. The data is stored in S3, and then loaded into Redshift using the COPY command.

Is there any way to interact with a GUI rather than a command line to set this up?

Yes, there is a CLI interface for Amazon AWS. However, you can use the boto library on Python for handling this stuff.

Would I have to install Anaconda or some other Python distribution before getting started?

Not a compulsion.  Anaconda is a wrapper of numerical and scientific libraries. You can install them on your own or intall Anaconda.

Check out PySpark, which allows you to do handle BigData in Python.