First time buyer, questions before I screw up
Yes, if you don't want the hassle of registering or giving your details to a website an ATM would be the better choice. Note that ATMs usually charge more than an online exchange.

They give you a bitcoin address and its QR code as well as a private key and its QR code. 

The private key is all you need to have control and access of your coins on that address. The private key can be used to calculate the public key and the latter can be used to calculate the bitcoin address (which as I mentioned is given already for convenience).

Two very good mobile wallet apps are Mycelium and Airbitz. You can use the wallet app to scan the private key QR code and have complete access to that address (ie. send coins elsewhere). If you scan the address QR code you can check the balance of that address but not spend any coins from it.