Token vs contract - are they the same? The Ethereum ERC20 token is also the contract. The token can never live without the contract. The name of the contract is the name of the token. In the code, the keyword is contract and not token.

I may hold 4 ETH but does that mean I'm holding 4 ETH contracts? If that is the case, why call them tokens at all? 

Is everyone exchanging tokens or contracts?

For an ICO, there is an ICO contract created to limit supply and issue tokens. If the contract is the token and the ICO contract goes away after the ICO, how do the tokens exist?
On ethereum, all tokens are contracts, but not all contracts are tokens.

ERC20 is a specific type of contract that is recognized by everyone to be a token contract. It has a base definition and structure that is defined in the ERC20 spec regarding how transfers, balances, approvals, issuance, etc. work.

An ICO or anyone issuing a token will write an ERC20 contract that specifies things such as the initial supply, price of the token (if it is being crowdsold), bonus rounds, etc. This contract is deployed on the ethereum blockchain, and functions as the token. Any given user's token balance is simply a value that the contract says that user controls.

Ethereum is native to the platform, and does not have an underlying contract. It is not an ERC20 token.

You can also have non-ERC20 contracts, such as the lottery contracts, DAO contract, Ethereum Name Service contracts, etc. A contract can be any variety of programs in solidity.