Why do we refer to previous transactions in each transaction? Isn't it enough to just have the Input Address, Output Address, Amount and Signature for each transaction?
We don't refer to previous transactions, but rather to specific outputs of previous transactions. When transactions sign over money to a recipient, they create a uniquely identifiable "Unspent Transaction Output" or UTXO. Transaction inputs in turn spend these "UTXO" to fuel transactions. Since addresses can be reused, addresses alone are insufficient to precisely identify the bitcoins that are being spent.

If you only had "Input Address, Output Address, Amount and Signature for each transaction". This would be trivially broken:
Let's say Alice keeps all her funds in address A1, and pays Bob 0.1BTC in rent to address B1.
Bob pays Mallet to fix Alice's heater from B1, using the funds that he received from Alice.
When Alice pays Bob next month again from address A1 to address B1, Mallet can replay Bob's transaction to himself in order to get paid again.