What stops this style of attack on the Bitcoin network?
Then when my chain (which is essentially empty with just my mining address full of Bitcoin) is as long/longer than the current consensus I can broadcast it and nodes/miners should take the longer chain which will be mine.


It won't.

To compare the "length" of chains for the purpose of determining which of two valid chains is to be accepted, Bitcoin Core (and I hope all other implementations) look at the amount of work that went into them. Roughly speaking, this means the sum of the difficulties of all the blocks in it.

The only way you'll have more work than the existing chain, is by doing more mining than what went into the existing chain, in a short amount of time. In order to not be outrun by the existing network while you're performing your attack, you'll need (significantly) more hashrate than all the rest of the network combined has. So, what you're talking about is a 51% attack.

That said, there is a related, and far less serious attack possible. You could mine thousands of short low-difficulty branches, that are all valid, but won't ever be accepted, and broadcast them. If nodes would accept them, they'd eventually go out of memory from storing all your headers. The solution to this is the client checkpoints that MCCCS talks about in his answer. Note that the scope of checkpoints has been reduced several times, as they used to protect against many types of attacks that are now prevented through measures that don't rely on hardcoded knowledge about the network.