Why wouldn't miners change the protocol? (e.g. change difficulty) How about a scenario where a few miners change their protocols by setting difficulty to something easy. Let's take it to the extreme and say they are able to accept any block. 

They would keep publishing blocks which they think are valid and those would get rejected by the network. What if more miners start joining that scheme though (by changing their protocols)? 

Because they are able to publish blocks extremely quickly (because they set their difficulty low) they might actually be able to compete with the whole network and their branch might overtake the network branch even if they are not that many of them.

What am I missing here?
The rest of the network, which uses the existing difficulty parameters, will reject the new blocks generated by your modified miners. There will be a longer blockchain, but there will not be a longer valid blockchain according to the established rules.