Why verifying later blockchain is slower? I'm truly lost about how bitcoin works.

I understand a blockchain contains all transactions ever made. So, what things need to be verified while I'm downloading the whole blockchain? If it's just the hash of each block, the processing/verification speed of earlier blocks should be same as later blocks, right? 

Only better guess I have, is that if anything need to be indexed, then that would explain why later blocks are slower. So is this the case?
Earlier blocks had fewer transactions, and so were smaller. Later blocks have many transactions, and so are much larger.