Trying to run Timer1 in ATmega2560 with assembly
Here:

ldi r20, 0b00000010 ; CTC mode, int clk;
sts TCCR1A, r20     
ldi r20, 0b000000101 ; prescaler /1024
sts TCCR1B, r20


you are setting the bit WGM11 on TCCR1A. The timer then runs in mode 2 (PWM, Phase Correct, 9-bit) rather than mode 4 (CTC).