Arduino Uno's Timer maximum frequency using Timer Compare Interrupt, not Timer Output Pins (e.g. OC0A)
How can one explain that using this code I get this toggle frequency on the digital pin ?

Code takes time to execute. Your ISR, when compiled, looks like:
ISR(TIMER0_COMPA_vect)
{
 124:   1f 92           push    r1
 126:   0f 92           push    r0
 128:   0f b6           in  r0, 0x3f    ; 63
 12a:   0f 92           push    r0
 12c:   11 24           eor r1, r1
 12e:   8f 93           push    r24
 130:   9f 93           push    r25
    PORTB ^= 1 &lt;&lt; 1;  // Toggle D9; this allows to measure the toggle frequency, however this could be replaced by some code to generate a custom binary signal
 132:   85 b1           in  r24, 0x05   ; 5
 134:   92 e0           ldi r25, 0x02   ; 2
 136:   89 27           eor r24, r25
 138:   85 b9           out 0x05, r24   ; 5
}
 13a:   9f 91           pop r25
 13c:   8f 91           pop r24
 13e:   0f 90           pop r0
 140:   0f be           out 0x3f, r0    ; 63
 142:   0f 90           pop r0
 144:   1f 90           pop r1
 146:   18 95           reti

That's 18 assembly instructions, each needing between 1 and 4 clock cycles to execute.  Referencing the datasheet you can count the exact cycles needed and sum them up, yielding 31 clock cycles.
Toggling at maximum speed that would be 16MHz / 31 = 516kHz.
And then you have the rest of the code running between each call to the interrupt, so it's slower than that.

How can I generate a custom binary signal over a digital pin at the highest possible frequency ?

By not using interrupts, and writing a tight loop in assembly language.