STM32F7: activate USART Receiver Timeout Interrupt (USART_RTOR.RTOF)
I simply missed the USART_CR2_RTOEN flag. Only the USART_CR1_RTOIE flag is mentioned in the manual as necessary for the USART_ISR_RTOF to trigger an interrupt but USART_CR2_RTOEN is necessary to first generate the USART_ISR_RTOF flag.

My initialization now looks like this:

// Set RTOR.RTO = 960 (0,1s * 9600baud = 960):
MODIFY_REG(USART6-&gt;RTOR, USART_RTOR_RTO, 960);

// Set CR2.RTOEN (enable Receiver Timeout):
SET_BIT(USARTx-&gt;CR2, USART_CR2_RTOEN); // &lt;- this was missing

// Reset RTOF flag:
WRITE_REG(USART6-&gt;ICR, USART_ICR_RTOCF);

// Set RTOIE (enable RTO interrupt):
SET_BIT(USARTx-&gt;CR1, USART_CR1_RTOIE);


Note: For some strange reasons, the USART_ISR_RTOF receiver timeout flag is set directly after the reception of one byte without a timeout together with the USART_ISR_RXNE interrupt flag. So you need to reset the USART_ISR_RTOF flag at the reception of every byte to not get an interrupt pending directly after the RXNE interrupt. My USART6 interrupt routine now looks like this (with the usage of the ST low level drivers):

void USART6_IRQHandler(void)
{
  if (LL_USART_IsActiveFlag_RXNE(USART6)) // USART6.ISR.RXNE set =&gt; Byte received
  {
    // Byte in USART6 RX register

    // Reading USART_RDR register clears RXNE flag.
    // This reads the USART6.RDR[7:0] register and handles the data
    handleRXByte(LL_USART_ReceiveData8(USART6));
    LL_USART_ClearFlag_RTO(USART6); // for some reason this is already set
  }
  else if (LL_USART_IsActiveFlag_RTO(USART6)) // USART6.ISR.RTOF set
  {
    // RX inter-byte timeout

    // reset RTOF:
    LL_USART_ClearFlag_RTO(USART6);

    // handle timeout (reset receiver state):
    serialHandleRXTimeout();
  }
}