Is there a counter mode of encryption where the key is being changed for every block of data? I thought like encrypting an IV concatenated with a CTR but instead of XOR-ing this output with the plaintext this output should be used as a temporary key for another encryption block that actually converts the plain- to ciphertext. In other words in a regular CTR mode of operation the XOR would be replaced by a full encryption algorithm and the OTP would be its key.
There are some ciphers which do this, most notably E0, a 128-bit stream cipher based on four LFSRs that is used in older versions of the Bluetooth protocol. As vulnerabilities have been found in E0, a scheme was devised to reduce the security impact called two-level E0. Normally, the cipher encrypts 2745 bits (one Bluetooth frame) at a time. With two-level E0, every 128 bits of the keystream is used to seed another instance of E0 which generate only 2745 bits of keystream from that key.



Note that this is not a special mode of encryption, but the use of true stream ciphers. This difference is minor, as counter mode is merely a way to convert a block cipher into a stream cipher. Most block ciphers (and many stream ciphers) have a key schedule, which is an expensive setup phase whenever a new key is used, making it impractical to frequently switch keys. The stream cipher E0 does not have a complicated key schedule. The 128-bit key is used directly as the cipher's inner state (specifically, it is split up into 25, 31, 33, and 39 bits, each of which is used as the initial state for each of the cipher's four LFSRs). This allows it to change keys frequently without a significant performance impact.



Please note that E0, even two-level E0, is not a particularly secure cipher. The key must be changed very regularly or known-plaintext attacks can allow for key recovery. These weaknesses stem from both problems in the design of two-level E0, as well as the fact that E0 is built off of LFSRs, which are non-cryptographic algorithms. Using AES in CTR mode will be far more secure than any type of E0.