mixing round functions and key schedules what is the security of an encryption algorithm that takes the key schedule of algorithm X and the round function of algorithm Y?
should it need a new security proof? or it relies on X and Y's security?
Assuming the 2 algorithms play well together you would end up with a composite of the cryptographic properties of both algorithms - the key schedule would have the properties of X's key schedule and the round function would have the properties of Y's round function. Taking them out of context of their original algorithm won't (usually) change their cryptographic properties.
Many encryption algorithms have been built like this, and the scheme tends to work. That said, you still need to make sure they play well together but that's easier than reviewing an entirely new algorithm.
Update
A list of algorithms using components from other cryptosystems is as follows:

TwoFish


Twofish's distinctive features are the use of pre-computed key-dependent S-boxes, and a relatively complex key schedule. One half of an n-bit key is used as the actual encryption key and the other half of the n-bit key is used to modify the encryption algorithm (key-dependent S-boxes). Twofish borrows some elements from other designs; for example, the pseudo-Hadamard transform[3] (PHT) from the SAFER family of ciphers. Twofish has a Feistel structure like DES. Twofish also employs a Maximum Distance Separable matrix.


SAFER


In cryptography, SAFER (Secure And Fast Encryption Routine) is the name of a family of block ciphers designed primarily by James Massey (one of the designers of IDEA) on behalf of Cylink Corporation. The early SAFER K and SAFER SK designs share the same encryption function, but differ in the number of rounds and the key schedule. More recent versions — SAFER+ and SAFER++ — were submitted as candidates to the AES process and the NESSIE project respectively. All of the algorithms in the SAFER family are unpatented and available for unrestricted use.


RC6


In cryptography, RC6 (Rivest cipher 6) is a symmetric key block cipher derived from RC5. It was designed by Ron Rivest, Matt Robshaw, Ray Sidney, and Yiqun Lisa Yin to meet the requirements of the Advanced Encryption Standard (AES) competition. The algorithm was one of the five finalists, and also was submitted to the NESSIE and CRYPTREC projects. It was a proprietary algorithm, patented by RSA Security.


DEAL


DEAL is a Feistel network which uses DES as the round function. It has a 128-bit block size and a variable key size of either 128, 192, or 256 bits. For key sizes of 128 and 192 bits, the cipher uses 6 rounds, increasing to 8 for the 256-bits size. The scheme has a comparable performance to Triple DES, and was relatively slow compared to many other AES candidates.


CAST-256


CAST-256 uses the same elements as CAST-128, including S-boxes, but is adapted for a block size of 128 bits – twice the size of its 64-bit predecessor. (A similar construction occurred in the evolution of RC5 into RC6). Acceptable key sizes are 128, 160, 192, 224 or 256 bits. CAST-256 is composed of 48 rounds, sometimes described as 12 &quot;quad-rounds&quot;, arranged in a generalized Feistel network.


Camellia


Like other AES candidates, E2 operates on blocks of 128 bits, using a key of 128, 192, or 256 bits. It uses a 12-round Feistel network. E2 has an input transformation and output transformation that both use modular multiplication, but the round function itself consists only of XORs and S-box lookups. The single 8×8-bit S-box is constructed from the composition of an affine transformation with the discrete exponentiation x127 over the finite field GF(28). NTT adopted many of E2's special characteristics in Camellia, which has essentially replaced E2.