Can I replace the 433mhz modules on my homemade remote controller with a nrf24 transreciever
Not directly, no.  But you can design a system to use the nRF24, you can even make it compatible with the many existing RC systems that use nRF24 or similar radios.

The main issue is that your 433 MHz radio is a crude OOK system where the encoder injects pulses via a single line and the decoder must accept pulses via a single line, distinguish valid ones from noise, and unpack them into some meaning.

In contrast, the nRF24 is a tunable digital packet radio transceiver with an SPI interface.  Various registers are set to configure the mode of operation.  Data is written into a packet buffer, sent as a complete message of some number of bytes, and received into a buffer on the other end where it can be read out over the SPI.

To make use of the nRF24 you'll need an MCU that can implement the nRF24's SPI-based interface.  Most with 5 free signals (sometimes 4, bare minimum 3) can be made to do this.  There are for example quite a few relevant Arduino projects you can find with some web searching, and probably a few for other chips like the STM32F103.