Is Android's system recovery a firmware thing or Android thing? Can factory reset be prevented? I'm researching on how to protect my phone from being able to be reseted by thieves. Doing some research online I figured out that the main method of putting a stolen phone to work again is to do a factory reset through rebooting and using some combinations of volume + power buttons to activate the factory reset.
I'd like to know how this reset works. I've seen that on a Moto G 2nd gen, it had options like 'factory reset' or 'recover from SD card'.
First of all, this program, let's call program A that starts when you press the volume + power is running from firmware or kernel? Can it be erased?
I'd guess it's firmware, but it'll only accept a new system from an SD card if it's signed. So, the option a thief would use is the factory reset.
Supposing that I cannot block the program A from running and neither modifying it to supress the factory reset option, is it possible that I can make it factory reset to a custom OS, not the installed one? This custom OS could not work or have some monitoring programs.
Android recovery is part of boot.img located in /boot partition. Hold key press signal for bootloader and recovery mode are hardcoded in android bootloader (ABL). ABL listens to keypress signal and boots bootloader or recovery mode. In Qualcomm Snapdragon devices, if ABL is corrupted, then Qualcomm's Extended Bootloader (XBL) fallsback to EDL mode for flashing device partitions.
See the bootflow of device with Qualcomm's SoC:

In this image, it shows kernel boots recovery which is not the case anymore. In A/B slots, recovery is part of boot.img so it can be directly booted by ABL.
Android has weak implemented physical security. Someone with physical access can access bootloader and recovery mode by hard restart of the device and pressing key combinations. Hard restart is hardwired to the SoC. By pressing power button for 15 seconds, SoC reboots and this feature is independent of the OS but reboot policy like hold press delay timer is hardcoded in firmware of SoC.
If factory reset protection (FRP) has no unpatched vulnerabilities, it will prevent reuse of stolen device after factory reset without authenticating your account. But OEMs implement it poorly and its bypassing methods become quickly available.
A good security design is when an unauthorised person shouldn't even have authorisation to factory reset the device. Android 8+ devices are fully capable of authentication for bootloader mode and recovery mode. Modern android devices are equipped with TEE which wakes on boot and in Qualcomm devices, it is awaken by XBL. TEE can perform biometric authentication or can use PIN authentication of screen lock even if the host OS is not loaded yet. Why AOSP and OEMs are not enforcing it doesn't have a justifying execuse.
Whether your bootloader is locked or unlocked, there's no anti-erasure protection. But atleast with bootloader locked, fully patched FRP can't be bypassed.