How do I detect what kinds of signal(s) this device receives/sends
U3 is clearly associated with the antenna that you can see embedded in the PCB. It could be something like the TI CC1101, using a completely proprietary protocol.

This is a very flexible chip, so to reverse-engineer it, I would start by "snooping" its SPI control bus with a logic analyzer to try to figure out what register settings the application is using.