How to listen to radio communications from a transmitter?
You don't tell us anything in your profile about your level of experience or education, so I'll give a very generic answer.

The best tool to start with would be a spectrum analyzer, which can tell you what frequency the system operates on, and also allows you to learn something about the overall patterns of activity, and can offer clues about the modulation techniques being used.

But to really dig into it and completely reverse-engineer the system, I would recommend using a software-defined radio (SDR) testbed. This is a simple hardware RF front end and digitizer that converts the signal into digital data, and all subsequent filtering, demodulating and decoding is done in software. This makes is easy to try different parameters and algorithms until you have all of the details.