How do I wire a 9 pin serial connector into a RJ11 4pin connector?
You may be able to trace the connections out on the board. If it goes to a standard RS-232 interface chip such as a MAX232 or MC1488/9 then you have at least part of the answer. 

If it's a 3-wire interface (pretty common) there will be only three wires required- ground, which you should be able to trace out easily. Data input which will go somewhere on the board (other than ground) and won't measure much, if any, voltage when the unit is powered, and Data output, which should measure significant voltage (mark state/idle voltage on RS-232 is usually -9 to -15V). 

That's assuming it's RS232. If it's some other serial protocol, you're not going to have much luck figuring it out, I fear. Also, if it can affect security there will probably some protection against spoofing the signals, cryptographic or otherwise, for obvious reasons.