Can I configure the modem on my G5 to only allow calls from certain numbers to ring through?
I can think of a variety of methods to do what you want, but none of them are easy solutions. You'll probably need to write some code. This question should be tagged PBX, but that is too obscure for this site.

You'll probably want minicom installed to control the G5 modem. Get it via fink or macports. If you can use minicom to check the number without answering, then you are mostly done. If it doesn't match, ignore it. If it does match, you'll need to find some way for minicom to answer/forward/pass the incoming call out the modem out line. I'm not yet sure how to do this.

If minicom needs to answer the call in order to determine where it came from, then the situation is as above, but you might need two modems. If it is a call for the security center, pass it out one modem. If it isn't, pass it out the other, which goes to your home phone I presume.

If minicom cannot determine the number, or if can but has to answer in order to check it and this is problematic, then I would get a caller ID display, put it in next to the G5, and use a webcam to read the numbers off the caller ID. OpenCV would help with this part, or any script-able optical character recognition (OCR) software. You can get gocr from fink or macports too, and that might be sufficient. Determine the number and if it should go to the security console, have minicom patch it through.

But, why do you need to call in?

If your G5 is at home and online, it might be easier to not call into the security system, but SMS or email yourself, have the G5 fetching the mail, and taking action based on that (perhaps that action is using minicom to connect to the security system, but you don't need a phone line to the outside world in this case).