Is it possible to have "in-out" communication in a single 56k modem?
The linked article doesn't explain how the system works, how it's connected or what the unwanted caller hears. By the sound of things it is connected in parallel with the line. There are a few things to be aware of:

Caller ID is transmitted between the first and second ring cadence, as far as I know. See Wikipedia's Caller ID for confirmation of this. This means that the phone will ring once before the modem can read the caller ID.
Caller ID is typically spoofed by the baddies. You can't rely on it to be accurate.
Blacklisting could take forever if the marketers keep changing their Caller ID.

On the Amazon page you linked to there is a review by Gordo who says:

I bought this modem to connect to a Raspberry Pi for a caller ID call blocking application.
The good: Raspberry Pi Linux OS recognized the modem without having to install additional drivers.
The bad: Caller ID function worked only about 10% of the time. Most of the time, incoming calls would just show RING without Caller ID while other phones in the house displayed Caller ID properly. But it worked occasionally. Maddening.

He goes on to describe more trouble and changed it for something else.
It doesn't look like a good solution to me. I think an answering machine that asked a question such as, &quot;This is Rafael. To speak to me please dial in the first four letters of the name of the company I work for / town I was born in / mother's name, etc.&quot;, and then accepted or rejected the call would be much better.

I may consider a CAPTCHA approach, like you mentioned. But for that to work, it would be even more necessary to wire the blocker in parallel, not in series. Do you know if this is possible with a single modem?

For the CAPTCHA method the problems become more severe. The phone has to be disconnected by the controller, the CAPTCHA has to answer the call, decide if it's OK or not, if it is, then hold the call, send out ring voltage to the phone, detect phone pickup and transfer the call. No modem involvement at all but a lot more electronics.
A simpler solution might be to use an answering machine with the &quot;CAPTCHA&quot; prompt or even &quot;Please state your name&quot;. Leave it with the sound turned up and enjoy the nuisance callers. I used to answer them, ask them to hold on and put the phone in front of the radio. It helps take them out of circulation for a while.