Is there any reason that any speaker connected to any computer couldn't be used for listening/surveillance?
I take your question to mean "can software use the speakers in computers at the moment as microphones".

And not to mean "can we change computer electronics to use their speakers as microphones".

Currently, computer speakers are generally (almost always) driven by an amplifier, with no electronic circuitry to read that speaker signal. So there is no mechanism there to 'listen' through the speaker using the existing electronics.

As you suspected, amplification itself is a one-way street. It is possible to design more elaborate circuits that aren't one-way. But that's not what your desktop or laptop PC has driving the speaker in it.