Trouble replacing a reed switch with a wired sensor along long stretches of wire
The sensor input is working with ac voltage.

Wires that are not working have a higher capacity that triggers the alarm. The weird not releasing issue is caused by the sensor input hysteresis.  

No easy fix, just to change the wire. 

Edit: 

Actually there is but it works only if not battery powered , use the sensor to drive a reed relay placed near the alarm input or a mosfet transistor but this depends on the input.