postgresql query hangs indefinitely over VPN - new firewall
We solved the problem. Technically, we never figured out exactly why it was happening, but UDP packets were getting received by the VPN server in an incorrect order, or sometimes not at all. The firewall didn't give us any indication that it was doing this, so as best we can figure it was an unlogged side-effect of CheckPoint's multiple layers of threat protection.

We switched over to using OpenVPN over TCP instead of UDP, and that seems to have solved the problem. Hopefully we don't suffer any adverse effects from that in the long-term!