Reset TCP/IP Stack without Rebooting?
I don't think there is a way to restart the TCP stack, but the question should be why TCP becomes unresponsive, it must be hitting some limits in the stack.
If the service creates new connections to answer incoming requests, it is possible you're hitting the default 16K limit on ephemeral ports and the even more problematic 30s (MSL*2) timeout on TIME_WAIT connections until they can be reused.
You can use sysctl to tweak those:
sudo sysctl net.inet.tcp.msl=1000 # default 15000
sudo sysctl net.inet.ip.portrange.first=16384 # default 49152

The default TCP queues are also rather small (128 waiting sockets):
sudo sysctl kern.ipc.somaxconn=2048 # default 128

If you are also getting port scans or DoS to unused ports you can reduce backscatter by enabling blackhole mode and limiting ICMP/RST responses:
sudo sysctl net.inet.tcp.blackhole=2 # default 0
sudo sysctl net.inet.udp.blackhole=1 # default 0
sudo sysctl net.inet.icmp.icmplim=50 # default 250

Once you are happy with the sysctls you can persist them in /etc/sysctl.conf.
Use something like sudo vim /etc/sysctl.conf to create it.
I would also suggest to install HAProxy on your server in front of your services and use it to apply rate limiting and filtering for backend connections. You can even do packet inspection like dropping connections with invalid SNI headers. The easiest way to install haproxy on the Mac is through Homebrew.