title: Python TCP Payload Duplication - Passing through data to multiple endpoints concurrently
body: <p>this is my first post here!</p>
  
  <p>My goal is to duplicate the payload of a unidirectional TCP stream and send this payload to multiple endpoints concurrently. I have a working prototype written in Python, however I am new to Python, and to Socket programming. Ideally the solution is capable of running in both Windows and *nix environments.</p>
  
  <p>This prototype works, however it creates a new send TCP connection for each Buffer length (currently set to 4096 bytes). The main problem with this is I will eventually run out of local ports to send from, and ideally I would like the data to pass from each single incoming TCP stream to one single TCP stream out (for each endpoint). The incoming data can vary from less than 1024 bytes to hundreds of megabytes.</p>
  
  <p>At the moment a new outgoing TCP stream is initiated for every 4096 bytes. I am not sure if the problem is in my implementation of threading, or if I have missed something else really obvious.</p>
  
  <p>In my research I have found that select() could help, however I am not sure if it would be appropriate because I may need to process some of the incoming data and respond to the sending client for certain cases in the future.</p>
  
  <p>Here is the code I have so far (some of the code variations I have tried are commented out):</p>
  
  <pre><code>#!/usr/bin/python
  #One way TCP payload duplication
  import sys
  import threading
  from socket import *
  bufsize = 4096
  host= ''
  
  # Methods:  
  #handles sending the data to the endpoints  
  def send(endpoint,port,data):
      sendSocket = socket(AF_INET, SOCK_STREAM)
      #sendSocket.setblocking(1)
      sendSocket.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
      #sendport = sendSocket.getsockname
      #print sendport
      try:
          sendSocket.connect((endpoint, port))
          sendSocket.sendall(data)
      except IOError as msg:
          print "Send Failed. Error Code: " + str(msg[0]) + ' Message: ' + msg[1]
          sys.exit()  
  
  #handles threading for sending data to endpoints
  def forward(service, ENDPOINT_LIST, port, data):
      #for each endpoint in the endpoint list start a new send thread
      for endpoint in ENDPOINT_LIST:
          print "Forwarding data for %s from %s:%s to %s:%s" % (service,host,port,endpoint,port)
          #send(endpoint,port,data)
          ethread = threading.Thread(target=send, args=(endpoint,port,data))
          ethread.start()
  
  #handles threading for incoming clients 
  def clientthread(conn,service,ENDPOINT_LIST,port):
      while True:
          #receive data form client
          data = conn.recv(bufsize)
          if not data:
              break
          cthread = threading.Thread(target=forward, args=(service, ENDPOINT_LIST, port, data))
          cthread.start()
      #no data? then close the connection
      conn.close()
  
  #handles listening to sockets for incoming connections
  def listen(service, ENDPOINT_LIST, port):
      #create the socket
      listenSocket = socket(AF_INET, SOCK_STREAM)
      #Allow reusing addresses - I think this is important to stop local ports getting eaten up by never-ending tcp streams that don't close
      listenSocket.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
      #try to bind the socket to host and port
      try:
          listenSocket.bind((host, port))
      #display an error message if you can't
      except IOError as msg:
          print "Bind Failed. Error Code: " + str(msg[0]) + ' Message: ' + msg[1]
          sys.exit()
      #start listening on the socket
      listenSocket.listen(10)
      print "Service %s on port %s is listening" %(service,port)
      while True:
          #wait to accept a connection
          conn, addr = listenSocket.accept()
          print 'Connected to ' + addr[0] + ':' + str(addr[1]) + ' on port ' + str(port)
          #start new thread for each connection
          lthread = threading.Thread(target=clientthread , args=(conn,service,ENDPOINT_LIST,port))
          lthread.start()
      #If no data close the connection
      listenSocket.close()
  
  service = "Dumb-one-way-tcp-service-name1"  
  ENDPOINT_LIST = ["192.168.1.100","192.168.1.200"]
  port = 55551    
  listen(service,ENDPOINT_LIST,port)
  </code></pre>
  
  <p>I have looked into other libraries to try to achieve my goal, including using:</p>
  
  <ul>
  <li>Twisted </li>
  <li>Asyncore </li>
  <li>Scapy</li>
  </ul>
  
  <p>However I found them quite complicated for my modest needs and programming skill level.</p>
  
  <p>If anyone has any suggestions on how I could refine the approach I have, or any other ways this goal could be achieved, please let me know!</p>