title: Sending Mouse and keyboard inputs from server to client over socket in Python?
body: <p>I have my desktop stream working the server can see the client desktop, what I basically do is grab screenshots from client and send it to server continuously and displaying it using <code>cv2.imshow()</code>.<br>
  Now I want a way to send mouse and keyboards inputs from server to the client so that it can control the client remotely. I know <code>pyautogui</code> module could be used to simulate moving, click and dragging mouse as well as pressing keys.
  One way I thought is to send the coordinates in fractions of the resolution.</p>
  
  <p><strong>Example:</strong><br>
  If your picturebox is 800x600 and you click at location (200, 300), send (0.25, 0.5).<br>
  On the other machine, you receive (0.25, 0.5) and you multiply them by the resolution. On a 1024x768, you move the mouse pointer to location (256, 384).</p>
  
  <p>But I need a method to get the mouse and keyboard input from the <code>opencv</code> window and send it to the client over the socket and then simulate it on client side.</p>
  
  <p>Any other method instead of using <code>opencv</code> to display the frames on server would be accepted as well, as long as I am able to work with keyboard and mouse inputs and improves the performance.</p>
  
  <p><strong>Note</strong>: I am also not able to get the screenshot of the UAC prompt using <code>Imagegrab.grab()</code> even if I run the script as an administrator I don't know why
  this is happening.</p>
  
  <p>Below is my code for client and server exceptions which are not being handled as I am just trying out that the things work.</p>
  
  <p>Any <strong>suggestions</strong> to make my stream faster are welcomed as well :)</p>
  
  <p>server.py:</p>
  
  <pre><code>import socket
  import struct
  import cv2
  import numpy
  
  host = "192.168.0.5"
  port = 4444
  
  s = socket.socket(socket.AF_INET,socket.SOCK_STREAM) #create socket
  s.bind((host,port))
  s.listen(5)
  c , addr = s.accept()
  print("Connection recieved from {}".format(addr))
  
  cv2.namedWindow("Remote Desktop",cv2.WINDOW_NORMAL)
  while True:
      bs = c.recv(8)
      (length,) = struct.unpack("&gt;Q",bs) # get length of image
      data = b''
      while(len(data)&lt;length):  # get image data
          recv_data = length - len(data)
          if(recv_data&gt;4096):
              data += c.recv(4096)
          else:
              data += c.recv(recv_data)
  
      with open("demo.jpg","wb") as f:
          d = f.write(data)
          #print(d)
      img = cv2.imread("demo.jpg")
      cv2.imshow("Remote Desktop",img)
      key = cv2.waitKey(1)
      if key == 27:
          break
  
  cv2.destroyAllWindows()
  s.close()
  </code></pre>
  
  <p>Here is my client.py:</p>
  
  <pre><code>import socket
  import struct
  from PIL import ImageGrab
  import time
  
  host = "192.168.0.5"
  port = 4444
  s = socket.socket(socket.AF_INET,socket.SOCK_STREAM) #create socket
  s.connect((host,port)) #connect to server
  print("Connection Established")
  time.sleep(5)
  
  while True:
          img = ImageGrab.grab()
          img.save("demo.jpg","JPEG")
          with open("demo.jpg","rb") as f:
              data = f.read()    # read the image
              s.sendall(struct.pack("&gt;Q",len(data))) #send the length of image in bytes
              s.sendall(data)     # send the data
  
  s.close()
  </code></pre>
  
  <p>So how can I send mouse and keyboard inputs effectively over socket in <code>python</code>? Thanks for your help.</p>