title: Distributed game server using tornado
body: <p>We  are developing a <a href="http://servicestargame.com" rel="nofollow noreferrer">gamified environment</a> which facilitates learning of 'boring' material. Recently, we have launched with the 1.0 release that comprises of the following modules:</p>
  
  <ol>
  <li>Django based web portal for learning courses, viewing leaderboard (scoring), personal activity and achievements.</li>
  <li>Tornado based real-time feeds and notification service. We are utilized <a href="https://github.com/mrjoes/sockjs-tornado" rel="nofollow noreferrer">sockjs-tornado</a> developed by <a href="https://github.com/mrjoes" rel="nofollow noreferrer">mrjoes</a>. We took benefit from shared session: if user logged in to django site, he automatically opened a channel with our Comet backend.</li>
  <li>Tornado based asynchronous multiplayer game server. Again, we have used <a href="https://github.com/mrjoes/sockjs-tornado" rel="nofollow noreferrer">sockjs-tornado</a> for connecting and serving players (sockets), synchronizing game loop events across the network.</li>
  </ol>
  
  <p>Now, I am working on scalability and high-availability features of the system. <strong>The main challenge</strong> is to distribute load across game cluster nodes properly and consistently (no data loss, fail-over in case of crash, re-balance if one of game servers (loop) suddenly crashes). <em>The main goal</em> is to make client interaction with game as seamless as possible: client shouldn't make additional actions: manually reconnecting if game crashed due to server fail. <strong>The figure</strong> depicts a possible architecture of distributed game server.</p>
  
  <p>In case some actions performed on .../game on Django site (CRUD actions), tornado-based Game Router notified  in real-time through a kind of oplog collection (capped collection in mongodb which is tailed by Game Router). If this is a new game, Game Router determines next host (game server) for the game by special algorithm (fastest heartbeat trip, or min number of live(active) games). Further, Game Router notifies appropriate Loop by publishing message to topic exchange.</p>
  
  <p>If any game server crashes, Game Router will be immediately notified and re-balances load by taking idle games in memcached cluster. </p>
  
  <p>In case of fail, how client can seamlessly reconnect to new game server ? I don't want to open redundant channel (socket connection).
  Whether I am going right way or not ? Please, provide me with good approaches, critics, techniques. It would be nice if you share your minds by drawing diagrams.</p>
  
  <p>Thank you in advance.<img src="https://i.stack.imgur.com/uBgl6.png" alt="enter image description here"></p>