title: IoT data system design: Google Pub/Sub vs Kafka vs Kinesis vs PubNub for IoT data ingestion?
body: <p>I'm trying to build an IoT + data analytics system and I'm having trouble deciding on what technology or service to use for ingestion.</p>
  
  <p>A high level description of the end goal is:</p>
  
  <ol>
  <li>IoT devices push data to an IoT gateway (using Zigbee, Z-wave, Bluetooth, etc)</li>
  <li>IoT gateway (which is connected to the internet) pushes data to a pub/sub system</li>
  <li>Backend services process the data coming out of the pub/sub system, updates dashboards and sends out alerts</li>
  </ol>
  
  <p>My question is what kind of pubsub system should we use if we only need ~10 second response time? (E.g. The following is acceptable: IoT device senses event and then about 10 seconds later it shows up on a user's dashboard or sets off an alert)</p>
  
  <p>Below are some questions I have:</p>
  
  <ol>
  <li>I see PubNub advertised a lot for use in IoT. My understanding is that PubNub is basically a very fast pubsub system that guarantees less than 1/4 second delivery- is this a correct understanding of it? But PubNub is a bit expensive compared to using Google's Pub/Sub or maintaining our own Kafka.</li>
  <li>Is Google Pub/Sub in a sense similar to PubNub, because unlike being a single self-managed Kafka cluster running in a single data center, Google Pub/Sub has its own network because it's a part of Google? (similar to how PubNub is a "data-streams-network"?)</li>
  <li>If I use Kafka, should the producer be in the gateway devices?
  
  <ol>
  <li>If the producer isn't in the gateway devices, then should the Kafka producers be on our servers and have a REST API to accept messages from the gateway devices?</li>
  <li>If the Kafka producer IS in the gateway devices, does there need to be anything special in front of the Kafka brokers for them to accept messages from the gateway devices?</li>
  </ol></li>
  <li>PubNub can be used to send commands back to the IoT devices. Can this also be done with Google Pub/Sub or Kafka?
  
  <ol>
  <li>To push commands out to the IoT devices With Kafka, would every Gateway device need a consumer that is waiting for messages from the topics it's subscribed to? (e.g. the commands)</li>
  </ol></li>
  </ol>
  
  <p>Also, not sure if it's worth mentioning, but currently, the team is just me and maybe 2 other full stack developers. We've read up on Kafka and Zookeeper but none of us have gone past rolling out a tutorial example of it.</p>