title: ingesting millions of telemetry events per day - Redis vs kafka vs another queue/message broker my best best for ingestion
body: <p>I will shortly be ingesting millions of telemetry events per day. It is important that we don't lose these telemetry events as they represent events like sales which is used for billing and charging customers. Today we are at tens of thousands of events per day but expect to 100x in the coming months.</p>
  <p>Today we simply have a rest endpoint where clients send us telemetry events as POST requests. The service that hosts the endpoint (node express) has some logic which detects the telemetry payload type and performs actions based off that (e.g. write to database, send an sms).</p>
  <p>What I am thinking would be nice is to have a FIFO queue which a very lightweight REST API(s) can push telemetry events into, then have a horizontally scalable bunch of microservices which pull telemetry and can perform the appropriate action from there*. A telemetry event should only be processed/consumed once by a single service.</p>
  <p>So my question now falls to, what technology(s) should I choose? Loosing a telemetry event is a pretty big deal. At this point in time, when the clients sending us telemetry events receive a HTTP 200 (successful) from our system (saying we received it) we have no way of ever getting them to send us telemetry again - i.e. once we receive a telemetry event we must not lose it.  Further reading and processing a telemetry event more than once is also a terrible thing to do.</p>
  <p>From reading the REDIS docs it seems I can simply use the FIFO queue - push and pop operations are atomic, so this looks good. What I am concerned about is:  (i) would all my microservices workers that process the telemetry events need to be continuously polling the redis queue, or can it be more event driven (notification of a new queue item)  (ii) my other points of failure - redis going down (AOF file can help) or things I haven’t thought about yet  (iii) how would this queue scale if we started handling more amount of data as the queue seems to be a choke point (can you shard the queue or would one just use kafka or maybe something like rsmq**?)  </p>
  <p>EDIT: A reason I'm particularly interested in redis is I have other needs for it - caching database queryies/tables, expiring keys for a notification service.</p>
  <p>Thank you</p>
  <p>*I could even perhaps have different queues for each type of telemetry.</p>
  <p>**<a href="http://smrchy.github.io/rsmq/" rel="nofollow noreferrer">http://smrchy.github.io/rsmq/</a></p>