title: Is it possible to use redis_store output plugin for fluentd to handle huge amount of logs?
body: <p>I'm trying to configure fluentd to send logs to redis on different server (the path is fluentbit-fluentd-redis-logstash-elastic), but I can't figure out some performance issues. If I save logs only on disc, everything is ok, all logs are saved (thousands per sec), but if I add redis_store section, the same amount of data makes fluentd much slower and the memory is still growing until next restart (day or two).
  I know that it's probably because of input is faster than output, but how to handle that, how to send so many data to redis? Add more memory is not solution it can only add some time. Is it because redis can't receive data from so many threads (but redis is not overloaded, there is no queue)? I dont know if that is network issue (but in this plugin i cant try another protocol?), cpu issue (cpu is on cca 70%), or the plugin itself?
  If I add redis plugin, the communication with redis is just so slow that fluentd is not fast enough and add data to memory.</p>
  <p>Config</p>
  <pre><code>&lt;system&gt;
    workers 4
    root_dir /fluentd/log/buffer/
  &lt;/system&gt;
  
  &lt;worker 0-3&gt;
  &lt;source&gt;
   @type forward
    bind 0.0.0.0
    port 9880
  &lt;/source&gt;
  
  &lt;label @TEST&gt;
  &lt;match test.**&gt; 
   @type forest
   subtype copy
   &lt;template&gt;
   &lt;store&gt;
      @type file
      @id &quot;test-#{worker_id}&quot;
      @log_level debug
      path &quot;fluentd/log/test-#{worker_id}.*.log&quot;
      append true
   &lt;buffer&gt;
      flush_mode interval
      flush_interval 3
      flush_at_shutdown true
    &lt;/buffer&gt;
   &lt;format&gt;
    @type single_value
    message_key log
   &lt;/format&gt;
   &lt;/store&gt;
   &lt;store&gt;
       @type redis_store
       host server_ip
       port 6379
       key test
       store_type list
   &lt;buffer&gt;
      flush_mode interval
      flush_interval 3
      flush_at_shutdown true
      flush_thread_count 4
    &lt;/buffer&gt;
   &lt;/store&gt;
    &lt;/template&gt;
  &lt;/match&gt;
  &lt;/label&gt;
  &lt;/worker&gt;
  </code></pre>
  <p>Any tips how to make throughput to redis better?
  Thank you</p>