title: grpc unary-stream with redis pubsub - degradation with too many clients
body: <p>We have a python grpc (grpcio with asyncio) server which performs server side streaming of data consumed from redis PUB/SUB (using aioredis 2.x) , combining up to 25 channels per stream. With low traffic everything works fine, as soon as we reach 2000+ concurrent streams , the delivery of messages start falling behind.</p>
  <p>Some setup details and what we tried so far:</p>
  <ul>
  <li><p>The client connections to GRPC are loadbalanced over kubernetes cluster with Ingress-NGINX controller, and it seems scaling (we tried 9 pods with 10 process instances each) doesn't help at all (loadbalancing is distributed evenly).</p>
  </li>
  <li><p>We are running a five node redis 7.x cluster with 96 threads per replica.</p>
  </li>
  <li><p>Connecting to redis with CLI client while GRPC falls behind - individual channels are on time while GRPC streams are falling behind</p>
  </li>
  <li><p>Messages are small in size (40B) with a variable rate anywhere between 20-200 per second on each stream.</p>
  </li>
  <li><p>Aioredis seems to be opening a new connection for each pubsub subscriber even if we're using capped connection pool for each grpc instance.</p>
  </li>
  <li><p>Memory/CPU utilisation is not dramatic as well as Network I/O, so we're not getting bottlenecked there</p>
  </li>
  <li><p>Tried identical setup with a very similar grpc server written in Rust, with similar results</p>
  </li>
  </ul>