title: GCP Dataflow: System Lag for streaming from Pub/Sub IO
body: <p>We use "System Lag" to check the health of our Dataflow jobs. For example if we see an increase in system lag, we will try to see how to bring this metric down. There are few question regarding this metric. </p>
  
  <ul>
  <li>1) What does system lag exactly means?</li>
  </ul>
  
  <blockquote>
    <p>The maximum time that an item of data has been awaiting processing</p>
  </blockquote>
  
  <p>Above is what we see in GCP Console when we hit information icon. What does an item of data mean in this case? Stream processing has concept of Windowing, event time vs processing time, watermark, etc. When is an item considered awaiting to be processed? For example is it simply when the message arrives regardless of its state?</p>
  
  <ul>
  <li>2) What is the optimum threshold for this metric?</li>
  </ul>
  
  <p>We try to keep this metric as low as possible, but we don't have any recommendation on how low we should keep it. For example do we have some recommendation such as keeping system lag between 20s to 30s is optimum. </p>
  
  <ul>
  <li>3)How does system lag implicates sinks</li>
  </ul>
  
  <p>How does system lag affect latency of the event itself? </p>