title: Is branch divergence really so bad?
body: <p>I've seen many questions scattered across the Internet about branch divergence, and how to <em>avoid</em> it. However, even after reading dozens of articles on how CUDA works, <strong>I can't seem to see how avoiding branch divergence helps in most cases</strong>. Before anyone jumps on on me with claws outstretched, allow me to describe what I consider to be "most cases".</p>
  
  <p>It seems to me that most instances of branch divergence involve a number of truly distinct blocks of code. For example, we have the following scenario:</p>
  
  <pre><code>if (A):
    foo(A)
  else:
    bar(B)
  </code></pre>
  
  <p>If we have two threads that encounter this divergence, thread 1 will execute first, taking path A. Following this, thread 2 will take path B. In order to remove the divergence, we might change the block above to read like this:</p>
  
  <pre><code>foo(A)
  bar(B)
  </code></pre>
  
  <p>Assuming it is safe to call <code>foo(A)</code> on thread 2 and <code>bar(B)</code> on thread 1, one might expect performance to improve. However, here's the way I see it:</p>
  
  <p>In the first case, threads 1 and 2 execute in serial. Call this two clock cycles.</p>
  
  <p>In the second case, threads 1 and 2 execute <code>foo(A)</code> in parallel, then execute <code>bar(B)</code> in parallel. This still looks to me like two clock cycles, the difference is that in the former case, if <code>foo(A)</code> involves a read from memory, I imagine thread 2 can begin execution during that latency, which results in latency hiding. <em>If this is the case, the branch divergent code is faster.</em></p>