title: Code alignment in one object file is affecting the performance of a function in another object file
body: <p>I'm familiar with data alignment and performance but I'm rather new to aligning code.  I started programming in x86-64 assembly recently with NASM and have been comparing performance using code alignment. As far as I can tell NASM inserts <code>nop</code> instructions to achieve code alignment.</p>
  
  <p>Here is a function I have been trying this on a Ivy Bridge system</p>
  
  <pre><code>void triad(float *x, float *y, float *z, int n, int repeat) {
      float k = 3.14159f;
      int(int r=0; r&lt;repeat; r++) {
          for(int i=0; i&lt;n; i++) {
              z[i] = x[i] + k*y[i];
          }
      }
  }
  </code></pre>
  
  <p>The assembly I'm using for this is below. If I don't specify the alignment my performance compared to the peak is only about 90%. However, when I align the code before the loop as well as both inner loops to 16 bytes the performance jumps to 96%. So clearly the code alignment in this case makes a difference. </p>
  
  <p>But here is the strangest part. If I align the innermost loop to 32 bytes it makes no difference in the performance of this function, however, in another version of this function using intrinsics in a separate object file I link in its performance jumps from 90% to 95%!</p>
  
  <p>I did an object dump (using <code>objdump -d -M intel</code>) of the version aligned to 16 bytes (I posted the result to the end of this question) and 32 bytes and they are identical! It turns out that the inner most loop is aligned to 32 bytes anyway in both object files. But there must be some difference. </p>
  
  <p>I did a hex dump of each object file and there is one byte in the object files that differ. The object file aligned to 16 bytes has a byte with <code>0x10</code> and the object file aligned to 32 bytes has a byte with <code>0x20</code>. <strong>What exactly is going on! Why does code alignment in one object file affect the performance of a function in another object file? How do I know what is the optimal value to align my code to?</strong></p>
  
  <p>My only guess is that when the code is relocated by the loader that the 32 byte aligned object file affects the other object file using intrinsics. You can find the code to test all this at <a href="https://stackoverflow.com/questions/25899395/obtaining-peak-bandwidth-on-haswell-in-the-l1-cache-only-getting-62">Obtaining peak bandwidth on Haswell in the L1 cache: only getting 62%</a></p>
  
  <p>The NASM code I am using:</p>
  
  <pre><code>global triad_avx_asm_repeat
  ;RDI x, RSI y, RDX z, RCX n, R8 repeat
  pi: dd 3.14159
  align 16
  section .text
      triad_avx_asm_repeat:
      shl             rcx, 2  
      add             rdi, rcx
      add             rsi, rcx
      add             rdx, rcx
      vbroadcastss    ymm2, [rel pi]
      ;neg                rcx 
  
  align 16
  .L1:
      mov             rax, rcx
      neg             rax
  align 16
  .L2:
      vmulps          ymm1, ymm2, [rdi+rax]
      vaddps          ymm1, ymm1, [rsi+rax]
      vmovaps         [rdx+rax], ymm1
      add             rax, 32
      jne             .L2
      sub             r8d, 1
      jnz             .L1
      vzeroupper
      ret
  </code></pre>
  
  <p>Result from <code>objdump -d -M intel test16.o</code>. The disassembly is identical if I change <code>align 16</code> to <code>align 32</code> in the assembly above just before <code>.L2</code>. However, the object files still differ by one byte.</p>
  
  <pre><code>test16.o:     file format elf64-x86-64
  
  
  Disassembly of section .text:
  
  0000000000000000 &lt;pi&gt;:
     0:   d0 0f                   ror    BYTE PTR [rdi],1
     2:   49                      rex.WB
     3:   40 90                   rex xchg eax,eax
     5:   90                      nop
     6:   90                      nop
     7:   90                      nop
     8:   90                      nop
     9:   90                      nop
     a:   90                      nop
     b:   90                      nop
     c:   90                      nop
     d:   90                      nop
     e:   90                      nop
     f:   90                      nop
  
  0000000000000010 &lt;triad_avx_asm_repeat&gt;:
    10:   48 c1 e1 02             shl    rcx,0x2
    14:   48 01 cf                add    rdi,rcx
    17:   48 01 ce                add    rsi,rcx
    1a:   48 01 ca                add    rdx,rcx
    1d:   c4 e2 7d 18 15 da ff    vbroadcastss ymm2,DWORD PTR [rip+0xffffffffffffffda]        # 0 &lt;pi&gt;
    24:   ff ff 
    26:   90                      nop
    27:   90                      nop
    28:   90                      nop
    29:   90                      nop
    2a:   90                      nop
    2b:   90                      nop
    2c:   90                      nop
    2d:   90                      nop
    2e:   90                      nop
    2f:   90                      nop
  
  0000000000000030 &lt;triad_avx_asm_repeat.L1&gt;:
    30:   48 89 c8                mov    rax,rcx
    33:   48 f7 d8                neg    rax
    36:   90                      nop
    37:   90                      nop
    38:   90                      nop
    39:   90                      nop
    3a:   90                      nop
    3b:   90                      nop
    3c:   90                      nop
    3d:   90                      nop
    3e:   90                      nop
    3f:   90                      nop
  
  0000000000000040 &lt;triad_avx_asm_repeat.L2&gt;:
    40:   c5 ec 59 0c 07          vmulps ymm1,ymm2,YMMWORD PTR [rdi+rax*1]
    45:   c5 f4 58 0c 06          vaddps ymm1,ymm1,YMMWORD PTR [rsi+rax*1]
    4a:   c5 fc 29 0c 02          vmovaps YMMWORD PTR [rdx+rax*1],ymm1
    4f:   48 83 c0 20             add    rax,0x20
    53:   75 eb                   jne    40 &lt;triad_avx_asm_repeat.L2&gt;
    55:   41 83 e8 01             sub    r8d,0x1
    59:   75 d5                   jne    30 &lt;triad_avx_asm_repeat.L1&gt;
    5b:   c5 f8 77                vzeroupper 
    5e:   c3                      ret    
    5f:   90                      nop
  </code></pre>