title: Why does gcc -O3 handle avx256 compare intrinsic differently than gcc -O0 and clang?
body: <p>I want to set two integer vectors and compare them with SIMD, and later on use this mask for a blend operation on packed floats. I produced the following code:</p>
  
  <pre><code>#include &lt;immintrin.h&gt;
  #include &lt;stdio.h&gt;
  #include &lt;string.h&gt;
  
  
  int main(){
      __m256i is =  _mm256_setr_epi32(1, 2, 3, 4, 5, 6, 7, 8);
      __m256i js =  _mm256_set1_epi32(1);               // integer bit-patterns
      __m256 mask = _mm256_cmp_ps(is,js, _CMP_EQ_OQ);   // compare as subnormal floats
  
      float val[8];
      memcpy(val, &amp;mask, sizeof(val));
      printf("%f %f %f %f %f %f %f %f \n", val[0], val[1], val[2], val[3], val[4], val[5], val[6], val[7]);
  }
  </code></pre>
  
  <p>which works fine with <code>gcc -mavx main.c</code> as well as <code>clang -mavx main.c</code> and <code>clang -O3 -mavx main.c</code>.</p>
  
  <p>(Editor's note: it'll break with <code>-ffast-math</code> when cmpps treats those denormal inputs as <code>0.0</code> so all the compares are true.  You want AVX2 <code>_mm256_cmp_epi32</code> to do an <em>integer</em> compare, and <code>_mm256_castsi256_ps</code> the result.  But that's unrelated to the question about <code>gcc -O0</code> and clang allowing implicit conversion from <code>__m256i</code> to <code>__m256</code>)</p>
  
  <p>However, when I use <code>gcc -O3 -mavx main.c</code> I get the following error message:</p>
  
  <pre><code>main.c: In function ‘main’:
  main.c:9:33: error: incompatible type for argument 1 of ‘_mm256_cmp_ps’
      9 |     __m256 mask = _mm256_cmp_ps(is,js, _CMP_EQ_OQ);
        |                                 ^~
        |                                 |
        |                                 __m256i {aka __vector(4) long long int}
  In file included from /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/include/immintrin.h:51,
                   from main.c:1:
  /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/include/avxintrin.h:404:23: note: expected ‘__m256’ {aka ‘__vector(8) float’} but argument is of type ‘__m256i’ {aka ‘__vector(4) long long int’}
    404 | _mm256_cmp_ps (__m256 __X, __m256 __Y, const int __P)
        |                ~~~~~~~^~~
  main.c:9:36: error: incompatible type for argument 2 of ‘_mm256_cmp_ps’
      9 |     __m256 mask = _mm256_cmp_ps(is,js, _CMP_EQ_OQ);
        |                                    ^~
        |                                    |
        |                                    __m256i {aka __vector(4) long long int}
  In file included from /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/include/immintrin.h:51,
                   from main.c:1:
  /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/include/avxintrin.h:404:35: note: expected ‘__m256’ {aka ‘__vector(8) float’} but argument is of type ‘__m256i’ {aka ‘__vector(4) long long int’}
    404 | _mm256_cmp_ps (__m256 __X, __m256 __Y, const int __P)
        |                            ~~~~~~~^~~
  
  </code></pre>
  
  <p>I notice two things. First of all, the compiler seems to treat <code>is</code> as <code>__m256i {aka __vector(4) long long int}</code> whereas it contains 8 ints. Secondly, the compiler is correct to complain, because the intel intrinsics guide <a href="https://software.intel.com/sites/landingpage/IntrinsicsGuide/" rel="nofollow noreferrer">1</a> shows the arguments as <code>__m256</code>. I'm now confused why this code even worked at the beginning. And if it is indeed correct because the integers are casted to floats, then I don't understand why it doesn't work with <code>gcc -O3</code>. </p>
  
  <p>I did not want to use <code>_mm256_cmpeq_epi32</code> which returns an <code>__m256i</code> and there (seems to be no) is no <code>blend_ps</code> instruction that accepts such a mask. </p>
  
  <p><strong>Why do the compilers behave differently, and what is the <em>correct</em> way to do this operation?</strong></p>
  
  <hr>
  
  <p><strong>Compiler versions</strong></p>
  
  <pre><code>$ gcc -v
  Using built-in specs.
  COLLECT_GCC=gcc
  COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/lto-wrapper
  Target: x86_64-pc-linux-gnu
  Configured with: /build/gcc/src/gcc/configure --prefix=/usr --libdir=/usr/lib --libexecdir=/usr/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-pkgversion='Arch Linux 9.3.0-1' --with-bugurl=https://bugs.archlinux.org/ --enable-languages=c,c++,ada,fortran,go,lto,objc,obj-c++,d --enable-shared --enable-threads=posix --with-system-zlib --with-isl --enable-__cxa_atexit --disable-libunwind-exceptions --enable-clocale=gnu --disable-libstdcxx-pch --disable-libssp --enable-gnu-unique-object --enable-linker-build-id --enable-lto --enable-plugin --enable-install-libiberty --with-linker-hash-style=gnu --enable-gnu-indirect-function --enable-multilib --disable-werror --enable-checking=release --enable-default-pie --enable-default-ssp --enable-cet=auto gdc_include_dir=/usr/include/dlang/gdc
  Thread model: posix
  gcc version 9.3.0 (Arch Linux 9.3.0-1) 
  </code></pre>
  
  <pre><code>$ clang -v
  clang version 10.0.0 
  Target: x86_64-pc-linux-gnu
  Thread model: posix
  InstalledDir: /usr/bin
  Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-pc-linux-gnu/8.4.0
  Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-pc-linux-gnu/9.3.0
  Found candidate GCC installation: /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/8.4.0
  Found candidate GCC installation: /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/9.3.0
  Found candidate GCC installation: /usr/lib/gcc/x86_64-pc-linux-gnu/8.4.0
  Found candidate GCC installation: /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0
  Found candidate GCC installation: /usr/lib64/gcc/x86_64-pc-linux-gnu/8.4.0
  Found candidate GCC installation: /usr/lib64/gcc/x86_64-pc-linux-gnu/9.3.0
  Selected GCC installation: /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/9.3.0
  Candidate multilib: .;@m64
  Candidate multilib: 32;@m32
  Selected multilib: .;@m64
  Found CUDA installation: /opt/cuda, version 10.1
  </code></pre>
  
  <p>[1] <a href="https://software.intel.com/sites/landingpage/IntrinsicsGuide/" rel="nofollow noreferrer">https://software.intel.com/sites/landingpage/IntrinsicsGuide/</a></p>