title: GCC 8.3 ARM Assembly error when building Linux kernel 3.14.17 in Buildroot 2020.02 setup
body: <p>Migrating from Buildroot 2015.08.03 with GCC 4.9.3/Make 3.81/gblibc 2.20 to a Buildroot 2020.02 setup to support GCC > 8.x/Make 4.1/glibc 2.30.<br>
  Encountered an issue with GCC 8.3 compiler when building a Linux 3.14.17 ARM Cortex-A9 kernel.  Using GCC 7.5, the file in question builds (other issues debugging).  Also builds fine with the GCC 4.9.3 compiler.
  Details below - more can be provided.
  Seems to be an issue with intermediate GCC generation, causing ASM to generate incorrect code??  Has anyone encountered this?<br>
  Tried different Kernel build options, no changes.  </p>
  
  <p>Thanks for your expertise and wisdom.
  Stephen Beckwith</p>
  
  <p>Host System:  VMWare Fusion setup on MAC OSX Mojave (10.14.6) on a 2014 vintage 15” Macbook Pro with a Core i7, 16GB RAM, 512GB SSD.<br>
  VM gets 4 cores (HT enabled) and 8GB RAM
  Sources located on an external USB-3 HDD, where the VM is booted from and it’s Filesystems is located.
  VM is Ubuntu 18.04.01 - updated in Mid-March</p>
  
  <pre><code>sbeckwit@ubuntu:~$ uname -a
  Linux ubuntu 5.3.0-42-generic #34~18.04.1-Ubuntu SMP Fri Feb 28 13:42:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
  
  sbeckwit@ubuntu:~$ gcc --version
  gcc (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
  Copyright (C) 2017 Free Software Foundation, Inc.
  This is free software; see the source for copying conditions.  There is NO
  warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
  
  sbeckwit@ubuntu:/lib32$ ldd --version
  ldd (Ubuntu GLIBC 2.27-3ubuntu1) 2.27
  Copyright (C) 2018 Free Software Foundation, Inc.
  This is free software; see the source for copying conditions.  There is NO
  warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
  Written by Roland McGrath and Ulrich Drepper.
  </code></pre>
  
  <p><strong>The setup:</strong>
  Our previous setup is a Buildroot 2015.08.03 setup.
  Using the original Buildroot setup - we use a Buildroot Built Compiler, version 4.9.3, glibc 2.20, Make 3.81.
   - This system builds our Embedded Linux setup, including kernel/uboot/busybox/utilites/applications.<br>
   - In this setup, everything builds just fine.
   - This system runs in a server environment on older RHEL 5.x based system.</p>
  
  <p>The Linux Kernel we are building is a 3.14.17 kernel, with some 3rd party patches for specific SOC (EMULEX PIlot4 BMC, now owned by ASPEED - this is a dual-core ARM Cortex-A9 ARM SoC)
   - Build options are “stock” with the exception of the adding the -save-temps flag to get the .s output files.</p>
  
  <p><strong>New Setup:</strong>
  We are migrating to a newer Buildroot setup in order to mitigate recent CVE issues related to GLIBC and GCC.  Therefore, we have put into place new buildroot setup:
  Buildroot 2020.02:
   -  This supports building the following:
      GCC 7.5, GCC 8.3, GCC 9.2
      Glibc 2.30 (the one we’ve chosen for now)
      Uses Make 4.1</p>
  
  <p>Initially we built a GCC 9.2 Compiler and ran into the following error:</p>
  
  <pre><code>/tmp/ccxaGNai.s: Assembler messages:
  /tmp/ccxaGNai.s:2262: Error: .err encountered
  arm-buildroot-linux-gnueabi-gcc.br_real: warning: /home/sbeckwit/sp_dev/bowie_dev/src/include: linker input file unused because linking not done
    CC      security/keys/key.o
  scripts/Makefile.build:309: recipe for target 'kernel/fork.o' failed
  </code></pre>
  
  <ul>
  <li>similar messages for kernel/exit.o and fs/fat/dir.o</li>
  </ul>
  
  <p>Built a GCC 8.3 Compiler and ran into the exact same errors.</p>
  
  <p>Built a GCC 7.5 Compiler and this error no longer exists (though we have other issues)</p>
  
  <p>It was determined that in order to comply with Corporate Security policy, we are required to have GCC 8.x or above.</p>
  
  <p>Therefore:
   - I re-setup the GCC 7.5 and obtained the .i/.s output files (using the -save-temps switch in the Kernel Makefile)
   - I then switched to the GCC 8.3 and obtained the same files.</p>
  
  <p><strong>GCC 8.3 Information:</strong></p>
  
  <pre><code>sbeckwit@ubuntu:~/sp_dev/sp_archives/toolchains/arm8.x-toolchain/bin$ ./arm-arm-linux-gnueabi-gcc-8.3.0 --version
  arm-arm-linux-gnueabi-gcc-8.3.0.br_real (Buildroot 2020.02) 8.3.0
  Copyright (C) 2019 Free Software Foundation, Inc.
  This is free software; see the source for copying conditions.  There is NO
  warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
  </code></pre>
  
  <p><strong>Error:</strong></p>
  
  <pre><code>fork.s: Assembler messages:
  fork.s:2277: Error: .err encountered
  scripts/Makefile.build:308: recipe for target 'kernel/fork.o' failed
  </code></pre>
  
  <p>This is in the function:  mm_release()</p>
  
  <pre><code>.LBB1932:
      .loc 8 110 35 is_stmt 0 view .LVU654
      mov r2, sp
      bic r2, r2, #8128
      bic r2, r2, #63
  .LBE1932:
  .LBE1933:
      .loc 1 784 278 view .LVU655
      mov r0, r5
  .LVL102:
      .loc 1 784 303 is_stmt 1 view .LVU656
      .loc 1 784 39 is_stmt 0 view .LVU657
      ldr r1, [r2, #8]
  .LVL103:
      .loc 1 784 326 view .LVU658
      sub r1, r1, #1
  .LVL104:
      .loc 1 784 351 is_stmt 1 view .LVU659
      .loc 1 784 379 view .LVU660
      .loc 1 784 947 view .LVU661
      .syntax divided
  @ 784 "kernel/fork.c" 1
      .ifnc r0,r0 ; .err ; .endif
      **.ifnc r3,r2 ; .err ; .endif**
      .ifnc r1,r1 ; .err ; .endif
      bl  __put_user_4
  @ 0 "" 2
  .LVL105:
  </code></pre>
  
  <p>highlighted line is line 2277 from the error message</p>
  
  <p>When I grep’d through the .s files generated for the 8.3 build, looking for the “.ifnc” in the current files.  Yet, this file is the only file where this .ifnc has “different” registers:  all other instances show the same register.  Why is this one different.</p>
  
  <p>From the fork.i file for this function:</p>
  
  <pre><code># 747 "kernel/fork.c"
  void mm_release(struct task_struct *tsk, struct mm_struct *mm)
  {
  
  
   if (__builtin_expect(!!(tsk-&gt;robust_list), 0)) {
    exit_robust_list(tsk);
    tsk-&gt;robust_list = ((void *)0);
   }
  
  
  
  
  
  
   if (__builtin_expect(!!(!list_empty(&amp;tsk-&gt;pi_state_list)), 0))
    exit_pi_state_list(tsk);
  
  
   uprobe_free_utask(tsk);
  
  
   do { } while (0);
  # 777 "kernel/fork.c"
   if (tsk-&gt;clear_child_tid) {
    if (!(tsk-&gt;flags &amp; 0x00000400) &amp;&amp;
        (*(volatile int *)&amp;(&amp;mm-&gt;mm_users)-&gt;counter) &gt; 1) {
  
  
  
  
     ({ might_fault(); ({ unsigned long __limit = current_thread_info()-&gt;addr_limit - 1; const typeof(*(tsk-&gt;clear_child_tid)) *__tmp_p = (tsk-&gt;clear_child_tid); register const typeof(*(tsk-&gt;clear_child_tid)) __r2 asm("r2") = (0); register const typeof(*(tsk-&gt;clear_child_tid)) *__p asm("r0") = __tmp_p; register unsigned long __l asm("r1") = __limit; register int __e asm("r0"); switch (sizeof(*(__p))) { case 1: __asm__ __volatile__ ( ".ifnc " "%0" "," "r0" " ; .err ; .endif\n\t" ".ifnc " "%2" "," "r2" " ; .err ; .endif\n\t" ".ifnc " "%3" "," "r1" " ; .err ; .endif\n\t" "bl    __put_user_" "1" : "=&amp;r" (__e) : "0" (__p), "r" (__r2), "r" (__l) : "ip", "lr", "cc"); break; case 2: __asm__ __volatile__ ( ".ifnc " "%0" "," "r0" " ; .err ; .endif\n\t" ".ifnc " "%2" "," "r2" " ; .err ; .endif\n\t" ".ifnc " "%3" "," "r1" " ; .err ; .endif\n\t" "bl  __put_user_" "2" : "=&amp;r" (__e) : "0" (__p), "r" (__r2), "r" (__l) : "ip", "lr", "cc"); break; case 4: __asm__ __volatile__ ( ".ifnc " "%0" "," "r0" " ; .err ; .endif\n\t" ".ifnc " "%2" "," "r2" " ; .err ; .endif\n\t" ".ifnc " "%3" "," "r1" " ; .err ; .endif\n\t" "bl  __put_user_" "4" : "=&amp;r" (__e) : "0" (__p), "r" (__r2), "r" (__l) : "ip", "lr", "cc"); break; case 8: __asm__ __volatile__ ( ".ifnc " "%0" "," "r0" " ; .err ; .endif\n\t" ".ifnc " "%2" "," "r2" " ; .err ; .endif\n\t" ".ifnc " "%3" "," "r1" " ; .err ; .endif\n\t" "bl  __put_user_" "8" : "=&amp;r" (__e) : "0" (__p), "r" (__r2), "r" (__l) : "ip", "lr", "cc"); break; default: __e = __put_user_bad(); break; } __e; }); });
     sys_futex(tsk-&gt;clear_child_tid, 1,
       1, ((void *)0), ((void *)0), 0);
    }
    tsk-&gt;clear_child_tid = ((void *)0);
   }
  
  
  
  
  
   if (tsk-&gt;vfork_done)
    complete_vfork_done(tsk);
  }
  </code></pre>
  
  <p>ASSUME:
   - Given that the GCC 9.2.0 produces the same error, we are assuming that the same issue exists in this compiler as well.  </p>
  
  <p><strong>GCC 7.5 Information:</strong></p>
  
  <pre><code>sbeckwit@ubuntu:~/sp_dev/sp_archives/toolchains/arm7.5-toolchain/bin$ ./arm-buildroot-linux-gnueabi-gcc --version
  arm-buildroot-linux-gnueabi-gcc.br_real (Buildroot 2020.02) 7.5.0
  Copyright (C) 2017 Free Software Foundation, Inc.
  This is free software; see the source for copying conditions.  There is NO
  warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
  </code></pre>
  
  <p>using the same generated error for reference:</p>
  
  <pre><code>fork.s: Assembler messages:
  fork.s:2277: Error: .err encountered
  scripts/Makefile.build:308: recipe for target 'kernel/fork.o' failed
  </code></pre>
  
  <p>This is in the function:  mm_release()</p>
  
  <pre><code>.LBE1879:
      .loc 1 784 0
      mov r2, r3
  .LVL102:
      mov r0, r5
  .LVL103:
      ldr r1, [r1, #8]
      sub r1, r1, #1
  .LVL104:
      .syntax divided
  @ 784 "kernel/fork.c" 1
      .ifnc r0,r0 ; .err ; .endif
      **.ifnc r2,r2 ; .err ; .endif**
      .ifnc r1,r1 ; .err ; .endif
      bl  __put_user_4
  @ 0 "" 2
  .LVL105:
      .arm
      .syntax unified
  </code></pre>
  
  <p>highlighted line correlates to the error above in the 8.3.0 compiler</p>
  
  <p>From the fork.i file for this function:</p>
  
  <pre><code># 747 "kernel/fork.c"
  void mm_release(struct task_struct *tsk, struct mm_struct *mm)
  {
  
  
   if (__builtin_expect(!!(tsk-&gt;robust_list), 0)) {
    exit_robust_list(tsk);
    tsk-&gt;robust_list = ((void *)0);
   }
  
  
  
  
  
  
   if (__builtin_expect(!!(!list_empty(&amp;tsk-&gt;pi_state_list)), 0))
    exit_pi_state_list(tsk);
  
  
   uprobe_free_utask(tsk);
  
  
   do { } while (0);
  # 777 "kernel/fork.c"
   if (tsk-&gt;clear_child_tid) {
    if (!(tsk-&gt;flags &amp; 0x00000400) &amp;&amp;
        (*(volatile int *)&amp;(&amp;mm-&gt;mm_users)-&gt;counter) &gt; 1) {
  
  
  
  
     ({ might_fault(); ({ unsigned long __limit = current_thread_info()-&gt;addr_limit - 1; const typeof(*(tsk-&gt;clear_child_tid)) *__tmp_p = (tsk-&gt;clear_child_tid); register const typeof(*(tsk-&gt;clear_child_tid)) __r2 asm("r2") = (0); register const typeof(*(tsk-&gt;clear_child_tid)) *__p asm("r0") = __tmp_p; register unsigned long __l asm("r1") = __limit; register int __e asm("r0"); switch (sizeof(*(__p))) { case 1: __asm__ __volatile__ ( ".ifnc " "%0" "," "r0" " ; .err ; .endif\n\t" ".ifnc " "%2" "," "r2" " ; .err ; .endif\n\t" ".ifnc " "%3" "," "r1" " ; .err ; .endif\n\t" "bl    __put_user_" "1" : "=&amp;r" (__e) : "0" (__p), "r" (__r2), "r" (__l) : "ip", "lr", "cc"); break; case 2: __asm__ __volatile__ ( ".ifnc " "%0" "," "r0" " ; .err ; .endif\n\t" ".ifnc " "%2" "," "r2" " ; .err ; .endif\n\t" ".ifnc " "%3" "," "r1" " ; .err ; .endif\n\t" "bl  __put_user_" "2" : "=&amp;r" (__e) : "0" (__p), "r" (__r2), "r" (__l) : "ip", "lr", "cc"); break; case 4: __asm__ __volatile__ ( ".ifnc " "%0" "," "r0" " ; .err ; .endif\n\t" ".ifnc " "%2" "," "r2" " ; .err ; .endif\n\t" ".ifnc " "%3" "," "r1" " ; .err ; .endif\n\t" "bl  __put_user_" "4" : "=&amp;r" (__e) : "0" (__p), "r" (__r2), "r" (__l) : "ip", "lr", "cc"); break; case 8: __asm__ __volatile__ ( ".ifnc " "%0" "," "r0" " ; .err ; .endif\n\t" ".ifnc " "%2" "," "r2" " ; .err ; .endif\n\t" ".ifnc " "%3" "," "r1" " ; .err ; .endif\n\t" "bl  __put_user_" "8" : "=&amp;r" (__e) : "0" (__p), "r" (__r2), "r" (__l) : "ip", "lr", "cc"); break; default: __e = __put_user_bad(); break; } __e; }); });
     sys_futex(tsk-&gt;clear_child_tid, 1,
       1, ((void *)0), ((void *)0), 0);
    }
    tsk-&gt;clear_child_tid = ((void *)0);
   }
  
  
  
  
  
   if (tsk-&gt;vfork_done)
    complete_vfork_done(tsk);
  }
  </code></pre>
  
  <p>The kernel code is the same.  The Build options are the same.
  Difference is the compiler used, all other infrastructure utilities (i.e. Make 4.1) are the same.</p>