title: cross compiling with sysroot from device
body: <p>I mounted target device's (Beaglebone Black, ARM arch) root with sshfs, to a folder on host, and trying to cross compile using <code>--sysroot</code> option of GCC.</p>
  
  <p>But there's a problem - some include header files are located on target device not just at /usr/include folder, but in its subfolder <code>arm-linux-gnueabihf</code>. The same there's /usr/lib/arm-linux-gnueabihf folder too.</p>
  
  <p>"Hello world" sample fails to compile</p>
  
  <pre><code>[gmixaz:/work] $ $CC --sysroot=/work/sysroot h.c
  In file included from /work/sysroot/usr/include/stdio.h:27:0,
                   from h.c:1:
  /work/sysroot/usr/include/features.h:364:25: fatal error: sys/cdefs.h: No such file or directory
   #  include &lt;sys/cdefs.h&gt;
  </code></pre>
  
  <p>because <code>sys/cdefs.h</code> is located in /usr/include/arm-linux-gnueabihf, while compiler expects it in /usr/include</p>
  
  <p>My question is why do I have that subfolder <code>arm-linux-gnueabihf</code> in /usr/include? What was the rationale to put part of include files (and .so files in /usr/lib/arm-linux-gnueabihf) to that subfolder?</p>
  
  <p>And how shall I address this issue when cross compiling with --sysroot option - do I need to specify the subfolder with -I and -L compiler options, or there's some better solution? I though only --sysroot should be enough but it isn't. Is that 'by design'?</p>
  
  <p>After adding the folders to command line I got another problem:</p>
  
  <pre><code>[gmixaz:/work] 1 $ $CC --sysroot /work/sysroot3 -I /work/sysroot3/usr/include/arm-linux-gnueabihf -L /work/sysroot3/usr/lib/arm-linux-gnueabihf h.c
  /usr/xcc/arm-cortexa9_neon-linux-gnueabihf/lib/gcc/arm-cortexa9_neon-linux-gnueabihf/6.3.1/../../../../arm-cortexa9_neon-linux-gnueabihf/bin/ld.bfd: cannot find crt1.o: No such file or directory
  /usr/xcc/arm-cortexa9_neon-linux-gnueabihf/lib/gcc/arm-cortexa9_neon-linux-gnueabihf/6.3.1/../../../../arm-cortexa9_neon-linux-gnueabihf/bin/ld.bfd: cannot find crti.o: No such file or directory
  collect2: error: ld returned 1 exit status
  </code></pre>
  
  <p>What's going on? When building crosstool-NG I tried to match target arch info. Cross-compiling GCC on host:</p>
  
  <pre><code>[gmixaz:/work] 1 $ $CC -v
  Using built-in specs.
  COLLECT_GCC=/usr/xcc/arm-cortexa9_neon-linux-gnueabihf/bin/arm-cortexa9_neon-linux-gnueabihf-gcc
  COLLECT_LTO_WRAPPER=/usr/xcc/arm-cortexa9_neon-linux-gnueabihf/libexec/gcc/arm-cortexa9_neon-linux-gnueabihf/6.3.1/lto-wrapper
  Target: arm-cortexa9_neon-linux-gnueabihf
  Configured with: /dockcross/crosstool/toolchain/.build/src/gcc-linaro-6.3-2017.02/configure --build=x86_64-build_pc-linux-gnu --host=x86_64-build_pc-linux-gnu --target=arm-cortexa9_neon-linux-gnueabihf --prefix=/usr/xcc/arm-cortexa9_neon-linux-gnueabihf --with-sysroot=/usr/xcc/arm-cortexa9_neon-linux-gnueabihf/arm-cortexa9_neon-linux-gnueabihf/sysroot --enable-languages=c,c++ --with-cpu=cortex-a8 --with-fpu=neon --with-float=hard --with-pkgversion='crosstool-NG ' --enable-__cxa_atexit --disable-libmudflap --disable-libgomp --disable-libssp --disable-libquadmath --disable-libquadmath-support --disable-libsanitizer --disable-libmpx --with-gmp=/dockcross/crosstool/toolchain/.build/arm-cortexa9_neon-linux-gnueabihf/buildtools --with-mpfr=/dockcross/crosstool/toolchain/.build/arm-cortexa9_neon-linux-gnueabihf/buildtools --with-mpc=/dockcross/crosstool/toolchain/.build/arm-cortexa9_neon-linux-gnueabihf/buildtools --with-isl=/dockcross/crosstool/toolchain/.build/arm-cortexa9_neon-linux-gnueabihf/buildtools --enable-lto --with-host-libstdcxx='-static-libgcc -Wl,-Bstatic,-lstdc++,-Bdynamic -lm' --enable-threads=posix --enable-plugin --enable-gold --with-libintl-prefix=/dockcross/crosstool/toolchain/.build/arm-cortexa9_neon-linux-gnueabihf/buildtools --disable-multilib --with-local-prefix=/usr/xcc/arm-cortexa9_neon-linux-gnueabihf/arm-cortexa9_neon-linux-gnueabihf/sysroot --enable-long-long
  Thread model: posix
  gcc version 6.3.1 20170109 (crosstool-NG ) 
  </code></pre>
  
  <p>GCC on target:</p>
  
  <pre><code>debian@beaglebone:~$ cc -v
  Using built-in specs.
  COLLECT_GCC=cc
  COLLECT_LTO_WRAPPER=/usr/lib/gcc/arm-linux-gnueabihf/6/lto-wrapper
  Target: arm-linux-gnueabihf
  Configured with: ../src/configure -v --with-pkgversion='Debian 6.3.0-18+deb9u1' --with-bugurl=file:///usr/share/doc/gcc-6/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-6 --program-prefix=arm-linux-gnueabihf- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-libitm --disable-libquadmath --enable-plugin --enable-default-pie --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-6-armhf/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-6-armhf --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-6-armhf --with-arch-directory=arm --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable-sjlj-exceptions --with-arch=armv7-a --with-fpu=vfpv3-d16 --with-float=hard --with-mode=thumb --enable-checking=release --build=arm-linux-gnueabihf --host=arm-linux-gnueabihf --target=arm-linux-gnueabihf
  Thread model: posix
  gcc version 6.3.0 20170516 (Debian 6.3.0-18+deb9u1) 
  </code></pre>
  
  <p>What I'm missing in the toolchain configuration?</p>
  
  <p><strong>ADDED:</strong> I have found that following file is responsible for libraries being placed to that subfolder on target:</p>
  
  <pre><code>debian@beaglebone:~$ cat /etc/ld.so.conf.d/arm-linux-gnueabihf.conf
  # Multiarch support
  /lib/arm-linux-gnueabihf
  /usr/lib/arm-linux-gnueabihf
  </code></pre>
  
  <p>Now I'm trying to understand what is correct way to attach that folders to cross compiler toolchain.</p>
  
  <p>Also, I guess that something similar is set for include header files, just interesting where's that setting for GCC on target?</p>
  
  <p><strong>ADDED:</strong> Similar issue discussed for RPi cross compilation: <a href="https://github.com/raspberrypi/tools/issues/42" rel="noreferrer">https://github.com/raspberrypi/tools/issues/42</a></p>