I am developing a [Net-SNMP](http://www.net-snmp.org/) subagent whose final target will be an ARM board, so I'm using [CMake](https://cmake.org/) to make it easier to manage building the native and cross-compiled versions. I started testing it on my host platform (x86\_64) and it builds and works fine with this **CMakeLists.txt** file:

```bash
cmake_minimum_required (VERSION 2.6)

project (snmp_agent C)

set(snmp_agent_VERSION_MAJOR 1)
set(snmp_agent_VERSION_MINOR 0)

# Defines path to the net-snmp-config script
set(NETSNMPCONFIG "${CMAKE_FIND_ROOT_PATH}/usr/bin/net-snmp-config")

# Gets compiling flags and libs linked to Net-SNMP
execute_process(COMMAND "${NETSNMPCONFIG}" "--base-cflags" OUTPUT_VARIABLE NETSNMPCFLAGS)
execute_process(COMMAND "${NETSNMPCONFIG}" "--agent-libs" OUTPUT_VARIABLE NETSNMPLIBS)

# Removes leading/trailing spaces from net-snmp-config output
string(STRIP ${NETSNMPCFLAGS} NETSNMPCFLAGS)
string(STRIP ${NETSNMPLIBS} NETSNMPLIBS)

# Prints compilation and linker flags used in Net-SNMP package
message("Net-SNMP package CFLAGS: ${NETSNMPCFLAGS}")
message("Net-SNMP package LIBS: ${NETSNMPLIBS}")

# Setting libs and compilation flags variables
set(LIBS "${NETSNMPLIBS}")
set(STRICT_FLAGS "-Wall -Wstrict-prototypes")
set(CFLAGS "-I. ${STRICT_FLAGS} ${NETSNMPCFLAGS}")

# Sets prefix for files created by 'mib2c' for the wanted MIB
set(ENT_PHYSICAL_ENTRY "scalars/entPhysicalEntry")

# Source files created by 'mib2c' and then user customized
set(USER_SRCS
    ${ENT_PHYSICAL_ENTRY}.c
  )

# Setting subagent sources
set(SRCS ${USER_SRCS}
    ${CMAKE_PROJECT_NAME}.c
  )

# Finds the required Net-SNMP lib paths and assigns them to variables
find_library(NETSNMPAGENT "netsnmpagent")
message("Found ${NETSNMPAGENT}")
find_library(NETSNMPMIBS "netsnmpmibs")
message("Found ${NETSNMPMIBS}")
find_library(NETSNMP "netsnmp")
message("Found ${NETSNMP}")

# Sets the flags to be used for compiling and linking the executable
set_source_files_properties(${SRCS} COMPILE_FLAGS ${CFLAGS})
add_executable(${CMAKE_PROJECT_NAME} ${SRCS})
target_link_libraries(${CMAKE_PROJECT_NAME} ${NETSNMPAGENT} ${NETSNMPMIBS} ${NETSNMP})

```

I use a separate build dir so I don't mix build and source files, which stay along with the **CMakeLists.txt** file, so the the output of these commands...

```bash
cd ~/git/snmp_agent # CMakeLists.txt is in here along with source files
mkdir build
cd build
cmake ..

```

...is as follows:

```bash
claudio@slackdev:~/git/snmp_agent/build$ cmake ..
-- The C compiler identification is GNU 5.3.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
Net-SNMP package CFLAGS: -DNETSNMP_ENABLE_IPV6 -fno-strict-aliasing -O2 -fPIC -Ulinux -Dlinux=linux -D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2 -I/usr/lib64/perl5/CORE -I/usr/include/libnl3 -I/usr/include
Net-SNMP package LIBS: -L/usr/lib64 -lnetsnmpmibs -lsensors -lpci -ldl -lnetsnmpagent -lwrap -lnsl -Wl,-E -Wl,-rpath,/usr/lib64/perl5/CORE -lnetsnmp -lcrypto -lnl-3 -lm
Found /usr/lib64/libnetsnmpagent.so
Found /usr/lib64/libnetsnmpmibs.so
Found /usr/lib64/libnetsnmp.so
-- Configuring done
-- Generating done
-- Build files have been written to: /home/claudio/git/snmp_agent/build

```

Running `make`, it builds just fine:

```bash
claudio@slackdev:~/git/snmp_agent/build$ make
Scanning dependencies of target snmp_agent
[ 25%] Building C object CMakeFiles/snmp_agent.dir/scalars/entPhysicalEntry.c.o
[ 50%] Building C object CMakeFiles/snmp_agent.dir/snmp_agent.c.o
[100%] Linking C executable snmp_agent
[100%] Built target snmp_agent

```

This `snmp_agent` runs as an agentX subagent, connecting to the main `snmpd` master agent and properly answering SNMP requests against `entPhysicalEntry` scalar objects.

Ok, so far so good. The problem is when I try to cross-compile this so it does the same thing on my final target, which is an ARMv7 Allwinner A20 board. I already have a cross-toolchain available for it - which I created using [Crosstool-NG](http://crosstool-ng.github.io/). It is in the search PATH and its tools begin with the prefix `armv7-a20_allwinner-linux-gnueabihf`. In order to use it, I set up the following [toolchain file](https://cmake.org/Wiki/CMake_Cross_Compiling#The_toolchain_file), which was named `armv7-a20_allwinner-linux-gnueabihf.cmake`:

```bash
# the name of the target operating system
SET(CMAKE_SYSTEM_NAME Linux)

# which C and C++ compiler to use
SET(CMAKE_C_COMPILER   armv7-a20_allwinner-linux-gnueabihf-gcc)
SET(CMAKE_CXX_COMPILER armv7-a20_allwinner-linux-gnueabihf-g++) 

# here is the target environment located
SET(CMAKE_FIND_ROOT_PATH  "$ENV{HOME}/arm_rootfs") 

# adjust the default behaviour of the FIND_XXX() commands:
# search headers and libraries in the target environment, search 
# programs in the host environment
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)

```

Using a different build dir for the ARM version, CMake should pick up the right paths pointing to the ARM root filesystem, which is a copy of what will be on the target board's SDcard:

```bash
cd ~/git/snmp_agent # CMakeLists.txt and armv7-a20_allwinner-linux-gnueabihf.cmake are here
mkdir build-arm
cd build-arm
cmake -DCMAKE_TOOLCHAIN_FILE=../armv7-a20_allwinner-linux-gnueabihf.cmake ..

```

Note that apparently CMake runs fine, as the paths found are all relative to the ARM root filesystem:

```bash
claudio@slackdev:~/git/snmp_agent/build-arm$ cmake -DCMAKE_TOOLCHAIN_FILE=../cmake_defs/armv7-a20_allwinner-linux-gnueabihf.cmake ..
-- The C compiler identification is GNU 6.3.0
-- Check for working C compiler: /home/claudio/x-tools/armv7-a20_allwinner-linux-gnueabihf/bin/armv7-a20_allwinner-linux-gnueabihf-gcc
-- Check for working C compiler: /home/claudio/x-tools/armv7-a20_allwinner-linux-gnueabihf/bin/armv7-a20_allwinner-linux-gnueabihf-gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
Net-SNMP package CFLAGS: -DNETSNMP_ENABLE_IPV6 -fno-strict-aliasing -O2 -march=armv7-a -mfpu=vfpv3-d16 -mfloat-abi=hard -Ulinux -Dlinux=linux -D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2 -I/usr/lib/perl5/CORE -I/usr/include/libnl3 -I/usr/include
Net-SNMP package LIBS: -L/usr/lib -lnetsnmpmibs -lsensors -lpci -ldl -lnetsnmpagent -lwrap -lnsl -Wl,-E -Wl,-rpath,/usr/lib/perl5/CORE -lnetsnmp -lcrypto -lnl-3 -lm
Found /home/claudio/arm_rootfs/usr/lib/libnetsnmpagent.so
Found /home/claudio/arm_rootfs/usr/lib/libnetsnmpmibs.so
Found /home/claudio/arm_rootfs/usr/lib/libnetsnmp.so
-- Configuring done
-- Generating done
-- Build files have been written to: /home/claudio/git/snmp_agent/build-arm

```

But when running `make` it falls apart while looking for `/usr/include/gnu/stubs.h`. If we try to repeat `make` with `VERBOSE=1`, we can see the gcc invocation is using paths relative to the host machine:

```bash
claudio@slackdev:~/git/snmp_agent/build-arm$ make VERBOSE=1
/usr/bin/cmake -H/home/claudio/git/snmp_agent -B/home/claudio/git/snmp_agent/build-arm --check-build-system CMakeFiles/Makefile.cmake 0
/usr/bin/cmake -E cmake_progress_start /home/claudio/git/snmp_agent/build-arm/CMakeFiles /home/claudio/git/snmp_agent/build-arm/CMakeFiles/progress.marks
make -f CMakeFiles/Makefile2 all
make[1]: Entering directory 'snmp_agent/build-arm'
make -f CMakeFiles/snmp_agent.dir/build.make CMakeFiles/snmp_agent.dir/depend
make[2]: Entering directory 'snmp_agent/build-arm'
cd /home/claudio/git/snmp_agent/build-arm && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /home/claudio/git/snmp_agent /home/claudio/git/snmp_agent /home/claudio/git/snmp_agent/build-arm /home/claudio/git/snmp_agent/build-arm /home/claudio/git/snmp_agent/build-arm/CMakeFiles/snmp_agent.dir/DependInfo.cmake --color=
make[2]: Leaving directory 'snmp_agent/build-arm'
make -f CMakeFiles/snmp_agent.dir/build.make CMakeFiles/snmp_agent.dir/build
make[2]: Entering directory 'snmp_agent/build-arm'
[ 25%] Building C object CMakeFiles/snmp_agent.dir/scalars/entPhysicalEntry.c.o
/home/claudio/x-tools/armv7-a20_allwinner-linux-gnueabihf/bin/armv7-a20_allwinner-linux-gnueabihf-gcc     -I. -Wall -Wstrict-prototypes -DNETSNMP_ENABLE_IPV6 -fno-strict-aliasing -O2 -march=armv7-a -mfpu=vfpv3-d16 -mfloat-abi=hard -Ulinux -Dlinux=linux -D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2 -I/usr/lib/perl5/CORE -I/usr/include/libnl3 -I/usr/include -o CMakeFiles/snmp_agent.dir/scalars/entPhysicalEntry.c.o   -c /home/claudio/git/snmp_agent/scalars/entPhysicalEntry.c
In file included from /usr/include/features.h:392:0,
                 from /usr/include/stdio.h:27,
                 from /usr/include/net-snmp/net-snmp-includes.h:14,
                 from /home/claudio/git/snmp_agent/scalars/entPhysicalEntry.c:7:
/usr/include/gnu/stubs.h:7:27: fatal error: gnu/stubs-32.h: No such file or directory
 # include <gnu/stubs-32.h>
                           ^
compilation terminated.
CMakeFiles/snmp_agent.dir/build.make:62: recipe for target 'CMakeFiles/snmp_agent.dir/scalars/entPhysicalEntry.c.o' failed
make[2]: *** [CMakeFiles/snmp_agent.dir/scalars/entPhysicalEntry.c.o] Error 1
make[2]: Leaving directory 'snmp_agent/build-arm'
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/snmp_agent.dir/all' failed
make[1]: *** [CMakeFiles/snmp_agent.dir/all] Error 2
make[1]: Leaving directory 'snmp_agent/build-arm'
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2

```

The strange thing here is the `/usr/include/gnu/stubs.h` from the ARM root filesystem has nothing to do with `stubs-32.h`:

```c
/* This file is automatically generated.
   This file selects the right generated file of `__stub_FUNCTION' macros
   based on the architecture being compiled for.  */

#if !defined __ARM_PCS_VFP
# include <gnu/stubs-soft.h>
#endif
#if defined __ARM_PCS_VFP
# include <gnu/stubs-hard.h>
#endif

```

But if you look at the same file from the host system - which is a x86\_64 machine - we can guess why it tried to find `stubs-32.h` (remember we are trying to compile for ARM, so it won't find the `__x86_64__` symbol definition):

```c
/* This file is automatically generated.
   This file selects the right generated file of `__stub_FUNCTION' macros
   based on the architecture being compiled for.  */

#if !defined __x86_64__
# include <gnu/stubs-32.h>
#endif
#if defined __x86_64__ && defined __LP64__
# include <gnu/stubs-64.h>
#endif
#if defined __x86_64__ && defined __ILP32__
# include <gnu/stubs-x32.h>
#endif

```

Why is that happening, since the toolchain file clearly specifies that libraries and includes are to be searched ONLY in the path set by CMAKE\_FIND\_ROOT\_PATH?

**UPDATE (problem not totally solved yet!):**

After I had accepted the answer from @Tsyvarev I was double checking my **CMakeLists.txt** file and I found while trying to make it work I had made a hack by manually setting CMAKE\_FIND\_ROOT\_PATH as prefix of each compiler include switches (-I) returned by **net-snmp-config**, which obviously is not the ideal thing to do. That, along with CMAKE\_SYSROOT, makes it work, but CMAKE\_SYSROOT alone doesn't prefix the include paths by itself: 

```bash
# This is the manually hacked line:
set(NETSNMPCFLAGS "-DNETSNMP_ENABLE_IPV6 -fno-strict-aliasing -O2 -march=armv7-a -mfpu=vfpv3-d16 -mfloat-abi=hard -Ulinux -Dlinux=linux -D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I${CMAKE_FIND_ROOT_PATH}/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2 -I${CMAKE_FIND_ROOT_PATH}/usr/lib/perl5/CORE -I${CMAKE_FIND_ROOT_PATH}/usr/include/libnl3 -I${CMAKE_FIND_ROOT_PATH}/usr/include")

set(LIBS "${NETSNMPLIBS}")

set(STRICT_FLAGS "-Wall -Wstrict-prototypes")
set(CFLAGS "-I. ${STRICT_FLAGS} ${NETSNMPCFLAGS}")

```