title: Why is cmake finding 32-bit libraries instead of 64-bit libs on a 64-bit system?
body: <h1>Problem Description</h1>
  <p>I'm working on porting several codebases over from Red Hat 5 to 6, and I've run into a cmake issue that I'm completely stumped by.</p>
  <p>Cmake is consistently finding the 32-bit versions of libraries under <code>/usr/lib</code> instead of the 64-bit versions under <code>/usr/lib64</code> on an RHEL6 system, while it correctly detects the lib64 versions on the an RHEL5 system.</p>
  <h1>Minimal Example</h1>
  <p>For example, I have a very minimal <code>CMakeLists.txt</code> file:</p>
  <pre><code>cmake_minimum_required(VERSION 2.8)
  
  find_library(XTEST X11)
  message(&quot;Found X11 at ${XTEST}&quot;)
  </code></pre>
  <p>On an RHEL6 system, running <code>cmake</code> on this results in:</p>
  <pre><code>$ cmake ..
  -- The C compiler identification is GNU 4.4.7
  -- The CXX compiler identification is GNU 4.4.7
  -- Check for working C compiler: /usr/bin/cc
  -- Check for working C compiler: /usr/bin/cc -- works
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Check for working CXX compiler: /usr/bin/c++
  -- Check for working CXX compiler: /usr/bin/c++ -- works
  -- Detecting CXX compiler ABI info
  -- Detecting CXX compiler ABI info - done
  Found X11 at /usr/lib/libX11.so
  -- Configuring done
  -- Generating done
  -- Build files have been written to: $HOME/software/64bit_problem/build
  </code></pre>
  <p>(The key part here is the <code>Found X11 at /usr/lib/libX11.so</code> line)</p>
  <p>However, if I do the same thing on an RHEL5 system, it correctly detects the <code>/usr/lib64/</code> version: (Note that I'm clearing out the <code>CMakeCache.txt</code> and other temporary cmake-files in between runs.)</p>
  <pre><code>$ cmake ..
  -- The C compiler identification is GNU 4.1.2
  -- The CXX compiler identification is GNU 4.1.2
  -- Check for working C compiler: /usr/bin/cc
  -- Check for working C compiler: /usr/bin/cc -- works
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Check for working CXX compiler: /usr/bin/c++
  -- Check for working CXX compiler: /usr/bin/c++ -- works
  -- Detecting CXX compiler ABI info
  -- Detecting CXX compiler ABI info - done
  Found X11 at /usr/lib64/libX11.so
  -- Configuring done
  -- Generating done
  -- Build files have been written to: $HOME/software/64bit_problem/build
  </code></pre>
  <h1>Troubleshooting Info</h1>
  <p>The <code>/usr/lib64</code> versions of the libraries exist on both systems.  Here's the listing on the RHEL6 system:</p>
  <pre><code>$ ls /usr/lib64/libX11.so*
  /usr/lib64/libX11.so.6  /usr/lib64/libX11.so.6.3.0
  </code></pre>
  <p>And on the RHEL5 system:</p>
  <pre><code>$ ls /usr/lib64/libX11.so*
  /usr/lib64/libX11.so.6  /usr/lib64/libX11.so.6.3.0
  </code></pre>
  <p>Just to confirm that <code>/usr/lib</code> is indeed the 32-bit versions (also, it's not a symlink to another location):</p>
  <pre><code>$ file /usr/lib/libX11.so*
  /usr/lib/libX11.so:       symbolic link to `libX11.so.6.3.0'
  /usr/lib/libX11.so.6:     symbolic link to `libX11.so.6.3.0'
  /usr/lib/libX11.so.6.3.0: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, stripped
  
  $ file /usr/lib64/libX11.so*
  /usr/lib64/libX11.so.6:     symbolic link to `libX11.so.6.3.0'
  /usr/lib64/libX11.so.6.3.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, stripped
  </code></pre>
  <p>This is probably due to an environment setting somewhere, but I'm a bit stumped as to where.  I don't have <code>LD_LIBRARY_PATH</code>, <code>LD_RUN_PATH</code>, or <code>LDFLAGS</code> set. Same for <code>CFLAGS</code> and <code>CXXFLAGS</code>.  My user environment should be identical, as my $HOME is an NFS share that's the same on both machines.</p>
  <p><code>/usr/lib</code> isn't in my <code>$PATH</code>, and regardless, limiting my path to a minimal subset doesn't seem to help:</p>
  <pre><code>$ export PATH=/bin:/usr/bin:$HOME/local/bin
  
  $ cmake ..
  &lt;snip&gt;
  Found X11 at /usr/lib/libX11.so
  -- Configuring done
  &lt;snip&gt;
  </code></pre>
  <p>@SergyA had the excellent suggestion of checking which environment variables were being accessed by using <code>ltrace</code>. <code>strace</code> didn't turn up anything diagnostic from what I could see, but <code>ltrace</code> shows the environment variable access nicely. Here's a quick summary:</p>
  <pre><code>$ ltrace -o ltrace_output cmake ..
  
  $ grep getenv ltrace_output
  getenv(&quot;PWD&quot;)                                                    = &quot;$HOME/software/64bit_problem&quot;
  getenv(&quot;PATH&quot;)                                                   = &quot;/bin:/usr/bin:$HOME/local/bin&quot;
  getenv(&quot;CMAKE_ROOT&quot;)                                             = NULL
  getenv(&quot;MAKEFLAGS&quot;)                                              = NULL
  getenv(&quot;VERBOSE&quot;)                                                = NULL
  getenv(&quot;CFLAGS&quot;)                                                 = NULL
  getenv(&quot;LDFLAGS&quot;)                                                = NULL
  getenv(&quot;LDFLAGS&quot;)                                                = NULL
  getenv(&quot;LDFLAGS&quot;)                                                = NULL
  getenv(&quot;CXXFLAGS&quot;)                                               = NULL
  getenv(&quot;LDFLAGS&quot;)                                                = NULL
  getenv(&quot;LDFLAGS&quot;)                                                = NULL
  getenv(&quot;LDFLAGS&quot;)                                                = NULL
  </code></pre>
  <h3><code>cmake</code>-specific Trobleshooting</h3>
  <p>The cmake version is identical on both machines (in fact, it's the same executable for reasons I'll omit for brevity):</p>
  <pre><code>$ cmake --version
  cmake version 2.8.11.2
  
  $ which cmake
  $HOME/local/bin/cmake
  </code></pre>
  <p>I've tried explicitly enabling <code>FIND_LIBRARY_USE_LIB64_PATHS</code>, but this doesn't seem to make a difference:</p>
  <pre><code>cmake_minimum_required(VERSION 2.8)
  
  set_property(GLOBAL PROPERTY FIND_LIBRARY_USE_LIB64_PATHS ON)
  
  find_library(XTEST X11)
  message(&quot;Found X11 at ${XTEST}&quot;)
  </code></pre>
  <p>As @Ravi mentioned, this is more likely due to some issue with <code>CMAKE_LIBRARY_PATH</code>, but it's not set, and changing it as either an environment variable or as a cmake variable doesn't seem to help.  However, I fully admit I am largely ignorant of the various cmake configuration variables, so it's very likely that I'm missing something obvious here.</p>
  <p>One key thing I've recently realized is that it's not all libraries... For example:</p>
  <pre><code>cmake_minimum_required(VERSION 2.8)
  
  find_library(PNGTEST png)
  message(&quot;Found libpng at ${PNGTEST}&quot;)
  </code></pre>
  <p>Finds <code>/usr/lib64/libpng.so</code> instead of <code>/usr/lib/libpng.so</code>.</p>
  <p>This leads me to think it's something cmake-specific, at any rate.</p>
  <h3><code>find_package</code> instead of <code>find_library</code></h3>
  <p>Given the library-specific issues I mentioned above, I thought to try finding the entire X11 package instead of a single library (which is really what the codebase I was working with should have done anyway).</p>
  <p>However, I'm getting even more confusing results... It seems to be detecting a mix of 64-bit and 32-bit libraries?  For example:</p>
  <pre><code>cmake_minimum_required(VERSION 2.8)
  
  FIND_PACKAGE(X11 REQUIRED)
  
  message(&quot;X11_LIBRARIES: ${X11_LIBRARIES}&quot;)
  </code></pre>
  <p>We'll get:</p>
  <pre><code>$ cmake ..
  &lt;snip&gt;
  -- Looking for XOpenDisplay in /usr/lib/libX11.so;/usr/lib/libXext.so
  -- Looking for XOpenDisplay in /usr/lib/libX11.so;/usr/lib/libXext.so - not found
  &lt;snip&gt;
  -- Found X11: /usr/lib/libX11.so
  X11_LIBRARIES: /usr/lib64/libSM.so;/usr/lib64/libICE.so;/usr/lib/libX11.so;/usr/lib/libXext.so
  -- Configuring done
  &lt;snip&gt;
  </code></pre>
  <p>However, if we have a look at these specific libraries in <code>X11_LIBRARIES</code>, they're a mix of 32-bit and 64-bit versions!</p>
  <pre><code>$ file /usr/lib64/libSM.so.6.0.1
  /usr/lib64/libSM.so.6.0.1: ELF 64-bit LSB shared object, x86-64, &lt;snip&gt;
  
  $ file /usr/lib64/libICE.so.6.3.0
  /usr/lib64/libICE.so.6.3.0: ELF 64-bit LSB shared object, x86-64, &lt;snip&gt;
  
  $ file /usr/lib/libX11.so.6.3.0
  /usr/lib/libX11.so.6.3.0: ELF 32-bit LSB shared object, Intel 80386, &lt;snip&gt;
  
  $ file /usr/lib/libXext.so.6.4.0
  /usr/lib/libXext.so.6.4.0: ELF 32-bit LSB shared object, Intel 80386, &lt;snip&gt;
  </code></pre>
  <h1>In summary, what else should I try?</h1>
  <p>Am I missing a cmake-specific or possibly X11-specific configuration option?</p>