(This question is an indirect continuation of my previous question which has been solved.)Small summary of what I'm trying to doI've got a .so file named pin_sim.so for the architectural simulator snipersim (http://www.snipersim.org). This library basically implements a wrapper around Intel's PIN instrumentation library used by the simulation.This file is dynamically loaded before the start of the simulation by the main application. As the simulation target executable must be loaded with the same library versions as pin_sim.so, and pin_sim.so packages some versions of the libraries, before a simulation starts, it checks which version of libraries are loaded by pin_sim.so using ldd.ProblemOn my home desktop (running Linux Mint 17.2), this works correctly. ldd pin_sim.so returns:linux-vdso.so.1 =>  (0x00007ffcd75a3000)libpython2.7.so.1.0 => /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 (0x00007fdd19d08000)librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fdd19b00000)libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fdd198e6000)libsqlite3.so.0 => /usr/lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007fdd1962d000)libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fdd19429000)libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fdd19124000)libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fdd18e1e000)libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fdd18c08000)libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fdd18842000)/lib64/ld-linux-x86-64.so.2 (0x00007fdd1ae55000)libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fdd18624000)libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007fdd18421000)However, on a computing cluster I have access to which runs OpenSUSE 13.1, this command returns (no matter if I copy over the file, or re-compile it):not a dynamic executableIn addition, I am unable to run sniper correctly (the simulation never executes, which I assume has to do with pin_sim.so not loading correctly).If I compile the file in the cluster, and then copy to my home machine, ldd works correctly as before. All other files I've tried ldd on (even the main sniper binary) work correctly, only pin_sim.so fails.More informationfile pin_sim.so on my home machine (same on the cluster, if not recompiled):pin_sim.so: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, BuildID[sha1]=4ac6397fefb655c475b5a941cef726183d3e3710, not strippedfile pin_sim.so on the cluster (re-compiled):pin_sim.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not strippedobjdump -p pin_sim.so | grep NEEDED on both machines (even if recompiled on the cluster):NEEDED               libpython2.7.so.1.0NEEDED               librt.so.1NEEDED               libz.so.1NEEDED               libsqlite3.so.0NEEDED               libdl.so.2NEEDED               libstdc++.so.6NEEDED               libm.so.6NEEDED               libgcc_s.so.1NEEDED               libc.so.6NEEDED               ld-linux-x86-64.so.2readelf -d pin_sim.so | grep NEEDED on both machines (even if recompiled on the cluster):0x0000000000000001 (NEEDED)             Shared library: [libpython2.7.so.1.0]0x0000000000000001 (NEEDED)             Shared library: [librt.so.1]0x0000000000000001 (NEEDED)             Shared library: [libz.so.1]0x0000000000000001 (NEEDED)             Shared library: [libsqlite3.so.0]0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]0x0000000000000001 (NEEDED)             Shared library: [ld-linux-x86-64.so.2]ldd --version on my home machine:ldd (Ubuntu EGLIBC 2.19-0ubuntu6.6) 2.19ldd --version on the cluster:ldd (Gentoo 2.20-r2 p4) 2.20What I've tried / My thoughtsI'm not very familiar with the linux dynamic linking process, so I haven't tried much.I looked around the web, and I could only find references to ldd being mistaken on very old binaries (compiled before ldd support was added to the ELF format), or when running ldd on 32-bit binaries under a 64-bit system without the 32-bit system libraries installed. Since this binary was compiled by me, and is 64-bit, I have no idea why ldd might be failing.I'm thinking of switching the version checker in sniper from using ldd to using readelf/objdump instead, but I'm still surprised that ldd fails like this.Any ideas? I do not have root access in the cluster.Edit 1: StraceI did two syscall traces on the cluster machines. One on the sniper binary (where ldd succeeds) and one on the pin_sim.so binary (where ldd fails).I cut off the beginning, until the interesting part where they start to diverge.EDIT 2: Now strace -fstrace -f ldd sniper:faccessat(AT_FDCWD, ./sniper, R_OK)   = 0faccessat(AT_FDCWD, ./sniper, X_OK)   = 0faccessat(AT_FDCWD, /lib/ld-linux.so.2, X_OK) = -1 ENOENT (No such file or directory)faccessat(AT_FDCWD, /lib64/ld-linux-x86-64.so.2, X_OK) = 0pipe([3, 4])                            = 0rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0rt_sigprocmask(SIG_BLOCK, [INT CHLD], [], 8) = 0lseek(255, -52, SEEK_CUR)               = 5706clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x359287389d0) = 21112rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0rt_sigaction(SIGCHLD, {0x6f679b3710, [], SA_RESTORER|SA_RESTART, 0x35927cffc20}, {0x6f679b3710, [], SA_RESTORER|SA_RESTART, 0x35927cffc20}, 8) = 0close(4)                                = 0read(3, Process 21112 attached <unfinished ...>[pid 21112] close(255)                  = 0[pid 21112] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0[pid 21112] rt_sigaction(SIGTSTP, {SIG_DFL, [], SA_RESTORER, 0x35927cffc20}, {SIG_DFL, [], 0}, 8) = 0[pid 21112] rt_sigaction(SIGTTIN, {SIG_DFL, [], SA_RESTORER, 0x35927cffc20}, {SIG_DFL, [], 0}, 8) = 0[pid 21112] rt_sigaction(SIGTTOU, {SIG_DFL, [], SA_RESTORER, 0x35927cffc20}, {SIG_DFL, [], 0}, 8) = 0[pid 21112] rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x35927cffc20}, {SIG_DFL, [], SA_RESTORER, 0x35927cffc20}, 8) = 0[pid 21112] rt_sigaction(SIGQUIT, {SIG_DFL, [], SA_RESTORER, 0x35927cffc20}, {SIG_IGN, [], SA_RESTORER, 0x35927cffc20}, 8) = 0[pid 21112] rt_sigaction(SIGCHLD, {SIG_DFL, [], SA_RESTORER|SA_RESTART, 0x35927cffc20}, {0x6f679b3710, [], SA_RESTORER|SA_RESTART, 0x35927cffc20}, 8) = 0[pid 21112] rt_sigaction(SIGCHLD, {0x6f679b3710, [], SA_RESTORER|SA_RESTART, 0x35927cffc20}, {SIG_DFL, [], SA_RESTORER|SA_RESTART, 0x35927cffc20}, 8) = 0[pid 21112] rt_sigaction(SIGINT, {0x6f679ce1a0, [], SA_RESTORER, 0x35927cffc20}, {SIG_DFL, [], SA_RESTORER, 0x35927cffc20}, 8) = 0[pid 21112] dup2(4, 1)                  = 1[pid 21112] close(4)                    = 0[pid 21112] close(3)                    = 0[pid 21112] rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0[pid 21112] rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x35927cffc20}, {0x6f679ce1a0, [], SA_RESTORER, 0x35927cffc20}, 8) = 0[pid 21112] rt_sigaction(SIGQUIT, {SIG_DFL, [], SA_RESTORER, 0x35927cffc20}, {SIG_DFL, [], SA_RESTORER, 0x35927cffc20}, 8) = 0[pid 21112] rt_sigaction(SIGCHLD, {SIG_DFL, [], SA_RESTORER|SA_RESTART, 0x35927cffc20}, {0x6f679b3710, [], SA_RESTORER|SA_RESTART, 0x35927cffc20}, 8) = 0[pid 21112] execve(/lib64/ld-linux-x86-64.so.2, [/lib64/ld-linux-x86-64.so.2, --verify, ./sniper], [/* 51 vars */]) = 0[pid 21112] brk(0)                      = 0x2d9db43c910[pid 21112] open(./sniper, O_RDONLY|O_CLOEXEC) = 3[pid 21112] read(3, \177ELF\2\1\1\3\0\0\0\0\0\0\0\0\2\0>\0\1\0\0\0\321\307@\0\0\0\0\0..., 832) = 832[pid 21112] fstat(3, {st_mode=S_IFREG|0775, st_size=46853390, ...}) = 0[pid 21112] getcwd(/path/to/sniper/lib, 128) = 52[pid 21112] mmap(0x400000, 3837952, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0) = 0x400000[pid 21112] mmap(0x9a8000, 626688, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3a8000) = 0x9a8000[pid 21112] mmap(0xa41000, 32872, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xa41000[pid 21112] close(3)                    = 0[pid 21112] exit_group(0)               = ?[pid 21111] <... read resumed> , 128) = 0[pid 21112] +++ exited with 0 +++--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=21112, si_uid=169971, si_status=0, si_utime=0, si_stime=0} ---wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG, NULL) = 21112wait4(-1, 0x3e3bd435d80, WNOHANG, NULL) = -1 ECHILD (No child processes)rt_sigreturn()                          = 0close(3)                                = 0*** TRACE CONTINUES, no longer important because the other one has already errored out by this point ***strace ldd -f pin_sim.so:faccessat(AT_FDCWD, ./pin_sim.so, R_OK) = 0faccessat(AT_FDCWD, ./pin_sim.so, X_OK) = 0faccessat(AT_FDCWD, /lib/ld-linux.so.2, X_OK) = -1 ENOENT (No such file or directory)faccessat(AT_FDCWD, /lib64/ld-linux-x86-64.so.2, X_OK) = 0pipe([3, 4])                            = 0rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0rt_sigprocmask(SIG_BLOCK, [INT CHLD], [], 8) = 0lseek(255, -52, SEEK_CUR)               = 5706clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x36aaed959d0) = 21099rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0rt_sigaction(SIGCHLD, {0x798dd3a710, [], SA_RESTORER|SA_RESTART, 0x36aae35cc20}, {0x798dd3a710, [], SA_RESTORER|SA_RESTART, 0x36aae35cc20}, 8) = 0close(4)                                = 0read(3, Process 21099 attached <unfinished ...>[pid 21099] close(255)                  = 0[pid 21099] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0[pid 21099] rt_sigaction(SIGTSTP, {SIG_DFL, [], SA_RESTORER, 0x36aae35cc20}, {SIG_DFL, [], 0}, 8) = 0[pid 21099] rt_sigaction(SIGTTIN, {SIG_DFL, [], SA_RESTORER, 0x36aae35cc20}, {SIG_DFL, [], 0}, 8) = 0[pid 21099] rt_sigaction(SIGTTOU, {SIG_DFL, [], SA_RESTORER, 0x36aae35cc20}, {SIG_DFL, [], 0}, 8) = 0[pid 21099] rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x36aae35cc20}, {SIG_DFL, [], SA_RESTORER, 0x36aae35cc20}, 8) = 0[pid 21099] rt_sigaction(SIGQUIT, {SIG_DFL, [], SA_RESTORER, 0x36aae35cc20}, {SIG_IGN, [], SA_RESTORER, 0x36aae35cc20}, 8) = 0[pid 21099] rt_sigaction(SIGCHLD, {SIG_DFL, [], SA_RESTORER|SA_RESTART, 0x36aae35cc20}, {0x798dd3a710, [], SA_RESTORER|SA_RESTART, 0x36aae35cc20}, 8) = 0[pid 21099] rt_sigaction(SIGCHLD, {0x798dd3a710, [], SA_RESTORER|SA_RESTART, 0x36aae35cc20}, {SIG_DFL, [], SA_RESTORER|SA_RESTART, 0x36aae35cc20}, 8) = 0[pid 21099] rt_sigaction(SIGINT, {0x798dd551a0, [], SA_RESTORER, 0x36aae35cc20}, {SIG_DFL, [], SA_RESTORER, 0x36aae35cc20}, 8) = 0[pid 21099] dup2(4, 1)                  = 1[pid 21099] close(4)                    = 0[pid 21099] close(3)                    = 0[pid 21099] rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0[pid 21099] rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x36aae35cc20}, {0x798dd551a0, [], SA_RESTORER, 0x36aae35cc20}, 8) = 0[pid 21099] rt_sigaction(SIGQUIT, {SIG_DFL, [], SA_RESTORER, 0x36aae35cc20}, {SIG_DFL, [], SA_RESTORER, 0x36aae35cc20}, 8) = 0[pid 21099] rt_sigaction(SIGCHLD, {SIG_DFL, [], SA_RESTORER|SA_RESTART, 0x36aae35cc20}, {0x798dd3a710, [], SA_RESTORER|SA_RESTART, 0x36aae35cc20}, 8) = 0[pid 21099] execve(/lib64/ld-linux-x86-64.so.2, [/lib64/ld-linux-x86-64.so.2, --verify, ./pin_sim.so], [/* 51 vars */]) = 0[pid 21099] brk(0)                      = 0x2b0b251b090[pid 21099] open(./pin_sim.so, O_RDONLY|O_CLOEXEC) = 3[pid 21099] read(3, \177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240@\24\0\0\0\0\0..., 832) = 832[pid 21099] fstat(3, {st_mode=S_IFREG|0775, st_size=55707995, ...}) = 0[pid 21099] getcwd(/path/to/sniper/lib, 128) = 52[pid 21099] mmap(NULL, 12474024, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x2b0af187000[pid 21099] mprotect(0x2b0afa9b000, 2093056, PROT_NONE) = 0[pid 21099] mmap(0x2b0afc9a000, 716800, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x913000) = 0x2b0afc9a000[pid 21099] mmap(0x2b0afd49000, 145064, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x2b0afd49000[pid 21099] mprotect(0x3a0fbff6000, 4096, PROT_READ|PROT_WRITE|PROT_EXEC|PROT_GROWSDOWN) = -1 EACCES (Permission denied)[pid 21099] close(3)                    = 0[pid 21099] exit_group(1)               = ?[pid 21099] +++ exited with 1 +++<... read resumed> , 128)             = 0--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=21099, si_uid=169971, si_status=1, si_utime=0, si_stime=1} ---wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 1}], WNOHANG, NULL) = 21099wait4(-1, 0x3de445c7ac0, WNOHANG, NULL) = -1 ECHILD (No child processes)rt_sigreturn()                          = 0close(3)                                = 0rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0rt_sigaction(SIGINT, {0x798dd36e90, [], SA_RESTORER, 0x36aae35cc20}, {SIG_DFL, [], SA_RESTORER, 0x36aae35cc20}, 8) = 0rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x36aae35cc20}, {0x798dd36e90, [], SA_RESTORER, 0x36aae35cc20}, 8) = 0rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0faccessat(AT_FDCWD, /libx32/ld-linux-x32.so.2, X_OK) = -1 ENOENT (No such file or directory)fstat(1, {st_mode=S_IFCHR|0622, st_rdev=makedev(136, 0), ...}) = 0mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x36aaed99000write(1, \tnot a dynamic executable\n, 26 not a dynamic executable) = 26read(255, \nexit $result\n# Local Variables:..., 5758) = 52rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0exit_group(1)                           = ?+++ exited with 1 +++Edit 3readelf -Wl pin_sim.so:Elf file type is DYN (Shared object file)Entry point 0x1440a0There are 7 program headers, starting at offset 64Program Headers:  Type           Offset   VirtAddr           PhysAddr           FileSiz  MemSiz   Flg Align  LOAD           0x000000 0x0000000000000000 0x0000000000000000 0x913415 0x913415 R E 0x200000  LOAD           0x913990 0x0000000000b13990 0x0000000000b13990 0x0ae648 0x0d1d18 RW  0x200000  DYNAMIC        0x9bf0f8 0x0000000000bbf0f8 0x0000000000bbf0f8 0x0002a0 0x0002a0 RW  0x8  GNU_EH_FRAME   0x7fbee0 0x00000000007fbee0 0x00000000007fbee0 0x036cfc 0x036cfc R   0x4  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000 0x000000 RWE 0x10  GNU_RELRO      0x913990 0x0000000000b13990 0x0000000000b13990 0x0ac670 0x0ac670 R   0x1  PAX_FLAGS      0x000000 0x0000000000000000 0x0000000000000000 0x000000 0x000000     0x8 Section to Segment mapping:  Segment Sections...   00     .hash .dynsym .dynstr .gnu.version .gnu.version_d .gnu.version_r .rela.dyn .rela.plt .init .plt .text .fini .rodata .charmversion .eh_frame_hdr .eh_frame .gcc_except_table    01     .init_array .fini_array .jcr .data.rel.ro .dynamic .got .data .pinclientint .bss    02     .dynamic    03     .eh_frame_hdr    04        05     .init_array .fini_array .jcr .data.rel.ro .dynamic .got    06
ldd returns not a dynamic executable for a 64-bit ELF, even though objdump and readelf disagree
opensuse;dynamic linking
In second strace part that tells about x32 binaries is strange, but unlikely:faccessat(AT_FDCWD, /libx32/ld-linux-x32.so.2, X_OK) = -1 ENOENT (No such file or directory)Maybe cluster's /lib64/ld-linux-x86-64.so.2 dislikes your pin_sim.so somehow.To further troubleshoot the issue, try running /lib64/ld-linux-x86-64.so.2 --list /path/to/your/pin_sim.so on your cluster.Next time, don't forget to add -f to your strace arguments, it turns on full process tree tracing.