title: How to fix GCC compilation error when compiling >2 GB of code?
body: <p>I have a huge number of functions totaling around 2.8 GB of object code (unfortunately there's no way around, scientific computing ...)</p>
  <p>When I try to link them, I get (expected) <code>relocation truncated to fit: R_X86_64_32S</code> errors, that I hoped to circumvent by specifing the compiler flag <code>-mcmodel=medium</code>. All libraries that are linked in addition that I have control of are compiled with the <code>-fpic</code> flag.</p>
  <p>Still, the error persists, and I assume that some libraries I link to are not compiled with PIC.</p>
  <p>Here's the error:</p>
  <pre><code>/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/crt1.o: In function `_start':
  (.text+0x12): relocation truncated to fit: R_X86_64_32S against symbol `__libc_csu_fini'     defined in .text section in /usr/lib64/libc_nonshared.a(elf-init.oS)
  /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/crt1.o: In function `_start':
  (.text+0x19): relocation truncated to fit: R_X86_64_32S against symbol `__libc_csu_init'    defined in .text section in /usr/lib64/libc_nonshared.a(elf-init.oS)
  /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/crt1.o: In function `_start':
  (.text+0x20): undefined reference to `main'
  /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/crti.o: In function    `call_gmon_start':
  (.text+0x7): relocation truncated to fit: R_X86_64_GOTPCREL against undefined symbol      `__gmon_start__'
  /usr/lib/gcc/x86_64-redhat-linux/4.1.2/crtbegin.o: In function `__do_global_dtors_aux':
  crtstuff.c:(.text+0xb): relocation truncated to fit: R_X86_64_PC32 against `.bss' 
  crtstuff.c:(.text+0x13): relocation truncated to fit: R_X86_64_32 against symbol `__DTOR_END__' defined in .dtors section in /usr/lib/gcc/x86_64-redhat-linux/4.1.2/crtend.o
  crtstuff.c:(.text+0x19): relocation truncated to fit: R_X86_64_32S against `.dtors'
  crtstuff.c:(.text+0x28): relocation truncated to fit: R_X86_64_PC32 against `.bss'
  crtstuff.c:(.text+0x38): relocation truncated to fit: R_X86_64_PC32 against `.bss'
  crtstuff.c:(.text+0x3f): relocation truncated to fit: R_X86_64_32S against `.dtors'
  crtstuff.c:(.text+0x46): relocation truncated to fit: R_X86_64_PC32 against `.bss'
  crtstuff.c:(.text+0x51): additional relocation overflows omitted from the output
  collect2: ld returned 1 exit status
  make: *** [testsme] Error 1
  </code></pre>
  <p>And system libraries I link against:</p>
  <pre><code>-lgfortran -lm -lrt -lpthread
  </code></pre>
  <p>Any clues where to look for the problem?</p>
  <p><strong>EDIT:</strong></p>
  <p>First of all, thank you for the discussion...</p>
  <p>To clarify a bit, I have hundreds of functions (each approx 1 MB in size in separate object files) like this:</p>
  <pre><code>double func1(std::tr1::unordered_map&lt;int, double&gt; &amp; csc, 
               std::vector&lt;EvaluationNode::Ptr&gt; &amp; ti, 
               ProcessVars &amp; s)
  {
      double sum, prefactor, expr;
  
      prefactor = +s.ds8*s.ds10*ti[0]-&gt;value();
      expr =       ( - 5/243.*(s.x14*s.x15*csc[49300] + 9/10.*s.x14*s.x15*csc[49301] +
             1/10.*s.x14*s.x15*csc[49302] - 3/5.*s.x14*s.x15*csc[49303] -
             27/10.*s.x14*s.x15*csc[49304] + 12/5.*s.x14*s.x15*csc[49305] -
             3/10.*s.x14*s.x15*csc[49306] - 4/5.*s.x14*s.x15*csc[49307] +
             21/10.*s.x14*s.x15*csc[49308] + 1/10.*s.x14*s.x15*csc[49309] -
             s.x14*s.x15*csc[51370] - 9/10.*s.x14*s.x15*csc[51371] -
             1/10.*s.x14*s.x15*csc[51372] + 3/5.*s.x14*s.x15*csc[51373] +
             27/10.*s.x14*s.x15*csc[51374] - 12/5.*s.x14*s.x15*csc[51375] +
             3/10.*s.x14*s.x15*csc[51376] + 4/5.*s.x14*s.x15*csc[51377] -
             21/10.*s.x14*s.x15*csc[51378] - 1/10.*s.x14*s.x15*csc[51379] -
             2*s.x14*s.x15*csc[55100] - 9/5.*s.x14*s.x15*csc[55101] -
             1/5.*s.x14*s.x15*csc[55102] + 6/5.*s.x14*s.x15*csc[55103] +
             27/5.*s.x14*s.x15*csc[55104] - 24/5.*s.x14*s.x15*csc[55105] +
             3/5.*s.x14*s.x15*csc[55106] + 8/5.*s.x14*s.x15*csc[55107] -
             21/5.*s.x14*s.x15*csc[55108] - 1/5.*s.x14*s.x15*csc[55109] -
             2*s.x14*s.x15*csc[55170] - 9/5.*s.x14*s.x15*csc[55171] -
             1/5.*s.x14*s.x15*csc[55172] + 6/5.*s.x14*s.x15*csc[55173] +
             27/5.*s.x14*s.x15*csc[55174] - 24/5.*s.x14*s.x15*csc[55175] +
             // ...
             ;
  
          sum += prefactor*expr;
      // ...
      return sum;
  }
  </code></pre>
  <p>The object <code>s</code> is relatively small and keeps the needed constants x14, x15, ..., ds0, ..., etc. while <code>ti</code> just returns a double from an external library. As you can see, <code>csc[]</code> is a precomputed map of values which is also evaluated in separate object files (again hundreds with about ~1 MB of size each) of the following form:</p>
  <pre><code>void cscs132(std::tr1::unordered_map&lt;int,double&gt; &amp; csc, ProcessVars &amp; s)
  {
      {
      double csc19295 =       + s.ds0*s.ds1*s.ds2 * ( -
             32*s.x12pow2*s.x15*s.x34*s.mbpow2*s.mWpowinv2 -
             32*s.x12pow2*s.x15*s.x35*s.mbpow2*s.mWpowinv2 -
             32*s.x12pow2*s.x15*s.x35*s.x45*s.mWpowinv2 -
             32*s.x12pow2*s.x25*s.x34*s.mbpow2*s.mWpowinv2 -
             32*s.x12pow2*s.x25*s.x35*s.mbpow2*s.mWpowinv2 -
             32*s.x12pow2*s.x25*s.x35*s.x45*s.mWpowinv2 +
             32*s.x12pow2*s.x34*s.mbpow4*s.mWpowinv2 +
             32*s.x12pow2*s.x34*s.x35*s.mbpow2*s.mWpowinv2 +
             32*s.x12pow2*s.x34*s.x45*s.mbpow2*s.mWpowinv2 +
             32*s.x12pow2*s.x35*s.mbpow4*s.mWpowinv2 +
             32*s.x12pow2*s.x35pow2*s.mbpow2*s.mWpowinv2 +
             32*s.x12pow2*s.x35pow2*s.x45*s.mWpowinv2 +
             64*s.x12pow2*s.x35*s.x45*s.mbpow2*s.mWpowinv2 +
             32*s.x12pow2*s.x35*s.x45pow2*s.mWpowinv2 -
             64*s.x12*s.p1p3*s.x15*s.mbpow4*s.mWpowinv2 +
             64*s.x12*s.p1p3*s.x15pow2*s.mbpow2*s.mWpowinv2 +
             96*s.x12*s.p1p3*s.x15*s.x25*s.mbpow2*s.mWpowinv2 -
             64*s.x12*s.p1p3*s.x15*s.x35*s.mbpow2*s.mWpowinv2 -
             64*s.x12*s.p1p3*s.x15*s.x45*s.mbpow2*s.mWpowinv2 -
             32*s.x12*s.p1p3*s.x25*s.mbpow4*s.mWpowinv2 +
             32*s.x12*s.p1p3*s.x25pow2*s.mbpow2*s.mWpowinv2 -
             32*s.x12*s.p1p3*s.x25*s.x35*s.mbpow2*s.mWpowinv2 -
             32*s.x12*s.p1p3*s.x25*s.x45*s.mbpow2*s.mWpowinv2 -
             32*s.x12*s.p1p3*s.x45*s.mbpow2 +
             64*s.x12*s.x14*s.x15pow2*s.x35*s.mWpowinv2 +
             96*s.x12*s.x14*s.x15*s.x25*s.x35*s.mWpowinv2 +
             32*s.x12*s.x14*s.x15*s.x34*s.mbpow2*s.mWpowinv2 -
             32*s.x12*s.x14*s.x15*s.x35*s.mbpow2*s.mWpowinv2 -
             64*s.x12*s.x14*s.x15*s.x35pow2*s.mWpowinv2 -
             32*s.x12*s.x14*s.x15*s.x35*s.x45*s.mWpowinv2 +
             32*s.x12*s.x14*s.x25pow2*s.x35*s.mWpowinv2 +
             32*s.x12*s.x14*s.x25*s.x34*s.mbpow2*s.mWpowinv2 -
             32*s.x12*s.x14*s.x25*s.x35pow2*s.mWpowinv2 -
             // ...
      
         csc.insert(cscMap::value_type(192953, csc19295));
      }
  
      {
         double csc19296 =      // ... ;
  
         csc.insert(cscMap::value_type(192956, csc19296));
      }
  
      // ...
  }
  </code></pre>
  <p>That's about it. The final step then just consists in calling all those <code>func[i]</code> and summing the result up.</p>
  <p>Concerning the fact that this is a rather special and unusual case: Yes, it is. This is what people have to cope with when trying to do high precision computations for particle physics.</p>
  <p><strong>EDIT2:</strong></p>
  <p>I should also add that x12, x13, etc. are not really constants. They are set to specific values, all those functions are run and the result returned, and then a new set of x12, x13, etc. is chosen to produce the next value. And this has to be done 10<sup>5</sup> to 10<sup>6</sup> times...</p>
  <p><strong>EDIT3:</strong></p>
  <p>Thank you for the suggestions and the discussion so far... I'll try to roll the loops up upon code generation somehow, not sure how to this exactly, to be honest, but this is the best bet.</p>
  <p>BTW, I didn't try to hide behind &quot;this is scientific computing -- no way to optimize&quot;.<br />
  It's just that the basis for this code is something that comes out of a &quot;black box&quot; where I have no real access to and, moreover, the whole thing worked great with simple examples, and I mainly feel overwhelmed with what happens in a real world application...</p>
  <p><strong>EDIT4:</strong></p>
  <p>So, I have managed to reduce the code size of the <code>csc</code> definitions by about one forth by simplifying expressions in a computer algebra system (<a href="http://en.wikipedia.org/wiki/Mathematica" rel="nofollow noreferrer">Mathematica</a>). I see now also some way to reduce it by another order of magnitude or so by applying some other tricks before generating the code (which would bring this part down to about 100 MB) and I hope this idea works.</p>
  <p>Now related to your answers:</p>
  <p>I'm trying to roll the loops back up again in the <code>func</code>s, where a CAS won't help much, but I have already some ideas. For instance, sorting the expressions by the variables like <code>x12, x13,...</code>, parse the <code>csc</code>s with Python and generate tables that relate them to each other. Then I can at least generate these parts as loops. As this seems to be the best solution so far, I mark this as the best answer.</p>
  <p>However, I'd like to also give credit to VJo. GCC 4.6 indeed works <em>much</em> better, produces smaller code and is faster. Using the large model works at the code as-is. So technically this is the correct answer, but changing the whole concept is a much better approach.</p>
  <p>Thank you all for your suggestions and help. If anyone is interested, I'm going to post the final outcome as soon as I am ready.</p>
  <p><strong>REMARKS:</strong></p>
  <p>Just some remarks to some other answers: The code I'm trying to run does not originate in an expansion of simple functions/algorithms and stupid unnecessary unrolling. What actually happens is that the stuff we start with is pretty complicated mathematical objects and bringing them to a numerically <em>computable</em> form generates these expressions. The problem lies actually in the underlying physical theory. Complexity of intermediate expressions scales factorially, which is well known, but when combining all of this stuff to something physically measurable -- an observable -- it just boils down to only a handful of very small functions that form the basis of the expressions. (There is definitely something &quot;wrong&quot; in this respect with the general and <em>only</em> available <a href="http://en.wiktionary.org/wiki/ansatz#Noun" rel="nofollow noreferrer">ansatz</a> which is called &quot;perturbation theory&quot;) We try to bring this ansatz to another level, which is not feasible analytically anymore and where the basis of needed functions is not known. So we try to brute-force it like this. Not the best way, but hopefully one that helps with our understanding of the physics at hand in the end...</p>
  <p><strong>LAST EDIT:</strong></p>
  <p>Thanks to all your suggestions, I've managed to reduce the code size considerably, using Mathematica and a modification of the code generator for the <code>func</code>s somewhat along the lines of the top answer :)</p>
  <p>I have simplified the <code>csc</code> functions with Mathematica, bringing it down to 92 MB. This is the irreducible part. The first attempts took forever, but after some optimizations this now runs through in about 10 minutes on a single CPU.</p>
  <p>The effect on the <code>func</code>s was dramatic: The whole code size for them is down to approximately 9 MB, so the code now totals in the 100 MB range. Now it makes sense to turn optimizations on and the execution is quite fast.</p>
  <p>Again, thank you all for your suggestions, I've learned a lot.</p>