title: Numeric precision for log(1-exp(x))
body: <p>I'm doing some math with really big numbers (I'm using Python, but this question isn't Python specific). For one value, I have a formula that gives me <code>f(t) = Pr(X &lt; t)</code>. I want to use this formula to get <code>Pr(X &gt;= t) = 1 - f(t)</code>. Because <code>f(t)</code> returns values very close to zero, I've been using a log transform and storing <code>log( f(t) )</code> instead of <code>f(t)</code>. My <code>log( f(t) )</code> is on the order of -1e5 or so.</p>
  
  <p>For multiplying, this works quite well. <code>log( f(t) * g ) = log( f(t) ) + log(g)</code>. </p>
  
  <p>But, it's very difficult to compute <code>log( 1 - f(t) )</code> using only <code>log( f(t) )</code>; I could, of course, temporarily exponentiate the value I store and compute <code>log( 1 - exp( log( f(t) ) )</code>, but that will return <code>log( 1 - 0.0 ) = 0.0</code> because <code>log( f(t) )</code> is so close to zero.</p>
  
  <p>You might ask, "Why do you care? If it's close to zero, then 1 minus it is very close to 1." Well, that's a good point you've made. You're a smart cookie. </p>
  
  <p>The trouble is I want to use this to rank values, so I really care if one is <code>log(0.999)</code> and the other is <code>log(0.9999)</code>. You might also ask, "Well, why don't you just rank the <code>log( f(t) )</code> and then reverse the order to get the rankings for <code>log( 1 - f(t) )</code>." Again, I cannot help but point out how great your questions are. It really has been a pleasure talking to you.</p>
  
  <p>But here's the problem: I don't just want to rank by <code>1 - f(t)</code>; I actually want to rank based on <code>Pr(X &gt;= t) * g(t) = (1 - f(t)) g(t)</code>. After taking logs, I get <code>log( 1 - f(t) ) + log( g(t) )</code>; ranking based on <code>f(t)</code> alone won't give the correct answer.</p>
  
  <p>In the past I wrote a little Python function to compute <code>log(a + b)</code> from <code>log(a)</code> and <code>log(b)</code>:</p>
  
  <pre><code>def log_add(logA,logB):
      if logA == log(0):
          return logB
      if logA&lt;logB:
          return log_add(logB,logA)
      return log( 1 + math.exp(logB-logA) ) + logA
  </code></pre>
  
  <p>It helps by first normalizing them so that they're close together and then exponentiating when they're close together.</p>
  
  <p>Unfortunately, I wasn't able to get the same trick to work for my subtraction because there is no normalization factor that will bring <code>log(1)</code> and <code>log( f(t) )</code> close together because they're so far apart. </p>
  
  <p>Does anyone know how to solve this? It seems like such a classic kind of problem; I'd really expect/hope/pray that there is a clever function that operates on the bit level that can give me <code>log(1-x)</code> from <code>log(x)</code>. Also, if you know <em>how</em> it works, I'd really, really love to know. </p>
  
  <p>Cheers!
  Oliver</p>