how can LSTM output ever be negative with 0 initialization? From the lstm equations, e.g. as they appear on p406 of the Deep Learning Book, it looks to me like initializing with zeros (as is common practice), must always produce a strictly positive output. When $s_0 = 0$, then the $i$th output unit at the first time step can be written as 
$h_i = \sigma(A)tanh((\sigma(B)(\sigma(C))$ where $A$, $B$ and $C$ are linear functions on the current input and the previous output. But isn't then $h_i &gt; 0$ for all real numbers $A$, $B$ and $C$?
Cannot find the equations in your reference, so I take them from wikipedia. At first time step the following equation holds: $$h_{i} = \sigma(A)\tanh((\sigma(B)(\tanh(C))$$
where $A, B$ and $C$ are values calculated from linear functions. So it can be negative if $C$ is so.