Understanding how SHA-512 achieves its design goals
SHA-2, like SHA-1, is an ARX hash function: that is, it uses Addition, Rotation, and eXclusive-or for bit diffusion.  The purpose of each one is explained very simply and clearly by Khovratovich &amp; Nikolić in their paper Rotational Cryptanalysis of ARX, so I will simply quote them here:


  Addition provides diffusion and nonlinearity, while XOR does not.  Although the diffusion is relatively slow, it is compensated by a low price of addition in both software and hardware, so primitives with a relatively high number of additions (tens per byte) are still fast.  The intraword rotation removes disbalance between left and right bits (introduced by addition) and speeds up the diffusion.


So there is no "chaotic behavior interacting", but there is bit-diffusion, and that is what you want in a hash.  Sufficient bit-diffusion is called an "avalanche effect".  The message expansion mixes the bits between words, and the actual compression function is where the nonlinearity comes in, using these pre-mixed bits.

As to the second part of your question, $\operatorname{ch}(a,b,c)$ is the "choice" function: when $a$ is $\text{true}$, the result is equivalent to $b$, and when $a$ is $\text{false}$, the result is equivalent to $c$.  So $a$ acts as a selector between either $b$ or $c$.  $\operatorname{maj}(a,b,c)$ is the "majority" function, which takes the majority value as the final result.  So if two or three of the variables are $1$, then the result is $1$; otherwise, the result is $0$.  And usually there is also the $\operatorname{par}$ function, which is "parity" and uses XOR between the operands.

The truth density of each of these functions is 50%.  In other words, given completely random values for $a$, $b$, and $c$, you have a 50% chance of getting a $1$ or $0$ result.