What is the "shared secret" used for in IPSec VPN? Can somebody explain what the "shared secret" and "password" do when opening/creating a VPN tunnel?

In this specific case I setup a VPN to my Fritz!Box and I had to provide a shared secret (which was pregenerated and very long) and a password.

I'd like to understand how each of these two credentials is used in terms of encryption.
Most likely, this 'shared secret' was actually an IKE "preshared key"; it is used to authenticate the two sides (and, for IKEv1, is stirred into the keys).  It actually isn't used as a key (and hence someone learning that key cannot use it to listen in, unless they perform an active Man-in-the-Middle attack).

I suspect the password is the authentication credential to the remote operating system; it doesn't get involved with the encryption at all.

Now, if you're asking "how is this IKE preshared key" used, well, I'll try to outline it for you; the bottom line is that someone with the preshared key cannot listen in (or be able to decrypt previously captured sessions).  They would be able to perform a man-in-the-middle attack; that's because the preshared-key works as authentication data; someone with it can impersonate.

To make this even more complex, there are two different version of IKE (IKEv1 and IKEv2), and they use the preshared key somewhat differently.  Since I don't know which one you're using, I'll list how they both work separately.

For both versions of IKE, the negotiation happens in two phases; the differences (that you care about) occur in the first phase (which generates the IKE SAs; that is an encrypted control channel that the two sides use to coordinate things).

Here's how the first phase of IKEv1 works (assuming that you're using preshared key authentication, and omitting the parts not relevant to the key generation):


The two sides exchange nonces
The two sides perform a Diffie-Hellman exchange
The two sides each take the nonces, the Diffie-Hellman shared secret and the preshared key), and generate a set of IKE keys
They exchange IKE encrypted messages (to verify that both came up with the same IKE keys; if they used different IKE keys, they won't).


For IKEv2, this first phase looks like:


The two sides exchange nonces
The two sides perform a Diffie-Hellman exchange
The two sides each take the nonces, the Diffie-Hellman shared secret, and generate a set of IKE keys
Via IKE encrypted messages, they exchange authentication data.  For preshared key authentication, this is a complex (noninvertable) function of the preshared key, and the keying data.  The idea is that if there was a man-in-the-middle who didn't know the preshared key, the keying data between the actual two sides woudl differ, and the man-in-the-middle wouldn't be able to adjust this authentication tag to account for that).


Now, for both IKEv1 and IKEv2, they perform the second phase; they generate the IPSec SAs, which are the keys (and other data) used to actually encrypt the traffic.  To do this, they exchange SPI values and nonces, possibly do another Diffie-Hellman exchange, and they create the IPSec keys from some IKE keying data, the SPI values (and the Diffie-Hellman shared secret, if a Diffie-Hellman was used.

Now that the two sides have the IPSec SAs established, they can now send and receive encrypted traffic.

And, because a Diffie-Hellman operation was involved, someone listening into the traffic is unable to decrypt anything; even if they know what the "preshared key" was.



From the fritzbox website on "VPN for experts":


  IKE negotiation takes place in two phases. The first phase serves to
  establish an encrypted connection; in the second phase the actual VPN
  tunnel is established - already encrypted. Special authentication
  procedures ensure that no password or key is transmitted in plain text
  during the negotiation. A hacker tracing the establishment of the VPN
  connection using IPSec cannot procure any security-sensitive
  information.