How is the SPI determined/negotiated in phase 2 of IKE (v1)? While I am aware that the SPI (along with destination IP and protocol) is used to uniquely identify an incoming SA on a peer, for the life of me I am not able to understand how that particular SPI value is determined and agreed upon by both peers during IKE phase 2 message exchanges, for both the incoming and outgoing SA that are established on each of them. I have come across sources stating the below:


  The SPI is a 32-bit number that is chosen by the initiator to uniquely identify the outgoing IPSec SA that is generated as a
  result of this negotiation in its database of security associations.
  The responder, upon seeing the SPI, makes sure that it is not the
  same as one of the SPIs it is using and starts using it for its
  incoming IPSec SA. It also proposes an SPI for its outgoing (and the initiator's incoming) SA, which the initiator agrees to after
  checking.


The above scheme seems more like a trial and error than a negotiation. Logically, I would expect the responder to chose a value for the SPI that is not currently in use for its incoming SA, and send that over to the initiator to record in its SAD and encode the same in all future AH/ESP payloads it sends to the responder over this SA. My questions are as below:


Is the above description actually how it takes place?
If yes, what happens in case the responder receiving the SPI from the initiator finds out that the SPI value is already in use?
Is the above description actually how it takes place?


No; it works as you suspect.  Each side chooses the SPI to use for the incoming SA; that is, how the other side marks encrypted packets that are encrypted with this SA; in the initial quickmode message, the initiator specifies the SPI that will be used for encrypted packets going to the initiator, in the response packet, the responder specifies the SPI that will be used for encrypted packets going to the responder.  This is done specifically so that, in either case, the decryptor can select SPI values that are convenient to look up; for example, it may chose to place an index into a 'decryption table' into the lower 16 bits, say (and yes, there are implementations that do that).


  If yes, what happens in case the responder receiving the SPI from the initiator finds out that the SPI value is already in use?


While (as above) the assumption behind this question is wrong, there is a nontrivial answer here, and so I'll answer that.

As above, the initiator selects its SPI first (that is, the SPI for its incoming packets), and the responder then selects its SPI.

Because of how IKEv1 derives keys, if the responder selects the same SPI as the initiator, the keys for the two SAs (going in the opposite directions) will be the same; depending on what transforms are negotiated, this can be disastrous.  I don't believe there's any requirements, but I know of at least one implementation that will, as a responder, always select a different SPI then the initiator, and as an initiator, if it sees the responder selected the same SPI, automatically close the connection without encrypting anything.