where does the prime number taken in DH algorithm in IPSEC
It's in the SA payload of the initial packets.

It's attribute type 4 within the SA payload; in Main Mode of IKE (which is what you're looking at) the initiator will propose a list of groups, and the responder will pick one and send it back its selection in packet 2.

Now, what they exchange is a short integer value; both sides are expected to know the translation between the short values (current standard values are between 1 and 30) and the actual DH (or ECDH) group.  See this page for pointers to the definitions of the current standard groups.