where does the prime number taken in DH algorithm in IPSEC I am studying &amp; configuring IPSEC ikev1 and in between i am analysing the wireshark captures. I am using the linux kernel for TCP/IP stack and user-space i took ipsec-tools.

In the first two packet exchanges initiator is offering the proposals &amp; responder will respond to the supported proposal.

Next two packet exchanges(packet 3[initiator] &amp; 4[responder]) will perform DH key exchange.Here how the responder will came to know that which prime number the initiator uses &amp; how it will be exchanged.
It's in the SA payload of the initial packets.

It's attribute type 4 within the SA payload; in Main Mode of IKE (which is what you're looking at) the initiator will propose a list of groups, and the responder will pick one and send it back its selection in packet 2.

Now, what they exchange is a short integer value; both sides are expected to know the translation between the short values (current standard values are between 1 and 30) and the actual DH (or ECDH) group.  See this page for pointers to the definitions of the current standard groups.