What is the G function of FIPS-186-2? We are trying to follow FIPS 186-2 for creating a  pseudo random number generator for EAP AKA AT_MAC Calculation in SIM Authentication:


Choose a new, secret value for the seed-key, XKEY
In hexadecimal notation let t = 67452301 EFCDAB89 98BADCFE 10325476 C3D2E1F0. This is the initial value for H0|H1|H2|H3|H4 in the FIPS SHS [SHA-1]
Code:

For j = 0 to m - 1 do
    XSEED_j = 0 /* no optional user input */
    For i = 0 to 1 do
        XVAL = (XKEY + XSEED_j) mod 2^b
        w_i = G(t, XVAL)
        XKEY = (1 + XKEY + w_i) mod 2^b
        x_j = w_0|w_1



The algorithm requires the use of function G(t,c). We have gone through several SPECS but not found the correct definition of how to implement G(t,c).

At one place we have seen that G(t,c) is the same as SHA-1 except for some padding. But, it is not very clear. What is the G function in Implementation of FIPS-186-2 for EAP AKA AT_MAC Calculation in SIM Authentication?
The RNG used (arguably misused) as a PRF for key derivation in EAP-AKA and also EAP-SIM was defined as a small almost incidental part of FIPS 186-2 (2000) plus change notice 1 (2001). The URL in the 2006 RFCs http://csrc.nist.gov/publications/fips/fips186-2/fips186-2-change1.pdf is no longer valid, because 186-2 has been superseded by 186-3 (2009) and 186-4 (2013) and those newer versions no longer define the RNGs for digital signatures; instead they refer to a separate publication, first SP800-90 and now SP800-90A, whose focus is random generators.

However, 186-2 is still available in the CSRC archive at https://csrc.nist.gov/publications/fips/archive/fips186-2/fips186-2-change1.pdf . The original definition of two RNGs, with two applications that both involve generating numbers less than a number called 'q' (the subgroup order for the Digital Signature Algorithm DSA), is in Appendix 3 starting on page 16 (PDF page 19):


  The algorithms employ a one-way function G(t,c), where t is 160 bits, c is b bits (160 ≤ b ≤ 512) and G(t,c) is 160 bits. One way to construct G is via [a modification of] the Secure Hash Algorithm (SHA-1), as defined in the Secure Hash Standard (SHS). The 160-bit message digest output of the SHA-1 algorithm when message M is input is denoted by SHA-1(M). A second method for constructing G is to use the Data Encryption Standard (DES). The construction of G by these techniques is discussed in sections 3.3 and 3.4 of this appendix.
  
  3.1. ALGORITHM FOR [generating x0..m-1]
  Step 1. Choose a new, secret value for the seed-key, XKEY.
  Step 2. In hexadecimal notation let
  &nbsp;&nbsp;&nbsp;&nbsp;t = 67452301 EFCDAB89 98BADCFE 10325476 C3D2E1F0
  &nbsp;&nbsp;This is the initial value for H0 || H1 || H2 || H3 || H4 in the SHS.
  Step 3. For j = 0 to m - 1 do
  &nbsp;&nbsp;a. XSEEDj = optional user input.
  &nbsp;&nbsp;b. XVAL = (XKEY + XSEEDj) mod 2b.
  &nbsp;&nbsp;c. xj = G(t,XVAL) mod q.
  &nbsp;&nbsp;d. XKEY = (1 + XKEY + xj) mod 2b.


However, the change notice at PDF page 71 first alters section 3.1 to:


  Step 1. Choose a new, secret value for the seed-key, XKEY .
  Step 2. In hexadecimal notation let
  &nbsp;&nbsp;&nbsp;&nbsp;t = 67452301 EFCDAB89 98BADCFE 10325476 C3D2E1F0.
  &nbsp;&nbsp;This is the initial value for H0 || H1 || H2 || H3 || H4 in the SHS [FIPS 180-1].
  Step 3. For j = 0 to m - 1 do
  &nbsp;&nbsp;3.1 XSEED j = optional user input
  &nbsp;&nbsp;3.2 For i = 0 to 1 do
  &nbsp;&nbsp;&nbsp;&nbsp;a. XVAL = (XKEY + XSEED j) mod 2b
  &nbsp;&nbsp;&nbsp;&nbsp;b. wi = G(t, XVAL ).
  &nbsp;&nbsp;&nbsp;&nbsp;c. XKEY = (1 + XKEY + wi) mod 2b .
  &nbsp;&nbsp;3.3 xj = (w0 || w1) mod q


and then adds that this algorithm can be used for 'General Purpose Random Number Generation' by removing the 'mod q' term from the computation of xj. 

You can see this is now the algorithm you quote from Appendix A of rfc4187 (and Appendix B of rfc4186) except those set XSEEDj to zero (as stated in section 7 of each RFC) and change the notation because RFCs are ASCII-text (| for concatenation, ^ for superscript/exponentiation, and _ for subscript).

You want the G function based on SHA-1 specified in 3.3:


  G(t,c) may be constructed using steps (a) - (e) in section 7 of the Specifications for the Secure Hash Standard. Before executing these steps, {Hj} and M1 must be initialized as follows:  
  
  i. Initialize the {Hj} by dividing the 160 bit value t into five 32-bit segments as follows:
  &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;t = t0 || t1 || t2 || t3 || t4
  &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Then Hj = tj for j = 0 through 4.  
  
  ii. There will be only one message block, M1, which is initialized as follows: M1 = c || 0512-b (The first b bits of M1 contain c, and the remaining (512-b) bits are set to zero). Then steps (a) through (e) of section 7 are executed, and G(t,c) is the 160 bit string represented by the five words:
  &nbsp;&nbsp;&nbsp;&nbsp;H0 || H1 || H2 || H3 || H4 at the end of step (e).  


Now there's a snag. FIPS186-2, and rfc4187 and rfc4186, refer to an earlier version of the Secure Hash Standard, FIPS180-1 (1995), which has also been superseded, by 180-2 (2002,2004) 180-3 (2008) and 180-4 (2011, 2015), and as far as I can tell is NOT officially archived. But Google finds http://securityv.isu.edu/isl/fip180-1.html whose section 7 a-e are the per-block 'compression function' of SHA-1, matching the substance of 6.1.2 in newer standards though different in notation. The same logic is duplicated, perhaps a little more readably, in rfc3174 section 6.1.

The EAP RFCs specify the initial value of H (now called H(0)) is 't' which is the standard initial value for SHA-1 (section 6 in 180-1, 5.3.1 in newer versions), so this amounts to the same thing as SHA-1 for a single-block message EXCEPT WITH DIFFERENT PADDING than defined by 180.

And there's the rub. Implementations of SHA-1 you find in modern systems are almost always designed to do the standard padding, not the zero-padding you need. Unless you can get at the internals -- which will depend on exactly what implementation(s) or platform(s) you use (and that part probably does belong on Stackoverflow where the question was originally posted) -- you'll have to duplicate the code for SHA-1, with this (tiny) modification. Fortunately SHA-1 is used for so many things, and very many of them open source, this shouldn't be too hard; for one possibility, rfc4634 (which updates 3174) has C code under a license that (IMOIANAL) is equivalent to CC-BY.