How is the encoding of extraNonce done in the genesis block? You can see here the coinbase of the genesis block

txNew.vin[0].scriptSig = CScript() &lt;&lt; 486604799 &lt;&lt; CBigNum(4) &lt;&lt; vector&lt;unsigned char&gt;((const unsigned char*)pszTimestamp, (const unsigned char*)pszTimestamp + strlen(pszTimestamp));


Broken down into hex it looks like this

Size of nBits as varint

0x04


nBits themselves

0xffff001d


Satoshi's message in HEX

0x5468652054696d65732030332f4a616e2f32303039204368616e63656c6c6f72206f6e206272696e6b206f66207365636f6e64206261696c6f757420666f722062616e6b73


And between the nBits and Satoshi's message we have these three bytes which should be the extraNonce

0x010445


Which correspond to the code CBigNum(4), since 4 is an integer, it's 4 bytes, if this was encoded as a varint, it's value is &lt; 253 as such it would've taken only 1 byte, but as you see there are three bytes.
Upon asking the bitcoin developers on IRC, they told me it's because the BigNum library encodes bits differently.

I started poking around in the Bitcoin source code, specifically the Bignum wrapper, I found many overloaded constructors and operators, but ultimately failed to find how these three bytes are calculated and returned because of my little experience with C++.

However, I believe it's this function

void setulong(unsigned long n)
{
    if (!BN_set_word(this, n))
        throw bignum_error("CBigNum conversion from unsigned long : BN_set_word failed");
}


However it's still not returning these bytes. So my question is, how is the integer 4 which should still be 0x04 in hex encoded to be this sequence of bytes 0x010445(big-endian) so that I can do it in C?
Looks like you're misinterpreting the code.

CBigNum is just a class that stores numbers. CBigNum(4) simply stores the number 4, and CBigNum(SHA256(block)) would store the hash of a block as a number (after all, SHA256 gives us numbers). CBigNum does what it says: it stores a big number. As it has a function that exports the number as big endian, it's useful here, and the Bitcoin source will often use it to concatenate numbers in a data stream.

Let's take a quick look at the genesis block's script, without looking at the code. Please open the relevant wiki page: Script

04FFFF001D0104455468652054696D65732030332F4A616E2F32303039204368616E63656C6C6F72206F6E206272696E6B206F66207365636F6E64206261696C6F757420666F722062616E6B73


Part of this can be interpreted as :

04 -&gt; PUSH 4  bytes (FFFF001D)
01 -&gt; PUSH 1  byte  (04)
45 -&gt; PUSH 69 bytes (The Times 03/Jan/2009 Chancellor on brink of second bailout for banks)