Encoding a string into base64
Bugs

I found a couple of bugs in your program.


Your end-of-string handling is off for input strings smaller than 3 bytes.  This is because your end-of-string check is too late to stop the first byte from being processed.  So for the input "a", your function returns "YA==" instead of "YQ==".  See below for a solution for this problem.
If the input string contains characters with the high bit set, such as what could happen with UTF-8 input, then your function handles that incorrectly.  The culprit is that you use signed chars, and do bit shifting with those signed chars.  Imagine that input[n] is 0x80 here:

        i = (uint8_t)(input[n] &gt;&gt; 2);


What you would like to be calculated for i is 0x20.  But what you will actually calculate for i is 0xe0.  Later you will use 0xe0 as an index into your string and it will overflow the bounds.  You can fix this by moving your cast to the right place:

        i = ((uint8_t)input[n] &gt;&gt; 2);



One pass instead of two

Right now you are building up an array x of indices and then you use a second pass to convert the x array into the base64 string.  It would be quicker and simpler to just build up returnString directly and get rid of the intermediary x array.  For example, instead of:

        case 0:
            i = (uint8_t)(input[n] &gt;&gt; 2);
            x[xlen++] = i;
            break;


You could write:

        case 0:
            i = (uint8_t)(input[n] &gt;&gt; 2);
            returnString[xlen++] = index[i];
            break;


Simplifying the loop termination check

Right now, you have a complicated check that looks like this:

    if ((len - n) == ((padding==2)?2:3) &amp;&amp; padding &gt; 0) {
        n++;
        break;
    }


As mentioned earlier, this check was also insufficient for strings smaller than 3 bytes.  A simpler way of doing this is just to modify len to round down to the nearest multiple of 3 before starting the loop.  This does exactly what you wanted and also handles smaller than 3 byte strings:

    len -= (len % 3);
    for (n = 0; n &lt; len; n++) {
        switch (n % 3) {
            ...
        }
        // Complicated check removed.
    }


Getting rid of the switch

Once you have made len a multiple of three (above), you can actually get rid of the switch statement.  Instead of iterating over every byte of the input, you can now iterate by three bytes at a time and do all three switch cases one after the other.

Confusing bitmask constants

Your code is correct, but I had a hard time trying to figure out what some of your constants meant.  For example:

            o = (uint8_t)((input[n] &amp; 240) &gt;&gt; 4);


would make a lot more sense if you used the hex value like this:

            o = (uint8_t)((input[n] &amp; 0xf0) &gt;&gt; 4);


Now I know you are taking the top 4 bits and shifting them down.

Reusing common code

After the main loop when you handle the two possible padding cases, most of the code for the padding == 2 case is also done in the padding == 1 case.  You can combine some of the common code together so that you don't repeat the code twice.  This is a matter of preference though, so what you have is fine if you find it more readable.

Other things


You should check the return value of malloc.
Your input argument is unmodified so you can make it a const argument.
You don't really need both i and o although if you like them to be separate variables you can.




Putting it all together

Here's what your code would look like after all my suggested changes (and a lot of other tweaks that are stylistic/optional):

char *base64encode(const char *input)
{
    int      len      = strlen(input);
    int      leftover = len % 3;
    char    *ret      = malloc(((len/3) * 4) + ((leftover)?4:0) + 1);
    int      n        = 0;
    int      outlen   = 0;
    uint8_t  i        = 0;
    uint8_t *inp      = (uint8_t *) input;
    const char *index = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
                        "abcdefghijklmnopqrstuvwxyz"
                        "0123456789+/";

    if (ret == NULL)
        return NULL;

    // Convert each 3 bytes of input to 4 bytes of output.
    len -= leftover;
    for (n = 0; n &lt; len; n+=3) {
        i = inp[n] &gt;&gt; 2;
        ret[outlen++] = index[i];

        i  = (inp[n]   &amp; 0x03) &lt;&lt; 4;
        i |= (inp[n+1] &amp; 0xf0) &gt;&gt; 4;
        ret[outlen++] = index[i];

        i  = ((inp[n+1] &amp; 0x0f) &lt;&lt; 2);
        i |= ((inp[n+2] &amp; 0xc0) &gt;&gt; 6);
        ret[outlen++] = index[i];

        i  = (inp[n+2] &amp; 0x3f);
        ret[outlen++] = index[i];
    }

    // Handle leftover 1 or 2 bytes.
    if (leftover) {
        i = (inp[n] &gt;&gt; 2);
        ret[outlen++] = index[i];

        i = (inp[n]   &amp; 0x03) &lt;&lt; 4;
        if (leftover == 2) {
            i |= (inp[n+1] &amp; 0xf0) &gt;&gt; 4;
            ret[outlen++] = index[i];

            i  = ((inp[n+1] &amp; 0x0f) &lt;&lt; 2);
        }
        ret[outlen++] = index[i];
        ret[outlen++] = '=';
        if (leftover == 1)
            ret[outlen++] = '=';
    }
    ret[outlen] = '\0';
    return ret;
}