Convert UTF8 string to UTF32 string in C I'm doing some recreational programming in C (after spending some time in C++, but professionally using only PHP/JavaScript).
I wrote a UTF8 to UTF32 converter and just wanted to know if I made some obvious mistakes. (For example, is it a big no-no to malloc inside a function (because of possible memory leaks) – and if yes – how would you rather do it?)
#include &lt;stdlib.h&gt;
#include &lt;stdint.h&gt;

size_t utf8_strlen(uint8_t* text) {
    size_t i = 0;
    size_t num_chars = 0;

    while (text[i] != 0) {
        num_chars++;

        if ((text[i] &amp; 0b10000000) == 0) {
            // 1 byte code point, ASCII
            i += 1;
        }
        else if ((text[i] &amp; 0b11100000) == 0b11000000) {
            // 2 byte code point
            i += 2;
        }
        else if ((text[i] &amp; 0b11110000) == 0b11100000) {
            // 3 byte code point
            i += 3;
        }
        else {
            // 4 byte code point
            i += 4;
        }
    }

    return num_chars;
}

uint32_t* utf8_to_utf32(uint8_t* text) {
    size_t num_chars = utf8_strlen(text);
    uint32_t* c = malloc(sizeof(uint32_t) * num_chars);
    size_t i = 0;

    for (size_t n = 0; n &lt; num_chars; n++) {
        if ((text[i] &amp; 0b10000000) == 0) {
            // 1 byte code point, ASCII
            c[n] = (text[i] &amp; 0b01111111);
            i += 1;
        }
        else if ((text[i] &amp; 0b11100000) == 0b11000000) {
            // 2 byte code point
            c[n] = (text[i] &amp; 0b00011111) &lt;&lt; 6 | (text[i + 1] &amp; 0b00111111);
            i += 2;
        }
        else if ((text[i] &amp; 0b11110000) == 0b11100000) {
            // 3 byte code point
            c[n] = (text[i] &amp; 0b00001111) &lt;&lt; 12 | (text[i + 1] &amp; 0b00111111) &lt;&lt; 6 | (text[i + 2] &amp; 0b00111111);
            i += 3;
        }
        else {
            // 4 byte code point
            c[n] = (text[i] &amp; 0b00000111) &lt;&lt; 18 | (text[i + 1] &amp; 0b00111111) &lt;&lt; 12 | (text[i + 2] &amp; 0b00111111) &lt;&lt; 6 | (text[i + 3] &amp; 0b00111111);
            i += 4;
        }
    }

    return c;
}

Edit:
I've posted an updated version of the code to GitHub for anyone that is interested: https://github.com/s22h/cutils/blob/master/src/unicode.h
There are two main things to talk about: checking the input, and buffer handling (your malloc question).

It's a very bad idea to do things like array[i + 1], unless you're absolutely, 100% sure that this refers to memory that is both allocated and part of the string. If it's not allocated, your program is begging for a crash; if it's not part of the string, it's a vulnerability issue and every script kiddie will be able to read your process' memory.

Checking the input

Unless you're absolutely, 100% sure that your user will be supplying you with a valid, NULL-terminated UTF-8 string, you definitely need to do some checking. Note that when I say "user", I mean the developer who will be using your functions.

When writing APIs, it really helps to write, or at least think about, some usage code first, in order to get an idea of what developers would need and how your code will be used. This will help again later, when discussing your malloc question.

Now, think about your general use cases. The developer gets some input through a socket, file, hardware device, form field, whatever. Sockets, files and hardware are prone to losing some bytes due to many different reasons. Forms are generally safe, but you will come across a library that will leave off bytes of the last UTF-8 character when you give it a too small buffer, or not null-terminate the string when the last character fits exactly in the buffer (some C standard lib functions are guilty of the latter). Malicious users may also try to exploit the form input field.

Out of all these, only the form field will be naturally NULL-terminated (probably). With all the others, you get a stream of data that simply ends at some point. So you're forcing the user to manually add the NULL byte, which could possibly lead to reallocation... you can see it's not a good idea.

The developer, however, knows exactly how many bytes he got. Why not exploit this? Rewrite your utf8_strlen function like this:

size_t utf8_strlen(uint8_t* text, size_t nb_text, size_t* nb_valid);


Do pretty much the same thing you were doing, however:


Do not blindly increment i without testing whether i + NUM_BYTES_IN_NEXT_CHAR &lt;= nb_text
Stop immediately if that fails
Stop immediately at a null byte, even if you've used less than nb_text bytes
Stop when you've used nb_text bytes
You may check that every single byte conforms to the UTF-8 standard.
Depends on how thorough you want to be.  Best to do it anyways.
In short, only increment i when you're sure that you've got enough bytes left for the next character, and all bytes of the next character are valid UTF-8. Don't increment i at the terminating null byte and immediately stop.
In the end, set *nb_valid = i. This allows the user to check how many bytes were valid UTF-8. Note that text + *nb_valid would point to the byte immediately after the last valid UTF-8 character. If the string ended with a null byte, text + *nb_valid would point to it instead.
Be nice, and allow nb_valid to be a NULL pointer if the developer doesn't care about this information.
Return the number of valid characters parsed.


Now that you're sure you're only going through valid UTF-8, your utf8_to_utf32 can remain the same. Just add the needed parameters:

uint32_t* utf8_to_utf32(uint8_t* text, size_t nb_text, size_t* nb_valid) {
    size_t num_chars = utf8_strlen(text, nb_text, nb_valid);
    /* ... rest is the same ... */
}


... but do consider what the previous answers mentioned, especially the portability issue of left-shifts, if you plan to program arduinos or other microcontrollers.

Buffer handling

is the reason C is hard. But again, let's think about the use cases. At first, allocating the buffer for the user may seem very helpful, but actually it's rarely something that the user wants. Besides, it burdens them with freeing the buffer, which sucks.


Let's say I'm reading a file or some socket line by line. I'd actually want to allocate just one buffer, and re-use it for each line (reallocating or doing partial reads if I get a very long line). Thus, I'd like for utf8_to_utf32 to write to a particular buffer that I give it. Much better than having many, many allocations and deallocations.
Let's say I'm updating some user input. In most cases I'd already have a UTF-32 buffer where the old user input is stored, and I want to update it. Again, I'd like for utf8_to_utf32 to write to a particular buffer.
I may wish to concatenate or append text. Again, writing to a particular buffer.


So it seems you don't want to actually allocate the buffer, which is just dandy, since the best memory management is none at all. (Side note: the more experienced a C developer, the less you'll see them messing around with memory and allocations, for example see Casey Muratori [1] [2]
[3], a very experienced C developer).

The function signature changes to:

void utf8_to_utf32(uint8_t* text, size_t nb_text, uint32_t* out_buf, size_t* nb_valid) {
    /* don't allocate c, replace c with out_buf */
}


Great, but how will you ensure that there's enough space in out_buf? Two options, each with its pros and cons.

A. Let the user worry about it

You're already providing a function - utf8_strlen, that will let the developer know how many characters to allocate for the buffer (plus one more for the NULL-terminator). If they don't, then that's their problem. Many standard lib functions operate like this: for example mbstowcs.

B. You worry about it

Add another parameter to the function that tells you the size of out_buf. You should never write more than this number of characters. This is a bit safer, though the programmer can of course still lie. This will make your code a bit more complicated, and will allow for partial conversions, though this usually falls under the YAGNI principle. Again, many standard lib functions operate like this too: for example strncpy, snprintf

Pick your poison and keep it consistent

People are familiar with both styles, so choose whatever you prefer and keep it consistent if you decide to add more functions later.

Now the only question that remains is what to do with the return value, which is currently void. Looking at the standard lib functions, you'll see that they would usually return the number of characters written to out_buf.

Another reasonable thing to do would be to return 1 or 0, based on whether the input UTF-8 string was valid all the way through, or there was some invalid data. I'm suggesting it because the check would be

is_utf8_valid = (nb_text == *nb_valid) || (text[*nb_valid] == '\0');


which is something you don't want to bother the user with getting right.