UTF8 Codepoint decode and length I needed a function that would:


Decode, and return, the first character in an UTF8 encoded strings
Return the length of encoding with the special case that lenght of '\0' must be 0
Perfomance are important


I had no special requirement on what to do with invalid sequences so I opted for the following behaviour:


The first byte of an invalid sequences is considered as a single "character" (e.g. for the sequence "\xFF\x2F" it would return '\xFF' as value and 1 as length).
Overlong encoding are accepted


I wrote the following function:

static uint8_t LEN[] = {1,1,1,1,2,2,3,0};
static uint8_t MSK[] = {0,0,3,4,5,0,0,0};

static int utf8_cp(char *txt, int32_t *ch)
{
  int len = 0;
  int32_t val = 0;
  uint8_t first = (uint8_t)(*txt);

  len = (first &gt; 0) * (1 + ((first &amp; 0xC0) == 0xC0) * LEN[(first &gt;&gt; 3) &amp; 7]);
  val = first &amp; (0xFF &gt;&gt; MSK[len]);

  for (int k=len; k&gt;1; k--) {
    if ((*++txt &amp; 0xC0) != 0x80) {
      val = first;
      len = 1;
      break;
    }
    val = (val &lt;&lt; 6) | (*txt &amp; 0x3F);
  }

  *ch = val;
  return len;
}


So that a code like:

char *t; int l; int32_t c;

t = "aàも";
while(1) {
  l = utf8_cp(t, &amp;c);
  printf("'%s' len:%d cp:0x%05x\n", t, l, c);
  if (*t == 0) break;
  t += l;
} 


Produces:

'aàも' len:1 cp:0x00061
'àも' len:2 cp:0x000e0
'も' len:3 cp:0x03082
'' len:4 cp:0x2b014
'' len:0 cp:0x00000


To make it faster I thought about unrolling the for loop (but I wonder how much could I gain) and introducing, at the beginning, some if to handle ASCII character (but I fear that branching could be more costly that just making a bunch of operation).

I will appreciate any comment you may have and any suggestion for improvement.
Rather than use narrow types, use fastest ones

// uint8_t first = (uint8_t)(*txt);
unsigned first = (uint8_t)(*txt);
// or
uint_fast8_t first = (uint8_t)(*txt);


Rather than lookup a value to shift, look up the shifted value.

// static uint8_t MSK[] = {0,0,3,4,5,0,0,0};
//  val = first &amp; (0xFF &gt;&gt; MSK[len]);

static const uint8_t FF_MSK[] = {0xFF &gt;&gt;0, 0xFF &gt;&gt;0, 0xFF &gt;&gt;3, 
    0xFF &gt;&gt;4, 0xFF &gt;&gt;5, 0xFF &gt;&gt;0, 0xFF &gt;&gt;0, 0xFF &gt;&gt;0};
val = first &amp; FF_MSK[len];


Some modern compilers can make additional optimizations if the pointers are known to not overlap - use restrict and const where applicable.

// int utf8_cp(char *txt, int32_t *ch)
int utf8_cp(const char * restrict txt, int32_t *restrict ch)


Coding the companion function would aid in testing for both functions.

int utf8_cp_encode(int32_t *ch, char *txt);


As code does not detect invalid encoding like surrogates, redundant patterns and values above max_Unicode, I see little value in handling only a subset of invalid sequences.  Either detect them all (maybe in debug mode)  or skip detection.

Suggest doing a 32-byte (or 256-byte) lookup for performance.  Profile to find optimal.

// len = (first &gt; 0) * (1 + ((first &amp; 0xC0) == 0xC0) * LEN[(first &gt;&gt; 3) &amp; 7]);
len = (first &gt; 0) * LEN_32[first &gt;&gt; 3];
// or
len = LEN_256[first];


Could extend the above to do one lookup for both the len and val.