Can I slim this UTF8-encoding program?
Efficient file I/O

By default, files opened with fopen() are buffered, meaning that not every call to fread() or fwrite() will result in a system call. Instead, the C library has an internal buffer and will try to read and write larger chunks at a time. However, you are still paying for the overhead of a regular function call each time you call fread() and fwrite(). To avoid this, it is best that you read and write in large chunks in your own code as well.

While you could try to read in the whole file at once, or even use technique like mmap() to memory map the file, you can already get very good performance by reading and writing blocks of say 64 kilobytes at a time. This avoids using a lot of memory.
Of course, you have to handle the last block not being exactly 64 kilobytes large, but that is quite easy to deal with.

Furthermore, fread() and fwrite() allow you to specify the size of an element and the number of elements you want to read, this comes in handy to ensure you read in a whole number of 4-byte codepoints.

I would structure your code like so:

uint32_t bufferIn[16384]; // 16384 4-byte code points = 64 kB
char bufferOut[65536];

size_t countIn;

while ((countIn = fread(bufferIn, sizeof *bufferIn, sizeof bufferIn / sizeof *bufferIn, ptr)) &gt; 0) {
    // There are countIn codepoints in the buffer
    for (size_t i = 0; i &lt; countIn; i++) {
         uint32_t codepoint = ...; // Convert bufferIn[i] to native endian here.

         // Write UTF-8 to bufferOut here.
         // If bufferOut is almost full, fwrite() it and start writing to it from the start.
    }
}

// Flush the remaining bytes in bufferOut here.


Don't use floating point math for integer problems

Avoid using floating point math when you are dealing with integers. It is hard to get it right, and converting int to double, doing some math operation, and then converting back again can be quite slow.

There are several ways to get the highest set bit in an integer. If you want a portable one, I recommend using one of the bit twiddling hacks. Sometimes compilers will even recognize such a bit twiddling hack and convert it to a single CPU instruction if possible.