String compression implementation in C
A couple problems

You don't allocate enough space for the return buffer.  You need to allocate 2*len + 1 bytes to handle the worst cast scenario.  The +1 is for the null terminating byte.

If the count goes above 9, you will output a non-digit character instead of a digit.  If the count goes above 256, the digit will wrap around back to '1' and your compression will have failed to encode the original string.