Cracking the code interview: Problem 1 .6 basic string compression
A call to strlen(compress) is unnecessary. You already know the length of the compressed string; it is j.
You may (in fact, you should) terminate the loop as soon as j reaches the length of the original. Once it happened, the compressed string will for sure be longer than original.
nc + '0' assumes that nc is a single digit. The problem statement doesn't say anything about that.
The size of buffer[20] is pretty arbitrary. Since the function provides guarantees about the size of the compressed string, consider allocating strlen(original) bytes for compressed.