Encoding arbitrary data for stream transmission
Assuming the encoding format itself is not subject to review, it's a pretty good implementation. Things that could be improved (small stuff):


Chaining ?: is not necessary here, compare:

n += dchr[i] == 0x02 ? 2 : dchr[i] == 0x03 ? 2 : dchr[i] == 0x25 ? 2 : 1;


to:

n += (dchr[i] == 0x02 || dchr[i] == 0x03 || dchr[i] == 0x25) ? 2 : 1;

Prefer unsigned char* for work on raw binary sequences, char* means a string of text by convention
0x02, 0x03, and 0x25 all have meanings, so better give them meaningful names, e.g.

const unsigned char ENC_START = 0x02, ENC_END = 0x03, ENC_ESCAPE = 0x25;

In the test:

char* in = "1" "\x02" "%" "\x03" "3";
size_t n = encodedata(&amp;out, in, 6);


it's not clear where 6 came from, so I'd opt for:

char in[] = "1" "\x02" "%" "\x03" "3";
size_t n = encodedata(&amp;out, in, sizeof(in));



Moreover, unless the function signature you proposed is a requirement, I would opt for a buffered design that encodes data in chunks on the fly. It's a little more complicated because you would need to remember the state between calls -- how much input data was processed so far (think of fread). For stream transmission, you probably call encodedata, send the result, and throw it away immediately after. So elliding malloc and relieving the caller from encoding the whole stream at once are both welcome here. One of many possible designs may look like this:

struct encodestate
{
    void* in;
    size_t inlen;
    unsigned char out[256];  // use apprioprate chunk length for your stream
    size_t outlen;
};

/* Writes encoded data into state.out, updates state.outlen
   and for internal use updates state.in and state.inlen.

   Returns whether there's more input data to process (should be called again).
*/
bool encode(struct encodestate* state);


Example of use:

unsigned char in[] = "1" "\x02" "%" "\x03" "3";

for (struct encodestate state = {in, sizeof in}; encode(&amp;state); )
    send(somewhere, state.out, state.outlen);