Converting bytes to an escaped hexadecimal string
Use of assert

Some of the uses of assert here look semi-broken, at least to me. Part of the definition of assert is that when you compile with NDEBUG defined, assert becomes a nop. When it is enabled, a failed assertion immediately aborts the program. For testing inputs to a function, neither of those is generally desirable. For such purposes, I usually define an assure that always does its thing, and throws an exception when the test fails.

snprintf

At the very least, I'd try to wrap the ugliness of using snprintf up into a reasonably neat little function like to_hex that just converts a number into its hex representation:

std::string to_hex(int in) { 
    // snprintf ugliness here
    return std::string(buffer);
}


Alternatively, since you're converting inputs to strings, and accumulating those together into a buffer anyway, consider using an std::ostringstream.

Logic

In a few places, the logic seems...less than pleasing, at least to me. For example:

if (column &gt;= maxCols)


You've previously tested that maxCols % 4 == 0, and (assuming things work correctly) each item you add should be 4 more characters, so you should never see a result that's actually greater than maxCols--only less than maxCols, then equal to maxCols, then back to 0. This is harmless, and if you might want to support a maxCols that was not a multiple of 4, then it might even make good sense--but as things stand right now, it seems like there's a degree of...uncertainty about the real intent.

Reorganization

Right now you have code to ensure the maximum line length sprinkled liberally throughout the rest of the code. I'd be tempted to define a type that takes that as its single responsibility: when you create it, specify a maximum length. When you add characters to it, it tracks the length of the current "line", and when the maximum is reached (or exceeded) it adds a delimiter, and starts over. With this, the rest of the code gets a lot simpler in a hurry, and each piece of code has a much simpler, clearer purpose.