Format raw bytes in hex and ascii in manner like Wireshark does
I think probably the first thing to do is decide whether you're really using C, or really using C++.


If you're really using C, then you need to get rid of the std::string, and allocate space for your return differently (e.g., using malloc).
If you're really using C++, then I'd at least consider using iostreams and manipulators to do most of the work. They are fairly verbose, but most people using C++ expect to deal with iostreams rather than raw buffers and C-style string manipulation.

I think I'd break the code up into a few more functions. For example, I'd probably have format_hex, which would repeatedly call format_line, which might in turn call some format_hex_value (or something on that order) to write out each individual value.
I'd probably avoid using the line placeholder, for a couple of reasons. The first (and probably most important) is that as you've done things right now, the last line is (as I see things) somewhat malformed. You've surrounded the ASCII part of the display with | as delimiters, but for that to make sense, you want the delimiter right next to the data, so that part of the last line should look like: |123456789|, so you immediately know there's no more data after the 9 (where your current display looks like there are space characters after the 9).
I'd definitely expand the addresses on the left out to at least 4 characters, and maybe more. In theory, you should probably be thinking in terms of 16 characters (allowing 64-bit addresses), but in all honesty I've rarely seen much point in hex dump of multiple gigabytes of data (but I certainly have seen good uses for more than 256 bytes).


Putting those together, one possibility (going the C++ route) might come out something on this order:

#include &lt;cctype&gt;
#include &lt;iostream&gt;
#include &lt;iomanip&gt;
#include &lt;sstream&gt;

struct hex {
    unsigned value;
    int digits;
public:
    hex(unsigned value, int digits) : value(value), digits(digits) { }

    friend std::ostream &amp;operator&lt;&lt;(std::ostream &amp;os, hex const &amp;h) {
        auto oldfill = os.fill('0');

        // Yeah, iostreams get really verbose. Sorry.
        os &lt;&lt; std::hex 
           &lt;&lt; std::setw(h.digits) 
           &lt;&lt; std::uppercase 
           &lt;&lt; std::setprecision(h.digits)
           &lt;&lt; h.value;
        os.fill(oldfill);
        return os;
    }
};

void fmt_line(std::ostream &amp;os, char const *data, size_t offset, size_t len) { 
    os &lt;&lt; hex(offset, 8) &lt;&lt; ":";

    for (size_t i=0; i&lt;len; i++) {
        os &lt;&lt; " " &lt;&lt; hex(data[offset+i], 2);
    }

    os &lt;&lt; std::setw((16-len)*3 +2) &lt;&lt; "|";
    for (size_t i=0; i&lt;len; i++) {
        char ch = data[offset+i];
        os &lt;&lt; (std::isprint((unsigned char)ch) ? ch : '.');
    }
    os &lt;&lt; "|\n";
}

std::string format_hex(char const *data, size_t len) { 
    unsigned lines = len/16;
    std::stringstream out;

    for (size_t line=0; line&lt;lines; line++)
        fmt_line(out, data, line*16, 16);
    fmt_line(out, data, lines*16, len%16);
    return out.str();
}

int main() { 
    char input[] = "hello world\02khgavsd \xb \xa\x5Xasjhlasbdas jalsjdn\xd  0123456789";
    std::cout &lt;&lt; format_hex(input, sizeof(input));
}


This could use some further improvement. For one thing, it's probably not immediately obvious what some things like: os &lt;&lt; std::setw((16-len)*3 +2) &lt;&lt; "|"; are really doing, so they could undoubtedly use at least a little more work.