Returning a byte array from a method
I could be wrong, but I haven't seen other designs of serialization than :


Serializer with reflection (customizable through attributes/converter/mapper).

This design is more complicated, in a sense that you have to plan ahead every the possible details of serialization. It is also harder to wire everything up, while leaving room for customization.


  it would simply consist of a large if-else/switch branch


This really shouldn't happen unless you plan to do this : switch(element.GetType().Name). You should use reflection to pull every member or property(pick one), and use Attribute to filter amd format them. The only switch you might need to have is opposibly for the premitive types : string, int, enum, etc.


  it would force me to expose private members as internal/protected/public


You don't have to, as you can obtain information on any member of a class, be protected or private.
Serializable by exposing a serialize method

This design is more straight-forward where each element serializes its own stuffs and delegates the rest to the respective element.




I'm going to assume the document is composed of hierarchical elements. If that is case, copying and merging the bytes array could get very costly, more as we get closer to the root element. Since you mentionned that you're working with a "very specific binary representation" and writing to a "stream", it would be best to write directly on the stream, rather than copying and merging the bytes, and holding them around.

public abstract class Element
{
    // Removed for verbosity ...

    protected abstract void Serialize(BinaryWriter writer);
}

public class ElementX : Element
{
    public string Result { get; set; }
    public Element Data { get; set; }

    protected override void Serialize(BinaryWriter writer)
    {
        const string NewLine = "\r\n";
        const string Opening = "begin-elem-x";
        const string Closing = "end-elem-x";

        writer.Write(Opening);
        writer.Write(NewLine);
        writer.Write(Data);
        writer.Write(NewLine);
        writer.Write(Closing);
        writer.Write(Result);
    }
}

internal static class SerializationExtensions
{
    public static void Write(this BinaryWriter writer, Element element)
    {
        element.Serialize(writer);
    }
}