Serializing object graph to proprietary binary file
BinaryFormatter is not adding any extra information. It does not store "here comes a double", this must be stored/handled by the reading algorithm.

But to reproduce a certain byte order, you have to overwrite almost every serializing routine. At the end you gain nothing, comparing to your self-implementation. So it's not the extra-information, it's the order of elements, that might cause trouble when using the given BinaryFormatter.

Each class can be responsible for it's own serialization, if you have only one serialization. If you can always modify your own code, the classes and the serialization never go separate ways to separate libraries, then this might be a good way.

A more sophisticated approach would be a Visitor-pattern. This way you can implement multiple Serializers (Visitors), store in different formats, etc.

One way you have everything concerning a class in the class, the other way you have everything concerning one specific file format in one class. This you have to make up with yourself, it depends on how often you plan to modify or extent anything later on.

Serializing is a streaming process. You create only one Stream ( a File or MemoryStream), don't use one per item, and assemble it later on. You create one Formatter/Writer, and pass this to each item. Each item is storing it's stuff into the output stream. So an item with one property to store, should have one line of code in it's Serializer method.