How would I store a reference in a custom FieldType in Drupal 8?
So you're actually asking two questions.


Should I use a field type or a relation?


Both works. Field type requires much less storage (everything is in the field table), but it's hardcoded to one specific use case. The alternative is basically https://www.drupal.org/project/relation or you could also use something like https://www.drupal.org/project/paragraphs, https://www.drupal.org/project/field_collection if you only have a source -> target reference (relation can to n-n references). For all, you'd create a relation/paragraph/collection type with your additional fields. More flexible, can be configured and doesn't require custom code, but each  reference is another entity that has to be stored and loaded.

You will have to decide what's better for your case, depends on performance vs. flexibility requirements, how many variations (you probably don't want to create 10 field types..) and so on.


If a field type, how do I create it.


That's actually fairly easy in Drupal 8. You just extend from the default entity reference field type (EntityReferenceItem), override methods like schema() and propertyDefinitions() to define what properties you have and how they are stored. Then you write a widget and formatter for it, extending from the widget/formatter that's closest to what you want (e.g. autocomplete or select for the reference selection).

The advantage is, all the actual reference things will just work through the parent classes, you just have to implement your additional properties.