Entity Reference - adding fields to reference?
The Entity Reference module does not support this. You'll want to check out the Relation module. With this module the reference (relation) itself is a fieldable entity.

I do realize you'll have to write an update function to migrate your current references to relations, but aside from creating a custom table as @FriOne suggested, it's the only way to go as far as I know.