Additional field data stored outside of the field storage (in a separate table) doesn't load with the rest of the field
hook_field_load is being called, but only once after a cache clear. Either the loaded field or the whole node is cached somewhere. Remember to clear your caches!

I missed the initial load because I had yet to instrument the function with the typical bevy of dpm() and ddebug_backtrace() calls. Also the function itself isn't loading the data back properly, but that's outside the scope of the original question.