Drupal reference performance
There is no performance issue I see with reference field. Reference field holds lot of other things 


A table which keeps relationship between two entities which is exposed to other modules where references are saved as entity ids with no control of user input, whereas for text field you do not have much control unless you put validations and turn into almost what reference field is already doing.
Different contributed module are built around reference fields.
As an example you can expose reference fields as relationship to views module.
Text Field is definitely not a replacement for Reference field.
You would not like to get the performance boost for a reference field on the cost of its functionality.
If you have long list of entities to reference, keep the widget as autocomplete that can help in performance boost a bit.