Support integer or float values as an attribute on a Drupal Commerce "Add to Cart" form
I ran into the same problem (and asked a similar question which I ended up answering myself  here)

Basically with the knowledge you gave I was able to implement a new module which installs a widget for the "list integer" and "list float" types that uses a textfield for input and on POST it sanitizes and adds the value of the textbox to the "allowed_values" setting on the field allowing it to pass the standard list field validation and use the standard list field storage. 

This allows you to enter and display a "number" and use the value as a product attribute.  Best part is I did not have to alter another modules code, which addresses the majority of your concerns as:


  
  The core "options" module not playing nicely with float/int fields.
  


The options module defines the widget, since you are doing this, you don't need to worry about it.


  
  Other modules assuming the field is a list of data?
  


Since you defined your own widget and are not using the standard "options", other modules will not know it exists so you don't have to worry.


  
  The weirdness around implementing another modules hook.
  


Again, by adding the sanitized field into the allowed_values setting on POST you pass validation and this allows you to use the "LIST" modules standard implementation of hook_options_list and you don't have to modify someone else's code

As far as your first point on too much data, this is possible that you will have a problem here if you have a multitude of products OR if you are added and removing products all the time (which would continuously add to your "allowed_values" list). For the first point, you could probably also implement a field type in your module and do your own implementation of hook_options_list to follow your logic and just return all the values in your database (i don't have that much so didn't bother).  For the second point, you could implement hook_cron and clean up on the allowed values setting based on the values for the field in the database. 

hope this helps and if you have any questions let me know!