Best practices to store Python machine learning models What are the best practices to save, store, and share machine learning models?

In Python, we generally store the binary representation of the model, using pickle or joblib. Models, in my case, can be ~100Mo large. Also, joblib can save one model to multiple files unless you set compress=1 (https://stackoverflow.com/questions/33497314/sklearn-dumping-model-using-joblib-dumps-multiple-files-which-one-is-the-corre).

But then, if you want to control access rights to models, and be able to use models from different machines, what's the best way to store them?

I have a few choices:


Store them as files, and then put them in a repository using Git LFS
Store them in an SQL database as binary files:


For instance in Postgresql https://wiki.postgresql.org/wiki/BinaryFilesInDB
This is also the method recommended by the SQL Server team:


https://docs.microsoft.com/en-us/sql/advanced-analytics/tutorials/walkthrough-build-and-save-the-model
https://microsoft.github.io/sql-ml-tutorials/python/rentalprediction/step/3.html
https://blogs.technet.microsoft.com/dataplatforminsider/2016/10/17/sql-server-as-a-machine-learning-model-management-system


HDFS
You may have a look at nexus or dvc or datmo. 

There was recently a presentation at at meetup in berlin, zalandos AI data engineering meetup.