title: What are the pitfalls of using Dill to serialise scikit-learn/statsmodels models?
body: <p>I need to serialise scikit-learn/statsmodels models such that all the dependencies (code + data) are packaged in an artefact and this artefact can be used to initialise the model and make predictions. Using the pickle <code>module</code> is not an option because this will only take care of the data dependency (the code will not be packaged). So, I have been conducting experiments with <a href="https://pypi.python.org/pypi/dill">Dill</a>. To make my question more precise, the following is an example where I build a model and persist it. </p>
  
  <pre><code>from sklearn import datasets
  from sklearn import svm
  from sklearn.preprocessing import Normalizer
  import dill
  
  digits = datasets.load_digits()
  training_data_X = digits.data[:-5]
  training_data_Y = digits.target[:-5]
  test_data_X = digits.data[-5:]
  test_data_Y = digits.target[-5:]
  
  class Model:
      def __init__(self):
          self.normalizer = Normalizer()
          self.clf = svm.SVC(gamma=0.001, C=100.)
      def train(self, training_data_X, training_data_Y):
          normalised_training_data_X = normalizer.fit_transform(training_data_X)
          self.clf.fit(normalised_training_data_X, training_data_Y)
      def predict(self, test_data_X):
          return self.clf.predict(self.normalizer.fit_transform(test_data_X))  
  
  model = Model()
  model.train(training_data_X, training_data_Y)
  print model.predict(test_data_X)
  dill.dump(model, open("my_model.dill", 'w'))
  </code></pre>
  
  <p>Corresponding to this, here is how I initialise the persisted model (in a new session) and make a prediction. Note that this code does not explicitly initialise or have knowledge of the <code>class Model</code>.</p>
  
  <pre><code>import dill
  from sklearn import datasets
  
  digits = datasets.load_digits()
  training_data_X = digits.data[:-5]
  training_data_Y = digits.target[:-5]
  test_data_X = digits.data[-5:]
  test_data_Y = digits.target[-5:]
  
  with open("my_model.dill") as model_file:
      model = dill.load(model_file)
  
  print model.predict(test_data_X)
  </code></pre>
  
  <p>Has anyone used Dill isn this way?. The idea is for a data scientist to extend a <code>ModelWrapper class</code> for each model they implement and then build the infrastructure around this that persists the models, deploy the models as services and manage the entire lifecycle of the model.</p>
  
  <pre><code>class ModelWrapper(object):
      __metaclass__ = abc.ABCMeta
      def __init__(self, model):
          self.model = model
      @abc.abstractmethod
      def predict(self, input):
          return
      def dumps(self):
          return dill.dumps(self)
      def loads(self, model_string):
          self.model = dill.loads(model_string)
  </code></pre>
  
  <p>Other than the security implications (arbitrary code execution) and the requirement that modules like <code>scikit-learn</code> will have to be installed on the machine thats serving the model, are there and any other pitfalls in this approach? Any comments or words of advice would be most helpful.  </p>
  
  <p>I think that <a href="https://www.yhathq.com/">YHat</a> and <a href="https://dato.com/">Dato</a> have taken similar approach but rolled out there own implementations of Dill for similar purposes. </p>