title: Spark: save and load machine learning model on s3
body: <p>I would like to save and load machine learning model on s3.</p>
  
  <p>I did it:</p>
  
  <pre><code>val credentials = new ProfileCredentialsProvider()
  val hadoopConf = sc.hadoopConfiguration
  hadoopConf.set("fs.s3.impl", "org.apache.hadoop.fs.s3native.NativeS3FileSystem")
  hadoopConf.set("fs.s3.awsAccessKeyId", credentials.getCredentials.getAWSAccessKeyId)
  hadoopConf.set("fs.s3.awsSecretAccessKey", credentials.getCredentials.getAWSSecretKey)
  
  TrainValidationSplitModel.load(s"s3://model_path")
  </code></pre>
  
  <p>And it's working when I run it in local.</p>
  
  <p>However, when I run it in a cluster I got the following error :</p>
  
  <pre><code>Serialization trace:
  fields (org.apache.spark.sql.types.StructType)
  at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:101)
  at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:518)
  at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628)
  at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:366)
  at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:307)
  at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628)
  at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:312)
  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:324)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)
  
  Caused by: java.lang.IllegalArgumentException: Class is not registered: org.apache.spark.sql.types.StructField[]
  Note: To register this class use: kryo.register(org.apache.spark.sql.types.StructField[].class);
  at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:488)
  at com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:97)
  at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:517)
  at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:76)
  ... 10 more
  </code></pre>
  
  <p>You will probably say: "Easy, you just have to register the class org.apache.spark.sql.types.StructField using kryo.register(SomeClass.class);" </p>
  
  <p>But, after almost fifteen classes registrations. Kryo ask me to register a class which is private (access is retricted to the spark package).</p>
  
  <p>How can I solve this problem ?</p>