title: PySpark 1.5 & MSSQL jdbc
body: <p>I am using PySpark on Spark 1.5 on Cloudera YARN, using Python 3.3 on Centos 6 Machines. The SQL Server instance is SQL Server Enterprise 64bit. The SQL Server driver is listed below; sqljdbc4.jar; and I have added to my .bashrc </p>
  
  <pre><code>export SPARK_CLASSPATH="/var/lib/spark/sqljdbc4.jar"
  export PYSPARK_SUBMIT_ARGS="--conf spark.executor.extraClassPath="/var/lib/spark/sqljdbc4.jar" --driver-class-path="/var/lib/spark/sqljdbc4.jar" --jars="/var/lib/spark/sqljdbc4.jar" --master yarn --deploy-mode client"
  </code></pre>
  
  <p>And I can see confirmation when I launch Spark that </p>
  
  <pre><code>SPARK_CLASSPATH was detected (set to '/var/lib/spark/sqljdbc4.jar')
  </code></pre>
  
  <p>I have a dataframe that looks like this schema</p>
  
  <pre><code>root
   |-- daytetime: timestamp (nullable = true)
   |-- ip: string (nullable = true)
   |-- tech: string (nullable = true)
   |-- th: string (nullable = true)
   |-- car: string (nullable = true)
   |-- min_dayte: timestamp (nullable = true)
   |-- max_dayte: timestamp (nullable = true)
  </code></pre>
  
  <p>I have created an empty table already in my MS SQL server called 'dbo.shaping', where the 3 timestamp columns will be datetime2(7) and the others nvarchar(50).</p>
  
  <p>I try to export the dataframe from PySpark using this </p>
  
  <pre><code>properties = {"user": "&lt;username&gt;", "password": "&lt;password&gt;"} 
  
  df.write.format('jdbc').options(url='&lt;IP&gt;:1433/&lt;dbname&gt;', dbtable='dbo.shaping',driver="com.microsoft.sqlserver.jdbc.SQLServerDriver",properties=properties)
  </code></pre>
  
  <p>I get the following traceback error</p>
  
  <pre><code>Py4JError: An error occurred while calling o250.option. Trace:
  py4j.Py4JException: Method option([class java.lang.String, class java.util.HashMap]) does not exist
  at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:333)
  at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:342)
  at py4j.Gateway.invoke(Gateway.java:252)
  at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
  at py4j.commands.CallCommand.execute(CallCommand.java:79)
  at py4j.GatewayConnection.run(GatewayConnection.java:207)
  at java.lang.Thread.run(Thread.java:744)
  </code></pre>
  
  <p>Are my methods at least correct, and perhaps this error is related to writing the specific type of data, ie, I have an issue with the data construct and not my code?</p>