title: How to do authentication with Kerberos in Jupyter notebook
body: <p>I am using Jupyter Notebook on my Cloudera cluster, need to read data from hdfs. As the cluster is Kerberized so I need to get authenticated with my keytab, I ran the command inside my notebook:</p>
  
  <pre><code>kinit -kt keytab principal
  </code></pre>
  
  <p>no problem and I can see by klist the principal has got valid ticket</p>
  
  <p>However when I attempt to access hdfs data which I am authorized to:</p>
  
  <pre><code>df_load = sparkSession.read.csv('hdfs://cmanagerdev01/user/rxie/data.csv')
  </code></pre>
  
  <p>I received the following error:</p>
  
  <blockquote>
    <p>Py4JJavaError: An error occurred while calling o106.csv. :
    org.apache.hadoop.security.AccessControlException: SIMPLE
    authentication is not enabled.  Available:[TOKEN, KERBEROS]   at
    sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at
    sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
      at
    sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
      at
    org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
      at
    org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
      at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2110)
      at
    org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
      at
    org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
      at
    org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
      at
    org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
      at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426)     at
    org.apache.spark.sql.execution.datasources.DataSource$.org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary(DataSource.scala:714)
      at
    org.apache.spark.sql.execution.datasources.DataSource$$anonfun$15.apply(DataSource.scala:389)
      at
    org.apache.spark.sql.execution.datasources.DataSource$$anonfun$15.apply(DataSource.scala:389)
      at
    scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
      at
    scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
      at scala.collection.immutable.List.foreach(List.scala:381)  at
    scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
      at scala.collection.immutable.List.flatMap(List.scala:344)  at
    org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:388)
      at
    org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:239)
      at
    org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:227)
      at
    org.apache.spark.sql.DataFrameReader.csv(DataFrameReader.scala:596)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)     at
    py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)  at
    py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)    at
    py4j.Gateway.invoke(Gateway.java:282)     at
    py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
      at py4j.commands.CallCommand.execute(CallCommand.java:79)   at
    py4j.GatewayConnection.run(GatewayConnection.java:238)    at
    java.lang.Thread.run(Thread.java:748) Caused by:
    org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException):
    SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]    at
    org.apache.hadoop.ipc.Client.call(Client.java:1475)   at
    org.apache.hadoop.ipc.Client.call(Client.java:1412)   at
    org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
      at com.sun.proxy.$Proxy17.getFileInfo(Unknown Source)   at
    org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)     at
    org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
      at
    org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
      at com.sun.proxy.$Proxy18.getFileInfo(Unknown Source)   at
    org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2108)     ...
    28 more</p>
  </blockquote>