title: What's the delimiter equivalent of ^G when reading a CSV with spark?
body: <p>So, I really need help with a stupid thing but apparently I can't manage do it by myself.</p>
  
  <p>I have a set of rows in a file with this format (reading with <code>less</code> on OSX):</p>
  
  <pre><code>XXXXXXXX^GT^XXXXXXXX^G\N^G0^GDL^G\N^G2018-09-14 13:57:00.0^G2018-09-16 00:00:00.0^GCompleted^G\N^G\N^G1^G2018-09-16 21:41:02.267^G1^G2018-09-16 21:41:02.267^GXXXXXXX^G\N
  YYYYYYYY^GS^XXXXXXXX^G\N^G0^GDL^G\N^G2018-08-29 00:00:00.0^G2018-08-29 23:00:00.0^GCompleted^G\N^G\N^G1^G2018-09-16 21:41:03.797^G1^G2018-09-16 21:41:03.81^GXXXXXXX^G\N
  </code></pre>
  
  <p>So the delimiter is the <code>BEL</code> delimiter and I'm loading the CSV this way:</p>
  
  <pre><code>val df = sqlContext.read.format("csv")
    .option("header", "false")
    .option("inferSchema", "true")
    .option("delimiter", "\u2407")
    .option("nullValue", "\\N")
    .load("part0000")
  </code></pre>
  
  <p>But when I read it it just reads lines as just one column this way:</p>
  
  <pre><code>XXXXXXXXCXXXXXXXX\N0DL\N2018-09-15 00:00:00.02018-09-16 00:00:00.0Completed\N\N12018-09-16 21:41:03.25712018-09-16 21:41:03.263XXXXXXXX\N
  XXXXXXXXSXXXXXXXX\N0DL\N2018-09-15 00:00:00.02018-09-15 23:00:00.0Completed\N\N12018-09-16 21:41:03.3712018-09-16 21:41:03.373XXXXXXXX\N
  </code></pre>
  
  <p>It seems there is a <code>unkown character</code> (you see nothing only because I formatted it here on stackoverflow) in place of <code>^G</code>.</p>
  
  <p>UPDATE:
  could it be a limitation on spark for scala?
  If I run the code with scala this way:</p>
  
  <pre><code>val df = sqlContext.read.format("csv")
    .option("header", "false")
    .option("inferSchema", "true")
    .option("delimiter", "\\a")
    .option("nullValue", "\\N")
    .load("part-m-00000")
  
  display(df)
  </code></pre>
  
  <p>I get a big fat</p>
  
  <pre><code>java.lang.IllegalArgumentException: Unsupported special character for delimiter: \a
  </code></pre>
  
  <p>whereas if I run with python:</p>
  
  <pre><code>df = sqlContext.read.format('csv').options(header='false', inferSchema='true', delimiter = "\a", nullValue = '\\N').load('part-m-00000')
  
  display(df)
  </code></pre>
  
  <p>everything is fine!</p>