title: Reading a zst archive in Scala & Spark: native zStandard library not available
body: <p>I'm trying to read a zst-compressed file using Spark on Scala.</p>
  <pre><code> import org.apache.spark.sql._
   import org.apache.spark.sql.types._
   val schema = new StructType()
        .add(&quot;title&quot;, StringType, true)
        .add(&quot;selftext&quot;, StringType, true)
        .add(&quot;score&quot;, LongType, true)
        .add(&quot;created_utc&quot;, LongType, true)
        .add(&quot;subreddit&quot;, StringType, true)
        .add(&quot;author&quot;, StringType, true)
   val df_with_schema = spark.read.schema(schema).json(&quot;/home/user/repos/concepts/abcde/RS_2019-09.zst&quot;)
  
   df_with_schema.take(1)
  </code></pre>
  <p>Unfortunately this produces the following error:</p>
  <blockquote>
  <p>org.apache.spark.SparkException: Job aborted due to stage failure:
  Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0
  in stage 0.0 (TID 0) (192.168.0.101 executor driver):
  java.lang.RuntimeException: native zStandard library not available:
  this version of libhadoop was built without zstd support.</p>
  </blockquote>
  <p>My hadoop checknative looks as follows, but I understand from <a href="https://issues.apache.org/jira/browse/SPARK-34651" rel="nofollow noreferrer">here</a> that Apache Spark has its own ZStandardCodec.</p>
  <blockquote>
  <p>Native library checking:</p>
  <ul>
  <li>hadoop:  true /opt/hadoop/lib/native/libhadoop.so.1.0.0</li>
  <li>zlib:    true /lib/x86_64-linux-gnu/libz.so.1</li>
  <li>zstd  :  true /lib/x86_64-linux-gnu/libzstd.so.1</li>
  <li>snappy:  true /lib/x86_64-linux-gnu/libsnappy.so.1</li>
  <li>lz4:     true revision:10301</li>
  <li>bzip2:   true /lib/x86_64-linux-gnu/libbz2.so.1</li>
  <li>openssl: false EVP_CIPHER_CTX_cleanup</li>
  <li>ISA-L:   false libhadoop was built without ISA-L support</li>
  <li>PMDK:    false The native code was built without PMDK support.</li>
  </ul>
  </blockquote>
  <p>Any ideas are appreciated, thank you!</p>
  <p>UPDATE 1:
  As per this <a href="https://forum.huawei.com/enterprise/en/hive-orc-and-parquet-formats-use-zstd-compression-algorithm-by-default/thread/714175-893?from=latestPostsReplies" rel="nofollow noreferrer">post</a>, I've understood better what the message meant, namely that zstd is not enabled when compiling Hadoop by default, so one of possible solutions would be obviously building it with that flag enabled.</p>