title: YARN compression codecs with Apache Spark
body: <p>I'm running Apache Spark (1.3.0) on YARN. The <code>YARN_CONF_DIR</code> points to the YARN configuration. <code>core-site.xml</code> is loaded from that directory. It includes a setting for <code>io.compression.codecs</code>, and this list includes <code>com.hadoop.compression.lzo.LzoCodec</code>, which is not included in the pre-built Spark.</p>
  
  <p>This causes a <code>ClassNotFoundException</code> on startup. As a workaround we can run with a modified <code>core-site.xml</code> or we could include the required Jar file with <code>--jars</code>.</p>
  
  <p>Neither of these workarounds are very robust. I am not the administrator of the YARN cluster. The cluster administrator can make changes to <code>core-site.xml</code> without my knowing. If I'm using a modified copy, I will miss out on important changes. If I'm using <code>--jars</code>, my Spark application will break when a new codec is added.</p>
  
  <p>Are these codecs stored in a standard place? How could I add that place to my classpath? I'd like my application to survive config changes without anything to do on my part.</p>