anchor.diff:  ----
   
   [[snappy.compression.installation]]
  -.Install Snappy Support
  -
  -HBase does not ship with Snappy support because of licensing issues.
  -You can install Snappy binaries (for instance, by using +yum install snappy+ on CentOS) or build Snappy from source.
  +.Install Hadoop native Snappy Support
  +
  +Snappy support is bundled with Hadoop and is the default Snappy codec implementation.
  +It is not required that you make use of the Hadoop Snappy codec. Our Snappy codec
  +implemented with Xerial Snappy offers superior performance, and the AirCompressor
  +Snappy codec offers a pure Java option for use where native support is not available.
  +
  +That said, if you prefer the Hadoop codec option, you can install Snappy binaries (for
  +instance, by using +yum install snappy+ on CentOS) or build Snappy from source.
   After installing Snappy, search for the shared library, which will be called _libsnappy.so.X_ where X is a number.
   If you built from source, copy the shared library to a known location on your system, such as _/opt/snappy/lib/_.
   
  
positive.code: 
  In general, you need to weigh your options between smaller size and faster compression/decompression. Following are some general guidelines, expanded from a discussion at link:https://lists.apache.org/thread.html/481e67a61163efaaf4345510447a9244871a8d428244868345a155ff%401378926618%40%3Cdev.hbase.apache.org%3E[Documenting Guidance on compression and codecs].
  
  * If you have long keys (compared to the values) or many columns, use a prefix encoder.
    FAST_DIFF is recommended.
  * If the values are large (and not precompressed, such as images), use a data block compressor.
  * Use GZIP for [firstterm]_cold data_, which is accessed infrequently.
    GZIP compression uses more CPU resources than Snappy or LZO, but provides a higher compression ratio.
  * Use Snappy or LZO for [firstterm]_hot data_, which is accessed frequently.
    Snappy and LZO use fewer CPU resources than GZIP, but do not provide as high of a compression ratio.
  * In most cases, enabling Snappy or LZO by default is a good choice, because they have a low performance overhead and provide space savings.
  * Before Snappy became available by Google in 2011, LZO was the default.
    Snappy has similar qualities as LZO but has been shown to perform better.
  
  [[hadoop.native.lib]]
  === Making use of Hadoop Native Libraries in HBase
  
negative.code: [[compressor.install]]
  ==== Configure HBase For Compressors
  
  Compression codecs are provided either by HBase compressor modules or by Hadoop's native compression
  support. As described above you choose a compression type in table or column family schema or in
  site configuration using its short label, e.g. _snappy_ for Snappy, or _zstd_ for ZStandard. Which
  codec implementation is dynamically loaded to support what label is configurable by way of site
  configuration.
  
  [options="header"]
  |===
  |Algorithm label|Codec implementation configuration key|Default value
  //----------------------
  |BROTLI|hbase.io.compress.brotli.codec|org.apache.hadoop.hbase.io.compress.brotli.BrotliCodec
  |BZIP2|hbase.io.compress.bzip2.codec|org.apache.hadoop.io.compress.BZip2Codec
  |GZ|hbase.io.compress.gz.codec|org.apache.hadoop.hbase.io.compress.ReusableStreamGzipCodec
  |LZ4|hbase.io.compress.lz4.codec|org.apache.hadoop.io.compress.Lz4Codec
  |LZMA|hbase.io.compress.lzma.codec|org.apache.hadoop.hbase.io.compress.xz.LzmaCodec
  |LZO|hbase.io.compress.lzo.codec|com.hadoop.compression.lzo.LzoCodec
  |SNAPPY|hbase.io.compress.snappy.codec|org.apache.hadoop.io.compress.SnappyCodec
  |ZSTD|hbase.io.compress.zstd.codec|org.apache.hadoop.io.compress.ZStandardCodec
  |===
  
  The available codec implementation options are:
  
  [options="header"]
  |===
  |Label|Codec implementation class|Notes
  //----------------------
  |BROTLI|org.apache.hadoop.hbase.io.compress.brotli.BrotliCodec|
    Implemented with https://github.com/hyperxpro/Brotli4j[Brotli4j]
  |BZIP2|org.apache.hadoop.io.compress.BZip2Codec|Hadoop native codec
  |GZ|org.apache.hadoop.hbase.io.compress.ReusableStreamGzipCodec|
    Requires the Hadoop native GZ codec
  |LZ4|org.apache.hadoop.io.compress.Lz4Codec|Hadoop native codec
  |LZ4|org.apache.hadoop.hbase.io.compress.aircompressor.Lz4Codec|
    Pure Java implementation
  |LZ4|org.apache.hadoop.hbase.io.compress.lz4.Lz4Codec|
    Implemented with https://github.com/lz4/lz4-java[lz4-java]
  |LZMA|org.apache.hadoop.hbase.io.compress.xz.LzmaCodec|
    Implemented with https://tukaani.org/xz/java.html[XZ For Java]
  |LZO|com.hadoop.compression.lzo.LzoCodec|Hadoop native codec,
    requires GPL licensed native dependencies
  |LZO|org.apache.hadoop.io.compress.LzoCodec|Hadoop native codec,
    requires GPL licensed native dependencies
  |LZO|org.apache.hadoop.hbase.io.compress.aircompressor.LzoCodec|
    Pure Java implementation
  |SNAPPY|org.apache.hadoop.io.compress.SnappyCodec|Hadoop native codec
  |SNAPPY|org.apache.hadoop.hbase.io.compress.aircompressor.SnappyCodec|
    Pure Java implementation
  |SNAPPY|org.apache.hadoop.hbase.io.compress.xerial.SnappyCodec|
    Implemented with https://github.com/xerial/snappy-java[snappy-java]
  |ZSTD|org.apache.hadoop.io.compress.ZStandardCodec|Hadoop native codec
  |ZSTD|org.apache.hadoop.hbase.io.compress.aircompressor.ZStdCodec|
    Pure Java implementation, limited to a fixed compression level,
    not data compatible with the Hadoop zstd codec
  |ZSTD|org.apache.hadoop.hbase.io.compress.zstd.ZStdCodec|
    Implemented with https://github.com/luben/zstd-jni[zstd-jni],
    supports all compression levels, supports custom dictionaries
  |===
  
  Specify which codec implementation option you prefer for a given compression algorithm
  in site configuration, like so:
  [source]
  ----
  ...
  <property>
    <name>hbase.io.compress.lz4.codec</name>
    <value>org.apache.hadoop.hbase.io.compress.lz4.Lz4Codec</value>
  </property>
  ...
  ----
  
  .Compressor Microbenchmarks
  
  See https://github.com/apurtell/jmh-compression-tests
  
  256MB (258,126,022 bytes exactly) of block data was extracted from two HFiles containing Common
  Crawl data ingested using IntegrationLoadTestCommonCrawl, 2,680 blocks in total. This data was
  processed by each new codec implementation as if the block data were being compressed again for
  write into an HFile, but without writing any data, comparing only the CPU time and resource demand
  of the codec itself. Absolute performance numbers will vary depending on hardware and software
  particulars of your deployment. The relative differences are what are interesting. Measured time
  is the average time in milliseconds required to compress all blocks of the 256MB file. This is
  how long it would take to write the HFile containing these contents, minus the I/O overhead of
  block encoding and actual persistence.
  
  These are the results:
  
  [options="header"]
  |===
  |Codec|Level|Time (milliseconds)|Result (bytes)|Improvement
  //----------------------
  |AirCompressor LZ4|-|349.989 ± 2.835|76,999,408|70.17%
  |AirCompressor LZO|-|334.554 ± 3.243|79,369,805|69.25%
  |AirCompressor Snappy|-|364.153 ± 19.718|80,201,763|68.93%
  |AirCompressor Zstandard|3 (effective)|1108.267 ± 8.969|55,129,189|78.64%
  |Brotli|1|593.107 ± 2.376|58,672,319|77.27%
  |Brotli|3|1345.195 ± 27.327|53,917,438|79.11%
  |Brotli|6|2812.411 ± 25.372|48,696,441|81.13%
  |Brotli|10|74615.936 ± 224.854|44,970,710|82.58%
  |LZ4 (lz4-java)|-|303.045 ± 0.783|76,974,364|70.18%
  |LZMA|1|6410.428 ± 115.065|49,948,535|80.65%
  |LZMA|3|8144.620 ± 152.119|49,109,363|80.97%
  |LZMA|6|43802.576 ± 382.025|46,951,810|81.81%
  |LZMA|9|49821.979 ± 580.110|46,951,810|81.81%
  |Snappy (xerial)|-|360.225 ± 2.324|80,749,937|68.72%
  |Zstd (zstd-jni)|1|654.699 ± 16.839|56,719,994|78.03%
  |Zstd (zstd-jni)|3|839.160 ± 24.906|54,573,095|78.86%
  |Zstd (zstd-jni)|5|1594.373 ± 22.384|52,025,485|79.84%
  |Zstd (zstd-jni)|7|2308.705 ± 24.744|50,651,554|80.38%
  |Zstd (zstd-jni)|9|3659.677 ± 58.018|50,208,425|80.55%
  |Zstd (zstd-jni)|12|8705.294 ± 58.080|49,841,446|80.69%
  |Zstd (zstd-jni)|15|19785.646 ± 278.080|48,499,508|81.21%
  |Zstd (zstd-jni)|18|47702.097 ± 442.670|48,319,879|81.28%
  |Zstd (zstd-jni)|22|97799.695 ± 1106.571|48,212,220|81.32%
  |===
  
  .Compressor Support On the Master
  
  