anchor.diff:  [[compressor.install]]
   ==== Configure HBase For Compressors
   
  -Before HBase can use a given compressor, its libraries need to be available.
  -Due to licensing issues, only GZ compression is available to HBase (via native Java libraries) in a default installation.
  -Other compression libraries are available via the shared library bundled with your hadoop.
  -The hadoop native library needs to be findable when HBase starts.
  -See
  +Compression codecs are provided either by HBase compressor modules or by Hadoop's native compression
  +support. As described above you choose a compression type in table or column family schema or in
  +site configuration using its short label, e.g. _snappy_ for Snappy, or _zstd_ for ZStandard. Which
  +codec implementation is dynamically loaded to support what label is configurable by way of site
  +configuration.
  +
  +[options="header"]
  +|===
  +|Algorithm label|Codec implementation configuration key|Default value
  +//----------------------
  +|BROTLI|hbase.io.compress.brotli.codec|org.apache.hadoop.hbase.io.compress.brotli.BrotliCodec
  +|BZIP2|hbase.io.compress.bzip2.codec|org.apache.hadoop.io.compress.BZip2Codec
  +|GZ|hbase.io.compress.gz.codec|org.apache.hadoop.hbase.io.compress.ReusableStreamGzipCodec
  +|LZ4|hbase.io.compress.lz4.codec|org.apache.hadoop.io.compress.Lz4Codec
  +|LZMA|hbase.io.compress.lzma.codec|org.apache.hadoop.hbase.io.compress.xz.LzmaCodec
  +|LZO|hbase.io.compress.lzo.codec|com.hadoop.compression.lzo.LzoCodec
  +|SNAPPY|hbase.io.compress.snappy.codec|org.apache.hadoop.io.compress.SnappyCodec
  +|ZSTD|hbase.io.compress.zstd.codec|org.apache.hadoop.io.compress.ZStandardCodec
  +|===
  +
  +The available codec implementation options are:
  +
  +[options="header"]
  +|===
  +|Label|Codec implementation class|Notes
  +//----------------------
  +|BROTLI|org.apache.hadoop.hbase.io.compress.brotli.BrotliCodec|
  +  Implemented with https://github.com/hyperxpro/Brotli4j[Brotli4j]
  +|BZIP2|org.apache.hadoop.io.compress.BZip2Codec|Hadoop native codec
  +|GZ|org.apache.hadoop.hbase.io.compress.ReusableStreamGzipCodec|
  +  Requires the Hadoop native GZ codec
  +|LZ4|org.apache.hadoop.io.compress.Lz4Codec|Hadoop native codec
  +|LZ4|org.apache.hadoop.hbase.io.compress.aircompressor.Lz4Codec|
  +  Pure Java implementation
  +|LZ4|org.apache.hadoop.hbase.io.compress.lz4.Lz4Codec|
  +  Implemented with https://github.com/lz4/lz4-java[lz4-java]
  +|LZMA|org.apache.hadoop.hbase.io.compress.xz.LzmaCodec|
  +  Implemented with https://tukaani.org/xz/java.html[XZ For Java]
  +|LZO|com.hadoop.compression.lzo.LzoCodec|Hadoop native codec,
  +  requires GPL licensed native dependencies
  +|LZO|org.apache.hadoop.io.compress.LzoCodec|Hadoop native codec,
  +  requires GPL licensed native dependencies
  +|LZO|org.apache.hadoop.hbase.io.compress.aircompressor.LzoCodec|
  +  Pure Java implementation
  +|SNAPPY|org.apache.hadoop.io.compress.SnappyCodec|Hadoop native codec
  +|SNAPPY|org.apache.hadoop.hbase.io.compress.aircompressor.SnappyCodec|
  +  Pure Java implementation
  +|SNAPPY|org.apache.hadoop.hbase.io.compress.xerial.SnappyCodec|
  +  Implemented with https://github.com/xerial/snappy-java[snappy-java]
  +|ZSTD|org.apache.hadoop.io.compress.ZStandardCodec|Hadoop native codec
  +|ZSTD|org.apache.hadoop.hbase.io.compress.aircompressor.ZStdCodec|
  +  Pure Java implementation, limited to a fixed compression level,
  +  not data compatible with the Hadoop zstd codec
  +|ZSTD|org.apache.hadoop.hbase.io.compress.zstd.ZStdCodec|
  +  Implemented with https://github.com/luben/zstd-jni[zstd-jni],
  +  supports all compression levels, supports custom dictionaries
  +|===
  +
  +Specify which codec implementation option you prefer for a given compression algorithm
  +in site configuration, like so:
  +[source]
  +----
  +...
  +<property>
  +  <name>hbase.io.compress.lz4.codec</name>
  +  <value>org.apache.hadoop.hbase.io.compress.lz4.Lz4Codec</value>
  +</property>
  +...
  +----
  +
  +.Compressor Microbenchmarks
  +
  +See https://github.com/apurtell/jmh-compression-tests
  +
  +256MB (258,126,022 bytes exactly) of block data was extracted from two HFiles containing Common
  +Crawl data ingested using IntegrationLoadTestCommonCrawl, 2,680 blocks in total. This data was
  +processed by each new codec implementation as if the block data were being compressed again for
  +write into an HFile, but without writing any data, comparing only the CPU time and resource demand
  +of the codec itself. Absolute performance numbers will vary depending on hardware and software
  +particulars of your deployment. The relative differences are what are interesting. Measured time
  +is the average time in milliseconds required to compress all blocks of the 256MB file. This is
  +how long it would take to write the HFile containing these contents, minus the I/O overhead of
  +block encoding and actual persistence.
  +
  +These are the results:
  +
  +[options="header"]
  +|===
  +|Codec|Level|Time (milliseconds)|Result (bytes)|Improvement
  +//----------------------
  +|AirCompressor LZ4|-|349.989 ± 2.835|76,999,408|70.17%
  +|AirCompressor LZO|-|334.554 ± 3.243|79,369,805|69.25%
  +|AirCompressor Snappy|-|364.153 ± 19.718|80,201,763|68.93%
  +|AirCompressor Zstandard|3 (effective)|1108.267 ± 8.969|55,129,189|78.64%
  +|Brotli|1|593.107 ± 2.376|58,672,319|77.27%
  +|Brotli|3|1345.195 ± 27.327|53,917,438|79.11%
  +|Brotli|6|2812.411 ± 25.372|48,696,441|81.13%
  +|Brotli|10|74615.936 ± 224.854|44,970,710|82.58%
  +|LZ4 (lz4-java)|-|303.045 ± 0.783|76,974,364|70.18%
  +|LZMA|1|6410.428 ± 115.065|49,948,535|80.65%
  +|LZMA|3|8144.620 ± 152.119|49,109,363|80.97%
  +|LZMA|6|43802.576 ± 382.025|46,951,810|81.81%
  +|LZMA|9|49821.979 ± 580.110|46,951,810|81.81%
  +|Snappy (xerial)|-|360.225 ± 2.324|80,749,937|68.72%
  +|Zstd (zstd-jni)|1|654.699 ± 16.839|56,719,994|78.03%
  +|Zstd (zstd-jni)|3|839.160 ± 24.906|54,573,095|78.86%
  +|Zstd (zstd-jni)|5|1594.373 ± 22.384|52,025,485|79.84%
  +|Zstd (zstd-jni)|7|2308.705 ± 24.744|50,651,554|80.38%
  +|Zstd (zstd-jni)|9|3659.677 ± 58.018|50,208,425|80.55%
  +|Zstd (zstd-jni)|12|8705.294 ± 58.080|49,841,446|80.69%
  +|Zstd (zstd-jni)|15|19785.646 ± 278.080|48,499,508|81.21%
  +|Zstd (zstd-jni)|18|47702.097 ± 442.670|48,319,879|81.28%
  +|Zstd (zstd-jni)|22|97799.695 ± 1106.571|48,212,220|81.32%
  +|===
   
   .Compressor Support On the Master
   
  
positive.code: .Changes Take Effect Upon Compaction
  If you change compression or encoding for a ColumnFamily, the changes take effect during compaction.
  
  Some codecs take advantage of capabilities built into Java, such as GZip compression. Others rely on native libraries. Native libraries may be available as part of Hadoop, such as LZ4. In this case, HBase only needs access to the appropriate shared library.
  
  Other codecs, such as Google Snappy, need to be installed first.
  Some codecs are licensed in ways that conflict with HBase's license and cannot be shipped as part of HBase.
  
  This section discusses common codecs that are used and tested with HBase.
  No matter what codec you use, be sure to test that it is installed correctly and is available on all nodes in your cluster.
  Extra operational steps may be necessary to be sure that codecs are available on newly-deployed nodes.
  You can use the <<compression.test,compression.test>> utility to check that a given codec is correctly installed.
  
negative.code: 
  In general, you need to weigh your options between smaller size and faster compression/decompression. Following are some general guidelines, expanded from a discussion at link:https://lists.apache.org/thread.html/481e67a61163efaaf4345510447a9244871a8d428244868345a155ff%401378926618%40%3Cdev.hbase.apache.org%3E[Documenting Guidance on compression and codecs].
  
  * In most cases, enabling LZ4 or Snappy by default is a good choice, because they have a low
    performance overhead and provide reasonable space savings. A fast compression algorithm almost
    always improves overall system performance by trading some increased CPU usage for better I/O
    efficiency.
  * If the values are large (and not pre-compressed, such as images), use a data block compressor.
  * For [firstterm]_cold data_, which is accessed infrequently, depending on your use case, it might
    make sense to opt for Zstandard at its higher compression levels, or LZMA, especially for high
    entropy binary data, or Brotli for data similar in characteristics to web data. Bzip2 might also
    be a reasonable option but Zstandard is very likely to offer superior decompression speed.
  * For [firstterm]_hot data_, which is accessed frequently, you almost certainly want only LZ4,
    Snappy, LZO, or Zstandard at a low compression level. These options will not provide as high of
    a compression ratio but will in trade not unduly impact system performance.
  * If you have long keys (compared to the values) or many columns, use a prefix encoder.
    FAST_DIFF is recommended.
  * If enabling WAL value compression, consider LZ4 or SNAPPY compression, or Zstandard at
    level 1. Reading and writing the WAL is performance critical. That said, the I/O
    savings of these compression options can improve overall system performance.
  
  [[hadoop.native.lib]]
  === Making use of Hadoop Native Libraries in HBase
  