title: kafka + how to calculate the value of log.retention.byte
body: <p>One of the major role of <strong>log.retention.byte</strong> parameter is to avoid full size of the kafka disk , or in other words purging of data logs in order to avoid kafka disk full </p>
  
  <p>According to the following link: 
  <a href="https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_kafka-component-guide/content/kafka-broker-settings.html" rel="noreferrer">https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_kafka-component-guide/content/kafka-broker-settings.html</a></p>
  
  <p>log.retention.bytes – is The amount of data to retain in the log for each topic partition. By default, log size is unlimited.</p>
  
  <p>We can see also the Note -  that this is the limit for each partition, so multiply this value by the number of partitions to calculate the total data retained for the topic.</p>
  
  <p>In order to understanding it well Let’s give  little example ( hands-on is always much better)</p>
  
  <p>In kafka machine Under /var/kafka/kafka-logs we have the following topic partitions , while Topic name is - lop.avo.prt.prlop</p>
  
  <p>example of topics partitions under /var/kafka/kafka-logs</p>
  
  <pre><code>lop.avo.prt.prlop-1
  lop.avo.prt.prlop-2
  lop.avo.prt.prlop-3
  lop.avo.prt.prlop-4
  lop.avo.prt.prlop-5
  lop.avo.prt.prlop-6
  lop.avo.prt.prlop-7
  lop.avo.prt.prlop-8
  lop.avo.prt.prlop-9
  lop.avo.prt.prlop-10
  </code></pre>
  
  <p>and under each partition we have the following logs ( example )</p>
  
  <pre><code>4.0K    00000000000000023657.index
  268K    00000000000000023657.log
  4.0K    00000000000000023657.timeindex
  4.0K    00000000000000023854.index
  24K     00000000000000023854.log
  4.0K    00000000000000023854.timeindex
  </code></pre>
  
  <p>In the cluster we have 3 kafka machines ( 3 brokers ) 
  About kafka storage – each kafka include disk with size of 100G </p>
  
  <p>let’s say that we want to purge the logs in the topic when disk comes to 70% from the total disk , </p>
  
  <p>so now let’s try to calculate the value of <strong>log.retention.bytes</strong> according to the above info</p>
  
  <p>because we have 10 topic partitions and the we want to limit the total size of the disk to 70G</p>
  
  <p>then my assumption is to do the calculate as the following</p>
  
  <p>each partition will limit to 7G and 7G translating to bytes , so it is  7516192768  bytes</p>
  
  <p>7G X 10 = 70G  ( 70% from the total disk )</p>
  
  <p>So seems that <strong>log.retention.bytes</strong> should set to 7516192768 , in order to limit each partition to 7516192768 bytes</p>
  
  <p>Dose my assumption is logical?</p>
  
  <p>If not then what is the right calculation of - <strong>log.retention.bytes</strong>
  ?  , based on that kafka disk is 100G ,  and we have only 10 topic partitions under /var/kafka/kafka-logs</p>