title: Hadoop No space left on device erro when there is space available
body: <p>I have 5 Linux machines cluster. There are 3 data nodes and one master. At now about 50% hdfs storage is available on each data nodes. But I run a mapreduce job, It is failed with following error</p>
  
  <pre><code>2017-08-21 17:58:47,627 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for blk_6835454799524976171_3615612 bad datanode[0] 10.11.1.42:50010
  2017-08-21 17:58:47,628 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_6835454799524976171_3615612 in pipeline 10.11.1.42:50010, 10.11.1.43:50010: bad datanode 10.11.1.42:50010
  2017-08-21 17:58:51,785 ERROR org.apache.hadoop.mapred.Child: Error in syncLogs: java.io.IOException: No space left on device
  </code></pre>
  
  <p>While on each system <code>df -h</code> gives following information</p>
  
  <pre><code>Filesystem               Size  Used Avail Use% Mounted on
  devtmpfs                 5.9G     0  5.9G   0% /dev
  tmpfs                    5.9G   84K  5.9G   1% /dev/shm
  tmpfs                    5.9G  9.1M  5.9G   1% /run
  tmpfs                    5.9G     0  5.9G   0% /sys/fs/cgroup
  /dev/mapper/centos-root   50G  6.8G   44G  14% /
  /dev/sdb                 1.8T  535G  1.2T  31% /mnt/11fd6fcc-1f87-4f1e-a53c-54cc7117759c
  /dev/mapper/centos-home  412G  155G  59M  100% /home
  /dev/sda1                494M  348M  147M  71% /boot
  tmpfs                    1.2G   16K  1.2G   1% /run/user/42
  tmpfs                    1.2G     0  1.2G   0% /run/user/1000
  </code></pre>
  
  <p>As clear from above that my sdb dicsk (SDD) is only 31% used but centos-home is 100%. While hadoop is using local file system in mapreduce job when there is enough HDFS available? Where is the problem? I have search at google and found many such problem but no one covers my situation.</p>