Ubuntu 14.04 on a desktop  Source Drive: /dev/sda1: 5TB ext4 singledrive volume   Target Volume: /dev/mapper/archive-lvarchive: raid6 (mdadm) 18TB volume with lvmpartition and ext4There are roughly 15 million files to move, and some may be duplicates (I do not want to overwrite duplicates). Command used (from source directory) was: ls -U |xargs -i -t mv -n {} /mnt/archive/targetDir/{}This has been going on for a few days as expected, but I am getting the error in increasing frequency. When it started the target drive was about 70% full, now its about 90%. It used to be about 1/200 of the moves would state and error, now its about 1/5. None of the files are over 100Mb, most are around 100kSome info:$ df -hFilesystem                     Size  Used Avail Use% Mounted on/dev/sdb3                      155G  5.5G  142G   4% /none                           4.0K     0  4.0K   0% /sys/fs/cgroupudev                           3.9G  4.0K  3.9G   1% /devtmpfs                          797M  2.9M  794M   1% /runnone                           5.0M  4.0K  5.0M   1% /run/locknone                           3.9G     0  3.9G   0% /run/shmnone                           100M     0  100M   0% /run/user/dev/sdb1                       19G   78M   18G   1% /boot/dev/mapper/archive-lvarchive   18T   15T  1.8T  90% /mnt/archive/dev/sda1                      4.6T  1.1T  3.3T  25% /mnt/tmp$ df -iFilesystem                       Inodes    IUsed     IFree IUse% Mounted on/dev/sdb3                      10297344   222248  10075096    3% /none                            1019711        4   1019707    1% /sys/fs/cgroupudev                            1016768      500   1016268    1% /devtmpfs                           1019711     1022   1018689    1% /runnone                            1019711        5   1019706    1% /run/locknone                            1019711        1   1019710    1% /run/shmnone                            1019711        2   1019709    1% /run/user/dev/sdb1                       4940000      582   4939418    1% /boot/dev/mapper/archive-lvarchive 289966080 44899541 245066539   16% /mnt/archive/dev/sda1                     152621056  5391544 147229512    4% /mnt/tmpHere's my output:mv -n 747265521.pdf /mnt/archive/targetDir/747265521.pdf mv -n 61078318.pdf /mnt/archive/targetDir/61078318.pdf mv -n 709099107.pdf /mnt/archive/targetDir/709099107.pdf mv -n 75286077.pdf /mnt/archive/targetDir/75286077.pdf mv: cannot create regular file /mnt/archive/targetDir/75286077.pdf: No space left on devicemv -n 796522548.pdf /mnt/archive/targetDir/796522548.pdf mv: cannot create regular file /mnt/archive/targetDir/796522548.pdf: No space left on devicemv -n 685163563.pdf /mnt/archive/targetDir/685163563.pdf mv -n 701433025.pdf /mnt/archive/targetDir/701433025.pdI've found LOTS of postings on this error, but the prognosis doesn't fit. Such issues as your drive is actually full or you've run out of inodes or even your /boot volume is full. Mostly, though, they deal with 3rd party software causing an issue because of how it handles the files, and they are all constant, meaning EVERY move fails.Thanks.EDIT:here is a sample failed and succeeded file:FAILED (still on source drive)ls -lhs 702637545.pdf16K -rw-rw-r-- 1 myUser myUser 16K Jul 24 20:52 702637545.pdfSUCCEEDED (On target volume)ls -lhs /mnt/archive/targetDir/704886680.pdf104K -rw-rw-r-- 1 myUser myUser 103K Jul 25 01:22 /mnt/archive/targetDir/704886680.pdfAlso, while not all files fail, a file which fails will ALWAYS fail. If I retry it over and over it is consistent.EDIT: Some additional commands per request by @mjturner$ ls -ld /mnt/archive/targetDirdrwxrwxr-x 2 myUser myUser 1064583168 Aug 10 05:07 /mnt/archive/targetDir$ tune2fs -l /dev/mapper/archive-lvarchivetune2fs 1.42.10 (18-May-2014)Filesystem volume name:   <none>Last mounted on:          /mnt/archiveFilesystem UUID:          af7e7b38-f12a-498b-b127-0ccd29459376Filesystem magic number:  0xEF53Filesystem revision #:    1 (dynamic)Filesystem features:      has_journal ext_attr dir_index filetype needs_recovery extent 64bit flex_bg sparse_super huge_file uninit_bg dir_nlink extra_isizeFilesystem flags:         signed_directory_hash Default mount options:    user_xattr aclFilesystem state:         cleanErrors behavior:          ContinueFilesystem OS type:       LinuxInode count:              289966080Block count:              4639456256Reserved block count:     231972812Free blocks:              1274786115Free inodes:              256343444First block:              0Block size:               4096Fragment size:            4096Group descriptor size:    64Blocks per group:         32768Fragments per group:      32768Inodes per group:         2048Inode blocks per group:   128RAID stride:              128RAID stripe width:        512Flex block group size:    16Filesystem created:       Thu Jun 25 12:05:12 2015Last mount time:          Mon Aug  3 18:49:29 2015Last write time:          Mon Aug  3 18:49:29 2015Mount count:              8Maximum mount count:      -1Last checked:             Thu Jun 25 12:05:12 2015Check interval:           0 (<none>)Lifetime writes:          24 GBReserved blocks uid:      0 (user root)Reserved blocks gid:      0 (group root)First inode:              11Inode size:           256Required extra isize:     28Desired extra isize:      28Journal inode:            8Default directory hash:   half_md4Directory Hash Seed:      3ea3edc4-7638-45cd-8db8-36ab3669e868Journal backup:           inode blocks$ tune2fs -l /dev/sda1tune2fs 1.42.10 (18-May-2014)Filesystem volume name:   <none>Last mounted on:          /mnt/tmpFilesystem UUID:          10df1bea-64fc-468e-8ea0-10f3a4cb9a79Filesystem magic number:  0xEF53Filesystem revision #:    1 (dynamic)Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isizeFilesystem flags:         signed_directory_hash Default mount options:    user_xattr aclFilesystem state:         cleanErrors behavior:          ContinueFilesystem OS type:       LinuxInode count:              152621056Block count:              1220942336Reserved block count:     61047116Free blocks:              367343926Free inodes:              135953194First block:              0Block size:               4096Fragment size:            4096Reserved GDT blocks:      732Blocks per group:         32768Fragments per group:      32768Inodes per group:         4096Inode blocks per group:   256Flex block group size:    16Filesystem created:       Thu Jul 23 13:54:13 2015Last mount time:          Tue Aug  4 04:35:06 2015Last write time:          Tue Aug  4 04:35:06 2015Mount count:              3Maximum mount count:      -1Last checked:             Thu Jul 23 13:54:13 2015Check interval:           0 (<none>)Lifetime writes:          150 MBReserved blocks uid:      0 (user root)Reserved blocks gid:      0 (group root)First inode:              11Inode size:           256Required extra isize:     28Desired extra isize:      28Journal inode:            8Default directory hash:   half_md4Directory Hash Seed:      a266fec5-bc86-402b-9fa0-61e2ad9b5b50Journal backup:           inode blocks
How to fix intermittant No space left on device errors during mv when device has plenty of space?
linux;ubuntu;command line;ext4
Bug in the implementation of ext4 feature dir_index which you are using on your destination filesystem.Solution : recreate filesytem without dir_index.  Or disable feature using tune2fs (some caution required, see related link Novell SuSE 10/11: Disable H-Tree Indexing on an ext3 Filesystem which although relates to ext3 may need similar caution.(get a really good backup made of the filesystem)(unmount the filesystem)tune2fs -O ^dir_index /dev/fooe2fsck -fDvy /dev/foo(mount the filesystem)ext4: Mysterious No space left on device-errors ext4 has a feature called dir_index enabled by default, which is quite  susceptible to hash-collisions.......ext4 has the possibility to hash the filenames of its contents. This enhances performance, but has a small problem: ext4 does not grow its hashtable, when it starts to fill up. Instead it returns -ENOSPC or no space left on device.