Var is showing as full to many apps like Nagios, Puppet, and the LVM tools (pvs, vgs, etc)df -h output6.0G  4.3G  1.4G  77% /varvgs output/var/lock/lvm/V_rootvg:aux: open failed: No space left on device  Can't get lock for rootvg  Skipping volume group rootvglsof +L1 shows nothing under var anymore, so I don't think there are unlinked files which have yet to be cleared from the /var filesystem.  I don't understand why 1.4G free on a 6G filesystem is considered full.  I know some space is reserved by the system on each filesystem but that can't be it, it's too much space.  The filesystem is ext3 on Red Hat 5.dumpe2fs 1.41.12 (17-May-2010)Filesystem volume name:   <none>Last mounted on:          <not available>Filesystem UUID:          c8f44510-e8f7-4e2e-950a-1410b069910eFilesystem magic number:  0xEF53Filesystem revision #:    1 (dynamic)Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_fileFilesystem flags:         signed_directory_hash Default mount options:    user_xattr aclFilesystem state:         cleanErrors behavior:          ContinueFilesystem OS type:       LinuxInode count:              393216Block count:              1572864Reserved block count:     78627Free blocks:              1183083Free inodes:              388144First block:              0Block size:               4096Fragment size:            4096Reserved GDT blocks:      63Blocks per group:         32768Fragments per group:      32768Inodes per group:         8192Inode blocks per group:   512Filesystem created:       Mon Apr 29 13:12:02 2013Last mount time:          Wed Oct 23 19:10:44 2013Last write time:          Wed Oct 23 19:10:44 2013Mount count:              6Maximum mount count:      -1Last checked:             Mon Apr 29 13:12:02 2013Check interval:           0 (<none>)Reserved blocks uid:      0 (user root)Reserved blocks gid:      0 (group root)First inode:              11Inode size:           256Required extra isize:     28Desired extra isize:      28Journal inode:            8Default directory hash:   half_md4Directory Hash Seed:      8766dfd5-c802-4bc3-81cc-21869e810656Journal backup:           inode blocksJournal features:         journal_incompat_revokeJournal size:             32MJournal length:           8192Journal sequence:         0x0112568eJournal start:            3334
ext3 Filesystem shows full to most apps, but only 77% full to DF
filesystems;ext3;df
Looking at the comments others have helped you diagnose you're out of inodes. If you need to make a few available so you can get some basic access back to your system then you could delete the following files on a CentOS 5 install, assuming you can live without them.Example$ sudo rm -fr /var/log/*.[1-9]?(.gz)This will remove any of the previously backed up files in /var/log. This should buy you a few dozen inodes to start.Counting inodesusing dfI usually use the command df to determine the number available.$ df -i /Filesystem            Inodes   IUsed   IFree IUse% Mounted on/dev/mapper/VolGroup00-LogVol00                     59932672  807492 59125180    2% /using tune2fsYou can also use tune2fs. With it you'll need to give it the path to the LVM LV mapper.$ tune2fs -l /dev/mapper/VolGroup00-LogVol00 | grep -i inodeFilesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_fileInode count:              59932672Free inodes:              59126861Inodes per group:         32768Inode blocks per group:   1024First inode:              11Inode size:       128Journal inode:            8First orphan inode:       21561629Journal backup:           inode blocksFreed up some inodes, now what?With some breathing room you basically have a couple of options. I would start by trying to quickly get a list together of files that can be targeted for deletion, so you can begin to get more headroom. I'd focus on /tmp and /var some more potential files to be removed.If you have old versions of Java or anything installed under /usr/local or /opt I'd pick on those next.I'd start formulating a list of installed RPMs that can be uninstalledIf you've been using YUM to do updates on this server you can clear out its cache.$ sudo yum clean allLook into adding additional space.