What is a good limit to use on the quantity of files in a directory, and why?EDIT:Why shouldn't someone create a system that puts hundreds of thousands of files in the same directory?Why I ask:Someone set up a system that dumps files in a folder with a human readable date-time.My task it to create a system that gets the files for a select time period.Normally this wouldn't be a problem but the folder has 500,000 files in it and is growing and my system is expected to get them in real time.Parsing 500,000 files takes too long, so I think it is the responsibility of the person who built the system that inputs the files on the FTP server to create a directory structure such as having a sub folder for each day.
file quantity limit in a directory on a linux file server and why?
parsing;file structure;file systems
It depends upon the actual system (operating system, computer hardware, file system). Some [old Linux] file systems behaved badly (linear access time w.r.t. to number of directory entries). So it is generally preferable to have small sized directories of a few thousand entries at most (and this also makes the shell more happy: you might want to ls the directory). It is usually inefficient to have millions of very small files (e.g. a hundred bytes each), because often each file uses at least one file-system block.So I would suggest having files like dir001/file001.txt .... dir123/file345.txt ....Consider also using some other way of storing data: some indexed data file like GDBM, a Sqlite or PostGresql or MongoDb database, etc. And you might even mix the approach: use Sqlite for some meta-data about your files, and keep them in many directories. You might also have a seggregative approach: handle differently small contents and bigger ones (put the small contents in Sqlite or GDBM, and the large contents in files).