title: Efficiently listing files in a directory having very many entries
body: <p>I need to recursively read a directory structure, but I also need to perform an additional step once I have read through all entries for each directory. Therefore, I need to write my own recursion logic (and can't use the simplistic <code>filepath.Walk</code> routine). However, the <code>ioutil.ReadDir</code> and <code>filepath.Glob</code> routines only return slices. What if I'm pushing the limits of <em>ext4</em> or <em>xfs</em> and have a directory with files numbering into the billions? I would expect <em>golang</em> to have a function that returns an unsorted series of <code>os.FileInfo</code> (or, even better, raw strings) over a channel rather than a sorted slice. How do we efficiently read file entries in this case?</p>
  
  <p>All of the functions cited above seem to rely on <code>readdirnames</code> in <em>os/dir_unix.go</em>, and, for some reason, it just makes an array when it seems like it would've been easy to spawn a <em>gothread</em> and and push the values into a channel. There might have been sound logic to do this, but it's not clear what it is. I'm new to Go, so I also could've easy missed some principle that's obvious to everyone else.</p>
  
  <p>This is the sourcecode, for convenience:</p>
  
  <pre><code>func (f *File) readdirnames(n int) (names []string, err error) {
      // If this file has no dirinfo, create one.
      if f.dirinfo == nil {
          f.dirinfo = new(dirInfo)
          // The buffer must be at least a block long.
          f.dirinfo.buf = make([]byte, blockSize)
      }
      d := f.dirinfo
  
      size := n
      if size &lt;= 0 {
          size = 100
          n = -1
      }
  
      names = make([]string, 0, size) // Empty with room to grow.
      for n != 0 {
          // Refill the buffer if necessary
          if d.bufp &gt;= d.nbuf {
              d.bufp = 0
              var errno error
              d.nbuf, errno = fixCount(syscall.ReadDirent(f.fd, d.buf))
              if errno != nil {
                  return names, NewSyscallError("readdirent", errno)
              }
              if d.nbuf &lt;= 0 {
                  break // EOF
              }
          }
  
          // Drain the buffer
          var nb, nc int
          nb, nc, names = syscall.ParseDirent(d.buf[d.bufp:d.nbuf], n, names)
          d.bufp += nb
          n -= nc
      }
      if n &gt;= 0 &amp;&amp; len(names) == 0 {
          return names, io.EOF
      }
      return names, nil
  }
  </code></pre>