title: Rebuilding the Sphinx index fails
body: <p>We have a problem while rebuilding the Sphinx index for one of our MySQL databases. Rotating and rebuilding the index fails repeatedly, and the system continues to use the old index. The database size has doubled in the last months and it contains now tables with more than 10 million rows. During rebuilding of the Sphinx index with</p>
  
  <pre><code>indexer --config our_config.conf --rotate --all
  </code></pre>
  
  <p>(generated by the UltraSphinx plugin used here), we get the following error in the Sphinx daemon log file searchd.log</p>
  
  <pre><code>WARNING: rotating index 'main': preread failed: failed to open 
  ~/etc/sphinx//sphinx_index_main.new.spi: No such file or directory; 
  using old index
  WARNING: rotating index 'main': preread failure reported
  </code></pre>
  
  <p>The error messages occurred in increasing frequency as the database grew larger. Full indexing takes now about 1 1/2 hours, and the index files together are larger than one GB. We updated the Sphinx Version from 0.9.8.1 to 2.0.4 to solve this problem, but it still exists. The server has about 4 GB memory, and the configuration file begins with</p>
  
  <pre><code>indexer {
    mem_limit = 1024M
  }
  
  searchd {
    read_timeout = 5
    max_children = 300
    log = ~/etc/sphinx/log/searchd.log
    port = 3313
    max_matches = 100000
    query_log = ~/etc/sphinx/log/query.log
    seamless_rotate = 1
    pid_file = ~/etc/sphinx/log/searchd.pid
    address = 127.0.0.1
  }
  ..
  </code></pre>
  
  <p>Has anyone an idea how the problem can be solved? I guess there seems to be some kind of index size limit which has to do with the 2 GB limit, wich is the maximal value for a signed 32 bit integer and maximal value for file sizes on certain file systems. The ext3 file system we use on our Ubuntu/Debian systems allows up to 16 GB for files, but certain libraries may limit the file size in practice to 2 GB (see <a href="http://sphinxsearch.com/forum/view.html?id=3013" rel="nofollow">here</a> and <a href="http://sphinxsearch.com/forum/view.html?id=4891" rel="nofollow">here</a>).</p>