title: Retrieving a large amount of data through curl and php
body: <p>I have a project where we have an internal mssql warehouse not available over the internet. The user needs to be able to run a query from a normal website which can query this internal warehouse and receive a report.</p>
  
  <p>To do this, I have created a script on an internal server (is available over the internet) which runs the queries on the mssql server (via the intranet) and returns a string of data (using php://output)</p>
  
  <p>In order to get this working on the website side, I have created a php file on the remote server which does a curl request to the internal server (with variables i've passed in), when the curl request finishes it passes the variable back in to the website which writes the contents to a file, then in turn I redirect the user to this file.</p>
  
  <p>This works perfectly with a few caveats...</p>
  
  <p>If the query that is being run is big (they can be absolutely massive) its possible the server can take a long time (over 3/4 hours) to run the queries (indexing on the internal warehouse is not handled by me). Although the curl request continues to run, I do find sometimes it fails.</p>
  
  <p>I have 2 options for the user, 1 is to wait for the request to finish, so an ajax request then goes off to do all this and when it is done they are redirected to the file.... this sometimes never returns... so the user is left waiting for it come back forever - i have tried to get around this by checking for the file on the file system every 5 seconds and if found then download it - this has worked to some extent... but again sometimes it still does fail.</p>
  
  <p>The second option is to email the report once finished allowing the user to leave the website and let it continue to do the curl request in the background, however again if the query takes a LONG time to run, I seem to get empty emails as if the curl request has timed out.</p>
  
  <p>I'm a bit lost on why it fails if the query takes a long time or why the javascript sometimes fails to report that it is complete. Any information to point me in the right direction would be useful - I dont think i'm far off having this working perfectly.</p>
  
  <p>The warehouse has over 100 Million records.</p>