title: Import old app engine logs into BigQuery
body: <p>I'd like to add old App Engine logs to my current BigQuery database of logs, but they're in an odd format.</p>
  
  <p>So I recently started adding (streaming) the app engine logs to BigQuery directly. But I have 90 days of old logs (from before I started adding them to BigQuery) that I downloaded using this sort of command:</p>
  
  <pre><code>&lt;&lt;path-to&gt;&gt;/appcfg.sh --num_days=0 --severity=0 --include_all request_logs &lt;&lt;war-location&gt;&gt; &lt;&lt;output-file&gt;&gt;
  </code></pre>
  
  <p>which can be seen here: <a href="https://cloud.google.com/appengine/docs/java/tools/uploadinganapp#Command_Line_Arguments" rel="nofollow">https://cloud.google.com/appengine/docs/java/tools/uploadinganapp#Command_Line_Arguments</a></p>
  
  <p>So I downloaded all these logs and got a text file that is around 3 GB. It is in a format I don't recognize (not CSV or JSON). It looks like this:</p>
  
  <pre><code>2601:9:6900:b700:c1ca:f9d7:7d6f:9677 - - [27/Apr/2015:13:00:39 -0700] "OPTIONS /mediaServer/&lt;&lt;fileURL&gt;&gt;" 200 109 - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.101 Safari/537.36" "&lt;&lt;domain&gt;&gt;" ms=12 cpu_ms=18 cpm_usd=0.000012 instance=00c61b117cd30a702faf40832448daeb1e98 app_engine_release=1.9.19
      1:1430164839.664000 [s~&lt;&lt;domain&gt;&gt;/1.383906932052420675].&lt;stdout&gt;: 27 Apr 2015 08:00:39, 664 INFO  &lt;&lt;servlet&gt;&gt;.doOptions(): entered
      : 
      : 
      1:1430164839.664000 [s~&lt;&lt;domain&gt;&gt;/1.383906932052420675].&lt;stdout&gt;: 27 Apr 2015 08:00:39, 664 INFO  &lt;&lt;servlet&gt;&gt;.doOptions(): exiting
      : 
      : 
  </code></pre>
  
  <p>So this is, I believe, one log (of which I have tens of thousands). The first entry is the standard App Engine log, followed by some number (in this case 2, but could be zero or more) of logs from my application.</p>
  
  <p>I haven't found anything online about adding these logs to BigQuery.
  I'm thinking I'll just have to parse it into CSV and then upload it. But if there was a way to directly import it to BigQuery, that'd be preferable.</p>
  
  <p>Any ideas?
  Thanks.</p>