title: Strange "Over quota: " error with no further diagnostics
body: <p>I'm running a parallel job to copy data from Heroku into Google Cloud Storage and eventually into Bigquery. The way I'm doing it right now is split the job of querying IDs in the range [61500000, 62000000) into say 40 taskqueue tasks, and inside each task handler is responsible for the subrange say [
  61500000, 61512500). Inside each taskqueue task handler, it spawns 3 goroutines to query our Heroku API parallely, and an additional goroutine doing an Insert to Google Cloud Storage. The way the 3 HTTP API input goroutines pump data to the GCS insert goroutine is through io.Pipe(). </p>
  
  <p>However, for some reason I can't get this to work except for toy workloads. Virtually every time there will be some shards failing with error:</p>
  
  <p>"Post <a href="https://www.googleapis.com/upload/storage/v1beta2/b/ethereal-fort-637.appspot.com/o?alt=json&amp;uploadType=multipart" rel="nofollow noreferrer">https://www.googleapis.com/upload/storage/v1beta2/b/ethereal-fort-637.appspot.com/o?alt=json&amp;uploadType=multipart</a>: Over quota: "</p>
  
  <p>returned from the storage.ObjectsInsertCall.Do().</p>
  
  <p>I checked everywhere for possible places where we hit the quota for billed apps:
  * urlfetch total limits developers.google.com/appengine/docs/quotas#UrlFetch
  * instance memory developers.google.com/appengine/docs/go/modules/#Go_Instance_scaling_and_class
  but still couldn't find the cause.
  Below I explain why I ruled out the above possibilities:</p>
  
  <ul>
  <li><p>urlfetch total limits
  urlfetch is used in the 3 goroutines to query our API server for JSON data. These 3 goroutines then process the data and send them to the GCS goroutine through io.Pipe(). The code looks something like</p>
  
  <p>cl := urlfetch.Client(c)
  resp, err := cl.Get("pic-collage.com/...")
  if err != nil {
    if appengine.IsOverQuota(err) {
      c.Errorf("collageJSONByID over quota: %v", err)
    }
    return err
  }</p></li>
  </ul>
  
  <p>However, while we see numerous "POST www.googleapis.com/upload/storage/v1beta2...: Over quota: " errors, we never ever see the logs "collageJSONByID ..." related to urlfetches to our Heroku server.</p>
  
  <ul>
  <li>instance memory
  We are using the B1 instance for our jobs which has 128MB of RAM. Throughout our runs, we see from our appengine console our memory usage is constantly well below 30MB for each and every instance.</li>
  </ul>
  
  <p>I also applied the fix for the caching inside serviceaccounts described in <a href="https://stackoverflow.com/questions/23919834/over-quota-when-using-gcs-json-api-from-app-engine">&quot;Over quota&quot; when using GCS json-api from App Engine</a> , but the problem persists.</p>
  
  <p>I wonder is it possible for us to get more information about the specific App Engine quota we are exceeding? Or perhaps there are other hidden quotas for Google Cloud Storage that are not mentioned in the docs ?</p>