title: How to change xcom in Airflow to accomodate large data?
body: <p>I am using the following code in my Airflow operator:</p>
  <pre><code>import json
  import pandas as pd
  from airflow.exceptions import AirflowException
  from airflow.hooks.http_hook import HttpHook
  from airflow.models import BaseOperator
  from airflow.utils.decorators import apply_defaults
  from airflow.contrib.hooks.gcs_hook import GoogleCloudStorageHook
  
  class HttpToGoogleCloudStorageOperator(BaseOperator):
    template_fields = ['endpoint', 'data', 'headers', ]
    template_ext = ()
    ui_color = '#f4a460'
  
  @apply_defaults
  def __init__(self,
               endpoint,
               project_id,
               table_id,
               data=None,
               headers=None,
               auth=None,
               http_conn_id='http_default',
               *args, **kwargs):
      super(HttpToGoogleCloudStorageOperator, self).__init__(*args, **kwargs)
      self.table_id = table_id
      self.http_conn_id = http_conn_id
      self.method = &quot;GET&quot;
      self.endpoint = endpoint
      self.headers = headers or {}
      self.auth = auth
      self.data = data or {}
  
  def execute(self, context):
      http = HttpHook(self.method, http_conn_id=self.http_conn_id)
      self.log.info(&quot;Calling HTTP method &quot; + self.endpoint)
      response = http.run(self.endpoint, self.data,  self.headers,auth=self.auth)                            
      self.log.info(&quot;Got response&quot;)
  </code></pre>
  <p>Unfortunately the data returned is too large (about 5k) to fit in the standard xcom and I get this error:</p>
  <p><strong>{taskinstance.py:1059} ERROR - (_mysql_exceptions.DataError) (1406, &quot;Data too long for column 'value' at row 1&quot;)</strong></p>
  <p>Is there a way I can tell http_hook to use a different xcom, or (even better) not use xcom at all?  I have looked around and I do not see a solution.</p>
  <p>Thanks for any tips or pointers.</p>
  <p>Edit: Here is how I call the operator.  Note that nowhere do I specify xcom.</p>
  <pre><code>query_load_task = HttpToGoogleCloudStorageOperator(
    task_id=&quot;query_load_task&quot;,
    endpoint=endpoint,
    project_id=&quot;my_gcp_poroject_id&quot;,
    table_id=&quot;dataset.table&quot;,
    data=None,
    auth=(username, password))
  </code></pre>