airflow/dags/util.py
def generate_bq_payload(workflow_id: str, dataset: str, filename: str = None,
                        gcs_bucket: str = None, url: str = None) -> dict:
    """Creates the payload object required for the BQ ingestion operator.

    workflow_id: ID of the datasource workflow. Should match ID defined in
                 DATA_SOURCES_DICT.
    dataset: Name of the BQ dataset to write the data to.
    filename: Name of gcs file to get the data from. May also be a
              comma-separated list of files.
    gcs_bucket: GCS bucket to read from. Defaults to the GCS_LANDING_BUCKET env
                var.
    url: The URL used for ingestion. This should be deprecated in favor of
         writing any metadata to GCS during the GCS step. It's temporarily
         necessary since ACS directly requests metadata during BQ upload."""
    message = get_required_attrs(workflow_id, gcs_bucket=gcs_bucket)
    message['dataset'] = dataset
    if filename is not None:
        message['filename'] = filename
    if url is not None:
        message['url'] = url
    return {'message': message}
Creates the payload object required for the BQ ingestion operator.

    workflow_id: ID of the datasource workflow. Should match ID defined in
                 DATA_SOURCES_DICT.
    dataset: Name of the BQ dataset to write the data to.
    filename: Name of gcs file to get the data from. May also be a
              comma-separated list of files.
    gcs_bucket: GCS bucket to read from. Defaults to the GCS_LANDING_BUCKET env
                var.
    url: The URL used for ingestion. This should be deprecated in favor of
         writing any metadata to GCS during the GCS step. It's temporarily
         necessary since ACS directly requests metadata during BQ upload.
Creates the payload object required for the BQ ingestion operator.
workflow_id: ID of the datasource workflow. Should match ID defined in
DATA_SOURCES_DICT.
dataset: Name of the BQ dataset to write the data to.
filename: Name of gcs file to get the data from. May also be a
comma-separated list of files.
gcs_bucket: GCS bucket to read from. Defaults to the GCS_LANDING_BUCKET env
var.
url: The URL used for ingestion. This should be deprecated in favor of
writing any metadata to GCS during the GCS step. It's temporarily
necessary since ACS directly requests metadata during BQ upload.
Creates the payload object required for the BQ ingestion operator . workflow_id : ID of the datasource workflow . Should match ID defined in DATA_SOURCES_DICT . dataset : Name of the BQ dataset to write the data to . filename : Name of gcs file to get the data from . May also be a comma - separated list of files . gcs_bucket : GCS bucket to read from . Defaults to the GCS_LANDING_BUCKET env var . url : The URL used for ingestion . This should be deprecated in favor of writing any metadata to GCS during the GCS step . It ' s temporarily necessary since ACS directly requests metadata during BQ upload .
def generate_bq_payload(workflow_id: str, dataset: str, filename: str = None,
                        gcs_bucket: str = None, url: str = None) -> dict:
    message = get_required_attrs(workflow_id, gcs_bucket=gcs_bucket)
    message['dataset'] = dataset
    if filename is not None:
        message['filename'] = filename
    if url is not None:
        message['url'] = url
    return {'message': message}
def generate_bq_payload ( workflow_id : str , dataset : str , filename : str = None , gcs_bucket : str = None , url : str = None ) -> dict : message = get_required_attrs ( workflow_id , gcs_bucket = gcs_bucket ) message [ 'dataset' ] = dataset if filename is not None : message [ 'filename' ] = filename if url is not None : message [ 'url' ] = url return { 'message' : message }
Creates the payload object required for the BQ ingestion operator.
Creates the payload object required for the BQ ingestion operator .
"""Creates the payload object required for the BQ ingestion operator.

    workflow_id: ID of the datasource workflow. Should match ID defined in
                 DATA_SOURCES_DICT.
    dataset: Name of the BQ dataset to write the data to.
    filename: Name of gcs file to get the data from. May also be a
              comma-separated list of files.
    gcs_bucket: GCS bucket to read from. Defaults to the GCS_LANDING_BUCKET env
                var.
    url: The URL used for ingestion. This should be deprecated in favor of
         writing any metadata to GCS during the GCS step. It's temporarily
         necessary since ACS directly requests metadata during BQ upload."""
workflow_id
str
dataset
str
filename
str
gcs_bucket
str
url
str