title: Azure Data Factory Mapping Data Flow to CSV sink results in zero-byte files
body: <p>I'm sharpening up my Azure Data Factory chops, comparing Copy Activity performance vs Mapping Data Flows writing to a single CSV file in Azure Blob Storage.</p>
  
  <p>When I write to a Single CSV through an Azure Blob Storage Linked Service (azureBlobLinkedService) through a Dataset (azureBlobSingleCSVFileNameDataset), using a Copy Activity get the output in the blob storage container that I expect.  For example, an output file of MyData.csv in the container MyContainer under the folder /output/csv/singleFiles.</p>
  
  <p>When I write to a Single CSV through the same Blob Storage Linked Service, but through a different DataSet (azureBlobSingleCSVNoFileNameDataset), using a Mapping Data Flow I get the following:</p>
  
  <ul>
  <li>MyContainer/output/csv/singleFiles (zero length file)</li>
  <li>MyContainer/output/csv/singleFiles/MyData.csv (contains the data that I expect)</li>
  </ul>
  
  <p>I don't understand why I'm getting the zero length files produced when using a Mapping Data Flow.</p>
  
  <p>Here are my source files:</p>
  
  <p>linkedService/azureBlobLinkedService</p>
  
  <pre><code>{
      "name": "azureBlobLinkedService",
      "type": "Microsoft.DataFactory/factories/linkedservices",
      "properties": {
          "type": "AzureBlobStorage",
          "parameters": {
              "azureBlobConnectionStringSecretName": {
                  "type": "string"
              }
          },
          "annotations": [],
          "typeProperties": {
              "connectionString": {
                  "type": "AzureKeyVaultSecret",
                  "store": {
                      "referenceName": "AzureKeyVaultLinkedService",
                      "type": "LinkedServiceReference"
                  },
                  "secretName": "@{linkedService().azureBlobConnectionStringSecretName}"
              }
          }
      }
  }
  </code></pre>
  
  <p>dataset/azureBlobSingleCSVFileNameDataset</p>
  
  <pre><code>{
      "name": "azureBlobSingleCSVFileNameDataset",
      "properties": {
          "linkedServiceName": {
              "referenceName": "azureBlobLinkedService",
              "type": "LinkedServiceReference",
              "parameters": {
                  "azureBlobConnectionStringSecretName": {
                      "value": "@dataset().azureBlobConnectionStringSecretName",
                      "type": "Expression"
                  }
              }
          },
          "parameters": {
              "azureBlobConnectionStringSecretName": {
                  "type": "string"
              },
              "azureBlobSingleCSVFileName": {
                  "type": "string"
              },
              "azureBlobSingleCSVFolderPath": {
                  "type": "string"
              },
              "azureBlobSingleCSVContainerName": {
                  "type": "string"
              }
          },
          "annotations": [],
          "type": "DelimitedText",
          "typeProperties": {
              "location": {
                  "type": "AzureBlobStorageLocation",
                  "fileName": {
                      "value": "@dataset().azureBlobSingleCSVFileName",
                      "type": "Expression"
                  },
                  "folderPath": {
                      "value": "@dataset().azureBlobSingleCSVFolderPath",
                      "type": "Expression"
                  },
                  "container": {
                      "value": "@dataset().azureBlobSingleCSVContainerName",
                      "type": "Expression"
                  }
              },
              "columnDelimiter": ",",
              "escapeChar": "\\",
              "firstRowAsHeader": true,
              "quoteChar": "\""
          },
          "schema": []
      },
      "type": "Microsoft.DataFactory/factories/datasets"
  }
  </code></pre>
  
  <p>pipeline/Azure SQL Table to Blob Single CSV Copy Pipeline (this produces expected results)</p>
  
  <pre><code>{
      "name": "Azure SQL Table to Blob Single CSV Copy Pipeline",
      "properties": {
          "activities": [
              {
                  "name": "Copy Azure SQL Table to Blob Single CSV",
                  "type": "Copy",
                  "dependsOn": [],
                  "policy": {
                      "timeout": "7.00:00:00",
                      "retry": 0,
                      "retryIntervalInSeconds": 30,
                      "secureOutput": false,
                      "secureInput": false
                  },
                  "userProperties": [],
                  "typeProperties": {
                      "source": {
                          "type": "AzureSqlSource",
                          "queryTimeout": "02:00:00"
                      },
                      "sink": {
                          "type": "DelimitedTextSink",
                          "storeSettings": {
                              "type": "AzureBlobStorageWriteSettings"
                          },
                          "formatSettings": {
                              "type": "DelimitedTextWriteSettings",
                              "quoteAllText": true,
                              "fileExtension": ".csv"
                          }
                      },
                      "enableStaging": false
                  },
                  "inputs": [
                      {
                          "referenceName": "azureSqlDatabaseTableDataset",
                          "type": "DatasetReference",
                          "parameters": {
                              "azureSqlDatabaseConnectionStringSecretName": {
                                  "value": "@pipeline().parameters.sourceAzureSqlDatabaseConnectionStringSecretName",
                                  "type": "Expression"
                              },
                              "azureSqlDatabaseTableSchemaName": {
                                  "value": "@pipeline().parameters.sourceAzureSqlDatabaseTableSchemaName",
                                  "type": "Expression"
                              },
                              "azureSqlDatabaseTableTableName": {
                                  "value": "@pipeline().parameters.sourceAzureSqlDatabaseTableTableName",
                                  "type": "Expression"
                              }
                          }
                      }
                  ],
                  "outputs": [
                      {
                          "referenceName": "azureBlobSingleCSVFileNameDataset",
                          "type": "DatasetReference",
                          "parameters": {
                              "azureBlobConnectionStringSecretName": {
                                  "value": "@pipeline().parameters.sinkAzureBlobConnectionStringSecretName",
                                  "type": "Expression"
                              },
                              "azureBlobSingleCSVFileName": {
                                  "value": "@pipeline().parameters.sinkAzureBlobSingleCSVFileName",
                                  "type": "Expression"
                              },
                              "azureBlobSingleCSVFolderPath": {
                                  "value": "@pipeline().parameters.sinkAzureBlobSingleCSVFolderPath",
                                  "type": "Expression"
                              },
                              "azureBlobSingleCSVContainerName": {
                                  "value": "@pipeline().parameters.sinkAzureBlobSingleCSVContainerName",
                                  "type": "Expression"
                              }
                          }
                      }
                  ]
              }
          ],
          "parameters": {
              "sourceAzureSqlDatabaseConnectionStringSecretName": {
                  "type": "string"
              },
              "sourceAzureSqlDatabaseTableSchemaName": {
                  "type": "string"
              },
              "sourceAzureSqlDatabaseTableTableName": {
                  "type": "string"
              },
              "sinkAzureBlobConnectionStringSecretName": {
                  "type": "string"
              },
              "sinkAzureBlobSingleCSVContainerName": {
                  "type": "string"
              },
              "sinkAzureBlobSingleCSVFolderPath": {
                  "type": "string"
              },
              "sinkAzureBlobSingleCSVFileName": {
                  "type": "string"
              }
          },
          "annotations": []
      },
      "type": "Microsoft.DataFactory/factories/pipelines"
  }
  </code></pre>
  
  <p>dataset/azureBlobSingleCSVNoFileNameDataset: (no filename in dataset required by mapping data flow, set in the mapping data flow)</p>
  
  <pre><code>{
      "name": "azureBlobSingleCSVNoFileNameDataset",
      "properties": {
          "linkedServiceName": {
              "referenceName": "azureBlobLinkedService",
              "type": "LinkedServiceReference",
              "parameters": {
                  "azureBlobConnectionStringSecretName": {
                      "value": "@dataset().azureBlobConnectionStringSecretName",
                      "type": "Expression"
                  }
              }
          },
          "parameters": {
              "azureBlobConnectionStringSecretName": {
                  "type": "string"
              },
              "azureBlobSingleCSVFolderPath": {
                  "type": "string"
              },
              "azureBlobSingleCSVContainerName": {
                  "type": "string"
              }
          },
          "annotations": [],
          "type": "DelimitedText",
          "typeProperties": {
              "location": {
                  "type": "AzureBlobStorageLocation",
                  "folderPath": {
                      "value": "@dataset().azureBlobSingleCSVFolderPath",
                      "type": "Expression"
                  },
                  "container": {
                      "value": "@dataset().azureBlobSingleCSVContainerName",
                      "type": "Expression"
                  }
              },
              "columnDelimiter": ",",
              "escapeChar": "\\",
              "firstRowAsHeader": true,
              "quoteChar": "\""
          },
          "schema": []
      },
      "type": "Microsoft.DataFactory/factories/datasets"
  }
  </code></pre>
  
  <p>dataflow/azureSqlDatabaseTableToAzureBlobSingleCSVDataFlow</p>
  
  <pre><code>{
      "name": "azureSqlDatabaseTableToAzureBlobSingleCSVDataFlow",
      "properties": {
          "type": "MappingDataFlow",
          "typeProperties": {
              "sources": [
                  {
                      "dataset": {
                          "referenceName": "azureSqlDatabaseTableDataset",
                          "type": "DatasetReference"
                      },
                      "name": "readFromAzureSqlDatabase"
                  }
              ],
              "sinks": [
                  {
                      "dataset": {
                          "referenceName": "azureBlobSingleCSVNoFileNameDataset",
                          "type": "DatasetReference"
                      },
                      "name": "writeToAzureBlobSingleCSV"
                  }
              ],
              "transformations": [
                  {
                      "name": "enrichWithRuntimeMetadata"
                  }
              ],
              "script": "\nparameters{\n\tsourceConnectionSecretName as string,\n\tsinkConnectionStringSecretName as string,\n\tsourceObjectName as string,\n\tsinkObjectName as string,\n\tdataFactoryName as string,\n\tdataFactoryPipelineName as string,\n\tdataFactoryPipelineRunId as string,\n\tsinkFileNameNoPath as string\n}\nsource(allowSchemaDrift: true,\n\tvalidateSchema: false,\n\tisolationLevel: 'READ_UNCOMMITTED',\n\tformat: 'table') ~&gt; readFromAzureSqlDatabase\nreadFromAzureSqlDatabase derive({__sourceConnectionStringSecretName} = $sourceConnectionSecretName,\n\t\t{__sinkConnectionStringSecretName} = $sinkConnectionStringSecretName,\n\t\t{__sourceObjectName} = $sourceObjectName,\n\t\t{__sinkObjectName} = $sinkObjectName,\n\t\t{__dataFactoryName} = $dataFactoryName,\n\t\t{__dataFactoryPipelineName} = $dataFactoryPipelineName,\n\t\t{__dataFactoryPipelineRunId} = $dataFactoryPipelineRunId) ~&gt; enrichWithRuntimeMetadata\nenrichWithRuntimeMetadata sink(allowSchemaDrift: true,\n\tvalidateSchema: false,\n\tpartitionFileNames:[($sinkFileNameNoPath)],\n\tpartitionBy('hash', 1),\n\tquoteAll: true) ~&gt; writeToAzureBlobSingleCSV"
          }
      }
  }
  </code></pre>
  
  <p>pipeline/Azure SQL Table to Blob Single CSV Data Flow Pipeline (this produces the expected results, plus zero-byte file at folder paths.)</p>
  
  <pre><code>{
      "name": "Azure SQL Table to Blob Single CSV Data Flow Pipeline",
      "properties": {
          "activities": [
              {
                  "name": "Copy Sql Database Table To Blob Single CSV Data Flow",
                  "type": "ExecuteDataFlow",
                  "dependsOn": [],
                  "policy": {
                      "timeout": "7.00:00:00",
                      "retry": 0,
                      "retryIntervalInSeconds": 30,
                      "secureOutput": false,
                      "secureInput": false
                  },
                  "userProperties": [],
                  "typeProperties": {
                      "dataflow": {
                          "referenceName": "azureSqlDatabaseTableToAzureBlobSingleCSVDataFlow",
                          "type": "DataFlowReference",
                          "parameters": {
                              "sourceConnectionSecretName": {
                                  "value": "'@{pipeline().parameters.sourceAzureSqlDatabaseConnectionStringSecretName}'",
                                  "type": "Expression"
                              },
                              "sinkConnectionStringSecretName": {
                                  "value": "'@{pipeline().parameters.sinkAzureBlobConnectionStringSecretName}'",
                                  "type": "Expression"
                              },
                              "sourceObjectName": {
                                  "value": "'@{concat('[', pipeline().parameters.sourceAzureSqlDatabaseTableSchemaName, '].[', pipeline().parameters.sourceAzureSqlDatabaseTableTableName, ']')}'",
                                  "type": "Expression"
                              },
                              "sinkObjectName": {
                                  "value": "'@{concat(pipeline().parameters.sinkAzureBlobSingleCSVContainerName, '/', pipeline().parameters.sinkAzureBlobSingleCSVFolderPath, '/', \npipeline().parameters.sinkAzureBlobSingleCSVFileName)}'",
                                  "type": "Expression"
                              },
                              "dataFactoryName": {
                                  "value": "'@{pipeline().DataFactory}'",
                                  "type": "Expression"
                              },
                              "dataFactoryPipelineName": {
                                  "value": "'@{pipeline().Pipeline}'",
                                  "type": "Expression"
                              },
                              "dataFactoryPipelineRunId": {
                                  "value": "'@{pipeline().RunId}'",
                                  "type": "Expression"
                              },
                              "sinkFileNameNoPath": {
                                  "value": "'@{pipeline().parameters.sinkAzureBlobSingleCSVFileName}'",
                                  "type": "Expression"
                              }
                          },
                          "datasetParameters": {
                              "readFromAzureSqlDatabase": {
                                  "azureSqlDatabaseConnectionStringSecretName": {
                                      "value": "@pipeline().parameters.sourceAzureSqlDatabaseConnectionStringSecretName",
                                      "type": "Expression"
                                  },
                                  "azureSqlDatabaseTableSchemaName": {
                                      "value": "@pipeline().parameters.sourceAzureSqlDatabaseTableSchemaName",
                                      "type": "Expression"
                                  },
                                  "azureSqlDatabaseTableTableName": {
                                      "value": "@pipeline().parameters.sourceAzureSqlDatabaseTableTableName",
                                      "type": "Expression"
                                  }
                              },
                              "writeToAzureBlobSingleCSV": {
                                  "azureBlobConnectionStringSecretName": {
                                      "value": "@pipeline().parameters.sinkAzureBlobConnectionStringSecretName",
                                      "type": "Expression"
                                  },
                                  "azureBlobSingleCSVFolderPath": {
                                      "value": "@pipeline().parameters.sinkAzureBlobSingleCSVFolderPath",
                                      "type": "Expression"
                                  },
                                  "azureBlobSingleCSVContainerName": {
                                      "value": "@pipeline().parameters.sinkAzureBlobSingleCSVContainerName",
                                      "type": "Expression"
                                  }
                              }
                          }
                      },
                      "compute": {
                          "coreCount": 8,
                          "computeType": "General"
                      }
                  }
              }
          ],
          "parameters": {
              "sourceAzureSqlDatabaseConnectionStringSecretName": {
                  "type": "string"
              },
              "sourceAzureSqlDatabaseTableSchemaName": {
                  "type": "string"
              },
              "sourceAzureSqlDatabaseTableTableName": {
                  "type": "string"
              },
              "sinkAzureBlobConnectionStringSecretName": {
                  "type": "string"
              },
              "sinkAzureBlobSingleCSVContainerName": {
                  "type": "string"
              },
              "sinkAzureBlobSingleCSVFolderPath": {
                  "type": "string"
              },
              "sinkAzureBlobSingleCSVFileName": {
                  "type": "string"
              }
          },
          "annotations": []
      },
      "type": "Microsoft.DataFactory/factories/pipelines"
  }
  </code></pre>