I have started working with NiFi. I am working on a use case to load data into Hive. I get a CSV file and then I use SplitText to split the incoming flow-file into multiple flow-files(split record by record). Then I use ConvertToAvro to convert the split CSV file into an AVRO file. After that, I put the AVRO files into a directory in HDFS and I trigger the "LOAD DATA" command using ReplaceText + PutHiveQL processor.

I'm splitting the file record by record because to get the partition value(since LOAD DATA doesn't support dynamic partitioning). The flow looks like this:

**GetFile** (CSV) --- **SplitText** (split line count :1 and header line count : 1) --- **ExtractText** (Use RegEx to get partition fields' values and assign to attribute) --- **ConvertToAvro** (Specifying the Schema) --- **PutHDFS** (Writing to a HDFS location) --- **ReplaceText** (LOAD DATA cmd with partition info) --- **PutHiveQL**

The thing is, since I'm splitting the CSV file into each record at a time, it generates too many avro files. For ex, if the CSV file has 100 records, it creates 100 AVRO files. Since I want to get the partition values, I have to split them by one record at a time. I want to know is there any way, we can achieve this thing without splitting record by record. I mean like batching it. I'm quite new to this so I am unable to crack this yet. Help me with this.

PS: Do suggest me if there is any alternate approach to achieve this use case.