I have two large files ~9GB. CSV File 1 has columns A, B, C, D, E and CSV File 2 has columns B, C, F, G. The desired output is A, B, C, D, E, F, G. All I have been able to find is joining on similar columns and concatenating with the same columns, however here some match, and some do not. A sample output would look something along these lines:A   B   C   D   E   F   G1   2   3   4   5   6   7NaN 1   2   NaN 1   2   1So if the value doesn't exist for that column, as in it doesn't exist, I just want it to have a NaN value. I hope I have explained the problem well enough. Thanks!Edit: Normally I would do this in Python but these massive files make it considerably more annoying iterating over chunks and then concatenating at the end. There appears to be a more straightforward way using bash that I am unaware of. Thanks!
Concatenate CSV with some shared columns
text processing;command line;awk;sed;csv
This works based on the following facts:(a) All fields are strictly tab separated(b) Common columns in both files (B and C) have the same value$ join --nocheck-order -eNaN -13 -22 -t$'\t' -o 1.1 1.2 1.3 1.4 1.5 2.3 2.4 b.txt c.txtA   B   C   D   E   F   G1   2   3   4   5   6   7NaN 1   2   NaN 1   2   1Files Sample:$ cat b.txtA   B   C   D   E1   2   3   4   5    1   2       1$ cat c.txtB   C   F   G2   3   6   71   2   2   1Join Options:-13 -22 : Join based on file1 column3 (C) = file2 column2 (C)-t$'\t' : tab delimiter for input and output-o      : Output format. 1.1 means file1, column1, and so on.-e      : Fill empty values with NaNFor more info see man join and even better info joinAlternative Solution with AWKPS: Bear with me in awk, i'm an awk new learner.$ awk -F\t '{a[1]=;{for (i=1;i<=NF;i++) if (i==6 ||i==7) continue;else \if ($i!=) a[1]=a[1]FS$i;else a[1]=a[1]FSNaN;print a[1]}}' <(paste b.txt c.txt)Update for comma separated input fieldsAs advised in your comments , since csv files are separated by comma, this solution will separate input fields by comma and will output the results using tabs to be more readable. awk 'BEGIN {FS=,;OFS=\t}{a[1]=;{for (i=1;i<=NF;i++) if (i==6 ||i==7) continue;else \if ($i!=) a[1]=a[1]OFS$i;else a[1]=a[1]OFSNaN;print a[1]}}' <(paste b.txt c.txt)If you need output to be printed also with comma , just replace the begine section with {FS=OFS=,} Though is still unclear what you intent to do with common columns / different values.  You can remove the part if (i==6 ||i==7) continue;else to see if the results fits your needs. This condition check actually skips field 6 (B column of file2) and field7 (C column of file2) since those two columns of file 2 had been considered as identical to columns of file 1 till now.For the join solution:Replace -t$'\t' with -t',' to read comma separated fieldsFor the common columns you can play with this output format: join --nocheck-order -eNaN -13 -22 -t',' -o 1.1 1.2 2.1 1.3 2.2 1.4 1.5 2.3 2.4 b.txt c.txt