Essentially, I am looking for an encoding-type agnostic version of the column -t command in linux.I need to convert delimited files to flat and then write to an outfile. The specific delimiter varies (, | \t).Sometimes, but not always, the fields contain text characters with diacriticals.Since column -t is only compatible with UTF-8 encoding type, it chokes on these funny characters:I tried using commands like file -bi to determine encoding type so I could recode to UTF-8, but they only provide a best guess (which fails when the diacriticals start at row 100,000.)So far I have come across the following:This script in awk converts delimited to fixed length columns, but I need variable column widths:$ awk -F@ '{for(i=1;i<=NF;i++){printf %-20s, $i};printf \n}' input.csvThis script in perl finds max length for each column and sets each column to max length, but I also need the number of columns to be variable:perl -F'\t' -lane 'BEGIN {    open I, < inputfile;    for (<I>) {        chomp;        @x = split /\t/;        (length ($x[0]) > $f0) && ($f0 = length ($x[0]) + 1);        (length ($x[1]) > $f1) && ($f1 = length ($x[1]) + 1);        (length ($x[2]) > $f2) && ($f2 = length ($x[2]) + 1);    }    close I;}print $F[0] .   x ($f0 - length ($F[0])), $F[1] .   x ($f1 - length     ($F[1])), $F[2] .   x ($f2 - length ($F[2]));' inputfileDoes anyone have a script that:Is executable in the shellDoes what column -t does (deals with varying column widths, deals with varying numbers of columns, allows you to specify your delimiter)AND is encoding-type agnostic?
Command like `column -t` but encoding-type agnostic?
unicode;csv;text;file format