I'm trying to remove some characters from file(UTF-8). I'm using tr for this purpose:tr -cs '[[:alpha:][:space:]]' ' ' <testdata.dat File contains some foreign characters (like  or ). tr doesn't seem to understand them: it treats them as non-alpha and removes too.I've tried changing some of my locale settings:LC_CTYPE=C LC_COLLATE=C tr -cs '[[:alpha:][:space:]]' ' ' <testdata.datLC_CTYPE=ru_RU.UTF-8 LC_COLLATE=C tr -cs '[[:alpha:][:space:]]' ' ' <testdata.datLC_CTYPE=ru_RU.UTF-8 LC_COLLATE=ru_RU.UTF-8 tr -cs '[[:alpha:][:space:]]' ' ' <testdata.datUnfortunately, none of these worked.How can I make tr understand Unicode?
How to make tr aware of non-ascii(unicode) characters?
linux;text processing;unicode;tr
That's a known (1, 2, 3, 4, 5, 6) limitation of the GNU implementation of tr.It's not as much that it doesn't support foreign, non-English or non-ASCII characters, but that it doesn't support multi-byte characters.Those Cyrillic characters would be treated OK, if written in the iso8859-5 (single-byte per character) character set (and your locale was using that charset), but your problem is that you're using UTF-8 where non-ASCII characters are encoded in 2 or more bytes.GNU's got a plan to fix that and work is under way but not there yet.FreeBSD or Solaris tr don't have the problem.In the mean time, for most use cases of tr, you can use GNU sed or GNU awk which have been internationalised.For instance, your:tr -cs '[[:alpha:][:space:]]' ' 'could be written:gsed -E 's/( |[^[:space:][:alpha:]])+/ /'or:gawk -v RS='( |[^[:space:][:alpha:]])+' '{printf %s, sep $0; sep= }'To convert between lower and upper case (tr '[:upper:]' '[:lower:]'):gsed 's/[[:upper:]]/\l&/g'(that l is a lowercase L, not the 1 digit).or:gawk '{print tolower($0)}'For portability, perl is another alternative:perl -C -pe 's/([^[:space:][:alpha:]]| )+/ /g'perl -C -pe '$_=lc$_'If you know the data can be represented in a single-byte character set, then you can process it in that charset:(export LC_ALL=ru_RU.iso88595 iconv -f utf-8 |   tr -cs '[:alpha:][:space:]' ' ' |   iconv -t utf-8) < Russian-file.utf8