Format conversion of localization files | 3txt to xliff I'm trying to build an xliff for localization from 3 specific files: one contains a list of IDs, the other a list of source strings and the last, a list of translated strings.

Basically, each file contains 200,000 strings, and the process is taking so much time. How can I speed up this loop?

I use sed to replace &lt; with &amp;lt; and the same for &gt;. If you have better ideas, please tell me.

FILE_ID=$1
FILE_SOURCE=$2
FILE_TARGET=$3
TOT_STRING=$(wc -l &lt; $FILE_ID)
time=$(date +"%Y-%m-%d %H:%M")

echo "&lt;?xml version='1.0' encoding='utf-8'?&gt;"
echo "&lt;xliff xmlns=\"urn:oasis:names:tc:xliff:document:1.1\" version=\"1.1\"&gt;"
echo " &lt;file source-language=\"en-US\" datatype=\"plaintext\"&gt;"
echo "    &lt;body&gt;"
echo "      &lt;trans-unit xml:space=\"preserve\" id=\"1\" approved=\"no\" restype=\"x-gettext-domain-header\"&gt;"
echo "        &lt;source&gt;Project-Id-Version: 1.0"
echo " Report-Msgid-Bugs-To: email@example.com"
echo "POT-Creation-Date: $time+0200"
echo "PO-Revision-Date: $time+0200"
echo "Last-Translator: JohnnyKing"
echo "Language-Team: JohnnyKing"
echo "MIME-Version: 1.0"
echo "Content-Type: text/plain; charset=UTF-8"
echo "Content-Transfer-Encoding: 8bit"
echo "X-Generator: csv2xliff.sh"
echo "&lt;/source&gt;"
echo "        &lt;target state=\"translated\"&gt;Project-Id-Version: 1.0"
echo " Report-Msgid-Bugs-To: email@example.com"
echo "POT-Creation-Date: $time+0200"
echo "PO-Revision-Date: $time+0200"
echo "Last-Translator: JohnnyKing"
echo "Language-Team: JohnnyKing"
echo "MIME-Version: 1.0"
echo "Content-Type: text/plain; charset=UTF-8"
echo "Content-Transfer-Encoding: 8bit"
echo "X-Generator: csv2xliff.sh"
echo "&lt;/target&gt;"
echo "      &lt;/trans-unit&gt;"

    COUNTER=1

        while [  "$COUNTER" -le "$TOT_STRING" ]; do

        ROW_ID=$(sed -n $(( $COUNTER ))p $FILE_ID)
        ROW_SOURCE=$(sed -n $(( COUNTER ))p $FILE_SOURCE)
        ROW_TARGET=$(sed -n $(( COUNTER ))p $FILE_TARGET)

        if [ "$ROW_SOURCE" = "$ROW_TARGET" ]; then
            echo "      &lt;trans-unit xml:space=\"preserve\" id=\"$ROW_ID\"&gt;"
            echo "        &lt;source&gt;$(echo $ROW_SOURCE | sed 's/&lt;/\&amp;lt;/g' | sed 's/&gt;/\&amp;gt;/g')&lt;/source&gt;"
            echo "        &lt;target&gt;&lt;/target&gt;"
            echo "      &lt;/trans-unit&gt;"
        else
            echo "      &lt;trans-unit xml:space=\"preserve\" id=\"$ROW_ID\" approved=\"yes\"&gt;"
            echo "        &lt;source&gt;$(echo $ROW_SOURCE | sed 's/&lt;/\&amp;lt;/g' | sed 's/&gt;/\&amp;gt;/g')&lt;/source&gt;"
            echo "        &lt;target state=\"translated\"&gt;$(echo $ROW_TARGET | sed 's/&lt;/\&amp;lt;/g' | sed 's/&gt;/\&amp;gt;/g')&lt;/target&gt;"
            echo "      &lt;/trans-unit&gt;"
        fi

        COUNTER=$(( $COUNTER + 1 ))
        done

echo "       &lt;/trans-unit&gt;"
echo "    &lt;/body&gt;"
echo "  &lt;/file&gt;"
echo "&lt;/xliff&gt;"

exit
Improving speed

For each line, running 3 sed commands to extract the n-th line from 3 files,
and then running further 2-4 sed commands is of course slow.

My first recommendation would be to implement this in another scripting language, say, Python.

If you really want to do this in Bash, you could:


Combine the 3 files into one, with their lines interleaved. That is, take the 1st line from each file, then the 2nd from each file, and so on. And then in each iteration of your loop, read 3 lines.
Instead of transforming the &lt; and &gt; by running sed for each line, run sed just once for the entire input


If we can assume that all 3 files have the same number of lines,
then you can create the input with lines interleaved like this:

paste -d '\n' "$FILE_ID" "$FILE_SOURCE" "$FILE_TARGET" 


When replacing &lt; and &gt; using sed, you can do it with a single sed command using multiple -e flags, like this:

sed -e 's/&lt;/\&amp;lt;/g' -e 's/&gt;/\&amp;gt;/g'


Putting it together:

paste -d '\n' "$FILE_ID" "$FILE_SOURCE" "$FILE_TARGET" | sed -e 's/&lt;/\&amp;lt;/g' -e 's/&gt;/\&amp;gt;/g' | \
for ((COUNTER = 1; COUNTER &lt;= TOT_STRING; ++COUNTER)); do
    read ROW_ID
    read ROW_SOURCE
    read ROW_TARGET
    # ...
done


Simplify

This is unnecessarily complicated:


ROW_ID=$(sed -n $(( $COUNTER ))p $FILE_ID)
ROW_SOURCE=$(sed -n $(( COUNTER ))p $FILE_SOURCE)
ROW_TARGET=$(sed -n $(( COUNTER ))p $FILE_TARGET)



You can write much simpler:

ROW_ID=$(sed -n ${COUNTER}p $FILE_ID)
ROW_SOURCE=$(sed -n ${COUNTER}p $FILE_SOURCE)
ROW_TARGET=$(sed -n ${COUNTER}p $FILE_TARGET)


Counting loops in Bash

Instead of this:


COUNTER=1
while [ "$COUNTER" -le "$TOT_STRING" ]; do
    # do something
    COUNTER=$(( $COUNTER + 1 ))
done



This is equivalent, but cleaner and simpler:

for ((COUNTER = 1; COUNTER &lt;= TOT_STRING; ++COUNTER)); do
    # do something
done


Naming

TOT_STRING is a strange name for a variable with an integer value.

Printing large text

Instead of this:


echo "&lt;?xml version='1.0' encoding='utf-8'?&gt;"
echo "&lt;xliff xmlns=\"urn:oasis:names:tc:xliff:document:1.1\" version=\"1.1\"&gt;"
echo " &lt;file source-language=\"en-US\" datatype=\"plaintext\"&gt;"
echo "    &lt;body&gt;"
echo "      &lt;trans-unit xml:space=\"preserve\" id=\"1\" approved=\"no\" restype=\"x-gettext-domain-header\"&gt;"
echo "        &lt;source&gt;Project-Id-Version: 1.0"
echo " Report-Msgid-Bugs-To: email@example.com"
echo "POT-Creation-Date: $time+0200"
echo "PO-Revision-Date: $time+0200"
echo "Last-Translator: JohnnyKing"
echo "Language-Team: JohnnyKing"
echo "MIME-Version: 1.0"
echo "Content-Type: text/plain; charset=UTF-8"
echo "Content-Transfer-Encoding: 8bit"
echo "X-Generator: csv2xliff.sh"
echo "&lt;/source&gt;"
echo "        &lt;target state=\"translated\"&gt;Project-Id-Version: 1.0"
echo " Report-Msgid-Bugs-To: email@example.com"
echo "POT-Creation-Date: $time+0200"
echo "PO-Revision-Date: $time+0200"
echo "Last-Translator: JohnnyKing"
echo "Language-Team: JohnnyKing"
echo "MIME-Version: 1.0"
echo "Content-Type: text/plain; charset=UTF-8"
echo "Content-Transfer-Encoding: 8bit"
echo "X-Generator: csv2xliff.sh"
echo "&lt;/target&gt;"
echo "      &lt;/trans-unit&gt;"



A simpler way to write:

cat &lt;&lt; EOF
&lt;?xml version='1.0' encoding='utf-8'?&gt;
&lt;xliff xmlns="urn:oasis:names:tc:xliff:document:1.1" version="1.1"&gt;
 &lt;file source-language="en-US" datatype="plaintext"&gt;
    &lt;body&gt;
      &lt;trans-unit xml:space="preserve" id="1" approved="no" restype="x-gettext-domain-header"&gt;
        &lt;source&gt;Project-Id-Version: 1.0
 Report-Msgid-Bugs-To: email@example.com
POT-Creation-Date: +0200
PO-Revision-Date: +0200
Last-Translator: JohnnyKing
Language-Team: JohnnyKing
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-Generator: csv2xliff.sh
&lt;/source&gt;
        &lt;target state="translated"&gt;Project-Id-Version: 1.0
 Report-Msgid-Bugs-To: email@example.com
POT-Creation-Date: +0200
PO-Revision-Date: +0200
Last-Translator: JohnnyKing
Language-Team: JohnnyKing
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-Generator: csv2xliff.sh
&lt;/target&gt;
      &lt;/trans-unit&gt;
EOF


Indentation

The indentation here is odd:


echo "X-Generator: csv2xliff.sh"
echo "&lt;/target&gt;"
echo "      &lt;/trans-unit&gt;"

    COUNTER=1

        while [  "$COUNTER" -le "$TOT_STRING" ]; do

        ROW_ID=$(sed -n $(( $COUNTER ))p $FILE_ID)
        ROW_SOURCE=$(sed -n $(( COUNTER ))p $FILE_SOURCE)
        ROW_TARGET=$(sed -n $(( COUNTER ))p $FILE_TARGET)

        if [ "$ROW_SOURCE" = "$ROW_TARGET" ]; then
            echo "      &lt;trans-unit xml:space=\"preserve\" id=\"$ROW_ID\"&gt;"
            echo "        &lt;source&gt;$(echo $ROW_SOURCE | sed 's/&lt;/\&amp;lt;/g' | sed 's/&gt;/\&amp;gt;/g')&lt;/source&gt;"
            echo "        &lt;target&gt;&lt;/target&gt;"
            echo "      &lt;/trans-unit&gt;"
        else
            echo "      &lt;trans-unit xml:space=\"preserve\" id=\"$ROW_ID\" approved=\"yes\"&gt;"
            echo "        &lt;source&gt;$(echo $ROW_SOURCE | sed 's/&lt;/\&amp;lt;/g' | sed 's/&gt;/\&amp;gt;/g')&lt;/source&gt;"
            echo "        &lt;target state=\"translated\"&gt;$(echo $ROW_TARGET | sed 's/&lt;/\&amp;lt;/g' | sed 's/&gt;/\&amp;gt;/g')&lt;/target&gt;"
            echo "      &lt;/trans-unit&gt;"
        fi

        COUNTER=$(( $COUNTER + 1 ))
        done

echo "       &lt;/trans-unit&gt;"
echo "    &lt;/body&gt;"
echo "  &lt;/file&gt;"
echo "&lt;/xliff&gt;"



It would be more natural this way:

echo "X-Generator: csv2xliff.sh"
echo "&lt;/target&gt;"
echo "      &lt;/trans-unit&gt;"

COUNTER=1

while [  "$COUNTER" -le "$TOT_STRING" ]; do
    ROW_ID=$(sed -n $(( $COUNTER ))p $FILE_ID)
    ROW_SOURCE=$(sed -n $(( COUNTER ))p $FILE_SOURCE)
    ROW_TARGET=$(sed -n $(( COUNTER ))p $FILE_TARGET)

    if [ "$ROW_SOURCE" = "$ROW_TARGET" ]; then
        echo "      &lt;trans-unit xml:space=\"preserve\" id=\"$ROW_ID\"&gt;"
        echo "        &lt;source&gt;$(echo $ROW_SOURCE | sed 's/&lt;/\&amp;lt;/g' | sed 's/&gt;/\&amp;gt;/g')&lt;/source&gt;"
        echo "        &lt;target&gt;&lt;/target&gt;"
        echo "      &lt;/trans-unit&gt;"
    else
        echo "      &lt;trans-unit xml:space=\"preserve\" id=\"$ROW_ID\" approved=\"yes\"&gt;"
        echo "        &lt;source&gt;$(echo $ROW_SOURCE | sed 's/&lt;/\&amp;lt;/g' | sed 's/&gt;/\&amp;gt;/g')&lt;/source&gt;"
        echo "        &lt;target state=\"translated\"&gt;$(echo $ROW_TARGET | sed 's/&lt;/\&amp;lt;/g' | sed 's/&gt;/\&amp;gt;/g')&lt;/target&gt;"
        echo "      &lt;/trans-unit&gt;"
    fi

    COUNTER=$(( $COUNTER + 1 ))
done

echo "       &lt;/trans-unit&gt;"
echo "    &lt;/body&gt;"
echo "  &lt;/file&gt;"
echo "&lt;/xliff&gt;"