Trim files by removing bytes from the beginning in bash I have files in a store folder, which I want to trim to a particular length by removing bytes from the beginning. My app uses a temp folder for the files manipulation. All files from the previous operations must be taken out form temp, so I use the folder for temporary storing the trimmed files. I also log the errors.
#!/bin/bash

set -euo pipefail

data_service='/path/to/data-service'
store=&quot;${data_service}/data/store&quot;
temp=&quot;${data_service}/data/temp&quot;
log=&quot;${data_service}/logs/log.txt&quot;
max_size=$(( 200000 * 24 ))

{

# Checks if there are any files in 'temp'. It should be empty.
if [ &quot;$( ls -A &quot;${temp}&quot; )&quot; ]
then
    echo &quot;$( date +&quot;%Y-%m-%d %H:%M:%S&quot; ) [ERROR] Temp folder is not empty!&quot; 1&gt;&amp;2
    exit 1
fi

# Loops over all the files in 'store'
for file_path in &quot;${store}/&quot;*
do

    # Trim bigger then 'max_size' files from 'store' to 'temp'
    if [ &quot;$( wc -c &lt; &quot;${file_path}&quot; )&quot; -gt &quot;${max_size}&quot; ]
    then
        file_name=&quot;$( basename &quot;${file_path}&quot; )&quot;
        tail -c &quot;${max_size}&quot; &quot;${file_path}&quot; &gt; &quot;${temp}/${file_name}&quot;
    fi

done

# Move all the truncated files back to 'store'
if [ &quot;$( ls -A &quot;${temp}&quot; )&quot; ]
then
    mv &quot;${temp}/&quot;* &quot;${store}/&quot;
fi

} 2&gt;&gt; &quot;${log}&quot;


Any potential problems or ways to improve the code?
Because these are my first steps in writing in bash, the specific things I googled for this code were:

Multiplication and assigning the result to a variable:
max_size=$(( 200000 * 24 ))


Check if there are files in a folder:
if [ &quot;$( ls -A &quot;$temp&quot; )&quot; ]; then .... fi


Get the date and time in a formatted string
$( date +&quot;%Y-%m-%d %H:%M:%S&quot; )


Get the byte size of a file:
&quot;$( wc -c &lt; &quot;$file_path&quot; )&quot;


Copy the last N bytes of a file
tail -c &quot;$max_size&quot; &quot;$file_path&quot; &gt; &quot;$temp/$file_name&quot;


Group commands and redirect STDERR to a log file
{ ... } 2&gt;&gt; log-file


Echo to STDERR of the group
echo 'Error message' 1&gt;&amp;2
Some suggestions
Get the byte size of a file
Instead of:
&quot;$( wc -c &lt; &quot;$file_path&quot; )&quot;

use stat:
stat -c %s &quot;$file_path&quot;

=&gt; better query the file system than count the number of bytes in files, some of which can be pretty large
Redirect or copy console output to a file
Instead of enclosing the whole block between brackets, I would set up file redirection from the beginning eg:
#!/bin/bash
LOG_FILE=&quot;/tmp/test.log&quot;
exec &gt; &gt;(tee -a &quot;${LOG_FILE}&quot;) 2&gt;&amp;1

# this should show on console AND in the log file
echo &quot;Hello world $(date)&quot;

based on this example
This is more flexible, because once it's set up you don't have to bother about appending your commands, and you can disable or redefine the behavior on the fly if desired.
And the benefit is that since we are redirecting both stdout and stderr, any errors will show up in the log file as well.
Misc
I think parsing the output of ls to count files in the directory is a bit primitive. Let me think but hopefully someone will come up with a better way.