title: Bash: Find and delete duplicate files from different folders base on name and size of each file
body: <p>I have merged several old folders of mp3 together via MusicBrainz Picard <a href="https://picard.musicbrainz.org/" rel="nofollow noreferrer">https://picard.musicbrainz.org/</a><br>
  It did an amazing job identifying mp3 and properly organise them in a structured folder which I want.</p>
  
  <p>But now I am finding several duplicates in the following format:  </p>
  
  <pre><code>size    file_path_and_name  
  3130248 &lt;artist name&gt;/&lt;artist name&gt; - &lt;song name&gt;.mp3  
  3164554 &lt;artist name&gt;/&lt;artist name&gt; - &lt;song name&gt; (1).mp3  
  3337687 &lt;artist name&gt;/&lt;artist name&gt; - &lt;song name&gt; (2).mp3  
  3130248 &lt;artist name&gt;/&lt;artist name&gt; - &lt;song name&gt; (3).mp3  
  </code></pre>
  
  <p>I tried to use <code>fdupes</code>, but it is not working for me because the mp3 are not exactly the same files.<br>
  Often, each "duplicate" is a slightly different version, sometimes the quality is different, sometimes the file is corrupted sometimes the song is for a different album.</p>
  
  <p>I don't need to have multiple copies of the same file for each album.</p>
  
  <p>What I was planning in doing is to identify the biggest file  in size (<em>biggest always better! ;)</em> ) among the duplicates and delete the rest.</p>
  
  <p>I am still learning bash so I decided to go a step at the time, but I got blocked.<br>
  Any help would be appreciated.</p>
  
  <p>Here is what i have so far:</p>
  
  <pre class="lang-sh prettyprint-override"><code>#!/bin/bash
  tempfile="duplicates.tmp"
  tempfilesorted="duplicatessorted.tmp"
  
  # with the following I am identifying every duplicate and create a document with the list
  find . -regex '.* ([0-99]).*' &gt; $tempfile
  # in $tempfile we would find something like
  # ./&lt;artist_1&gt;/&lt;artist_1&gt; - &lt;song_name&gt; (1).mp3
  # ./&lt;artist_2&gt;/&lt;artist_2&gt; - &lt;song_name&gt; (1).mp3
  # ./&lt;artist_1&gt;/&lt;artist_1&gt; - &lt;song_name&gt; (2).mp3
  
  # I am removing the (xx) numbering
  sed -i -e 's/ ([0-99])//g' $tempfile
  # in $tempfile we would find something like
  # ./&lt;artist_1&gt;/&lt;artist_1&gt; - &lt;song_name&gt;.mp3
  # ./&lt;artist_2&gt;/&lt;artist_2&gt; - &lt;song_name&gt;.mp3
  # ./&lt;artist_1&gt;/&lt;artist_1&gt; - &lt;song_name&gt;.mp3
  
  # sorting and leave the unique entrances
  sort $tempfile | uniq -u &gt; $tempfilesorted
  # example of result in $tempfilesorted
  # ./&lt;artist_1&gt;/&lt;artist_1&gt; - &lt;song_name&gt;.mp3
  # ./&lt;artist_2&gt;/&lt;artist_2&gt; - &lt;song_name&gt;.mp3
  
  # adding * at the end of the file name before the extension
  sed -i -r 's/.mp3/*.mp3/' $tempfilesorted
  # example of result in $tempfilesorted
  # ./&lt;artist_1&gt;/&lt;artist_1&gt; - &lt;song_name&gt;*.mp3
  # ./&lt;artist_2&gt;/&lt;artist_2&gt; - &lt;song_name&gt;*.mp3
  
  # my plan from here was simple:
  # identify the bigger size and delete the remaining ones
  while IFS= read -r line; do
  #    ls -S "$line"
      echo "processing: $line"
  done &lt;  $tempfilesorted
  </code></pre>
  
  <p>Any suggestion?</p>
  
  <hr>
  
  <p><strong>UPDATE</strong></p>
  
  <p>Thanks to @Aaron I have updated the previous script with the following solution:</p>
  
  <pre class="lang-sh prettyprint-override"><code>#!/bin/bash
  tempfile="duplicates.tmp"
  tempfilesorted="duplicatessorted.tmp"
  
  # with the following I am identifying every duplicate and create a document with the list
  find . -regex '.* ([0-99]).*' &gt; $tempfile
  
  # I am removing the (x) numbering
  sed -i -e 's/ ([0-9])//g' $tempfile
  
  # sorting and leave the unique entrances
  sort $tempfile &gt; $tempfilesorted
  # TO DO, I have no idea but the original solution using uniq:
  # sort $tempfile | uniq -u &gt; $tempfilesorted
  # was skipping few lines, I will find a solution and update
  
  # adding * at the end of the file name before the extension
  sed -i -r 's/.mp3/*.mp3/' $tempfilesorted
  # TO DO: modify so it works for any type of file, not just mp3
  
  # Identify and rename temporary the bigger file
  while IFS= read -r line; do
      echo "processing: $line"
      {
         # read the biggest file from the first line
         IFS=$'\007' read -r biggest_file its_size
         #mv -f "$biggest_file" "${biggest_file/ ([0-9]*).mp3/$'\177'.mp3}"
         mv -f "$biggest_file" "${biggest_file/ ([0-9]*)./OWpfUWNOdC.}"
  
         # read other files
         while IFS=$'\007' read -r other_file its_size; do
              rm "${other_file}"
         done
      } &lt; &lt;(find . -wholename "$line" -printf '%p\007%s\n' | sort -grt $'\007' -k 2)
  done &lt;  $tempfilesorted
  
  # removing the added random string that was used to identify the bigger file
  #find -type f -wholename "*OWpfUWNOdC*" -execdir  rename  's/$'\177'.mp3/.mp3/' '{}' \;
  find . -type f -wholename "*OWpfUWNOdC*" -execdir  rename  's/OWpfUWNOdC././' '{}' \;
  </code></pre>
  
  <p><em>information of why some amendments from the original proposed code by @Aaron :</em></p>
  
  <ol>
  <li>In the following scenario</li>
  </ol>
  
  <pre><code> size    file_path_and_name  
  3130248 &lt;artist name&gt;/&lt;artist name&gt; - &lt;song name&gt;.mp3  
  3130248 &lt;artist name&gt;/&lt;artist name&gt; - &lt;song name&gt; (1).mp3 
  </code></pre>
  
  <p>the original solution was deleting both files.
  To solve this instead of moving the file to itself I have temporary renamed the file with a random string and in a second run finding the string and removing it.</p>
  
  <ol start="2">
  <li>I have modified the <code>find . -name</code> to <code>find . -wholename</code> because there were spaces in the file names</li>
  </ol>
  
  <hr>
  
  <p><strong>strange stuff still happening:</strong></p>
  
  <p>If I try to sort and uniq the following file, I will lose few entrances.<br>
  <a href="https://pastebin.com/8yvied22" rel="nofollow noreferrer">https://pastebin.com/8yvied22</a><br>
  Any ideas?</p>
  
  <hr>
  
  <p><strong>help with the following</strong></p>
  
  <p>As you can see in the code, I have tried to use the following symbol <code>$'\177'</code> instead of a random string.<br>
  I am able to find the files, but I cannot rename them.<br>
  You can find my proposed solution commented in the updated code.<br>
  Any suggestion? </p>