Delete duplicate files I have this function which looks through ALL files found in a directory (including sub folders) and deleted the duplicated file.

private static void DeleteDuplicateFiles(string directoryPath)
    {
        Console.WriteLine("Finding files...");

        // get all files
        var allFiles = Directory.GetFiles(directoryPath, "*.*", SearchOption.AllDirectories);
        Console.WriteLine("Files found: {0}", allFiles.Length);

        // get duplicate files
        var dupFiles = allFiles.Select(f =&gt;
        {
            using (var fs = new FileStream(f, FileMode.Open, FileAccess.Read))
            {
                return new
                {
                    FileName = f,
                    FileHash = BitConverter.ToString(SHA1.Create().ComputeHash(fs))
                };
            }
        }).GroupBy(f =&gt; f.FileHash).Select(g =&gt; new { FileHash = g.Key, Files = g.Select(z =&gt; z.FileName).ToList() })
        .SelectMany(f =&gt; f.Files.Skip(1)).ToList();
        Console.WriteLine("Duplicate files: {0}", dupFiles.Count);

        // delete deuplicate files
        dupFiles.ForEach(File.Delete);

        Console.WriteLine("{0} files were deleted", dupFiles.Count);

    }


This algorithm isn't the most efficient obviously, but what I have in mind is to:


Group files by their size (so files with unique size wont get processed)
Calculate the hash of the remaining files.


Is this the best solution or is there a better way of doing it?
To search files you're using Directory.GetFiles(), nothing wrong with that but, especially in conjunction with SearchOptions.AllDirectories, it may take some time to search for them all. You should switch to Directory.EnumerateFiles() because you can consume enumeration immediately and search will progress on-the-fly (in this case you do not give any progress feedback then you do not need to know the total number of files to process in advance).

It has few advantages:


You can start processing immediately, reducing the total perceived execution time.
You use less memory than reading the whole list (not such an issue nowadays).
It's easy to be parallelized.


And we then arrive to the second point: parallelization. It's always tricky to make I/O bound code parallel but in this case you have two main tasks:


Read file content.
Calculate hash.


Even a very simple parallel approach will benefit from overlapping next read operation with previous hash calculation. Roughly something like this:

var allFiles = Directory.EnumerateFiles(directoryPath, "*.*", SearchOption.AllDirectories);

var hashedFiles = allFiles.AsParallel()
    .Select(x =&gt; new { Path = x, Hash = CalculateHash(x) });


You may want to play with options to limit the maximum number of concurrent reads. Also note that I moved the hash calculation in its own separate function. I'd do the same also for the other tasks, it's true that this method is pretty small but you will gain in clarity, proof of concept:

public static void DeleteDuplicateFiles(string searchPath)
{
    foreach (var file in FindDuplicateFiles(searchPath))
        TryDeleteFile(file);
}

private static IEnumerable&lt;string&gt; FindDuplicateFiles(string searchPath)
{
    var hashedFiles = FindAllFiles(searchPath)
        .AsParallel()
        .Select(x =&gt; new { Path = x, Hash = CalculateHash(x) });

    return FindDuplicates(hashedFiles);
}


And so on, I think you get what I mean. Some of those functions may also be reused and to understand overall algorithm you just need to read few lines of code, no need to dig into implementation details.

I hope you noted I used a TryDeleteFile() function instead of directly calling File.Delete(). This is because I/O operations may fail and error may be transitory (for example another program or Windows Shell may keep temporarily keep the file open (without sharing for deletion) but it may be deleted few seconds later. In this function you simply try to delete the file and if you get an IOException with HRESULT for file in use then you simply wait and retry few times.

One last thing: do not mix your logic with presentation. This code can be easily reused outside a console application. You may also want to provide a different output (for example coloring messages) but if you mix them together then any change will be a pain. Introduce an interface IProgressWriter for example like this:

public interface IProgressWriter
{
    void Trace(string message);
    void Error(string messsage);
}


In this case I wrote a somehow generic one but you might want a specialized one (for example to later switch to a GUI application without touching a single line of code in your logic). Note that you can do it both with an interface and with events, like this:

public sealed class DuplicatedFilesSleuth
{
    public string SearchPath { get; set; }

    public void FindDuplicates() { ... }

    public void DeleteDuplicates() { ... }

    public event EventHandler&lt;DuplicatedFilesSleuthEventArgs&gt; Hashing;

    public event EventHandler&lt;DuplicatedFilesSleuthEventArgs&gt; Deleting;
}


Note that, as t3chb0t pointed out in a comment, if you do not expose callable methods but data then you should use IProgress&lt;T&gt;/Progress&lt;T&gt; instead of another custom interface.