File duplicate finder In an attempt to clean out my picture collection I made a script to find all duplicate files

import datetime
import hashlib
import json
import typing
from collections import defaultdict
from pathlib import Path


def filesize(path: Path) -&gt; int:
    """returns the filesize"""
    return path.stat().st_size


def hash_first(
    path: Path, hash_function=hashlib.md5, blocksize: int = 65536
) -&gt; str:
    """returns the hash of the first block of the file"""
    with path.open("rb") as afile:
        return hash_function(afile.read(blocksize)).hexdigest()


def hash_all(path: Path, hash_function=hashlib.md5, blocksize=65536) -&gt; str:
    """returns the hash of the whole file"""
    with path.open("rb") as afile:
        hasher = hash_function()
        buf = afile.read(blocksize)
        while buf:
            hasher.update(buf)
            buf = afile.read(blocksize)
    return hasher.hexdigest()


def compare_files(
    comparison: typing.Callable[[Path], typing.Hashable], duplicates: dict
) -&gt; dict:
    """
    Subdivides each group along `comparison`
    discards subgroups with less than 2 items
    """
    results: defaultdict = defaultdict(list)
    for old_hash, files in duplicates.items():
        for file in files:
            results[(*old_hash, comparison(file))].append(file)

    return {
        filehash: files
        for filehash, files in results.items()
        if len(files) &gt; 1
    }


def find_duplicates(
    rootdir: Path,
    comparisons: typing.Iterable[
        typing.Callable[[Path], typing.Hashable]
    ] = (),
):
    """
    Finds duplicate files in `rootdir` and its subdirectories

    Returns a list with subgroups of identical files

    Groups the files along the each of the comparisons in turn.
    Subgroups with less than 2 items are discarded.

    Each of the `comparisons` should be a callable that accepts a
    `pathlib.Path` as only argument, and returns a hashable value

    if `comparisons` is not defined, compares along:
        1. file size
        2. MD5 hash of the first block (65536 bytes)
        3. MD5 hash of the whole file
    """
    if not comparisons:
        comparisons = filesize, hash_first, hash_all

    duplicates = {(): filter(Path.is_file, rootdir.glob("**/*"))}
    for comparison in comparisons:
        duplicates = compare_files(comparison, duplicates)
    return [tuple(map(str, files)) for files in duplicates.values()]


if __name__ == "__main__":
    pic_dir = r"./testdata"
    results = find_duplicates(rootdir=Path(pic_dir))

    # print(json.dumps(results, indent=2))

    timestamp = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
    output_file = Path("./output") / f"{timestamp}.json"
    with output_file.open("w") as fh:
        json.dump(results, fh, indent=2)


Any remarks on this code?

The code is formatted with black (linelength 79), with mypy pylint and pylama as linters.

If I try to specify the dict as argument and return value to compare_files to typing.Dict[typing.Tuple[typing.Hashable, ...], typing.Iterable[Path]], mypy complains: Incompatible types in assignment (expression has type "Dict[Tuple[Hashable, ...], Iterable[Path]]", variable has type "Dict[Tuple[], Iterator[Path]]")
Some suggestions:


I would usually inline single line functions unless what they do is hard to understand from just reading the command. filesize might be one such candidate.
I believe that not specifying the block size can be faster, by allowing Python to use whatever block size it thinks is appropriate. This would need some actual testing though.
Your function arguments are only partially typed. You might want to use a stricter mypy configuration, such as the following, in your setup.cfg:

[mypy]
check_untyped_defs = true
disallow_untyped_defs = true
ignore_missing_imports = true
no_implicit_optional = true
warn_redundant_casts = true
warn_return_any = true
warn_unused_ignores = true

MD5 was optimized for cryptographic use, not for speed. Another Q&amp;A explores options for fast non-cryptographic hashing.
I would avoid converting digests to strings - it might be slightly easier for debugging purposes, but data type conversions are in general very costly.
Optional parameters mean you have to test at least two things rather than one. Your code only ever calls the hash_* functions without optional parameters, so you might as well either inline them.
On a related note, static values which are used in multiple places are perfect for pulling out as constants. The hashing function and block size would be obvious candidates for this.
Mutable default parameter values are an accident waiting to happen. The default pattern for this is to use a default of None and to assign a default if foo is None.
r-strings like r"./testdata" are meant for regular expressions. Since pic_dir is not used as such it should probably be a regular string.
duplicates is at first a list of potential duplicates and later is trimmed in stages. This makes the code hard to follow.
Things like the directory/directories to include should be arguments. argparse can deal with this easily.
To make this more scriptable I would print the result on standard output. It's trivial to then redirect it to a file.