Run dir scan as fast and efficient as possible / clean code I'm not a very experience python programmer. But I want to make my code as fast and efficient as possible as well as write it clean and therefore got the advice to share it here. So I'm having the following code, which works well but isn't very fast as I have dirs bigger than 4TB and I'm executing that code in my network. So I'm looking for advice, to read out all data in one path instead of doing redundant second scanning of dirs and files. Any advice would be appreciated! 
I'm also thinking of how to implement multiprocessing but I think that wouldn't help much as a lot is IO.

def get_size_and_fcount_pathlib(scan_path):
    """Gets the total size of given dir and counts how many folders and files are in the given
    path directory and return a file count, folder count and all types as a sum"""
    root_directory = Path(scan_path)
    total_size = 0
    all_types_count = 0
    file_count = 0
    folder_count = 0
    for f in root_directory.glob('**/*'):
        if f.is_file():
            file_count += 1
            total_size += f.stat().st_size
            if not str(f.name).startswith("."):
                all_types_count += 1
        if f.is_dir():
            folder_count += 1
    size_gb = ", ".join(map(str, (round(total_size/1000/1000/1000, 2), 'GB'))).replace(', ', '')

    print('Amount of all types searched: {}'.format(all_types_count))
    print('Amount of files searched:     {}'.format(file_count))
    print('Amount of folders searched:   {}'.format(folder_count))
    print('Directory size in GB:         {}'.format(size_gb))

    file_count_collection = [size_gb, all_types_count, file_count, folder_count]
    return file_count_collection
Naming things

Be consistent in the way you name things. This applies to the words chosen, whether you abbreviate things or not, whether you separate multiple words or not. There are some names you are using that are not consistent. For example:

def get_size_and_fcount_pathlib(scan_path):


You start with fully spelled out words separated by underscores, and then suddenly there is fcount, two words without separators and with one word being abbreviated. I would start by writing this out consistently as:

def get_size_and_file_count_pathlib(scan_path):


If you want to reduce the size of the name, I would rather omit redundant parts of it. Is pathlib really adding anything to this? If get_size_and_file_count() gives you enough context to deduce what the function does, I would go for that.

Another issue is with this variable:

file_count_collection = [size_gb, all_types_count, file_count, folder_count]


It says it is a file count collection, but it also contains the total size. I would change this to size_and_file_count_collection, to make it clear it also contains the size, and this also better matches the name of the function. Again, if you want to make it shorter, I would then drop _collection from the name, since size_and_file_count already implies it is more than one bit of information.

Arguably, size_and_file_count is also not completely descriptive since it also includes directory count and a count of all types of filesystem entities.
There might be an even better name for this, perhaps directory_statistics?

Use a class to represent structured data

The file_count_collection is just a list. Unless you read the code of this function, it is not obvious in which order you stored the size and counts in this list. It is better to create a class that represents this collection of data, where each piece will get its own name, like so:

class size_and_file_count:
   def __init__(self, size_gb, all_types_count, file_count, folder_count):
       self.size_gb = size_gb
       self.all_types_count = all_types_count
       self.file_count = file_count
       self.folder_count = folder_count


Then, in get_size_and_file_count(), you can replace the last two lines with:

return size_and_file_count(size_gb, all_types_count, file_count, folder_count)


And in the code that calls that function, instead of writing:

result = get_size_and_file_count("...")
file_count = result[2]


You can now write:

result = get_size_and_file_count("...")
file_count = result.file_count


Don't convert values too early

You are calculating the total size, and then converting it to a value in gigabytes (with only 2 decimals), and only storing that final result. What if I want to get the size of a directory that contains less than 10 megabytes of data? Your function will tell me it is 0.00 gigabytes big, which is not helpful. It is better to store the size in bytes, and only convert to gigabytes when printing the result, like so:

print('Directory size in GB:         {:.2f} GB'.format(total_size / 1.0e9))


And of course also store the total size in bytes in the returned collection.

Separate obtaining results from printing results

Your function does two things at once: it calculates the results and it prints them. It is best to separate the two issues, and have get_size_and_file_count() only calculate the results and return them, and then have a seperate function that takes a size_and_file_count variables as a parameter and prints the values to the screen.

If you have made a class size_and_file_count, you could make the function that prints the results a member function of that class.

What is all_types_count?

There are many different types of filesystem entries besides regular files and directories; there are fifos, sockets, symlinks, character devices and so on.
I would expect a variable named all_types_count to count all of them. However, in your code you actually disregard all things that are neither regular file nor directory, and instead all_types_count just counts those regular files whose name does not start with .. So actually your all_types_count is just a non_hidden_files_count. Either this was your intention, in which case you should rename the variable, or you actually wanted to count all filesystem items, in which case your count is wrong.

Optimizing the code

If you want to make the code as fast as possible, there are several ways to do that. First, I would start by replacing the call to glob() with os.walk(). Since you want all files and directories anyway, the glob function is likely incurring an unnecessary overhead of comparing each item it finds against the glob pattern you gave. Also, os.walk() already splits the results for each directory in a list of file names and list of directory names, making your life a bit easier. It would look like:

for root, dirs, files in os.walk(root_directory):
    folder_count += len(dirs)
    for f in files:
        if Path(root, f).is_file():
            ...


If you want more speed, then there are two other ways to consider, that can even be combined. First, write the function in C, and make a Python wrapper function around the C code. Second, you could try to paralellize the code, for example by scanning multiple directories in parallel. However, the latter is a lot of work, and there is a good chance it will not be faster at all, since you might actually not be CPU bound but I/O bound, and even if you are CPU bound, then the overhead of spawning multiple threads or tasks and synchronizing them might be more than the performance gain from the actual parallel execution of the code.

When you are interested in optimizing the code, start with benchmarking it, and then see if the suggestions actually help with performance. Also, before trying to parallelize the code, actually check if your Python code is actually using 100% CPU when scanning directories. If not, it's not worth going that route.