Tracking 1400+ client codes, multi-threading nightmare I am looking for a bit of help. I am having to cycle through 1400 clients code on my server, I have to check what Version of the software they are on and check for customization's to the code so when we apply updates, those aren't lost. I tried to do it serialized, as you might have guessed, that took forever. I then Tried running it parallel, as you might have expected, I am getting &quot;Os Error, too many open files&quot; I tried to resolve this by throwing in &quot;sleep(random())&quot; Which Works,  But this seems convoluted and slows down the process considerably, and it could also be my code is complete garbage, any insight would be great.
I am using git in CLI to get version, this seems to be heavy? Is there a better way to pull tags?
Here, I get a list of the clients, I do some editing to get the pathing
def full_check(client_list, cur_vers):
    for c in client_list:
        if sc.venue_check(c):
            url = sc.surl(c[0])
            if url is not None:
                path = f'/home/ubuntu/site-files/{url}'
                # Maximizing output, this cut run time by 90%
                # Hard to log though...

                nschools.append({
                    'path': path,
                    'cur_vers': cur_vers,
                    'client': c[0],
                    'schema_version': c[2]
                })
# Here, I am then looping through the paths I get, I put a sleep to stop it from bugging out
# about OS File open Limits
    for s in nschools:
        print(s['path'])
        sleep(random())
        t = threading.Thread(target=vers_check, args=(s['path'], s['cur_vers'], s['client'], s['schema_version']))
        t.start()

Here I am going through doing some version checking
def vers_check(path, cur_vers, client, schema_version):
    &quot;&quot;&quot;
    This Checks the Schools Release.
    TODO Add logging and Return Functionality.
    :param client:
    :param schema_version:
    :param path:
    :param school:
    :param cur_vers:
    :return:
    &quot;&quot;&quot;
    os.chdir(path)
    try:
        vers = subprocess.check_output([&quot;git&quot;, &quot;describe&quot;, &quot;--tags&quot;]).strip().decode('utf-8')

        raw = str(vers).strip('-')
        rawsplit = raw.split('-')
        st = rawsplit[0]

        if str(st) != cur_vers:
            cschools.append({
                'client': client,
                'schema_version': schema_version,
                'sw_version': st,
                'path': path,
                'is_update': False
            })
            log.log('warn', 'Version Check', f'The School {client} is on Software Version {st} and Schema Version {schema_version} Which is Out Of Date. Current Version is {cur_vers}')
        else:
            cschools.append({
                'client': client,
                'schema_version': schema_version,
                'sw_version': st,
                'path': path,
                'is_update': True
            })
            log.log('info', 'Version Check', f'The School is on Software Version {st} and Schema Version {schema_version} Which is Up To Date. Current Version is {cur_vers}')
    except subprocess.CalledProcessError as e:
        log.log('err', 'Failed Version Check', f'We were unable to get the version for {client}', e)
```
First, I'm pretty sure that it's impossible for
os.chdir(path)

to meaningfully apply different working directories per thread; and even if it were possible it would be a bad idea. subprocess supports a cwd parameter directly that you should use.
If you were opening files yourself, you would want to make a semaphore where the lock only applies to the section of the code where the file is opened. However, you aren't opening the files - it's probably your invocation of git that opens the most files.
I think the only practical way around this is to just limit the number of concurrent threads. You could either estimate this ahead of time, or keep spinning up threads until you hit the first OSError and start no new threads, running one-out-one-in until your work pool is complete.
If you deeply care about performance, subprocess-ing out to git should be replaced with in-process computation; either calling into a library or manually processing entries in .git (I have not researched either).