<p dir="auto">one of my folder contains multiple h5 files, and I tried to load them into dataframes and then concat these df into one.</p>
<p dir="auto">the python process crashes when the num_tasks&gt;1, if I debug thread by thread, it works, in another, it crashes simply when two threads run at the same time, even though they read different files.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from multiprocessing.pool import ThreadPool
import pandas as pd 

num_tasks=2
def readjob(x):
    path = x
    return pd.read_hdf(path,&quot;df&quot;,mode='r')

pool = ThreadPool(num_tasks)
results = pool.map(readjob,files)"><pre class="notranslate"><code class="notranslate">from multiprocessing.pool import ThreadPool
import pandas as pd 

num_tasks=2
def readjob(x):
    path = x
    return pd.read_hdf(path,"df",mode='r')

pool = ThreadPool(num_tasks)
results = pool.map(readjob,files)
</code></pre></div>
<p dir="auto">xref: <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="8870235" data-permission-text="Title is private" data-url="https://github.com/pandas-dev/pandas/issues/2397" data-hovercard-type="issue" data-hovercard-url="/pandas-dev/pandas/issues/2397/hovercard" href="https://github.com/pandas-dev/pandas/issues/2397">#2397</a><br>
xref <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="178136775" data-permission-text="Title is private" data-url="https://github.com/pandas-dev/pandas/issues/14263" data-hovercard-type="issue" data-hovercard-url="/pandas-dev/pandas/issues/14263/hovercard" href="https://github.com/pandas-dev/pandas/issues/14263">#14263</a> (example)</p>
<p dir="auto">I <em>think</em> that we should protect the file open/closes with a lock to avoid this problem.</p>