title: Why is h5py.h5f.create failing with OSError: unable to open file: name ="name",
body: <p>I am getting random failures to create the callback model weights file  when running Keras (with tensorflow) models in a loop with changing model parameters and input data. I am thinking that it might be something to do with the length of my directory name but if so, it seems like a bug as it only happens sometimes. Prior to the following error it wrote multiple files with same length directories. I am using long names for directories to make it easier to distinguish runs in tensorboard. I will show my basic code setup as pseudo code and then the random error that I am getting. I have a nested for loop that is changing model parameters as well as input data. The basic loop will work fine for hours and then randomly fail for the same error at some point in the loop. I would like to know if I am doing something wrong in my file name that is causing this. I would also like a work around so that when it fails, I can keep running and move on to the next file and skip the one that is failing. Some type of try/except but I don't know enough about h5py to know how to code that. I am running on Windows 10 (conda env), tensorflow-gpu 1.6.0, Keras 2.1.5, h5py 2.7.1, tensorboard 1.6.0. I also set up Windows 10 to handle long file names. This error seems to be coming directly from h5py (h5py\h5f.pyx). Also, the file actually gets created and written. I can load the file using h5py.File() and it is the correct size and has same groups and objects. Update: I included the os.makedirs() line in my code that I did not show before. I also added a check on the directory creation and ran the code again. It still failed in same way and it never triggered the isdir() check. Update 2: Wanted to point out that I am using multiprocessing because of memory leaks when using Keras with Tensorflow. This happens regardless of having K.clear_session() and tf.reset_default_graph(). I now believe this random error is related to multiprocessing as I have not observed this error yet when I eliminate the pooling process. </p>
  
  <pre><code>def main():
      for input_data in input_data_list:
          for model_parameters in model_parameters_list:
              # run model with different parameters on all data
              pool = multiprocessing.Pool(1)
              pool.apply(run_function,run_parameters...,model_func_name,
                         model_func_dict)
              pool.close()
  
  def run_function(run_parameters...,model_func_name,model_func_dict,...):
      # code to extract x_train,y_train, x_val, y_val etc not shown
      # model_def = long string representing model parameters example below
      # model_def =
      # 'basic_ff_nn4_mse_dr50_Nadam_LeakyReLU_kr_l2_ar_off_ns_0_BCtoA_all_2_2'
      # build and compile model
      model = model_func_name(**model_func_dict)
      # set up callbacks
      os.makedirs(models_dir + "{}_{}_{}_{}/".format(model_def, set_name, 
                   fold, set_num), exist_ok=True)
      tmp_path = models_dir + "{}_{}_{}_{}/".format(model_def, set_name, fold, 
                                                     set_num)
      best_weights_file = models_dir + "{}_{}_{}_{}/best_weights.hdf5".format(
          model_def, set_name, fold, set_num)
      best_model_weights = callbacks.ModelCheckpoint(best_weights_file,
                                                     save_best_only=True,
                                                     save_weights_only=True)
      log_dir = 'output/{}_{}/tf_logs/{}/{}/{}'.format(model_type, cur_time,
                                                       model_def, set_name,
                                                       'f' + str(fold))
      tensorboard = callbacks.TensorBoard(log_dir=log_dir,
                                          histogram_freq=0, write_graph=False,
                                          write_images=False, 
                                           write_grads=False)
      if not os.path.isdir(tmp_path):
          print('path not created = ',tmp_path)
      model_history = model.fit(x=x_train, y=y_train,
                                verbose=0,
                                batch_size=size_batches,
                                epochs=num_epochs,
                                validation_data=[x_val, y_val],
                                callbacks=[best_model_weights, tensorboard],
                                )
      K.clear_session()
      tf.reset_default_graph()
  
  multiprocessing.pool.RemoteTraceback:
  """
  Traceback (most recent call last):
    File "C:\ProgramData\Miniconda3\envs\tflow_g\lib\multiprocessing\pool.py", line 119, in worker
      result = (True, func(*args, **kwds))
    File "C:\Users\xxxxx\Dropbox (test)\Lab\VLL models\zakworkspace\cps\cps_main.py", line 1042, in run_joint_ff
      callbacks=[best_model_weights, tensorboard],
    File "C:\ProgramData\Miniconda3\envs\tflow_g\lib\site-packages\keras\models.py", line 963, in fit
      validation_steps=validation_steps)
    File "C:\ProgramData\Miniconda3\envs\tflow_g\lib\site-packages\keras\engine\training.py", line 1705, in fit
      validation_steps=validation_steps)
    File "C:\ProgramData\Miniconda3\envs\tflow_g\lib\site-packages\keras\engine\training.py", line 1255, in _fit_loop
      callbacks.on_epoch_end(epoch, epoch_logs)
    File "C:\ProgramData\Miniconda3\envs\tflow_g\lib\site-packages\keras\callbacks.py", line 77, in on_epoch_end
      callback.on_epoch_end(epoch, logs)
    File "C:\ProgramData\Miniconda3\envs\tflow_g\lib\site-packages\keras\callbacks.py", line 445, in on_epoch_end
      self.model.save_weights(filepath, overwrite=True)
    File "C:\ProgramData\Miniconda3\envs\tflow_g\lib\site-packages\keras\models.py", line 754, in save_weights
      with h5py.File(filepath, 'w') as f:
    File "C:\ProgramData\Miniconda3\envs\tflow_g\lib\site-packages\h5py\_hl\files.py", line 269, in __init__
      fid = make_fid(name, mode, userblock_size, fapl, swmr=swmr)
    File "C:\ProgramData\Miniconda3\envs\tflow_g\lib\site-packages\h5py\_hl\files.py", line 105, in make_fid
      fid = h5f.create(name, h5f.ACC_TRUNC, fapl=fapl, fcpl=fcpl)
    File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
    File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
    File "h5py\h5f.pyx", line 98, in h5py.h5f.create
  OSError: Unable to create file (unable to open file: name = 'output/TB_runs_03122018-031837/dump/models/basic_ff_nn4_mse_dr50_Nadam_LeakyReLU_kr_l2_ar_off_ns_0_BCtoA_all_2_2/best_weights.hdf5', errno = 22, error message = 'Invalid argument', flags = 13, o_flags = 302)
  """
  </code></pre>