title: Loading a custom dataset from json annotations files for Keras classification task
body: <p>I am new to deep learning and would like to implement a simple classification task using Keras. My dataset contains over 2000 images &amp; for each image I have a respective json file which contains the label for that image. Following is the code to load the json files &amp; create the X (image) &amp; Y (labels) arrays:</p>
  
  <pre><code>X = []
  Y = []
  with concurrent.futures.ProcessPoolExecutor() as executor:
      # Get a list of files to process
      str = jsonpath + '/*.json'
      #print(str)
      json_files = glob.glob(str)
      for jsonfile,y in zip(json_files, executor.map(create_array, json_files)):
          X.append(y[0])
          Y.append(y[1])
  </code></pre>
  
  <p>where the function create_array is defined as follows:</p>
  
  <pre><code>def create_array(jsonfile):
      array_list = []
      y_list = []
      with open(jsonfile) as f:
              data = json.load(f)
              name = data['annotation']['data_filename']
              img = cv2.imread(imgDIR + '/' + name) 
              array_list.append(img)      
              l = data['annotation']['data_annotation']['classification'][0]['classification_label']
              y_list.append(l)
  return array_list, y_list
  </code></pre>
  
  <p>It works for small no of images say 15, but for the entire set of 2000 images, the program gets automatically killed or sometimes it gives the error "MemoryError: out of memory".
  Is there an efficient way to do this? How can I speed up this data pre-processing part to give it as an input to the keras classification model?</p>