title: Tensorflow / keras: How to train CNN with custom image files (.FITS images)
body: <p>I'm currently trying to train a CNN (EfficientNet) using .FITS image files, but I believe this would apply to other image types also. This kind of image requires the library <strong>astropy</strong> in order to be opened, and in my case, in order to access the image data, I simply type:</p>
  <pre class="lang-py prettyprint-override"><code>from astropy.io import fits
  path = &quot;path/to/file.fits&quot;
  hdul = fits.open(path)
  image = hdul[1].data
  </code></pre>
  <p>This variable <code>image</code> will then have type <code>numpy.ndarray</code>. I first tried using keras' <code>image_dataset_from_directory</code> and, as expected, it didn't work out. I then checked out <code>tf.data</code> over here: <a href="https://www.tensorflow.org/tutorials/load_data/images#using_tfdata_for_finer_control" rel="nofollow noreferrer">https://www.tensorflow.org/tutorials/load_data/images#using_tfdata_for_finer_control</a>. I tried creating a similar pipeline and it all worked out up until the <code>decode_img</code> function. Since I'm not dealing with a jpeg I tried making a workaround so that I get:</p>
  <pre class="lang-py prettyprint-override"><code>data_dir = home/astro/train
  class_names = np.array(sorted([item.name for item in data_dir.glob('*')]))
  # class_names = [&quot;stars&quot;, &quot;galaxies&quot;]
  
  def get_label(file_path):
      parts = tf.strings.split(file_path, os.path.sep)
      one_hot = parts[-2] == class_names
      return tf.argmax(one_hot)
  
  def decode_img(img):
      hdul = fits.open(img)
      data = hdul[1].data
      data = data.reshape((data.shape[0], data.shape[1], 1))
      data = np.pad(data, [(0,0), (0,0), (0, 2)], 'constant') # padding to create 3 channels
      img = tf.convert_to_tensor(data, np.float32)
      return tf.image.resize(img, [img_height, img_width])
  
  def process_path(file_path):
      label = get_label(file_path)
      img = decode_img(file_path)
      return img, label
  </code></pre>
  <p>And it actually works out pretty well, in a way that when I print <code>process_path</code> I get two tensors, one for the image and one for the label with the correct shapes and values that I want.</p>
  <p><strong>THE PROBLEM:</strong></p>
  <p>By following the tutorial, when I get to:</p>
  <pre><code>AUTOTUNE = tf.data.experimental.AUTOTUNE
  train_ds = train_ds.map(process_path, num_parallel_calls=AUTOTUNE)
  val_ds = val_ds.map(process_path, num_parallel_calls=AUTOTUNE)
  </code></pre>
  <p>I get the following error:</p>
  <pre><code>TypeError                                 Traceback (most recent call last)
   in 
        1 AUTOTUNE = tf.data.experimental.AUTOTUNE
        2 
  ----&gt; 3 train_ds = train_ds.map(process_path, num_parallel_calls=AUTOTUNE)
        4 val_ds = val_ds.map(process_path, num_parallel_calls=AUTOTUNE)
  
  ~/anaconda3/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py in map(self, map_func, num_parallel_calls, deterministic)
     1700           num_parallel_calls,
     1701           deterministic,
  -&gt; 1702           preserve_cardinality=True)
     1703 
     1704   def flat_map(self, map_func):
  
  ~/anaconda3/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py in __init__(self, input_dataset, map_func, num_parallel_calls, deterministic, use_inter_op_parallelism, preserve_cardinality, use_legacy_function)
     4082         self._transformation_name(),
     4083         dataset=input_dataset,
  -&gt; 4084         use_legacy_function=use_legacy_function)
     4085     if deterministic is None:
     4086       self._deterministic = &quot;default&quot;
  
  ~/anaconda3/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py in __init__(self, func, transformation_name, dataset, input_classes, input_shapes, input_types, input_structure, add_to_graph, use_legacy_function, defun_kwargs)
     3369       with tracking.resource_tracker_scope(resource_tracker):
     3370         # TODO(b/141462134): Switch to using garbage collection.
  -&gt; 3371         self._function = wrapper_fn.get_concrete_function()
     3372         if add_to_graph:
     3373           self._function.add_to_graph(ops.get_default_graph())
  
  ~/anaconda3/lib/python3.7/site-packages/tensorflow/python/eager/function.py in get_concrete_function(self, *args, **kwargs)
     2937     &quot;&quot;&quot;
     2938     graph_function = self._get_concrete_function_garbage_collected(
  -&gt; 2939         *args, **kwargs)
     2940     graph_function._garbage_collector.release()  # pylint: disable=protected-access
     2941     return graph_function
  
  ~/anaconda3/lib/python3.7/site-packages/tensorflow/python/eager/function.py in _get_concrete_function_garbage_collected(self, *args, **kwargs)
     2904       args, kwargs = None, None
     2905     with self._lock:
  -&gt; 2906       graph_function, args, kwargs = self._maybe_define_function(args, kwargs)
     2907       seen_names = set()
     2908       captured = object_identity.ObjectIdentitySet(
  
  ~/anaconda3/lib/python3.7/site-packages/tensorflow/python/eager/function.py in _maybe_define_function(self, args, kwargs)
     3211 
     3212       self._function_cache.missed.add(call_context_key)
  -&gt; 3213       graph_function = self._create_graph_function(args, kwargs)
     3214       self._function_cache.primary[cache_key] = graph_function
     3215       return graph_function, args, kwargs
  
  ~/anaconda3/lib/python3.7/site-packages/tensorflow/python/eager/function.py in _create_graph_function(self, args, kwargs, override_flat_arg_shapes)
     3073             arg_names=arg_names,
     3074             override_flat_arg_shapes=override_flat_arg_shapes,
  -&gt; 3075             capture_by_value=self._capture_by_value),
     3076         self._function_attributes,
     3077         function_spec=self.function_spec,
  
  ~/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes)
      984         _, original_func = tf_decorator.unwrap(python_func)
      985 
  --&gt; 986       func_outputs = python_func(*func_args, **func_kwargs)
      987 
      988       # invariant: `func_outputs` contains only Tensors, CompositeTensors,
  
  ~/anaconda3/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py in wrapper_fn(*args)
     3362           attributes=defun_kwargs)
     3363       def wrapper_fn(*args):  # pylint: disable=missing-docstring
  -&gt; 3364         ret = _wrapper_helper(*args)
     3365         ret = structure.to_tensor_list(self._output_structure, ret)
     3366         return [ops.convert_to_tensor(t) for t in ret]
  
  ~/anaconda3/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py in _wrapper_helper(*args)
     3297         nested_args = (nested_args,)
     3298 
  -&gt; 3299       ret = autograph.tf_convert(func, ag_ctx)(*nested_args)
     3300       # If `func` returns a list of tensors, `nest.flatten()` and
     3301       # `ops.convert_to_tensor()` would conspire to attempt to stack
  
  ~/anaconda3/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py in wrapper(*args, **kwargs)
      256       except Exception as e:  # pylint:disable=broad-except
      257         if hasattr(e, 'ag_error_metadata'):
  --&gt; 258           raise e.ag_error_metadata.to_exception(e)
      259         else:
      260           raise
  
  TypeError: in user code:
  
      :17 process_path  *
          img = decode_img(file_path)
      :7 decode_img  *
          hdul = fits.open(img)
      /home/marcostidball/anaconda3/lib/python3.7/site-packages/astropy/io/fits/hdu/hdulist.py:154 fitsopen  *
          if not name:
      /home/marcostidball/anaconda3/lib/python3.7/site-packages/tensorflow/python/autograph/operators/logical.py:29 not_
          return _tf_not(a)
      /home/marcostidball/anaconda3/lib/python3.7/site-packages/tensorflow/python/autograph/operators/logical.py:35 _tf_not
          return gen_math_ops.logical_not(a)
      /home/marcostidball/anaconda3/lib/python3.7/site-packages/tensorflow/python/ops/gen_math_ops.py:5481 logical_not
          &quot;LogicalNot&quot;, x=x, name=name)
      /home/marcostidball/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:493 _apply_op_helper
          (prefix, dtypes.as_dtype(input_arg.type).name))
  
      TypeError: Input 'x' of 'LogicalNot' Op has type string that does not match expected type of bool.
  
  </code></pre>
  <p>Does anyone know of a way to fix this? I searched around for ways to train a CNN using directly a numpy array such as the one that I get before doing the tensor conversion and found some examples using MNIST with standalone keras. I'd like to apply the usual data augmentation and batch training, though, and I'm not sure if this is possible by following something such as what I've seen.</p>
  <p>Thanks a lot!</p>