text: """
  Implements classes and function for the PypeIt data model.
  
  .. data-container:
  
  DataContainer
  -------------
  
  :class:`DataContainer` objects provide a utility for
  enforcing a specific datamodel on an object, and provides convenience
  routines for writing data to fits files. The class itself is an
  abstract base class that cannot be directly instantiated. As a base
  class, :class:`DataContainer` objects are versatile, but they have
  their limitations.
  
  Derived classes must do the following:
  
      - Define a class attribute called ``datamodel``. See the examples
        below for their format.
      - Provide an :func:`__init__` method that defines the
        instantiation calling sequence and passes the relevant
        dictionary to this base-class instantiation.
      - Provide a :func:`_validate` method, if necessary, that
        processes the data provided in the `__init__` into a complete
        instantiation of the object. This method and the
        :func:`__init__` method are the *only* places where attributes
        can be added to the class.
      - Provide a :func:`_bundle` method that reorganizes the datamodel
        into partitions that can be written to one or more fits
        extensions. More details are provided in the description of
        :func:`DataContainer._bundle`.
      - Provide a :func:`_parse` method that parses information in one
        or more fits extensions into the appropriate datamodel. More
        details are provided in the description of
        :func:`DataContainer._parse`.
  
  .. note::
  
      The attributes of the class are *not required* to be a part of
      the ``datamodel``; however, it makes the object simpler when they
      are. Any attributes that are not part of the ``datamodel`` must
      be defined in either the :func:`__init__` or :func:`_validate`
      methods; otherwise, the class with throw an ``AttributeError``.
  
  Here are some examples of how to and how not to use them.
  
  Defining the datamodel
  ++++++++++++++++++++++
  
  The format of the ``datamodel`` needed for each implementation of a
  :class:`DataContainer` derived class is as follows.
  
  The datamodel itself is a class attribute (i.e., it is a member of
  the class, not just of an instance of the class). The datamodel is a
  dictionary of dictionaries: Each key of the datamodel dictionary
  provides the name of a given datamodel element, and the associated
  item (dictionary) for the datamodel element provides the type and
  description information for that datamodel element. For each
  datamodel element, the dictionary item must provide:
  
      - ``otype``: This is the type of the object for this datamodel
        item. E.g., for a float or a `numpy.ndarray`_, you would set
        ``otype=float`` and ``otype=np.ndarray``, respectively.
  
      - ``descr``: This provides a text description of the datamodel
        element. This is used to construct the datamodel tables in the
        pypeit documentation.
  
  If the object type is a `numpy.ndarray`_, you should also provide the
  ``atype`` keyword that sets the type of the data contained within the
  array. E.g., for a floating point array containing an image, your
  datamodel could be simply::
  
      datamodel = {'image' : dict(otype=np.ndarray, atype=float, descr='My image')}
  
  Currently, ``datamodel`` components are restricted to have ``otype``
  that are :obj:`tuple`, :obj:`int`, :obj:`float`, ``numpy.integer``,
  ``numpy.floating``, `numpy.ndarray`_, or `astropy.table.Table`_
  objects. E.g., ``datamodel`` values for ``otype`` *cannot* be
  :obj:`dict`.
  
  More advanced examples are given below.
  
  Basic container
  +++++++++++++++
  
  Here's how to create a derived class for a basic container that
  holds two arrays and a metadata parameter::
  
      import numpy as np
      import inspect
  
      from pypeit.datamodel import DataContainer
  
      class BasicContainer(DataContainer):
          datamodel = {'vec1': dict(otype=np.ndarray, atype=float, descr='Test'),
                       'meta1': dict(otype=str, decr='test'),
                       'arr1': dict(otype=np.ndarray, atype=float, descr='test')}
  
          def __init__(self, vec1, meta1, arr1):
              # All arguments are passed directly to the container
              # instantiation
              args, _, _, values = inspect.getargvalues(inspect.currentframe())
              super(BasicContainer, self).__init__({k: values[k] for k in args[1:]}) 
  
          def _bundle(self):
              # Use the base class _bundle Specify the extension
              return super(BasicContainer, self)._bundle(ext='basic')
  
  With this implementation:
  
      - You can instantiate the container so that number of data table
        rows would be the same (10)::
  
          data = BasicContainer(np.arange(10), 'length=10', np.arange(30).reshape(10,3))
  
      - After instantiating, access the data like this::
  
          # Get the data model keys
          keys = list(data.keys())
  
          # Access the datamodel as attributes ...
          print(data.vec1)
          # ... or as items
          print(data['meta1'])
  
      - Attributes and items are case-sensitive::
  
          # Faults because of case-sensitive attributes/items
          print(data.Vec1)
          print(data['MeTa1'])
  
      - The attributes of the container can only be part of the
        datamodel or added in either the :func:`DataContainer.__init__`
        or :func:`DataContainer._validate` methods::
  
          # Faults with KeyError
          data['newvec'] = np.arange(10)
          test = data['newvec']
  
          # Faults with AttributeError
          data.newvec = np.arange(10)
          test = data.newvec
  
      - The :class:`DataContainer` also enforces strict types for
        members of the datamodel::
  
          # Faults with TypeError
          data.vec1 = 3
          data.meta1 = 4.
  
      - Read/Write the data from/to a fits file. In this instantiation,
        the data is written to a `astropy.io.fits.BinTableHDU` object;
        the table has 10 rows because the shape of the arrays match
        this. The file I/O routines look like this::
  
          # Write to a file
          ofile = 'test.fits'
          data.to_file(ofile)
  
          # Write to a gzipped file
          ofile = 'test.fits.gz'
          data.to_file(ofile)
  
          # Test written data against input
          with fits.open(ofile) as hdu:
              print(len(hdu))
              # 2: The primary extension and a binary table with the data
  
              print(hdu[1].name)
              # BASIC: As set by the _bundle method
              
              print(len(hdu['BASIC'].data))
              # 10: The length of the data table
              
              print(hdu['BASIC'].columns.names)
              # ['vec1', 'arr1']: datamodel elements written to the table columns
              
              print(hdu['BASIC'].header['meta1'])
              # 'length=10': int, float, or string datamodel components are written to headers
  
      - If the shape of the first axis of the arrays (number of rows)
        do not match, the arrays are written as single elements of a
        table with one row::
  
          # Number of rows are mismatched 
          data = BasicContainer(np.arange(10), 'length=1', np.arange(30).reshape(3,10))
          data.to_file(ofile)
  
          with fits.open(ofile) as hdu:
              print(len(hdu))
              # 2: The primary extension and a binary table with the data
              print(len(hdu['BASIC'].data))
              # 1: All of the data is put into a single row
  
  Mixed Object Containers
  +++++++++++++++++++++++
  
  :class:`DataContainer` objects can also contain multiple arrays
  and/or `astropy.table.Table`_ objects. However, multiple Tables or
  combinations of arrays and Tables cannot be bundled into individual
  extensions. Here are two implementations of a mixed container, a good
  one and a bad one::
  
      import numpy as np
      import inspect
  
      from astropy.table import Table
  
      from pypeit.datamodel import DataContainer
  
      class GoodMixedTypeContainer(DataContainer):
          datamodel = {'tab1': dict(otype=Table, descr='Test'),
                       'tab1len': dict(otype=int, descr='test'),
                       'arr1': dict(otype=np.ndarray, descr='test'),
                       'arr1shape': dict(otype=tuple, descr='test')}
  
          def __init__(self, tab1, arr1):
              # All arguments are passed directly to the container
              # instantiation, but the list is incomplete
              args, _, _, values = inspect.getargvalues(inspect.currentframe())
              super(GoodMixedTypeContainer, self).__init__({k: values[k] for k in args[1:]}) 
  
          def _validate(self):
              # Complete the instantiation
              self.tab1len = len(self.tab1)
              # NOTE: DataContainer does allow for tuples, but beware because
              # they have to saved to the fits files by converting them to strings
              # and writing them to the fits header.  So the tuples should be
              # short!  See _bundle below, and in DataContainer.
              self.arr1shape = self.arr1.shape
  
          def _bundle(self):
              # Bundle so there's only one Table and one Image per extension
              return [{'tab1len': self.tab1len, 'tab1': self.tab1},
                      {'arr1shape': str(self.arr1shape), 'arr1': self.arr1}]
  
  
      class BadMixedTypeContainer(GoodMixedTypeContainer):
          def _bundle(self):
              # Use default _bundle method, which will try to put both tables
              # in the same extension.  NOTE: Can't use super here because
              # GoodMixedTypeContainer doesn't have an 'ext' argument
              return DataContainer._bundle(self, ext='bad')
  
  With this implementation:
  
      - To instantiate::
  
          x = np.arange(10)
          y = np.arange(10)+5
          z = np.arange(30).reshape(10,3)
  
          arr1 = np.full((3,3,3), -1)
          tab1 = Table(data=({'x':x,'y':y,'z':z}), meta={'test':'this'})
  
          data = GoodMixedTypeContainer(tab1, arr1)
  
      - Data access::
  
          print(data.tab1.keys())
          # ['x', 'y', 'z']
          print(assert data.tab1.meta['test'])
          # 'this'
          print(data.arr1shape)
          # (3,3,3)
  
      - Construct an `astropy.io.fits.HDUList`::
  
          hdu = data.to_hdu(add_primary=True)
  
          print(len(hdu))
          # 3: Includes the primary HDU, one with the table, and one with the array
  
          print([h.name for h in hdu])
          # ['PRIMARY', 'TAB1', 'ARR1']
  
      - Tuples are converted to strings::
  
          print(hdu['ARR1'].header['ARR1SHAPE'])
          # '(3,3,3)'
  
      - The tuples are converted back from strings when they're read
        from the HDU::
  
          _data = GoodMixedTypeContainer.from_hdu(hdu)
          print(_data.arr1shape)
          # (3,3,3)
  
      - Table metadata is also written to the header, which can be
        accessed with case-insensitive keys::
  
          print(hdu['TAB1'].header['TEST'])
          print(hdu['TAB1'].header['TesT'])
          # Both print: 'this'
  
      - However, it's important to note that the keyword case gets
        mangled when you read it back in. This has to do with the
        Table.read method and I'm not sure there's anything we can do
        about it without the help of the astropy folks. We recommend
        table metadata use keys that are in all caps::
  
          # Fails because of the string case
          print(_data.tab1.meta['test'])
          # This is okay
          print(_data.tab1.meta['TEST'])
  
      - The difference between the implementation of
        ``BadMixedTypeContainer`` and ``GoodMixedTypeContainer`` has to
        do with how the data is bundled into HDU extensions. The
        ``BadMixedTypeContainer`` will instantiate fine::
  
          data = BadMixedTypeContainer(tab1, arr1)
          print(data.tab1.keys())
          # ['x', 'y', 'z']
          print(data.tab1.meta['test'])
          # 'this'
  
      - But it will barf when you try to reformat/write the data
        because you can't write both a Table and an array to a single
        HDU::
  
          # Fails
          hdu = data.to_hdu()
  
  Complex Instantiation Methods
  +++++++++++++++++++++++++++++
  
  All of the :class:`DataContainer` above have had simple instatiation
  methods. :class:`DataContainer` can have more complex instantiation
  methods, but there are significant limitations to keep in made.
  Consider::
  
      class BadInitContainer(DataContainer):
          datamodel = {'inp1': dict(otype=np.ndarray, descr='Test'),
                       'inp2': dict(otype=np.ndarray, descr='test'),
                       'out': dict(otype=np.ndarray, descr='test'),
                       'alt': dict(otype=np.ndarray, descr='test')}
  
          def __init__(self, inp1, inp2, func='add'):
              args, _, _, values = inspect.getargvalues(inspect.currentframe())
              super(BadInitContainer, self).__init__({k: values[k] for k in args[1:]}) 
  
  
      class DubiousInitContainer(DataContainer):
          datamodel = {'inp1': dict(otype=np.ndarray, descr='Test'),
                       'inp2': dict(otype=np.ndarray, descr='test'),
                       'out': dict(otype=np.ndarray, descr='test'),
                       'alt': dict(otype=np.ndarray, descr='test')}
  
          def __init__(self, inp1, inp2, func='add'):
              # If any of the arguments of the init method aren't actually
              # part of the datamodel, you can't use the nominal two lines
              # used in all the other examples above.  You have to be specific
              # about what gets passed to super.__init__.  See the
              # BadInitContainer example above and the test below.  WARNING:
              # I'm not sure you would ever want to do this because it can
              # lead to I/O issues; see the _validate function.
              self.func = func
              super(DubiousInitContainer, self).__init__({'inp1': inp1, 'inp2':inp2})
  
          def _validate(self):
              # Because func isn't part of the data model, it won't be part of
              # self if the object is instantiated from a file.  So I have to
              # add it here.  But I don't know what the value of the attribute
              # was for the original object that was written to disk.  This is
              # why you likely always want anything that's critical to setting
              # up the object to be part of the datamodel so that it gets
              # written to disk.  See the testing examples for when this will
              # go haywire.
              if not hasattr(self, 'func'):
                  self.func = None
              if self.func not in [None, 'add', 'sub']:
                  raise ValueError('Function must be either \'add\' or \'sub\'.')
  
              # This is here because I don't want to overwrite something that
              # might have been read in from an HDU, particularly given that
              # func will be None if reading from a file!
              if self.out is None:
                  print('Assigning out!')
                  if self.func is None:
                      raise ValueError('Do not know how to construct out attribute!')
                  self.out = self.inp1 + self.inp2 if self.func == 'add' else self.inp1 - self.inp2
  
          # I'm not going to overwrite _bundle, so that the nominal approach
          # is used.
  
  
      class ComplexInitContainer(DataContainer):
          datamodel = {'inp1': dict(otype=np.ndarray, descr='Test'),
                       'inp2': dict(otype=np.ndarray, descr='test'),
                       'out': dict(otype=np.ndarray, descr='test'),
                       'alt': dict(otype=np.ndarray, descr='test'),
                       'func': dict(otype=str, descr='test')}
  
          def __init__(self, inp1, inp2, func='add'):
              # Since func is part of the datamodel now, we can use the normal
              # two intantiation lines.
              args, _, _, values = inspect.getargvalues(inspect.currentframe())
              super(ComplexInitContainer, self).__init__({k: values[k] for k in args[1:]}) 
  
          def _validate(self):
              if self.func not in ['add', 'sub']:
                  raise ValueError('Function must be either \'add\' or \'sub\'.')
              # This is here because I don't want to overwrite something that
              # might have been read in from an HDU, even though they should
              # nominally be the same!
              if self.out is None:
                  print('Assigning out!')
                  if self.func is None:
                      raise ValueError('Do not know how to construct out attribute!')
                  self.out = self.inp1 + self.inp2 if self.func == 'add' else self.inp1 - self.inp2
  
  
  With this implementation:
  
      - The instantiation of the ``BadInitContainer`` will fail because
        the init arguments all need to be part of the datamodel for it
        to work::
  
          x = np.arange(10)
          y = np.arange(10)+5
  
          data = BadInitContainer(x,y)
          # Fails with AttributeError
  
      - The following instantiation is fine because
        ``DubiousInitContainer`` handles the fact that some of the
        arguments to :func:`__init__` are not part of the datamodel::
  
          data = DubiousInitContainer(x,y)
          print(np.array_equal(data.out, data.inp1+data.inp2))
          # True
  
      - One component of the data model wasn't instantiated, so it will
        be None::
          
          print(data.alt is None)
          # True
  
      - The problem with ``DubiousInitContainer`` is that it has
        attributes that cannot be reinstantiated from what's written to
        disk. That is, :class:`DataContainers` aren't really fully
        formed objects unless all of its relevant attributes are
        components of the data model::
  
          _data = DubiousInitContainer.from_hdu(hdu)
          print(_data.func == DubiousInitContainer(x,y).func)
          # False
      
      - This is solved by adding func to the datamodel::
      
          data = ComplexInitContainer(x,y)
          _data = ComplexInitContainer.from_hdu(data.to_hdu(add_primary=True))
          print(data.func == _data.func)
          # True
  
  ----
  
  .. include common links, assuming primary doc root is up one directory
  .. include:: ../include/links.rst
  
  """
  import os
  import warnings
  
  from IPython import embed
  
  import numpy as np
  import inspect
  
  from astropy.io import fits
  from astropy.table import Table
  
  from pypeit import io
  from pypeit import masterframe
  from pypeit import msgs
  
  # TODO: There are methods in, e.g., doc/scripts/build_specobj_rst.py that output
  # datamodels for specific datacontainers.  It would be useful if we had
  # generalized methods in this base class that
  #   - construct an rst table that documents the datamodel.
  #   - construct an rst table that documents each Table component of the datamodel
  #     (like a nested DataContainer)
  
  class DataContainer:
      """
      Defines an abstract class for holding and manipulating data.
  
      The primary utilities of the class are:
          - Attributes can be accessed normally or as expected for a :obj:`dict`
          - Attributes and items are restricted to conform to a specified data model.
  
      This abstract class should only be used as a base class.
  
      Derived classes must do the following:
  
          - Define a datamodel
          - Provide an :func:`__init__` method that defines the
            instantiation calling sequence and passes the relevant
            dictionary to this base-class instantiation.
          - Provide a :func:`_validate` method, if necessary, that
            processes the data provided in the `__init__` into a
            complete instantiation of the object. This method and the
            :func:`__init__` method are the *only* places where
            attributes can be added to the class.
          - Provide a :func:`_bundle` method that reorganizes the
            datamodel into partitions that can be written to one or
            more fits extensions. More details are provided in the
            description of :func:`_bundle`.
          - Provide a :func:`_parse` method that parses information in
            one or more fits extensions into the appropriate datamodel.
            More details are provided in the description of
            :func:`_parse`.
  
      .. note::
  
          The attributes of the class are *not required* to be a part
          of the ``datamodel``; however, it makes the object simpler
          when they are. Any attributes that are not part of the
          ``datamodel`` must be defined in either the :func:`__init__`
          or :func:`_validate` methods; otherwise, the class with throw
          an ``AttributeError``.
  
      .. todo::
  
          Add a copy method
  
      Args:
          d (:obj:`dict`, optional):
              Dictionary to copy to the internal attribute dictionary.
              All of the keys in the dictionary *must* be elements of
              the ``datamodel``. Any attributes that are not part of
              the ``datamodel`` can be set in the :func:`__init__` or
              :func:`_validate` methods of a derived class. If None,
              the object is instantiated with all of the relevant data
              model attributes but with all of those attributes set to
              None.
  
      """
  
      # Define the class version
      version = None
      """
      Provides the string representation of the class version.
  
      This is currently put to minimal use so far, but will used for
      I/O verification in the future.
  
      Each derived class should provide a version to guard against data
      model changes during development.
      """
  
      hdu_prefix = None
      """
      If set, all HDUs generated for this DataContainer will have this
      prefix. This can be set independently for each DataContainer
      derived class; however, it always defaults to None for the base
      class. Be wary of nested DataContainer's!!
      """
  
      output_to_disk = None
      """
      If set, this limits the HDU extensions that are written to the
      output file. Note this is the name of the extension, including
      the hdu_prefix, not necessarily the names of specific datamodel
      components.
      """
  
      # Define the data model
      datamodel = None
      """
      Provides the class data model. This is undefined in the abstract
      class and should be overwritten in the derived classes.
  
      The format of the ``datamodel`` needed for each implementation of
      a :class:`DataContainer` derived class is as follows.
  
      The datamodel itself is a class attribute (i.e., it is a member
      of the class, not just of an instance of the class). The
      datamodel is a dictionary of dictionaries: Each key of the
      datamodel dictionary provides the name of a given datamodel
      element, and the associated item (dictionary) for the datamodel
      element provides the type and description information for that
      datamodel element. For each datamodel element, the dictionary
      item must provide:
  
          - ``otype``: This is the type of the object for this
            datamodel item. E.g., for a float or a `numpy.ndarray`_,
            you would set ``otype=float`` and ``otype=np.ndarray``,
            respectively.
  
          - ``descr``: This provides a text description of the
            datamodel element. This is used to construct the datamodel
            tables in the pypeit documentation.
  
      If the object type is a `numpy.ndarray`_, you should also provide
      the ``atype`` keyword that sets the type of the data contained
      within the array. E.g., for a floating point array containing an
      image, your datamodel could be simply::
  
          datamodel = {'image' : dict(otype=np.ndarray, atype=float, descr='My image')}
  
      More advanced examples are given in the top-level module documentation.
  
      Currently, ``datamodel`` components are restricted to have
      ``otype`` that are :obj:`tuple`, :obj:`int`, :obj:`float`,
      ``numpy.integer``, ``numpy.floating``, `numpy.ndarray`_, or
      `astropy.table.Table`_ objects. E.g., ``datamodel`` values for
      ``otype`` *cannot* be :obj:`dict`.
      """
      # TODO: Enable multiple possible types for the datamodel elements?
      # I.e., allow `otype` to be a tuple of the allowed object types? It
      # looks like this is already possible at least for some types, see
      # pypeit.tracepca.TracePCA.reference_row.
  
  
      def __init__(self, d=None):
          # Data model must be defined
          if self.datamodel is None:
              raise ValueError('Data model for {0} is undefined!'.format(self.__class__.__name__))
          if self.version is None:
              raise ValueError('Must define a version for the class.')
  
          # Ensure the dictionary has all the expected keys
          self.__dict__.update(dict.fromkeys(self.datamodel.keys()))
  
          # Initialize other internals
          self._init_internals()
  
          # Finalize the instantiation.
          # NOTE: The key added to `__dict__` by this call is always
          # `_DataContainer__initialised`, regardless of whether or not
          # the call to this `__init__` is from the derived class. This
          # is why I can check for `_DataContainer__initialised` is in
          # `self.__dict__`, even for derived classes. But is there a way
          # we could just check a boolean instead?
          self._init_key = '_DataContainer__initialised'
          self.__initialised = True
  
          # Include the provided data and build-out the data model, if
          # data were provided
          if d is not None:
  
              # Input dictionary cannot have keys that do not exist in
              # the data model
              if not np.all(np.isin(list(d.keys()), list(self.datamodel.keys()))):
                  raise AttributeError('Coding error: Initialization arguments do not match '
                                       'data model!')
  
              # Assign the values provided by the input dictionary
              #self.__dict__.update(d)  # This by-passes the data model checking
  
              ## Assign the values provided by the input dictionary
              for key in d:
                  # Nested DataContainer?
                  if obj_is_data_container(self.datamodel[key]['otype']):
                      if isinstance(d[key], dict):
                          setattr(self, key, self.datamodel[key]['otype'](**d[key]))
                      else:
                          setattr(self, key, d[key])
                  else:
                      setattr(self, key, d[key])
  
          # Validate the object
          self._validate()
  
      @classmethod
      def full_datamodel(cls, include_parent=True, include_children=True):
          """
          Expand out the datamodel into a single dict
          This needs to be a class method to access the datamodel without instantiation
  
          Args:
              include_parent (bool, optional):
                  If True, include the parent entry in additional to its pieces
              include_children (bool, optional):
                  If True, expand any items that are DataModel's
  
  
          Returns:
              dict: All the keys, items of the nested datamodel's
  
          """
          #
          full_datamodel = {}
          for key in cls.datamodel.keys():
              # Data container?
              if obj_is_data_container(cls.datamodel[key]['otype']):
                  if include_parent:
                      full_datamodel[key] = cls.datamodel[key]
                  if include_children:
                      # Now run through the others
                      sub_datamodel = cls.datamodel[key]['otype'].full_datamodel()
                      for key in sub_datamodel.keys():
                          # Check  this is not a duplicate
                          if key in full_datamodel.keys():
                              msgs.error("Duplicate key in DataModel.  Deal with it..")
                          # Assign
                          full_datamodel[key] = sub_datamodel[key]
                  else:
                      full_datamodel[key] = cls.datamodel[key]
              else:
                  full_datamodel[key] = cls.datamodel[key]
          #
          return full_datamodel
  
      def _init_internals(self):
          """
          Add internal variables to the object before initialization completes
  
          These should be set to None
          """
          pass
  
      def _validate(self):
          """
          Validate the data container.
  
          The purpose of this function is to check the input data
          provided by the instantiation and flesh out any details of
          the object.
  
          Derived classes should override this function, unless there
          is nothing to validate.
  
          Attributes can be added to the object in this function
          because it is called before the datamodel is frozen.
          """
          pass
  
      def _bundle(self, ext=None, transpose_arrays=False):
          """
          Bundle the data into a series of objects to be written to
          fits HDU extensions.
  
          The returned object must be a list. The list items should be
          one of the following:
  
              - a dictionary with a single key/item pair, where the key
                is the name for the extension and the item is the data
                to be written.
  
              - a single item (object) to be written, which will be
                written in the provided order without any extension
                names (although see the caveat in
                :func:`pypeit.io.dict_to_hdu` for dictionaries with
                single array or `astropy.table.Table`_ items).
  
          The item to be written can be a single array for an ImageHDU,
          an `astropy.table.Table`_ for a BinTableHDU, or a dictionary;
          see :func:`pypeit.io.write_to_hdu`.
  
          For how these objects parsed into the HDUs, see
          :func:`to_hdu`.
  
          The default behavior implemented by this base class just parses the
          attributes into a single dictionary based on the datamodel, under the
          expectation that it is all written to a single fits extension. Note that
          this will fault if the datamodel contains:
  
              - a dictionary object
              - more than one `astropy.table.Table`_, or
              - an `astropy.table.Table`_ and an array-like object
                (:obj:`list` or `numpy.ndarray`_)
  
          Certain **restrictions** apply to how the data can be bundled
          for the general parser implementation (:func:`_parse`) to
          work correctly. These restriction are:
  
              - The shape and orientation of any input arrays are
                assumed to be correct.
  
              - Datamodel keys for arrays or `astropy.table.Table`_
                objects written to an HDU should match the HDU
                extension name. Otherwise, the set of HDU extension
                names and datamodel keys **must** be unique.
  
              - Datamodel keys will be matched to header values:
                :func:`to_hdu` will write any items in a dictionary
                that is an integer, float, or string (specific numpy
                types or otherwise) to the header. This means header
                keys in **all** extensions should be unique and should
                not be the same as any extension name.
  
              - Datamodels can contain tuples, but these must be
                reasonably converted to strings such they are included
                in (one of) the HDU header(s).
  
          Args:
              ext (:obj:`str`, optional):
                  Name for the HDU extension. If None, no name is
                  given.
              transpose_arrays (:obj:`bool`, optional):
                  Transpose the arrays before writing them to the HDU.
                  This option is mostly meant to correct orientation of
                  arrays meant for tables so that the number of rows
                  match.
              
          Returns:
              :obj:`list`: A list of dictionaries, each list element is
              written to its own fits extension. See the description
              above.
          """
          d = {}
          for key in self.keys():
              if self[key] is not None and transpose_arrays \
                      and self.datamodel[key]['otype'] == np.ndarray:
                  d[key] = self[key].T
              elif self.datamodel[key]['otype'] == tuple:
                  # TODO: Anything with tuple type that is None will be
                  # converted to 'None'. Is that what we want, or do we
                  # want to set it to None so that it's not written?
                  d[key] = str(self[key])
              else:
                  d[key] = self[key]
          return [d] if ext is None else [{ext:d}]
  
      @classmethod
      def confirm_class(cls, name, allow_subclasses=False):
          """
          Confirm that the provided name of a datamodel class matches the calling
          class.
  
          Args:
              name (:obj:`str`):
                  The name of the class to check against.
              allow_subclasses (:obj:`bool`, optional):
                  Allow subclasses of the calling class to be included in the
                  check.
          Returns:
              :obj:`bool`: Class match confirmed if True
          """
          is_exact = name == cls.__name__
          if is_exact or not allow_subclasses:
              return is_exact
          # NOTE: Unlike pypeit.utils.all_subclasses, this does *not* recursively
          # check for all the subclasses.  We can do that, but it's not necessary
          # yet.
          return is_exact or name in [c.__name__ for c in cls.__subclasses__()]
  
      @classmethod
      def _parse(cls, hdu, ext=None, ext_pseudo=None, transpose_table_arrays=False, 
                 hdu_prefix=None, allow_subclasses=False):
          """
          Parse data read from one or more HDUs.
  
          This method is the counter-part to :func:`_bundle`, and
          parses data from the HDUs into a dictionary that can be used
          to instantiate the object.
  
          .. warning::
  
              - Beware that this may read data from the provided HDUs
                that could then be changed by :func:`_validate`.
                Construct your :func:`_validate` methods carefully!
  
              - Although :func:`_bundle` methods will likely need to be
                written for each derived class, this parsing method is
                very general to what :class:`DataContainer` can do.
                Before overwriting this function in a derived class,
                make sure and/or test that this method doesn't meet
                your needs, and then tread carefully regardless.
  
              - Because the `astropy.table.Table`_ methods are used
                directly, any metadata associated with the Table will
                also be included in the HDUs constructed by
                :func:`to_hdu`. However, the
                `astropy.table.Table.read`_ method always returns the
                metadata with capitalized keys. This means that,
                regardless of the capitalization of the metadata
                keywords when the data is written, **they will be
                upper-case when read by this function**!
  
              - Use ``allow_subclasses`` with care!  The expectation is that all
                the subclasses have exactly the same datamodel and version number
                as the main class.
  
          .. note::
  
              Currently, the only test for the object type (``otype``)
              given explicitly by the class datamodel is to check if
              the type should be a tuple. If it is, the parser reads
              the string from the header and evaluates it so that it's
              converted to a tuple on output. See the restrictions
              listed for :func:`_bundle`.
  
              All the other type conversions are implicit or based on
              the HDU type.
  
          Args:
              hdu (`astropy.io.fits.HDUList`_, `astropy.io.fits.ImageHDU`_, `astropy.io.fits.BinTableHDU`_):
                  The HDU(s) to parse into the instantiation dictionary.
              ext (:obj:`int`, :obj:`str`, :obj:`list`, optional):
                  One or more extensions with the data. If None, the
                  function trolls through the HDU(s) and parses the
                  data into the datamodel.
              ext_pseudo (:obj:`int`, :obj:`str`, :obj:`list`, optional):
                  Pseudonyms to use for the names of the HDU extensions instead of
                  the existing extension names.  Similar to the use of
                  ``hdu_prefix``, this is useful for parsing nested
                  :class:`DataContainer` objects; i.e., a child
                  :class:`DataContainer` may be assigned to an extension based on
                  the datamodel of its parent, meaning that it can't be properly
                  parsed by the parent's method.  If used, the number of extension
                  pseudonyms provided **must** match the list returned by::
                      
                      io.hdu_iter_by_ext(hdu, ext=ext, hdu_prefix=hdu_prefix)
  
              transpose_table_arrays (:obj:`bool`, optional):
                  Tranpose *all* the arrays read from any binary
                  tables. This is meant to invert the use of
                  ``transpose_arrays`` in :func:`_bound`.
              hdu_prefix (:obj:`str`, optional):
                  Only parse HDUs with extension names matched to this
                  prefix. If None, :attr:`hdu_prefix` is used. If the
                  latter is also None, all HDUs are parsed. See
                  :func:`pypeit.io.hdu_iter_by_ext`.
              allow_subclasses (:obj:`bool`, optional):
                  Allow subclasses of the calling class to successfully pass the
                  check on whether or not it can parse the provided HDUs.  That
                  is, if the class provided in the HDU header is "A", setting this
                  parameter to True means that parsing will continue as long as
                  "A" is a subclass of the calling class.  In ``PypeIt`` this is
                  needed, e.g., for :class:`~pypeit.sensfunc.SensFunc` to
                  successfully parse files written by either of its subclasses,
                  :class:`~pypeit.sensfunc.IRSensFunc` or 
                  :class:`~pypeit.sensfunc.UVISSensFunc`.  This is possible
                  because the datamodel is defined by the parent class and not
                  altered by the subclasses!  **Use with care! See function
                  warnings.**
                  
          Returns:
              :obj:`tuple`: Return three objects
  
                  - :obj:`dict`: Dictionary used to instantiate the object.
                  - :obj:`bool`: Describes datamodel version checking passed
                  - :obj:`bool`: Describes datamodel type checking passed
  
          Raises:
              TypeError:
                  Raised if ``ext``, or any of its elements if it's a
                  :obj:`list`, are not either strings or integers.
          """
          dm_version_passed = True
          dm_type_passed = True
  
          # Setup to iterate through the provided HDUs
          _ext, _hdu = io.hdu_iter_by_ext(hdu, ext=ext, hdu_prefix=hdu_prefix)
          _ext = np.atleast_1d(np.array(_ext, dtype=object))
  
          _ext_pseudo = _ext if ext_pseudo is None else np.atleast_1d(ext_pseudo)
  
          if len(_ext_pseudo) != len(_ext):
              raise ValueError(f'Length of provided extension pseudonym list must match number of '
                               f'extensions selected: {len(_ext)}.')
  
          str_ext = np.logical_not([isinstance(e, (int, np.integer)) for e in _ext_pseudo])
  
          # Construct instantiation dictionary
          _d = dict.fromkeys(cls.datamodel.keys())
  
          # Log if relevant data is found for this datamodel
          if np.all([_hdu[e].data is None for e in _ext]):
              # TODO: This is a KLUDGE. Not sure we should allow this...
              msgs.warn('Extensions to be read by {0} have no data!'.format(cls.__name__))
              # This is so that the returned booleans for reading the
              # data are not tripped as false!
              found_data = True
          else:
              found_data = False
  
          # NOTE: The extension and keyword comparisons are complicated
          # because the fits standard is to force these all to be
          # capitalized, while the datamodel doesn't
          # implement this restriction.
  
          # Handle hdu_prefix
          if hdu_prefix is not None:
              prefix = hdu_prefix
          else:
              prefix = '' if cls.hdu_prefix is None else cls.hdu_prefix
  
          # Save the list of hdus that have been parsed
          parsed_hdus = []
  
          # HDUs can have dictionary elements directly.
          keys = np.array(list(_d.keys()))
          prefkeys = np.array([prefix+key.upper() for key in keys])
          indx = np.isin(prefkeys, _ext_pseudo[str_ext])
  
          if np.any(indx):
              found_data = True
              for e in keys[indx]:
                  pseudo_hduindx = prefix+e.upper()
                  hduindx = _ext[np.where(_ext_pseudo == pseudo_hduindx)[0][0]]
                  # Add it to the list of parsed HDUs
                  parsed_hdus += [hduindx]
                  if obj_is_data_container(cls.datamodel[e]['otype']):
                      # Parse the DataContainer
                      # TODO: This only works with single extension
                      # DataContainers. Do we want this to be from_hdu
                      # instead and add chk_version to _parse?
                      # TODO: Should this propagate allow_subclasses?  It does
                      # not, for now.
                      _d[e], p1, p2, _ = cls.datamodel[e]['otype']._parse(_hdu[hduindx])
                      dm_version_passed &= p1
                      dm_type_passed &= p2
                  else:
                      # Parse the Image or Table data
                      dm_type_passed &= cls.confirm_class(_hdu[hduindx].header['DMODCLS'],
                                                          allow_subclasses=allow_subclasses)
                      dm_version_passed &= _hdu[hduindx].header['DMODVER'] == cls.version
                      # Grab it
                      _d[e] = _hdu[hduindx].data if isinstance(_hdu[hduindx], fits.ImageHDU) \
                          else Table.read(_hdu[hduindx]).copy()
  
          for e in _ext:
              if 'DMODCLS' not in _hdu[e].header.keys() or 'DMODVER' not in _hdu[e].header.keys() \
                      or not cls.confirm_class(_hdu[e].header['DMODCLS'],
                                               allow_subclasses=allow_subclasses):
                  # Can't be parsed
                  continue
              # Check for header elements, but do not over-ride existing items
              indx = np.isin([key.upper() for key in keys], list(_hdu[e].header.keys()))
              if np.any(indx):
                  found_data = True
                  parsed_hdus += [e if _hdu[e].name == '' else _hdu[e].name]
                  dm_type_passed &= cls.confirm_class(_hdu[e].header['DMODCLS'],
                                                      allow_subclasses=allow_subclasses)
                  dm_version_passed &= _hdu[e].header['DMODVER'] == cls.version
                  for key in keys[indx]:
                      if key in _d.keys() and _d[key] is not None:
                          continue
                      _d[key] = _hdu[e].header[key.upper()] if cls.datamodel[key]['otype'] != tuple \
                                  else eval(_hdu[e].header[key.upper()])
              if isinstance(e, (str, np.str_)) and e in prefkeys:
                  # Already parsed this above
                  continue
              # Parse BinTableHDUs
              if isinstance(_hdu[e], fits.BinTableHDU) \
                      and np.any(np.isin(list(cls.datamodel.keys()), _hdu[e].columns.names)):
                  parsed_hdus += [e if _hdu[e].name is None else _hdu[e].name]
                  found_data = True
                  # Datamodel checking
                  dm_type_passed &= cls.confirm_class(_hdu[e].header['DMODCLS'],
                                                      allow_subclasses=allow_subclasses)
                  dm_version_passed &= _hdu[e].header['DMODVER'] == cls.version
                  # If the length of the table is 1, assume the table
                  # data had to be saved as a single row because of shape
                  # differences.
                  single_row = len(_hdu[e].data) == 1
                  for key in _hdu[e].columns.names:
                      if key in cls.datamodel.keys():
                          _d[key] = _hdu[e].data[key][0] \
                                          if (single_row and _hdu[e].data[key].ndim > 1) \
                                          else _hdu[e].data[key]
                          if transpose_table_arrays:
                              _d[key] = _d[key].T
  
          # Two annoying hacks:
          #   - Hack to expunge charray which are basically deprecated and
          #     cause trouble.
          #   - Hack to force native byte ordering
          for key in _d:
              if isinstance(_d[key], np.chararray):
                  _d[key] = np.asarray(_d[key])
              elif isinstance(_d[key], np.ndarray) and _d[key].dtype.byteorder not in ['=', '|']:
                  _d[key] = _d[key].astype(_d[key].dtype.type)
  
          # Return
          return _d, dm_version_passed and found_data, dm_type_passed and found_data, \
                      np.unique(parsed_hdus).tolist()
  
  
      def __getattr__(self, item):
          """Maps values to attributes.
          Only called if there *isn't* an attribute with this name
          """
          try:
              return self.__getitem__(item)
          except KeyError as e:
              raise AttributeError('{0} is not an attribute of {1}!'.format(item,
                                      self.__class__.__name__)) from e
  
      def __setattr__(self, item, value):
          """
          Set the attribute value.
  
          Attributes are restricted to those defined by the datamodel.
          """
          # version is immutable
          if item == 'version':
              raise TypeError('Internal version does not support assignment.')
          # TODO: It seems like it would be faster to check a boolean
          # attribute. Is that possible?
          if '_DataContainer__initialised' not in self.__dict__:
              # Allow new attributes to be added before object is
              # initialized
              dict.__setattr__(self, item, value)
          else:
              # Otherwise, set as an item
              try:
                  self.__setitem__(item, value)
              except KeyError as e:
                  # Raise attribute error instead of key error
                  raise AttributeError('{0} is not part of the internals nor data model!'.format(item)) from e
  
      def __setitem__(self, item, value):
          """
          Access and set an attribute identically to a dictionary item.
  
          Items are restricted to those defined by the datamodel.
          """
          if item not in self.__dict__.keys():
              raise KeyError('Key {0} not part of the internals nor data model'.format(item))
          # Internal?
          if item not in self.keys():
              self.__dict__[item] = value
              return
          # Set datamodel item to None?
          if value is None:
              self.__dict__[item] = value
              return
          # Check data type
          if not isinstance(value, self.datamodel[item]['otype']):
              raise TypeError('Incorrect data type for {0}!  '.format(item) +
                              'Allowed type(s) are: {0}'.format(self.datamodel[item]['otype']))
          # Array?
          if 'atype' in self.datamodel[item].keys():
              if not isinstance(value.flat[0], self.datamodel[item]['atype']):
                  raise TypeError('Wrong data type for array: {}\n'.format(item)
                                  + 'Allowed type(s) for the array are: {}'.format(
                                      self.datamodel[item]['atype']))
          # Set
          self.__dict__[item] = value
  
      def __getitem__(self, item):
          """Get an item directly from the internal dict."""
          return self.__dict__[item]
  
      def keys(self):
          """
          Return the keys for the data objects only
  
          Returns:
              :obj:`dict_keys`: The iterable with the data model keys.
          """
          return self.datamodel.keys()
  
      def _primary_header(self, hdr=None):
          """
          Construct a primary header that is included with the primary
          HDU extension produced by :func:`to_hdu`.
  
          Additional data can be added to the header for individual HDU
          extensions in :func:`to_hdu` as desired, but this function
          **should not** add any elements from the datamodel to the
          header.
  
          Args:
              hdr (`astropy.io.fits.Header`_, optional):
                  Header for the primary extension. If None, set by
                  :func:`pypeit.io.initialize_header()`.
  
          Returns:
              `astropy.io.fits.Header`_: Header object to include in
              the primary HDU.
          """
          return io.initialize_header() if hdr is None else hdr.copy()
  
      def _base_header(self, hdr=None):
          """
          Construct a base header that is included with all HDU
          extensions produced by :func:`to_hdu` (unless they are
          overwritten by a nested :class:`DataContainer`).
  
          Additional data can be added to the header for individual HDU
          extensions in :func:`to_hdu` as desired, but this function
          **should not** add any elements from the datamodel to the
          header.
  
          Args:
              hdr (`astropy.io.fits.Header`_, optional):
                  Baseline header to add to all returned HDUs. If None,
                  set by :func:`pypeit.io.initialize_header()`.
  
          Returns:
              `astropy.io.fits.Header`_: Header object to include in
              all HDU extensions.
          """
          # Copy primary header to all subsequent headers
          _hdr = self._primary_header(hdr=hdr)
          # Add DataContainer class name and datamodel version number.
          # This is not added to the primary header because single output
          # files can contain multiple DataContainer objects.
          _hdr['DMODCLS'] = (self.__class__.__name__, 'Datamodel class')
          _hdr['DMODVER'] = (self.version, 'Datamodel version')
          return _hdr
  
      # TODO: Always have this return an HDUList instead of either that
      # or a normal list?
      def to_hdu(self, hdr=None, add_primary=False, primary_hdr=None,
                 limit_hdus=None, force_to_bintbl=False, hdu_prefix=None):
          """
          Construct one or more HDU extensions with the data.
  
          The organization of the data into separate HDUs is performed
          by :func:`_bundle`, which returns a list of objects. The type
          of each element in the list determines how it is handled. If
          the object is a dictionary with a single key/item pair, the
          key is used as the extension header. Otherwise, the objects
          are written to unnamed extensions. The object or dictionary
          item is passed to :func:`pypeit.io.write_to_hdu` to construct
          the HDU.
  
          Args:
              hdr (`astropy.io.fits.Header`, optional):
                  Baseline header to add to all returned HDUs. If None,
                  set by :func:`pypeit.io.initialize_header()`.
              add_primary (:obj:`bool`, optional):
                  If False, the returned object is a simple
                  :obj:`list`, with a list of HDU objects (either
                  `astropy.io.fits.ImageHDU`_ or
                  `astropy.io.fits.BinTableHDU`_). If true, the method
                  constructs an `astropy.io.fits.HDUList` with a
                  primary HDU, such that this call::
  
                      hdr = io.initialize_header()
                      hdu = fits.HDUList([fits.PrimaryHDU(header=primary_hdr)] + self.to_hdu(hdr=hdr))
  
                  and this call::
  
                      hdu = self.to_hdu(add_primary=True)
  
                  are identical.
              primary_hdr (`astropy.io.fits.Header`, optional):
                  Header to add to the primary if add_primary=True
              limit_hdus (:obj:`list`, optional):
                  Limit the HDUs that can be written to the items in this list
              force_to_bintbl (:obj:`bool`, optional):
                  Force any dict into a BinTableHDU (e.g. for
                  :class:`pypeit.specobj.SpecObj`)
              hdu_prefix (:obj:`str`, optional):
                  Prefix for the HDU names. If None, will use
                  :attr:`hdu_prefix`. If the latter is also None, no
                  prefix is added.
  
          Returns:
              :obj:`list`, `astropy.io.fits.HDUList`_: A list of HDUs,
              where the type depends on the value of ``add_primary``.
          """
          # Bundle the data
          data = self._bundle()
  
          # Initialize the primary header (only used if add_primary=True)
          _primary_hdr = self._primary_header(hdr=primary_hdr)
          # Check that the keywords in the primary header do not overlap
          # with any datamodel keys.
          if _primary_hdr is not None:
              hdr_keys = np.array([k.upper() for k in self.keys()])
              indx = np.isin(hdr_keys, list(_primary_hdr.keys()))
              # TODO: This is a hack to deal with PYP_SPEC, but this
              # needs to be cleaned up, as does masterframe more
              # generally...
              if np.sum(indx) > 1 or (np.sum(indx) == 1 and hdr_keys[indx] != 'PYP_SPEC'):
                  msgs.error('CODING ERROR: Primary header should not contain keywords that are the '
                             'same as the datamodel for {0}.'.format(self.__class__.__name__))
  
          # Initialize the base header
          _hdr = self._base_header(hdr=hdr)
          # Check that the keywords in the base header do not overlap
          # with any datamodel keys.
          if _hdr is not None \
                  and np.any(np.isin([k.upper() for k in self.keys()], list(_hdr.keys()))):
              msgs.error('CODING ERROR: Baseline header should not contain keywords that are the '
                         'same as the datamodel for {0}.'.format(self.__class__.__name__))
  
          # Construct the list of HDUs
          hdu = []
          for d in data:
              if isinstance(d, dict) and len(d) == 1:
                  ext = list(d.keys())[0]
                  # Allow for embedded DataContainer's
                  if isinstance(d[ext], DataContainer):
                      _hdu = d[ext].to_hdu(add_primary=False)
                      # NOTE: The lines below allow extension names to be
                      # overridden by the input dictionary keyword for
                      # DataContainers that are confined to a single HDU.
                      # This is necessary because `hdu_prefix` must be a
                      # class attribute to allow for its inclusion in the
                      # read methods (e.g., `_parse`)
                      if len(_hdu) == 1 and _hdu[0].name != ext:
                          _hdu[0].name = ext
                      hdu += _hdu
                  else:
                      hdu += [io.write_to_hdu(d[ext], name=ext, hdr=_hdr,
                                              force_to_bintbl=force_to_bintbl)]
              else:
                  hdu += [io.write_to_hdu(d, hdr=_hdr, force_to_bintbl=force_to_bintbl)]
          # Prefixes
          _hdu_prefix = (None if self.hdu_prefix is None else self.hdu_prefix) \
                          if hdu_prefix is None else hdu_prefix
          if _hdu_prefix is not None:
              for ihdu in hdu:
                  ihdu.name = _hdu_prefix+ihdu.name
          # Limit?
          if limit_hdus:
              hdu = [h for h in hdu if h.name in limit_hdus]
          # Return
          return fits.HDUList([fits.PrimaryHDU(header=_primary_hdr)] + hdu) if add_primary else hdu
  
      @classmethod
      def from_hdu(cls, hdu, ext=None, ext_pseudo=None, hdu_prefix=None, 
                   chk_version=True, allow_subclasses=False):
          """
          Instantiate the object from an HDU extension.
  
          This is primarily a wrapper for :func:`_parse`.
  
          Args:
              hdu (`astropy.io.fits.HDUList`_, `astropy.io.fits.ImageHDU`_, `astropy.io.fits.BinTableHDU`_):
                  The HDU(s) with the data to use for instantiation.
              ext (:obj:`str`, optional):
                  Passed to :func:`_parse`.
              ext_pseudo (:obj:`str`, optional):
                  Passed to :func:`_parse`.
              hdu_prefix (:obj:`str`, optional):
                  Passed to :func:`_parse`.
              chk_version (:obj:`bool`, optional):
                  If True, raise an error if the datamodel version or
                  type check failed. If False, throw a warning only.
              allow_subclasses (:obj:`bool`, optional):
                  Passed to :func:`_parse`.
          """
          # NOTE: We can't use `cls(d)`, where `d` is the dictionary returned by
          # `cls._parse`, because this will call the `__init__` method of the
          # derived class and we need to use the `__init__` of the base class
          # instead.  Instead, `d` is passed to `cls.from_dict`, which initiates
          # everything correctly.
          d, dm_version_passed, dm_type_passed, parsed_hdus \
                  = cls._parse(hdu, ext=ext, ext_pseudo=ext_pseudo, 
                               hdu_prefix=hdu_prefix,
                               allow_subclasses=allow_subclasses)
          # Check version and type?
          if not dm_type_passed:
              msgs.error('The HDU(s) cannot be parsed by a {0} object!'.format(cls.__name__))
          if not dm_version_passed:
              _f = msgs.error if chk_version else msgs.warn
              _f('Current version of {0} object in code (v{1})'.format(cls.__name__, cls.version)
                 + ' does not match version used to write your HDU(s)!')
          # Instantiate
          return cls.from_dict(d=d)
  
      @classmethod
      def from_dict(cls, d=None):
          """
          Instantiate from a dictionary.
  
          This is primarily to allow for instantiating classes from a
          file where the data has already been parsed. E.g., see how
          the :class:`~pypeit.tracepca.TracePCA` objects are
          instantiated in
          :class:`pypeit.edgetrace.EdgeTraceSet.from_hdu`. However,
          note that this does the bare minimum to instantiate the
          object. Any class-specific operations that are needed to
          complete the instantiation should be done by ovewritting this
          method; e.g., see :func:`pypeit.tracepca.TracePCA.from_dict`.
  
          Args:
              d (:obj:`dict`, optional):
                  Dictionary with the data to use for instantiation.
          """
          # NOTE: We can't use `cls(d)` here because this will call the `__init__`
          # method of the derived class and we need to use the `__init__` of the
          # base class instead. So instead, I get an empty instance of the derived
          # class using `__new__`, call the parent `__init__`, and then return the
          # result. The call to `DataContainer.__init__` is explicit to deal with
          # objects inheriting from both DataContainer and other base classes.
          self = super().__new__(cls)
          DataContainer.__init__(self, d=d)
          return self
  
      def to_file(self, ofile, overwrite=False, checksum=True, primary_hdr=None, hdr=None,
                  limit_hdus=None):
          """
          Write the data to a file.
  
          This is a convenience wrapper for :func:`to_hdu` and
          :func:`pypeit.io.write_to_fits`. The output is always placed
          in the 2nd extension; the first (primary) extension is always
          empty.
  
          Args:
              ofile (:obj:`str`):
                  Fits file for the data. File names with '.gz'
                  extensions will be gzipped; see
                  :func:`pypeit.io.write_to_fits`.
              primary_hdr (`astropy.io.fits.Header`, optional):
                  Primary header to add to first extension. Passed
                  directly to :func:`to_hdu`; see usage there.
              hdr (`astropy.io.fits.Header`, optional):
                  Baseline header to add to all returned HDUs. Passed
                  directly to :func:`to_hdu`; see usage there.
              overwrite (:obj:`bool`, optional):
                  Flag to overwrite any existing file.
              checksum (:obj:`bool`, optional):
                  Passed to `astropy.io.fits.HDUList.writeto`_ to add
                  the DATASUM and CHECKSUM keywords fits header(s).
              limit_hdus (:obj:`list`, optional):
                  Passed to :func:`to_hdu`; see usage there
          """
          io.write_to_fits(self.to_hdu(add_primary=True, primary_hdr=primary_hdr,
                                       limit_hdus=limit_hdus, hdr=hdr),
                           ofile, overwrite=overwrite, checksum=checksum, hdr=hdr)
  
      # TODO: This requires that master_key be an attribute... This
      # method is a bit too ad hoc for me...
      def to_master_file(self, master_filename=None, **kwargs):
          """
          Wrapper on to_file() that deals with masterframe naming and header
  
          This also sets master_key and master_dir internally if
          when master_filename is provided
  
          self.hdu_prefix and self.output_to_disk must be set (or None)
  
          Args:
              master_filename (str, optional):
                  Name of masterfile;  if provided, parsed for master_key, master_dir
                  If not provided, constructed from internal master_key, master_dir
              **kwargs: passed to to_file()
          """
          # Output file
          if master_filename is None:
              master_filename = masterframe.construct_file_name(self, self.master_key,
                                                                master_dir=self.master_dir)
          else:
              self.master_key, self.master_dir = masterframe.grab_key_mdir(
                  master_filename, from_filename=True)
          # Header
          if hasattr(self, 'process_steps'):
              steps = self.process_steps
          else:
              steps = None
          if hasattr(self, 'files'):
              raw_files = self.files
          else:
              raw_files = None
          hdr = masterframe.build_master_header(self, self.master_key, self.master_dir,
                                                steps=steps, raw_files=raw_files)
          # Finish
          self.to_file(master_filename, primary_hdr=hdr,
                       limit_hdus=self.output_to_disk, overwrite=True, **kwargs)
  
      # TODO: Add options to compare the checksum and/or check the package versions
      @classmethod
      def from_file(cls, ifile, verbose=True, chk_version=True):
          """
          Instantiate the object from an extension in the specified fits file.
  
          This is a convenience wrapper for :func:`from_hdu`.
          
          Args:
              ifile (:obj:`str`):
                  Fits file with the data to read
              verbose (:obj:`bool`, optional):
                  Print informational messages
              chk_version (:obj:`bool`, optional):
                  Passed to :func:`from_hdu`.
  
          Raises:
              FileNotFoundError:
                  Raised if the specified file does not exist.
          """
          if not os.path.isfile(ifile):
              raise FileNotFoundError('{0} does not exist!'.format(ifile))
  
          if verbose:
              msgs.info("Loading {} from {}".format(cls.__name__, ifile))
  
          # Do it
          with io.fits_open(ifile) as hdu:
              obj = cls.from_hdu(hdu, chk_version=chk_version)
              if hasattr(obj, 'head0'):
                  obj.head0 = hdu[0].header
              if hasattr(obj, 'filename'):
                  obj.filename = ifile
  
              # Master this and that
              if hasattr(cls, 'master_type'):
                  obj.master_key, obj.master_dir = masterframe.grab_key_mdir(ifile)
                  if hasattr(obj, 'head0'):
                      if 'MSTRTYP' in obj.head0.keys():
                          if obj.head0['MSTRTYP'] != cls.master_type:
                              msgs.error('Master Type read from header incorrect!  '
                                         'Found {0}; expected {1}'.format(obj.head0['MSTRTYP'],
                                                                          cls.master_type))
                      else:
                          msgs.warn('DataContainer has `master_type` attribute but is missing the '
                                    'MSTRTYP header keyword!')
          return obj
  
      def __repr__(self):
          """ Over-ride print representation
  
          Returns:
              str: Basics of the Data Container
          """
          repr = '<{:s}: '.format(self.__class__.__name__)
          # Image
          rdict = {}
          for attr in self.datamodel.keys():
              if hasattr(self, attr) and getattr(self, attr) is not None:
                  rdict[attr] = True
              else:
                  rdict[attr] = False
          repr += ' items={}'.format(rdict)
          repr = repr + '>'
          return repr
  
  
  def obj_is_data_container(obj):
      """
      Simple method to check whether an object is a data container
  
      Args:
          obj:
  
      Returns:
          bool:  True if it is
  
      """
      return inspect.isclass(obj) and issubclass(obj, DataContainer)
  