files: .. _classes:
  
  ===============
      Classes
  ===============
  
  
  Class Structure
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  
  Here's the overview of core classes::
  
      Common
      ├── Tree
      ├── Core
      │   ├── Plan
      │   ├── Story
      │   └── Test
      ├── Step
      │   ├── Discover
      │   ├── Execute
      │   ├── Finish
      │   ├── Prepare
      │   ├── Provision
      │   └── Report
      ├── Plugin
      │   ├── DiscoverPlugin
      │   │   ├── DiscoverFmf
      │   │   └── DiscoverShell
      │   ├── ExecutePlugin
      │   │   ├── ExecuteDetach
      │   │   └── ExecuteInternal
      │   ├── FinishPlugin
      │   │   └── FinishShell
      │   ├── PreparePlugin
      │   │   ├── PrepareAnsible
      │   │   ├── PrepareInstall
      │   │   └── PrepareShell
      │   ├── ProvisionPlugin
      │   │   ├── ProvisionConnect
      │   │   ├── ProvisionLocal
      │   │   ├── ProvisionMinute
      │   │   ├── ProvisionPodman
      │   │   └── ProvisionTestcloud
      │   └── ReportPlugin
      │       ├── ReportDisplay
      │       └── ReportHtml
      ├── Guest
      │   ├── GuestContainer
      │   ├── GuestLocal
      │   ├── GuestMinute
      │   └── GuestTestcloud
      ├── Run
      ├── Status
      └── Clean
  
  
  Attributes
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  
  Object hierarchy is following: ``Run`` -> ``Plans`` -> ``Steps``
  -> ``Plugins``, where the ``Run`` is on the top of this hierarchy.
  The objects have the ``parent`` attribute, that is pointing to
  the parent in which the current instance is contained.
  
  The ``node`` attribute of ``Test``, ``Plan`` and ``Story``
  instances references the original leaf node of the fmf metadata
  tree from which the respective test, plan or story have been
  created.
  
  In a similar way, the ``tree`` property of the ``Tree`` instance
  points to the original ``fmf.Tree`` from which it was initialized.
  
  
  Class Conversions
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  
  Various internal objects and classes often need to be converted
  from their Python nature to data that can be saved, loaded or
  exported in different form. To facilitate these conversions, three
  families of helper methods are provided, each with its own set of
  use cases.
  
  ``to_spec``/``from_spec``
  ------------------------------------------------------------------
  
  This family of methods works with tmt *specification*, i.e. raw
  user-provided data coming from fmf files describing plans, tests,
  stories, or from command-line options. ``from_spec()`` shall be
  called to spawn objects representing the user input, while
  ``to_spec()`` should produce output one could find in fmf files.
  
  The default implementation comes from ``tmt.utils.SpecBasedContainer``
  class, all classes based on user input data should include this
  class among their bases.
  
  .. code-block:: python
  
     # Create an fmf id object from raw data
     fmf_id = tmt.base.FmfId.from_spec({'url': ..., 'ref': ...})
  
  
  ``to_serialized``/``from_serialized``/``unserialize``
  ------------------------------------------------------------------
  
  This family of methods is aiming at runtime objects that may be
  saved into and loaded from tmt working files, i.e. files tmt uses
  to store a state in its workdir, like `step.yaml` or `guests.yaml`.
  
  Third member of this family, ``unserialize``, is similar to
  ``from_serialized`` - both create an object from its serialized form,
  only ``unserialize`` is capable of detecting the class to instantiate
  while for using ``from_serialized``, one must already know which
  class to work with. ``unserialize`` then uses ``from_serialized``
  under the hood to do the heavy lifting when correct class is
  identified.
  
  The default implementation comes from ``tmt.utils.SerializableContainer``
  class, all classes that are being saved and loaded during tmt run
  should include this class among their bases.
  
  See https://en.wikipedia.org/wiki/Serialization for more details
  on the concept of serialization.
  
  .. code-block:: python
  
      # tmt.steps.discover.shell.DiscoverShellData wishes to unserialize its
      # `tests` a list of `TestDescription` objects rather than a list of
      # dictionaries (the default implementation).
      @classmethod
      def from_serialized(cls, serialized: Dict[str, Any]) -> 'DiscoverShellData':
          obj = super().from_serialized(serialized)
  
          obj.tests = [TestDescription.from_serialized(
              serialized_test) for serialized_test in serialized['tests']]
  
          return obj
  
     # A step saving its state...
     content: Dict[str, Any] = {
         'status': self.status(),
         'data': [datum.to_serialized() for datum in self.data]
     }
     self.write('step.yaml', tmt.utils.dict_to_yaml(content))
  
     # ... and loading it back.
     # Note the use of unserialize(): step data may have been serialized from
     # various different classes (derived from tmt.steps.provision.Guest),
     # and unserialize() will detect the correct class.
     raw_step_data: Dict[Any, Any] = tmt.utils.yaml_to_dict(self.read('step.yaml'))
     self.data = [
         StepData.unserialize(raw_datum) for raw_datum in raw_step_data['data']
     ]
  
  
  ``to_dict``
  ------------------------------------------------------------------
  
  Very special helper method: its use cases are not related to any
  input or output data, and most of the time, when in need of
  iterating over object's keys and/or values, one can use ``keys()``,
  ``values()`` or ``items()`` methods. It is used as a source of data
  for serialization and validation, but it usually has no use outside
  of default implementations.
  
  .. warning::
  
     If you think of using ``to_dict()``, please, think again and be sure
     you know what are you doing. Despite its output being sometimes
     perfectly compatible with output of ``to_serialized()`` or ``to_spec()``,
     it is not generaly true, and using it instead of proper methods may lead
     to unexpected exceptions.
  
  .. code-block:: python
  
     # tmt.base.FmfId's specification is basically just a mapping,
     # therefore `to_dict()` is good enough to produce a specification.
     def to_spec(self) -> Dict[str, Any]:
          return self.to_dict()
  
  
  Essential Classes
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  
  .. automodule:: tmt
      :members:
      :undoc-members:
  
files: docs/classes.rst
files: .. _classes:
  
  ===============
      Classes
  ===============
  
  
  Class Structure
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  
  Here's the overview of core classes::
  
      Common
      ├── Tree
      ├── Core
      │   ├── Plan
      │   ├── Story
      │   └── Test
      ├── Step
      │   ├── Discover
      │   ├── Execute
      │   ├── Finish
      │   ├── Prepare
      │   ├── Provision
      │   └── Report
      ├── Plugin
      │   ├── DiscoverPlugin
      │   │   ├── DiscoverFmf
      │   │   └── DiscoverShell
      │   ├── ExecutePlugin
      │   │   ├── ExecuteDetach
      │   │   └── ExecuteInternal
      │   ├── FinishPlugin
      │   │   └── FinishShell
      │   ├── PreparePlugin
      │   │   ├── PrepareAnsible
      │   │   ├── PrepareInstall
      │   │   └── PrepareShell
      │   ├── ProvisionPlugin
      │   │   ├── ProvisionConnect
      │   │   ├── ProvisionLocal
      │   │   ├── ProvisionMinute
      │   │   ├── ProvisionPodman
      │   │   └── ProvisionTestcloud
      │   └── ReportPlugin
      │       ├── ReportDisplay
      │       └── ReportHtml
      ├── Guest
      │   ├── GuestContainer
      │   ├── GuestLocal
      │   ├── GuestMinute
      │   └── GuestTestcloud
      ├── Run
      ├── Status
      └── Clean
  
  
  Attributes
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  
  Object hierarchy is following: ``Run`` -> ``Plans`` -> ``Steps``
  -> ``Plugins``, where the ``Run`` is on the top of this hierarchy.
  The objects have the ``parent`` attribute, that is pointing to
  the parent in which the current instance is contained.
  
  The ``node`` attribute of ``Test``, ``Plan`` and ``Story``
  instances references the original leaf node of the fmf metadata
  tree from which the respective test, plan or story have been
  created.
  
  In a similar way, the ``tree`` property of the ``Tree`` instance
  points to the original ``fmf.Tree`` from which it was initialized.
  
  
  Essential Classes
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  
  .. automodule:: tmt
      :members:
      :undoc-members:
  
files: docs/classes.rst
files: 
  """ Base Metadata Classes """
  
  import copy
  import dataclasses
  import enum
  import functools
  import os
  import re
  import shutil
  import subprocess
  import sys
  import time
  from typing import Any, ClassVar, Dict, List, Optional, Tuple, Union, cast
  
  import click
  import fmf
  import fmf.base
  from click import confirm, echo, style
  from fmf.utils import listed
  from ruamel.yaml.error import MarkedYAMLError
  
  import tmt.export
  import tmt.identifier
  import tmt.steps
  import tmt.steps.discover
  import tmt.steps.execute
  import tmt.steps.finish
  import tmt.steps.prepare
  import tmt.steps.provision
  import tmt.steps.report
  import tmt.templates
  import tmt.utils
  from tmt.utils import FmfContextType, verdict
  
  if sys.version_info >= (3, 8):
      from typing import Literal, TypedDict
  else:
      from typing_extensions import Literal, TypedDict
  
  # Default test duration is 5m for individual tests discovered from L1
  # metadata and 1h for scripts defined directly in plans (L2 metadata).
  DEFAULT_TEST_DURATION_L1 = '5m'
  DEFAULT_TEST_DURATION_L2 = '1h'
  DEFAULT_ORDER = 50
  
  # How many already existing lines should tmt run --follow show
  FOLLOW_LINES = 10
  
  # Obsoleted test keys
  OBSOLETED_TEST_KEYS = "relevancy coverage".split()
  
  # Unofficial temporary test keys
  EXTRA_TEST_KEYS = (
      "extra-nitrate extra-hardware extra-pepa "
      "extra-summary extra-task id".split())
  
  # Unofficial temporary story keys
  EXTRA_STORY_KEYS = ("id".split())
  
  SECTIONS_HEADINGS = {
      'Setup': ['<h1>Setup</h1>'],
      'Test': ['<h1>Test</h1>',
               '<h1>Test .*</h1>'],
      'Step': ['<h2>Step</h2>',
               '<h2>Test Step</h2>'],
      'Expect': ['<h2>Expect</h2>',
                 '<h2>Result</h2>',
                 '<h2>Expected Result</h2>'],
      'Cleanup': ['<h1>Cleanup</h1>']
      }
  
  # Extra keys used for identification in Result class
  EXTRA_RESULT_IDENTIFICATION_KEYS = ['extra-nitrate', 'extra-task']
  
  
  #
  # fmf id types
  #
  # See https://fmf.readthedocs.io/en/latest/concept.html#identifiers for
  # formal specification.
  #
  
  # A "raw" fmf id as stored in fmf node data, i.e. as a mapping with various keys.
  # Used for a brief moment, to annotate raw fmf data before they are converted
  # into FmfId instances.
  _RawFmfId = TypedDict('_RawFmfId', {
      'url': Optional[str],
      'ref': Optional[str],
      'path': Optional[str],
      'name': Optional[str]
  })
  
  
  # An internal fmf id representation.
  @dataclasses.dataclass
  class FmfId(tmt.utils.SpecBasedContainer, tmt.utils.SerializableContainer):
      # The list of valid fmf id keys
      keys: ClassVar[List[str]] = ['url', 'ref', 'path', 'name']
  
      url: Optional[str] = None
      ref: Optional[str] = None
      path: Optional[str] = None
      name: Optional[str] = None
  
      def to_spec(self) -> Dict[str, Any]:
          """ Convert to a form suitable for saving in a specification file """
  
          return self.to_dict()
  
      @classmethod
      def from_spec(cls, raw: _RawFmfId) -> 'FmfId':
          """ Convert from a specification file or from a CLI option """
  
          return FmfId(**{key: raw.get(key, None) for key in cls.keys})
  
      def validate(self) -> Tuple[bool, str]:
          """
          Validate fmf id and return a human readable error
  
          Return a tuple (boolean, message) as the result of validation.
          The boolean specifies the validation result and the message
          the validation error. In case the FMF id is valid, return an empty
          string as the message.
          """
          # Validate remote id and translate to human readable errors
          try:
              # Simple asdict() is not good enough, fmf does not like keys that exist but are `None`.
              # Don't include those.
              fmf.base.Tree.node({
                  key: value for key, value in self.items()
                  if value is not None
                  })
          except fmf.utils.GeneralError as error:
              # Map fmf errors to more user friendly alternatives
              error_map: List[Tuple[str, str]] = [
                  ('git clone', f"repo '{self.url}' cannot be cloned"),
                  ('git checkout', f"git ref '{self.ref}' is invalid"),
                  ('directory path', f"path '{self.path}' is invalid"),
                  ('tree root', f"No tree found in repo '{self.url}', missing an '.fmf' directory?")
                  ]
  
              stringified_error = str(error)
  
              errors = [message for needle, message in error_map if needle in stringified_error]
              return (False, errors[0] if errors else stringified_error)
  
          return (True, '')
  
  #
  # Various types describing constructs as stored in "raw" fmf node data.
  # Used for a brief moment, to annotate raw fmf data before they are converted
  # into their internal representations.
  #
  
  #
  # A type describing the raw form of the core `link` attribute. See
  # https://tmt.readthedocs.io/en/stable/spec/core.html#link for its
  # formal specification. Internally, a link is represented by a `Link`
  # class instance, and types below describe the raw data coming from Fmf
  # nodes and CLI options.
  
  
  # Link relations.
  _RawLinkRelationName = Literal[
      'verifies', 'verified-by',
      'implements', 'implemented-by',
      'documents', 'documented-by',
      'blocks', 'blocked-by',
      'duplicates', 'duplicated-by',
      'parent', 'child',
      'relates',
      # Special case: not a relation, but it can appear where relations appear in
      # link data structures.
      'note'
  ]
  
  # Link target - can be either a string (like test case name or URL), or an fmf id.
  _RawLinkTarget = Union[str, _RawFmfId]
  
  # Basic "relation-aware" link - essentialy a mapping with one key/value pair.
  _RawLinkRelation = Dict[_RawLinkRelationName, _RawLinkTarget]
  
  # A single link can be represented as a string or FMF ID (meaning only target is specified),
  # or a "relation-aware" link aka mapping defined above.
  _RawLink = Union[
      str,
      _RawFmfId,
      _RawLinkRelation
  ]
  
  # Collection of links - can be either a single link, or a list of links, and all
  # link forms may be used together.
  _RawLinks = Union[
      _RawLink,
      List[_RawLink]
  ]
  
  
  # A type describing `adjust` content. See
  # https://tmt.readthedocs.io/en/stable/spec/core.html#adjust for its formal specification.
  #
  # The type is incomplete in the sense it does not describe all keys it may contain,
  # like `environment` or `provision`, and focuses only on the keys defined for `adjust`
  # itself.
  _RawAdjustRule = TypedDict(
      'AdjustRuleType',
      {
          'when': Optional[str],
          'continue': Optional[bool],
          'because': Optional[str]
          }
  )
  
  # A type describing content accepted by various require-like keys: - a string, fmf id,
  # or a list with a mixture of these two types.
  #
  # TODO: what other keys accept this kind of fmf data??
  _RawRequire = Union[
      str,
      _RawFmfId,
      List[Union[str, _RawFmfId]]
  ]
  
  
  class Core(tmt.utils.Common):
      """
      General node object
  
      Corresponds to given fmf.Tree node.
      Implements common Test, Plan and Story methods.
      Also defines L0 metadata and its manipulation.
      """
  
      # Core attributes (supported across all levels)
      summary: Optional[str] = None
      description: Optional[str] = None
      enabled: bool = True
      order: int = DEFAULT_ORDER
      link: Optional['Links'] = None
      id: Optional[str] = None
      tag: List[str] = []
      tier: Optional[str] = None
      adjust: Optional[List[_RawAdjustRule]] = None
  
      KEYS_SHOW_ORDER = [
          # Basic stuff
          'summary',
          'description',
          'enabled',
          'order',
          'id',
  
          # Filtering and more
          'tag',
          'tier',
          'link',
          'adjust',
          ]
  
      # Normalization methods
      _normalize_tag = tmt.utils.LoadFmfKeysMixin._normalize_string_list
  
      # TODO: remove when schema becomes mandatory, `order` shall never be `null`
      def _normalize_order(self, value: Optional[int]) -> int:
          if value is None:
              return DEFAULT_ORDER
          return int(value)
  
      def _normalize_link(self, value: _RawLinks) -> 'Links':
          return Links(data=value)
  
      def _normalize_adjust(
              self, value: Union[_RawAdjustRule, List[_RawAdjustRule]]) -> List[_RawAdjustRule]:
          if value is None:
              return []
          return [value] if not isinstance(value, list) else value
  
      def _normalize_tier(self, value: Optional[Union[int, str]]) -> Optional[str]:
          if value is None:
              return None
          return str(value)
  
      def __init__(self,
                   *,
                   node: fmf.Tree,
                   parent: Optional[tmt.utils.Common] = None,
                   **kwargs: Any) -> None:
          """ Initialize the node """
          kwargs.setdefault('logger', self)
          super().__init__(node=node, parent=parent, name=node.name, **kwargs)
  
          self.node = node
  
          # Verify id is not inherited from parent.
          if self.id and not tmt.identifier.key_defined_in_leaf(node, 'id'):
              raise tmt.utils.SpecificationError(
                  f"The 'id' key '{node.get('id')}' in '{self.name}' "
                  f"is inherited from parent, should be defined in a leaf.")
  
          # Store original metadata with applied defaults and including
          # keys which are not defined in the L1 metadata specification
          # Once the whole node has been initialized,
          # self._update_metadata() must be called to work correctly.
          self._metadata = self.node.data.copy()
  
      def __str__(self):
          """ Node name """
          return self.name
  
      def _update_metadata(self):
          """ Update the _metadata attribute """
          self._metadata.update(self.export(format_='dict'))
          self._metadata['name'] = self.name
  
      def _show_additional_keys(self):
          """ Show source files """
          if self.id is not None:
              echo(tmt.utils.format('id', self.id, key_color='magenta'))
          echo(tmt.utils.format(
              'sources', self.node.sources, key_color='magenta'))
  
      def _fmf_id(self):
          """ Show fmf identifier """
          echo(tmt.utils.format('fmf-id', self.fmf_id, key_color='magenta'))
  
      @property
      @functools.lru_cache(maxsize=None)
      def fmf_id(self):
          """ Return full fmf identifier of the node """
  
          def run(command):
              """ Run command, return output """
              cwd = fmf_root
              result = subprocess.run(
                  command.split(),
                  stdout=subprocess.PIPE,
                  stderr=subprocess.DEVNULL,
                  cwd=cwd)
              return result.stdout.strip().decode("utf-8")
  
          fmf_id = {'name': self.name}
  
          fmf_root = self.node.root
          # Prepare url (for now handle just the most common schemas)
          branch = run("git rev-parse --abbrev-ref --symbolic-full-name @{u}")
          try:
              remote_name = branch[:branch.index('/')]
          except ValueError:
              remote_name = 'origin'
          remote = run(f"git config --get remote.{remote_name}.url")
          fmf_id['url'] = tmt.utils.public_git_url(remote)
  
          # Get the ref (skip for master as it is the default)
          ref = run('git rev-parse --abbrev-ref HEAD')
          if ref != 'master':
              fmf_id['ref'] = ref
  
          # Construct path (if different from git root)
          git_root = run('git rev-parse --show-toplevel')
          if git_root != fmf_root:
              fmf_id['path'] = os.path.join(
                  '/', os.path.relpath(fmf_root, git_root))
  
          return fmf_id
  
      @classmethod
      def _save_context(cls, context):
          """ Save provided command line context for future use """
          super(Core, cls)._save_context(context)
  
          # Handle '.' as an alias for the current working directory
          names = cls._opt('names')
          if names is not None and '.' in names:
              root = context.obj.tree.root
              current = os.getcwd()
              # Handle special case when directly in the metadata root
              if current == root:
                  pattern = '/'
              # Prepare path from the tree root to the current directory
              else:
                  pattern = os.path.join('/', os.path.relpath(current, root))
                  # Prevent matching common prefix from other directories
                  pattern = f"{pattern}(/|$)"
              cls._context.params['names'] = tuple([
                  pattern if name == '.' else name for name in names])
  
      def name_and_summary(self):
          """ Node name and optional summary """
          if self.summary:
              return '{0} ({1})'.format(self.name, self.summary)
          else:
              return self.name
  
      def ls(self, summary=False):
          """ List node """
          echo(style(self.name, fg='red'))
          if summary and self.summary:
              echo(tmt.utils.format('summary', self.summary))
  
      def export(self, format_='yaml', keys=None):
          """ Export data into requested format (yaml or dict) """
          if keys is None:
              keys = self._keys()
  
          # Always include node name, add requested keys, ignore adjust
          data = dict(name=self.name)
          for key in keys:
              if key == 'adjust':
                  continue
  
              value = getattr(self, key)
  
              if key == 'link' and value:
                  # TODO: links must be saved in a form that can be than crunched by
                  # Links.__init__() method - it is tempting to use to_serialized()
                  # and from_unserialized(), but we don't use unserialization code
                  # when loading saved data back, so we can't go this way. Yet.
                  data[key] = cast('Links', value).to_spec()
  
              else:
                  data[key] = value
  
          # Choose proper format
          if format_ == 'dict':
              return data
          elif format_ == 'yaml':
              return tmt.utils.dict_to_yaml(data)
          else:
              raise tmt.utils.GeneralError(
                  f"Invalid export format '{format_}'.")
  
      def lint_keys(self, additional_keys):
          """ Return list of invalid keys used, empty when all good """
          known_keys = additional_keys + self._keys()
          return [key for key in self.node.get().keys() if key not in known_keys]
  
      def _lint_summary(self):
          """ Lint summary attribute """
          # Summary is advised with a resonable length
          if self.summary is None:
              verdict(None, "summary is very useful for quick inspection")
          elif len(self.summary) > 50:
              verdict(None, "summary should not exceed 50 characters")
          return True
  
      def has_link(self, needle: 'LinkNeedle') -> bool:
          """ Whether object contains specified link """
  
          if self.link is None:
              return False
  
          return self.link.has_link(needle)
  
  
  Node = Core
  
  
  class Test(tmt.utils.ValidateFmfMixin, tmt.utils.LoadFmfKeysMixin, Core):
      """ Test object (L1 Metadata) """
  
      # Basic test information
      contact: List[str] = []
      component: List[str] = []
  
      # Test execution data
      test: str
      path: Optional[str] = None
      framework: Optional[str] = None
      manual: bool = False
      require: List[Union[str, FmfId]] = []
      recommend: List[str] = []
      environment: Optional[tmt.utils.EnvironmentType] = {}
      duration: Optional[str] = DEFAULT_TEST_DURATION_L1
      result: str = 'respect'
  
      _normalize_contact = tmt.utils.LoadFmfKeysMixin._normalize_string_list
      _normalize_component = tmt.utils.LoadFmfKeysMixin._normalize_string_list
      _normalize_recommend = tmt.utils.LoadFmfKeysMixin._normalize_string_list
  
      def _normalize_require(
              self, value: Optional[_RawRequire]) -> List[Union[str, FmfId]]:
          if value is None:
              return []
          return [value] if isinstance(value, str) else value
  
      KEYS_SHOW_ORDER = [
          # Basic test information
          'summary',
          'description',
          'contact',
          'component',
          'id',
  
          # Test execution data
          'test',
          'path',
          'framework',
          'manual',
          'require',
          'recommend',
          'environment',
          'duration',
          'enabled',
          'order',
          'result',
  
          # Filtering attributes
          'tag',
          'tier',
          'link',
          ]
  
      @classmethod
      def from_dict(cls, mapping: Dict[str, Any], name: str, skip_validation: bool = False,
                    raise_on_validation_error: bool = False, **kwargs: Any) -> 'Test':
          """
          Initialize test data from a dictionary.
  
          Useful when data describing a test are stored in a mapping instead of an fmf node.
          """
  
          if not name.startswith('/'):
              raise tmt.utils.SpecificationError("Test name should start with a '/'.")
  
          node = fmf.Tree(mapping)
          node.name = name
  
          return cls(
              node=node,
              skip_validation=skip_validation,
              raise_on_validation_error=raise_on_validation_error,
              **kwargs)
  
      def __init__(
              self,
              *,
              node: fmf.Tree,
              skip_validation: bool = False,
              raise_on_validation_error: bool = False,
              **kwargs: Any) -> None:
          """
          Initialize test data from an fmf node or a dictionary
  
          The following two methods are supported:
  
              Test(node)
          """
  
          # Path defaults to the directory where metadata are stored or to
          # the root '/' if fmf metadata were not stored on the filesystem
          #
          # NOTE: default value of `path` attribute is not known when attribute
          # is declared, therefore we need to compute the default value and
          # assign it to attribute *before* calling superclass and its handy
          # node key extraction.
          try:
              directory = os.path.dirname(node.sources[-1])
              relative_path = os.path.relpath(directory, node.root)
              if relative_path == '.':
                  default_path = '/'
              else:
                  default_path = os.path.join('/', relative_path)
          except (AttributeError, IndexError):
              default_path = '/'
  
          self.path = default_path
  
          kwargs.setdefault('logger', self)
          super().__init__(
              node=node,
              skip_validation=skip_validation,
              raise_on_validation_error=raise_on_validation_error,
              **kwargs)
  
          # TODO: As long as validation is optional, a missing `test` key would be reported
          # as such but won't stop tmt from moving on.
          if self.test is None:
              raise tmt.utils.SpecificationError(
                  f"The 'test' attribute in '{self.name}' must be defined.")
  
          if self.framework == 'beakerlib':
              self.require.append('beakerlib')
  
          self._update_metadata()
  
      @staticmethod
      def overview(tree):
          """ Show overview of available tests """
          tests = [
              style(str(test), fg='red') for test in tree.tests()]
          echo(style(
              'Found {}{}{}.'.format(
                  listed(tests, 'test'),
                  ': ' if tests else '',
                  listed(tests, max=12)
                  ), fg='blue'))
  
      @staticmethod
      def create(name, template, path, force=False, dry=None):
          """ Create a new test """
          if dry is None:
              dry = Test._opt('dry')
  
          # Create directory
          if name == '.':
              directory_path = os.getcwd()
          else:
              directory_path = os.path.join(path, name.lstrip('/'))
              tmt.utils.create_directory(
                  directory_path, 'test directory', dry=dry)
  
          # Create metadata
          try:
              metadata_path = os.path.join(directory_path, 'main.fmf')
              tmt.utils.create_file(
                  path=metadata_path,
                  content=tmt.templates.TEST_METADATA[template],
                  name='test metadata',
                  dry=dry,
                  force=force)
          except KeyError:
              raise tmt.utils.GeneralError(f"Invalid template '{template}'.")
  
          # Create script
          script_path = os.path.join(directory_path, 'test.sh')
          try:
              content = tmt.templates.TEST[template]
          except KeyError:
              raise tmt.utils.GeneralError(f"Invalid template '{template}'.")
          tmt.utils.create_file(
              path=script_path, content=content,
              name='test script', dry=dry, force=force, mode=0o755)
  
      def show(self):
          """ Show test details """
          self.ls()
          for key in self.KEYS_SHOW_ORDER:
              value = getattr(self, key)
              if key == 'link':
                  value.show()
                  continue
              # No need to show the default order
              if key == 'order' and value == DEFAULT_ORDER:
                  continue
              if value not in [None, list(), dict()]:
                  echo(tmt.utils.format(key, value))
          if self.opt('verbose'):
              self._show_additional_keys()
              self._fmf_id()
          if self.opt('verbose', 0) >= 2:
              # Print non-empty unofficial attributes
              for key in sorted(self.node.get().keys()):
                  # Already asked to be printed
                  if key in self._keys():
                      continue
                  value = self.node.get(key)
                  if value not in [None, list(), dict()]:
                      echo(tmt.utils.format(key, value, key_color='blue'))
  
      def _lint_manual(self, test_path):
          """ Check that the manual instructions respect the specification """
          manual_test = os.path.join(test_path, self.test)
  
          # File does not exist
          if not os.path.exists(manual_test):
              return verdict(False, f"file '{self.test}' does not exist")
  
          # Check syntax for warnings
          warnings = tmt.export.check_md_file_respects_spec(manual_test)
          if warnings:
              for warning in warnings:
                  verdict(False, warning)
              return False
  
          # Everything looks ok
          return verdict(True, f"correct manual test syntax in '{self.test}'")
  
      def lint(self):
          """
          Check test against the L1 metadata specification.
  
          Return whether the test is valid.
          """
          self.ls()
          stripped_path = self.path.strip()
          test_path = self.node.root + stripped_path
  
          # Check test, path and summary (use bitwise '&' because 'and' is
          # lazy and would skip all verdicts following the first fail)
          valid = verdict(
              bool(self.test), 'test script must be defined')
          valid &= verdict(
              stripped_path.startswith('/'), 'directory path must be absolute')
          valid &= verdict(
              os.path.exists(test_path), 'directory path must exist')
          self._lint_summary()
  
          # Check for possible test case relevancy rules
          filename = self.node.sources[-1]
          metadata = tmt.utils.yaml_to_dict(self.read(filename))
          relevancy = metadata.pop('relevancy', None)
          if relevancy:
              # Convert into adjust rules if --fix enabled
              if self.opt('fix'):
                  metadata['adjust'] = tmt.convert.relevancy_to_adjust(relevancy)
                  self.write(filename, tmt.utils.dict_to_yaml(metadata))
                  verdict(None, 'relevancy converted into adjust')
              else:
                  valid = verdict(
                      False, 'relevancy has been obsoleted by adjust')
  
          # Check for possible coverage attribute
          coverage = metadata.pop('coverage', None)
          if coverage:
              valid = verdict(False, 'coverage has been obsoleted by link')
          # Check for unknown attributes
          # FIXME - Make additional attributes configurable
          # We don't want adjust in show/export so it is not yet in Test._keys
          invalid_keys = self.lint_keys(
              EXTRA_TEST_KEYS + OBSOLETED_TEST_KEYS + ['adjust'])
          if invalid_keys:
              valid = False
              for key in invalid_keys:
                  verdict(False, f"unknown attribute '{key}' is used")
          else:
              verdict(True, "correct attributes are used")
  
          # Check if the format of Markdown file respects the specification
          # https://tmt.readthedocs.io/en/latest/spec/tests.html#manual
          if self.manual:
              valid &= self._lint_manual(test_path)
  
          return valid
  
      def export(self, format_='yaml', keys=None):
          """
          Export test data into requested format
  
          In addition to 'yaml' and 'dict' it supports also a special
          format 'execute' used by the execute step which returns
          (test-name, test-data) tuples.
          """
  
          # Export to Nitrate test case management system
          if format_ == 'nitrate':
              tmt.export.export_to_nitrate(self)
  
          # Export to Polarion test case management system
          elif format_ == 'polarion':
              tmt.export.export_to_polarion(self)
  
          # Export the fmf identifier
          elif keys == 'fmf-id':
              if format_ == 'dict':
                  return self.fmf_id
              elif format_ == 'yaml':
                  return tmt.utils.dict_to_yaml(self.fmf_id)
              else:
                  raise tmt.utils.GeneralError(
                      f"Invalid test export format '{format_}'.")
  
          # Common node export otherwise
          else:
              return super().export(format_, keys)
  
  
  class Plan(tmt.utils.ValidateFmfMixin, tmt.utils.LoadFmfKeysMixin, Core):
      """ Plan object (L2 Metadata) """
  
      # `environment` and `environment-file` are NOT promoted to instance variables.
      context: FmfContextType = {}
      gate: List[str] = []
  
      _normalize_context = tmt.utils.LoadFmfKeysMixin._normalize_environment
      _normalize_gate = tmt.utils.LoadFmfKeysMixin._normalize_string_list
  
      extra_L2_keys = [
          'context',
          'environment',
          'environment-file',
          'gate',
          ]
  
      def __init__(
              self,
              *,
              node: fmf.Tree,
              run: Optional['Run'] = None,
              skip_validation: bool = False,
              raise_on_validation_error: bool = False,
              **kwargs: Any) -> None:
          """ Initialize the plan """
          kwargs.setdefault('logger', self)
          super().__init__(
              node=node,
              parent=run,
              skip_validation=skip_validation,
              raise_on_validation_error=raise_on_validation_error,
              **kwargs)
  
          # Save the run, prepare worktree and plan data directory
          self.my_run = run
          if self.my_run:
              # Skip to initialize the work tree if the corresponding option is
              # true. Note that 'tmt clean' consumes the option because it
              # should not initialize the work tree at all.
              if not run.opt(tmt.utils.PLAN_SKIP_WORKTREE_INIT):
                  self._initialize_worktree()
  
              self._initialize_data_directory()
  
          # Store 'environment' and 'environment-file' keys content
          self._environment = self._get_environment_vars(node)
          # Expand all environment variables in the node
          with tmt.utils.modify_environ(self.environment):
              self._expand_node_data(node.data)
  
          # Initialize test steps
          self.discover = tmt.steps.discover.Discover(
              plan=self, data=self.node.get('discover'))
          self.provision = tmt.steps.provision.Provision(
              plan=self, data=self.node.get('provision'))
          self.prepare = tmt.steps.prepare.Prepare(
              plan=self, data=self.node.get('prepare'))
          self.execute = tmt.steps.execute.Execute(
              plan=self, data=self.node.get('execute'))
          self.report = tmt.steps.report.Report(
              plan=self, data=self.node.get('report'))
          self.finish = tmt.steps.finish.Finish(
              plan=self, data=self.node.get('finish'))
  
          # Test execution context defined in the plan
          self._plan_context = self.node.get('context', dict())
  
          self._update_metadata()
  
      def _expand_node_data(self, data):
          """ Recursively expand variables in node data """
          if isinstance(data, str):
              return os.path.expandvars(data)
          elif isinstance(data, dict):
              for key, value in data.items():
                  data[key] = self._expand_node_data(value)
          elif isinstance(data, list):
              for i in range(len(data)):
                  data[i] = self._expand_node_data(data[i])
          return data
  
      @property
      def environment(self):
          """ Return combined environment from plan data and command line """
          if self.my_run:
              combined = self._environment.copy()
              # Command line variables take precedence
              combined.update(self.my_run.environment)
              # Include path to the plan data directory
              combined["TMT_PLAN_DATA"] = self.data_directory
              return combined
          else:
              return self._environment
  
      def _get_environment_vars(self, node):
          """ Get variables from 'environment' and 'environment-file' keys """
          # Environment variables from files
          environment_files = node.get("environment-file") or []
          if not isinstance(environment_files, list):
              raise tmt.utils.SpecificationError(
                  f"The 'environment-file' should be a list. "
                  f"Received '{type(environment_files).__name__}'.")
          combined = tmt.utils.environment_files_to_dict(
              environment_files, root=node.root)
  
          # Environment variables from key, make sure that values are string
          environment = dict([
              (key, str(value)) for key, value
              in node.get('environment', dict()).items()])
  
          # Combine both sources into one ('environment' key takes precendence)
          combined.update(environment)
          return combined
  
      def _initialize_worktree(self):
          """
          Prepare the worktree, a copy of the metadata tree root
  
          Used as cwd in prepare, execute and finish steps.
          """
          self.worktree = os.path.join(self.workdir, 'tree')
          tree_root = self.my_run.tree.root
  
          # Create an empty directory if there's no metadata tree
          if not tree_root:
              self.debug('Create an empty worktree (no metadata tree).', level=2)
              os.makedirs(self.worktree, exist_ok=True)
              return
  
          # Sync metadata root to the worktree
          self.debug(f"Sync the worktree to '{self.worktree}'.", level=2)
          self.run([
              "rsync", "-ar", "--exclude", ".git",
              f"{tree_root}/", self.worktree])
  
      def _initialize_data_directory(self):
          """
          Create the plan data directory
  
          This is used for storing logs and other artifacts created during
          prepare step, test execution or finish step and which are pulled
          from the guest for possible future inspection.
          """
          self.data_directory = os.path.join(self.workdir, "data")
          self.debug(
              f"Create the data directory '{self.data_directory}'.", level=2)
          os.makedirs(self.data_directory, exist_ok=True)
  
      def _fmf_context(self):
          """ Return combined context from plan data and command line """
          combined = self._plan_context.copy()
          combined.update(self._context.obj.fmf_context)
          return combined
  
      @staticmethod
      def edit_template(content):
          """ Edit the default template with custom values """
  
          content = tmt.utils.yaml_to_dict(content)
  
          # For each step check for possible command line data
          for step in tmt.steps.STEPS:
              options = Plan._opt(step)
              if not options:
                  continue
              step_data = []
  
              # For each option check for valid yaml and store
              for option in options:
                  try:
                      # FIXME: Ruamel.yaml "remembers" the used formatting when
                      #        using round-trip mode and since it comes from the
                      #        command-line, no formatting is applied resulting
                      #        in inconsistent formatting. Using a safe loader in
                      #        this case is a hack to make it forget, though
                      #        there may be a better way to do this.
                      try:
                          data = tmt.utils.yaml_to_dict(option, yaml_type='safe')
                          if not (data):
                              raise tmt.utils.GeneralError("Step data cannot be empty.")
                      except tmt.utils.GeneralError as error:
                          raise tmt.utils.GeneralError(
                              f"Invalid step data for {step}: '{option}'"
                              f"\n{error}")
                      step_data.append(data)
                  except MarkedYAMLError as error:
                      raise tmt.utils.GeneralError(
                          f"Invalid yaml data for {step}:\n{error}")
  
              # Use list only when multiple step data provided
              if len(step_data) == 1:
                  step_data = step_data[0]
              content[step] = step_data
  
          return tmt.utils.dict_to_yaml(content)
  
      @staticmethod
      def overview(tree):
          """ Show overview of available plans """
          plans = [
              style(str(plan), fg='red') for plan in tree.plans()]
          echo(style(
              'Found {}{}{}.'.format(
                  listed(plans, 'plan'),
                  ': ' if plans else '',
                  listed(plans, max=12)
                  ), fg='blue'))
  
      @staticmethod
      def create(name, template, path, force=False, dry=None):
          """ Create a new plan """
          # Prepare paths
          if dry is None:
              dry = Plan._opt('dry')
  
          (directory, plan) = os.path.split(name)
          directory_path = os.path.join(path, directory.lstrip('/'))
          has_fmf_ext = os.path.splitext(plan)[1] == '.fmf'
          plan_path = os.path.join(
              directory_path, plan + ('' if has_fmf_ext else '.fmf'))
  
          # Create directory & plan
          tmt.utils.create_directory(directory_path, 'plan directory', dry=dry)
          try:
              content = tmt.templates.PLAN[template]
          except KeyError:
              raise tmt.utils.GeneralError(
                  "Invalid template '{}'.".format(template))
  
          # Override template with data provided on command line
          content = Plan.edit_template(content)
  
          tmt.utils.create_file(
              path=plan_path, content=content,
              name='plan', dry=dry, force=force)
  
      def steps(self, enabled=True, disabled=False, names=False, skip=None):
          """
          Iterate over enabled / all steps
  
          Yields instances of all enabled steps by default. Use 'names' to
          yield step names only and 'disabled=True' to iterate over all.
          Use 'skip' to pass the list of steps to be skipped.
          """
          skip = skip or []
          for name in tmt.steps.STEPS:
              if name in skip:
                  continue
              step = getattr(self, name)
              if (enabled and step.enabled or disabled and not step.enabled):
                  yield name if names else step
  
      def show(self):
          """ Show plan details """
  
          # Summary and description first
          self.ls(summary=True)
          if self.description:
              echo(tmt.utils.format(
                  'description', self.description, key_color='green'))
  
          # Individual step details
          for step in self.steps(disabled=True):
              step.show()
  
          # Environment and context
          if self.environment:
              echo(tmt.utils.format(
                  'environment', self.environment, key_color='blue'))
          if self._fmf_context():
              echo(tmt.utils.format(
                  'context', self._fmf_context(), key_color='blue'))
  
          # The rest
          echo(tmt.utils.format('enabled', self.enabled, key_color='cyan'))
          if self.order != DEFAULT_ORDER:
              echo(tmt.utils.format('order', self.order, key_color='cyan'))
          if self.id:
              echo(tmt.utils.format('id', self.id, key_color='cyan'))
          if self.tag:
              echo(tmt.utils.format('tag', self.tag, key_color='cyan'))
          if self.tier:
              echo(tmt.utils.format('tier', self.tier, key_color='cyan'))
          self.link.show()
          if self.opt('verbose'):
              self._show_additional_keys()
  
      def _lint_execute(self):
          """ Lint execute step """
          execute = self.node.get('execute')
          if not execute:
              return verdict(False, "execute step must be defined with 'how'")
          if isinstance(execute, dict):
              execute = [execute]
  
          methods = [
              method.name
              for method in tmt.steps.execute.ExecutePlugin.methods()]
          correct = True
          for configuration in execute:
              how = configuration.get('how')
              if how not in methods:
                  name = configuration.get('name')
                  verdict(False,
                          f"unsupported execute method '{how}' in '{name}'")
                  correct = False
  
          return correct
  
      def _lint_discover(self):
          """ Lint discover step """
          # The discover step is optional
          discover = self.node.get('discover')
          if not discover:
              return True
          if isinstance(discover, dict):
              discover = [discover]
  
          methods = [
              method.name
              for method in tmt.steps.discover.DiscoverPlugin.methods()]
          correct = True
          for configuration in discover:
              how = configuration.get('how')
  
              if how not in methods:
                  verdict(False, f"unknown discover method '{how}'")
                  correct = False
                  continue
  
              # FIXME Add check for the shell discover method
              if how == 'shell':
                  continue
  
              if not self._lint_discover_fmf(configuration):
                  correct = False
          return correct
  
      @staticmethod
      def _lint_discover_fmf(discover):
          """ Lint fmf discover method """
          # Validate remote id and translate to human readable errors
          fmf_id_data = {
              key: value
              for key, value in discover.items()
              if key in ['url', 'ref', 'path']
              }
  
          valid, error = FmfId(**fmf_id_data).validate()
  
          if valid:
              name = discover.get('name')
              return verdict(True, f"fmf remote id in '{name}' is valid")
  
          return verdict(False, error)
  
      def lint(self):
          """
          Check plan against the L2 metadata specification
  
          Return whether the plan is valid.
          """
          self.ls()
  
          # Explore all available plugins
          tmt.plugins.explore()
  
          invalid_keys = self.lint_keys(
              list(self.steps(enabled=True, disabled=True, names=True)) +
              self.extra_L2_keys)
  
          if invalid_keys:
              for key in invalid_keys:
                  verdict(False, f"unknown attribute '{key}' is used")
          else:
              verdict(True, "correct attributes are used")
  
          return all([
              self._lint_summary(),
              self._lint_execute(),
              self._lint_discover(),
              len(invalid_keys) == 0])
  
      def go(self):
          """ Execute the plan """
          # Show plan name and summary (one blank line to separate plans)
          self.info('')
          self.info(self.name, color='red')
          if self.summary:
              self.verbose('summary', self.summary, 'green')
  
          # Additional debug info like plan environment
          self.debug('info', color='cyan', shift=0, level=3)
          self.debug('environment', self.environment, 'magenta', level=3)
          self.debug('context', self._fmf_context(), 'magenta', level=3)
  
          # Wake up all steps
          self.debug('wake', color='cyan', shift=0, level=2)
          for step in self.steps(disabled=True):
              self.debug(str(step), color='blue', level=2)
              try:
                  step.wake()
              except tmt.utils.SpecificationError as error:
                  # Re-raise the exception if the step is enabled (invalid
                  # step data), otherwise just warn the user and continue.
                  if step.enabled:
                      raise error
                  else:
                      step.warn(error)
  
          # Set up login and reboot plugins for all steps
          self.debug("action", color="blue", level=2)
          for step in self.steps(disabled=True):
              step.setup_actions()
  
          # Check if steps are not in stand-alone mode
          standalone = set()
          for step in self.steps():
              standalone_plugins = step.plugins_in_standalone_mode
              if standalone_plugins == 1:
                  standalone.add(step.name)
              elif standalone_plugins > 1:
                  raise tmt.utils.GeneralError(
                      f"Step '{step.name}' has multiple plugin configs which "
                      f"require running on their own. Combination of such "
                      f"configs is not possible.")
          if len(standalone) > 1:
              raise tmt.utils.GeneralError(
                  f'These steps require running on their own, their combination '
                  f'with the given options is not compatible: '
                  f'{fmf.utils.listed(standalone)}.')
          elif standalone:
              self._context.obj.steps = standalone
              self.debug(
                  f"Running the '{list(standalone)[0]}' step as standalone.")
  
          # Run enabled steps except 'finish'
          self.debug('go', color='cyan', shift=0, level=2)
          try:
              abort = False
              for step in self.steps(skip=['finish']):
                  step.go()
                  # Finish plan if no tests found (except dry mode)
                  if (step.name == 'discover' and not step.tests()
                          and not self.opt('dry')):
                      step.info(
                          'warning', 'No tests found, finishing plan.',
                          color='yellow', shift=1)
                      abort = True
                      return
          # Make sure we run 'finish' step always if enabled
          finally:
              if not abort and self.finish.enabled:
                  self.finish.go()
  
      def export(self, format_='yaml'):
          """
          Export plan data into requested format
  
          Supported formats are 'yaml' and 'dict'.
          """
          data = super().export(format_='dict')
  
          for key in self.extra_L2_keys:
              value = self.node.data.get(key)
              if value:
                  data[key] = value
  
          for step in tmt.steps.STEPS:
              value = self.node.data.get(step)
              if value:
                  data[step] = value
  
          # Choose proper format
          if format_ == 'dict':
              return data
          elif format_ == 'yaml':
              return tmt.utils.dict_to_yaml(data)
          else:
              raise tmt.utils.GeneralError(
                  f"Invalid plan export format '{format_}'.")
  
  
  class StoryPriority(enum.Enum):
      MUST_HAVE = 'must have'
      SHOULD_HAVE = 'should have'
      COULD_HAVE = 'could have'
      WILL_NOT_HAVE = 'will not have'
  
      # We need a custom "to string" conversion to support fmf's filter().
      def __str__(self) -> str:
          return self.value
  
  
  class Story(tmt.utils.ValidateFmfMixin, tmt.utils.LoadFmfKeysMixin, Core):
      """ User story object """
  
      example: List[str] = []
      story: str
      title: Optional[str] = None
      priority: Optional[StoryPriority] = None
  
      _normalize_example = tmt.utils.LoadFmfKeysMixin._normalize_string_list
  
      def _normalize_priority(self, value: Optional[str]) -> Optional[StoryPriority]:
          if value is None:
              return None
          return StoryPriority(value)
  
      KEYS_SHOW_ORDER = [
          'summary',
          'title',
          'story',
          'id',
          'priority',
          'description',
          'example',
          'enabled',
          'order',
          'tag',
          'tier',
          'link',
          ]
  
      def __init__(
              self,
              *,
              node: fmf.Tree,
              skip_validation: bool = False,
              raise_on_validation_error: bool = False,
              **kwargs: Any) -> None:
          """ Initialize the story """
          kwargs.setdefault('logger', self)
          super().__init__(node=node, skip_validation=skip_validation,
                           raise_on_validation_error=raise_on_validation_error, **kwargs)
          self._update_metadata()
  
      @property
      def documented(self):
          """ Return links to relevant documentation """
          return self.link.get('documented-by')
  
      @property
      def verified(self):
          """ Return links to relevant test coverage """
          return self.link.get('verified-by')
  
      @property
      def implemented(self):
          """ Return links to relevant source code """
          return self.link.get('implemented-by')
  
      def _match(
              self, implemented, verified, documented, covered,
              unimplemented, unverified, undocumented, uncovered):
          """ Return true if story matches given conditions """
          if implemented and not self.implemented:
              return False
          if verified and not self.verified:
              return False
          if documented and not self.documented:
              return False
          if unimplemented and self.implemented:
              return False
          if unverified and self.verified:
              return False
          if undocumented and self.documented:
              return False
          if uncovered and (
                  self.implemented and self.verified and self.documented):
              return False
          if covered and not (
                  self.implemented and self.verified and self.documented):
              return False
          return True
  
      @staticmethod
      def create(name, template, path, force=False, dry=None):
          """ Create a new story """
          if dry is None:
              dry = Story._opt('dry')
  
          # Prepare paths
          (directory, story) = os.path.split(name)
          directory_path = os.path.join(path, directory.lstrip('/'))
          has_fmf_ext = os.path.splitext(story)[1] == '.fmf'
          story_path = os.path.join(directory_path,
                                    story + ('' if has_fmf_ext else '.fmf'))
  
          # Create directory & story
          tmt.utils.create_directory(directory_path, 'story directory', dry=dry)
          try:
              content = tmt.templates.STORY[template]
          except KeyError:
              raise tmt.utils.GeneralError(
                  "Invalid template '{}'.".format(template))
  
          tmt.utils.create_file(
              path=story_path, content=content,
              name='story', dry=dry, force=force)
  
      @staticmethod
      def overview(tree):
          """ Show overview of available stories """
          stories = [
              style(str(story), fg='red') for story in tree.stories()]
          echo(style(
              'Found {}{}{}.'.format(
                  listed(stories, 'story'),
                  ': ' if stories else '',
                  listed(stories, max=12)
                  ), fg='blue'))
  
      def show(self):
          """ Show story details """
          self.ls()
          for key in self.KEYS_SHOW_ORDER:
              value = getattr(self, key)
              if key == 'link':
                  value.show()
                  continue
              if key == 'priority' and value is not None:
                  value = cast(StoryPriority, value).value
              if key == 'order' and value == DEFAULT_ORDER:
                  continue
              if value is not None and value != []:
                  wrap = False if key == 'example' else 'auto'
                  echo(tmt.utils.format(key, value, wrap=wrap))
          if self.opt('verbose'):
              self._show_additional_keys()
  
      def coverage(self, code, test, docs):
          """ Show story coverage """
          if code:
              code = bool(self.implemented)
              verdict(code, good='done ', bad='todo ', nl=False)
          if test:
              test = bool(self.verified)
              verdict(test, good='done ', bad='todo ', nl=False)
          if docs:
              docs = bool(self.documented)
              verdict(docs, good='done ', bad='todo ', nl=False)
          echo(self)
          return (code, test, docs)
  
      def export(self, format_='rst', title=True):
          """ Export story data into requested format """
  
          # Use common Core export unless 'rst' requested
          if format_ != 'rst':
              return super().export(format_=format_)
  
          output = ''
  
          # Title and its anchor
          if title:
              depth = len(re.findall('/', self.name)) - 1
              if self.title and self.title != self.node.parent.get('title'):
                  title = self.title
              else:
                  title = re.sub('.*/', '', self.name)
              output += f'\n.. _{self.name}:\n'
              output += '\n{}\n{}\n'.format(title, '=~^:-><'[depth] * len(title))
  
          # Summary, story and description
          if self.summary and self.summary != self.node.parent.get('summary'):
              output += '\n{}\n'.format(self.summary)
          if self.story != self.node.parent.get('story'):
              output += '\n*{}*\n'.format(self.story.strip())
          # Insert note about unimplemented feature (leaf nodes only)
          if not self.node.children and not self.implemented:
              output += '\n.. note:: This is a draft, '
              output += 'the story is not implemented yet.\n'
          if (self.description and
                  self.description != self.node.parent.get('description')):
              output += '\n{}\n'.format(self.description)
  
          # Examples
          if self.example and self.example != self.node.parent.get('example'):
              examples = tmt.utils.listify(self.example)
              first = True
              for example in examples:
                  if first:
                      output += '\nExamples::\n\n'
                      first = False
                  else:
                      output += '\n::\n\n'
                  output += tmt.utils.format(
                      '', example, wrap=False, indent=4,
                      key_color=None, value_color=None) + '\n'
  
          # Status
          if not self.node.children:
              status = []
              for coverage in ['implemented', 'verified', 'documented']:
                  if getattr(self, coverage):
                      status.append(coverage)
              output += "\nStatus: {}\n".format(
                  listed(status) if status else 'idea')
  
          return output
  
      def _lint_story(self):
          story = self.node.get('story')
          if not story:
              return verdict(False, "story is required")
          return True
  
      def lint(self):
          """
          Check story against the L3 metadata specification.
  
          Return whether the story is valid.
          """
          self.ls()
          invalid_keys = self.lint_keys(EXTRA_STORY_KEYS)
  
          if invalid_keys:
              for key in invalid_keys:
                  verdict(False, f"unknown attribute '{key}' is used")
          else:
              verdict(True, "correct attributes are used")
  
          return all([self._lint_summary(),
                      self._lint_story(),
                      len(invalid_keys) == 0])
  
  
  class Tree(tmt.utils.Common):
      """ Test Metadata Tree """
  
      def __init__(self, path='.', tree=None, context=None):
          """ Initialize tmt tree from directory path or given fmf tree """
          self._path = path
          self._tree = tree
          self._custom_context = context
  
      def _fmf_context(self):
          """ Use custom fmf context if provided, default otherwise """
          if self._custom_context is not None:
              return self._custom_context
          return super()._fmf_context()
  
      def _filters_conditions(self, nodes, filters, conditions, links: List['LinkNeedle'], excludes):
          """ Apply filters and conditions, return pruned nodes """
          result = []
          for node in nodes:
              filter_vars = copy.deepcopy(node._metadata)
              cond_vars = node._metadata
              # Add a lowercase version of bool variables for filtering
              bool_vars = {
                  key: [value, str(value).lower()]
                  for key, value in filter_vars.items()
                  if isinstance(value, bool)}
              filter_vars.update(bool_vars)
              # Conditions
              try:
                  if not all(fmf.utils.evaluate(condition, cond_vars, node)
                             for condition in conditions):
                      continue
              except fmf.utils.FilterError:
                  # Handle missing attributes as if condition failed
                  continue
              except Exception as error:
                  raise tmt.utils.GeneralError(
                      f"Invalid --condition raised exception: {error}")
              # Filters
              try:
                  if not all(fmf.utils.filter(filter_, filter_vars, regexp=True)
                             for filter_ in filters):
                      continue
              except fmf.utils.FilterError:
                  # Handle missing attributes as if filter failed
                  continue
              # Links
              try:
                  # Links are in OR relation
                  if links and all(not node.has_link(needle) for needle in links):
                      continue
              except BaseException:
                  # Handle broken link as not matching
                  continue
              # Exclude
              if any(node for expr in excludes if re.search(expr, node.name)):
                  continue
              result.append(node)
          return result
  
      @property
      def tree(self):
          """ Initialize tree only when accessed """
          if self._tree is None:
              try:
                  self._tree = fmf.Tree(self._path)
              except fmf.utils.RootError:
                  raise tmt.utils.MetadataError(
                      f"No metadata found in the '{self._path}' directory. "
                      f"Use 'tmt init' to get started.")
              except fmf.utils.FileError as error:
                  raise tmt.utils.GeneralError(f"Invalid yaml syntax: {error}")
              # Adjust metadata for current fmf context
              self._tree.adjust(fmf.context.Context(**self._fmf_context()))
          return self._tree
  
      @tree.setter
      def tree(self, new_tree):
          self._tree = new_tree
  
      @property
      def root(self):
          """ Metadata root """
          return self.tree.root
  
      def tests(self, keys=None, names=None, filters=None, conditions=None,
                unique=True, links: Optional[List['LinkNeedle']] = None, excludes=None):
          """ Search available tests """
          # Handle defaults, apply possible command line options
          keys = (keys or []) + ['test']
          names = names or []
          filters = (filters or []) + list(Test._opt('filters', []))
          conditions = (conditions or []) + list(Test._opt('conditions', []))
          links = (links or []) + [
              LinkNeedle.from_spec(value)
              for value in cast(List[str], Test._opt('links', []))
              ]
          excludes = (excludes or []) + list(Test._opt('exclude', []))
          # Used in: tmt run test --name NAME, tmt test ls NAME...
          cmd_line_names = list(Test._opt('names', []))
  
          def name_filter(nodes):
              """ Filter nodes based on names provided on the command line """
              if not cmd_line_names:
                  return nodes
              return [
                  node for node in nodes
                  if any(re.search(name, node.name) for name in cmd_line_names)]
  
          # Append post filter to support option --enabled or --disabled
          if Test._opt('enabled'):
              filters.append('enabled:true')
          if Test._opt('disabled'):
              filters.append('enabled:false')
  
          if Test._opt('source'):
              tests = [
                  Test(node=test) for test in self.tree.prune(
                      keys=keys, sources=cmd_line_names)]
          else:
              # First let's build the list of test objects based on keys & names.
              # If duplicate test names are allowed, match test name/regexp
              # one-by-one and preserve the order of tests within a plan.
              if not unique and names:
                  tests = []
                  for name in names:
                      selected_tests = [
                          Test(node=test) for test
                          in name_filter(self.tree.prune(keys=keys, names=[name]))]
                      tests.extend(
                          sorted(selected_tests, key=lambda test: test.order))
              # Otherwise just perform a regular key/name filtering
              else:
                  selected_tests = [
                      Test(node=test) for test
                      in name_filter(self.tree.prune(keys=keys, names=names))]
                  tests = sorted(selected_tests, key=lambda test: test.order)
  
          # Apply filters & conditions
          return self._filters_conditions(
              tests, filters, conditions, links, excludes)
  
      def plans(self, keys=None, names=None, filters=None, conditions=None,
                run=None, links: Optional[List['LinkNeedle']] = None, excludes=None):
          """ Search available plans """
          # Handle defaults, apply possible command line options
          keys = (keys or []) + ['execute']
          names = (names or []) + list(Plan._opt('names', []))
          filters = (filters or []) + list(Plan._opt('filters', []))
          conditions = (conditions or []) + list(Plan._opt('conditions', []))
          links = (links or []) + [
              LinkNeedle.from_spec(value)
              for value in cast(List[str], Plan._opt('links', []))
              ]
          excludes = (excludes or []) + list(Plan._opt('exclude', []))
  
          # Append post filter to support option --enabled or --disabled
          if Plan._opt('enabled'):
              filters.append('enabled:true')
          if Plan._opt('disabled'):
              filters.append('enabled:false')
  
          # For --source option use names as sources
          if Plan._opt('source'):
              sources = names
              names = None
          else:
              sources = None
  
          # Build the list, convert to objects, sort and filter
          plans = [
              Plan(node=plan, run=run) for plan
              in self.tree.prune(keys=keys, names=names, sources=sources)]
          return self._filters_conditions(
              sorted(plans, key=lambda plan: plan.order),
              filters, conditions, links, excludes)
  
      def stories(self, keys=None, names=None, filters=None, conditions=None,
                  whole=False, links: Optional[List['LinkNeedle']] = None, excludes=None):
          """ Search available stories """
          # Handle defaults, apply possible command line options
          keys = (keys or []) + ['story']
          names = (names or []) + list(Story._opt('names', []))
          filters = (filters or []) + list(Story._opt('filters', []))
          conditions = (conditions or []) + list(Story._opt('conditions', []))
          links = (links or []) + [
              LinkNeedle.from_spec(value)
              for value in cast(List[str], Story._opt('links', []))
              ]
          excludes = (excludes or []) + list(Story._opt('exclude', []))
  
          # Append post filter to support option --enabled or --disabled
          if Story._opt('enabled'):
              filters.append('enabled:true')
          if Story._opt('disabled'):
              filters.append('enabled:false')
  
          # For --source option use names as sources
          if Story._opt('source'):
              sources = names
              names = None
          else:
              sources = None
  
          # Build the list, convert to objects, sort and filter
          stories = [
              Story(node=story) for story
              in self.tree.prune(keys=keys, names=names, whole=whole, sources=sources)]
          return self._filters_conditions(
              sorted(stories, key=lambda story: story.order),
              filters, conditions, links, excludes)
  
      @staticmethod
      def init(path, template, force, **kwargs):
          """ Initialize a new tmt tree, optionally with a template """
          path = os.path.realpath(path)
          dry = Tree._opt('dry')
  
          # Check for existing tree
          try:
              tree = tmt.Tree(path)
              # Are we creating a new tree under the existing one?
              if path == tree.root:
                  echo(f"Tree '{tree.root}' already exists.")
              else:
                  # Are we creating a nested tree?
                  echo(f"Path '{path}' already has a parent tree root '{tree.root}'.")
                  if not force and not confirm("Do you really want to initialize a nested tree?"):
                      return
                  tree = None
          except tmt.utils.GeneralError:
              tree = None
  
          # Create a new tree
          if tree is None:
              if dry:
                  echo(f"Tree '{path}' would be initialized.")
              else:
                  try:
                      fmf.Tree.init(path)
                      tree = tmt.Tree(path)
                      path = tree.root
                  except fmf.utils.GeneralError as error:
                      raise tmt.utils.GeneralError(
                          f"Failed to initialize tree in '{path}': {error}")
                  echo(f"Tree '{tree.root}' initialized.")
  
          # Populate the tree with example objects if requested
          if template == 'empty':
              choices = listed(tmt.templates.INIT_TEMPLATES, join='or')
              echo(
                  f"To populate it with example content, "
                  f"use --template with {choices}.")
          else:
              echo(f"Applying template '{template}'.")
  
          if template == 'mini':
              tmt.Plan.create('/plans/example', 'mini', path, force, dry)
          elif template == 'base':
              tmt.Test.create('/tests/example', 'beakerlib', path, force, dry)
              tmt.Plan.create('/plans/example', 'base', path, force, dry)
          elif template == 'full':
              tmt.Test.create('/tests/example', 'shell', path, force, dry)
              tmt.Plan.create('/plans/example', 'full', path, force, dry)
              tmt.Story.create('/stories/example', 'full', path, force, dry)
  
  
  class Run(tmt.utils.Common):
      """ Test run, a container of plans """
  
      def __init__(self, id_=None, tree=None, context=None):
          """ Initialize tree, workdir and plans """
          # Use the last run id if requested
          self.config = tmt.utils.Config()
          if context.params.get('last'):
              id_ = self.config.last_run()
              if id_ is None:
                  raise tmt.utils.GeneralError(
                      "No last run id found. Have you executed any run?")
          if context.params.get('follow') and id_ is None:
              raise tmt.utils.GeneralError(
                  "Run id has to be specified in order to use --follow.")
          # Do not create workdir now, postpone it until later, as options
          # have not been processed yet and we do not want commands such as
          # tmt run discover --how fmf --help to create a new workdir.
          super().__init__(context=context)
          self._workdir_path = id_ or True
          self._tree = tree
          self._plans = None
          self._environment_from_workdir = dict()
          self._environment_from_options = None
          self.remove = self.opt('remove')
  
      def _use_default_plan(self):
          """ Prepare metadata tree with only the default plan """
          default_plan = tmt.utils.yaml_to_dict(tmt.templates.DEFAULT_PLAN)
          # The default discover method for this case is 'shell'
          default_plan['/plans/default']['discover']['how'] = 'shell'
          self.tree = tmt.Tree(tree=fmf.Tree(default_plan))
          self.debug("No metadata found, using the default plan.")
  
      def _save_tree(self, tree):
          """ Save metadata tree, handle the default plan """
          default_plan = tmt.utils.yaml_to_dict(tmt.templates.DEFAULT_PLAN)
          try:
              self.tree = tree if tree else tmt.Tree('.')
              self.debug(f"Using tree '{self.tree.root}'.")
              # Clear the tree and insert default plan if requested
              if Plan._opt("default"):
                  new_tree = fmf.Tree(default_plan)
                  new_tree.root = self.tree.root
                  self.tree.tree = new_tree
                  self.debug("Enforcing use of default plan")
              # Insert default plan if no plan detected
              if not list(self.tree.tree.prune(keys=['execute'])):
                  self.tree.tree.update(default_plan)
                  self.debug("No plan found, adding the default plan.")
          # Create an empty default plan if no fmf metadata found
          except tmt.utils.MetadataError:
              self._use_default_plan()
  
      @property
      def environment(self):
          """ Return environment combined from wake up and command line """
          # Gather environment variables from options only once
          if self._environment_from_options is None:
              self._environment_from_options = dict()
              # Variables gathered from 'environment-file' options
              self._environment_from_options.update(
                  tmt.utils.environment_files_to_dict(
                      (self.opt('environment-file') or []),
                      root=self.tree.root))
              # Variables from 'environment' options (highest priority)
              self._environment_from_options.update(
                  tmt.utils.environment_to_dict(self.opt('environment')))
  
          # Combine workdir and command line
          combined = self._environment_from_workdir.copy()
          combined.update(self._environment_from_options)
          return combined
  
      def save(self):
          """ Save list of selected plans and enabled steps """
          data = {
              'root': self.tree.root,
              'plans': [plan.name for plan in self._plans],
              'steps': list(self._context.obj.steps),
              'environment': self.environment,
              'remove': self.remove,
              }
          self.write('run.yaml', tmt.utils.dict_to_yaml(data))
  
      def load_from_workdir(self):
          """
          Load the run from its workdir, do not require the root in
          run.yaml to exist. Doest not load the fmf tree.
  
          Use only when the data in workdir is sufficient (e.g. tmt
          clean and status only require the steps to be loaded and
          their status).
          """
          self._save_tree(self._tree)
          self._workdir_load(self._workdir_path)
          try:
              data = tmt.utils.yaml_to_dict(self.read('run.yaml'))
          except tmt.utils.FileError:
              self.debug('Run data not found.')
              return
          self._environment_from_workdir = data.get('environment')
          self._context.obj.steps = set(data['steps'])
          plans = []
  
          # The root directory of the tree may not be available, create
          # an fmf node that only contains the necessary attributes
          # required for plan/step loading. We will also need a dummy
          # parent for these nodes, so we would correctly load each
          # plan's name.
          dummy_parent = fmf.Tree({'summary': 'unused'})
  
          for plan in data.get('plans'):
              node = fmf.Tree({'execute': None}, name=plan, parent=dummy_parent)
              plans.append(Plan(node=node, run=self, skip_validation=True))
  
          self._plans = plans
  
      def load(self):
          """ Load list of selected plans and enabled steps """
          try:
              data = tmt.utils.yaml_to_dict(self.read('run.yaml'))
          except tmt.utils.FileError:
              self.debug('Run data not found.')
              return
  
          # If run id was given and root was not explicitly specified,
          # create a new Tree from the root in run.yaml
          if self._workdir and 'root' in data and not self.opt('root'):
              if data['root']:
                  self._save_tree(tmt.Tree(data['root']))
              else:
                  # The run was used without any metadata, default plan
                  # was used, load it
                  self._use_default_plan()
  
          # Filter plans by name unless specified on the command line
          plan_options = ['names', 'filters', 'conditions', 'links', 'default']
          if not any(Plan._opt(option) for option in plan_options):
              self._plans = [
                  plan for plan in self.tree.plans(run=self)
                  if plan.name in data['plans']]
  
          # Initialize steps only if not selected on the command line
          step_options = 'all since until after before skip'.split()
          selected = any(self.opt(option) for option in step_options)
          if not selected and not self._context.obj.steps:
              self._context.obj.steps = set(data['steps'])
  
          # Store loaded environment
          self._environment_from_workdir = data.get('environment')
          self.debug(
              f"Loaded environment: '{self._environment_from_workdir}'.",
              level=3)
  
          # If the remove was enabled, restore it, option overrides
          self.remove = self.remove or data.get('remove', 'False')
          self.debug(f"Remove workdir when finished: {self.remove}", level=3)
  
      @property
      def plans(self):
          """ Test plans for execution """
          if self._plans is None:
              self._plans = self.tree.plans(run=self, filters=['enabled:true'])
          return self._plans
  
      def finish(self):
          """ Check overall results, return appropriate exit code """
          # We get interesting results only if execute or prepare step is enabled
          execute = self.plans[0].execute
          report = self.plans[0].report
          interesting_results = execute.enabled or report.enabled
  
          # Gather all results and give an overall summary
          results = [
              result
              for plan in self.plans
              for result in plan.execute.results()]
          if interesting_results:
              self.info('')
              self.info('total', Result.summary(results), color='cyan')
  
          # Remove the workdir if enabled
          if self.remove and self.plans[0].finish.enabled:
              self._workdir_cleanup(self.workdir)
  
          # Skip handling of the exit codes in dry mode and
          # when there are no interesting results available
          if self.opt('dry') or not interesting_results:
              return
  
          # Return 0 if test execution has been intentionally skipped
          if tmt.steps.execute.Execute._opt("dry"):
              raise SystemExit(0)
          # Return appropriate exit code based on the total stats
          stats = Result.total(results)
          if sum(stats.values()) == 0:
              raise SystemExit(3)
          if stats['error']:
              raise SystemExit(2)
          if stats['fail'] + stats['warn']:
              raise SystemExit(1)
          if stats['pass']:
              raise SystemExit(0)
          raise SystemExit(2)
  
      def follow(self):
          """ Periodically check for new lines in the log. """
          logfile = open(os.path.join(self.workdir, tmt.utils.LOG_FILENAME), 'r')
          # Move to the end of the file
          logfile.seek(0, os.SEEK_END)
          # Rewind some lines back to show more context
          location = logfile.tell()
          read_lines = 0
          while location >= 0:
              logfile.seek(location)
              location -= 1
              current_char = logfile.read(1)
              if current_char == '\n':
                  read_lines += 1
              if read_lines > FOLLOW_LINES:
                  break
  
          while True:
              line = logfile.readline()
              if line:
                  print(line, end='')
              else:
                  time.sleep(0.5)
  
      def go(self):
          """ Go and do test steps for selected plans """
          # Create the workdir and save last run
          self._save_tree(self._tree)
          self._workdir_load(self._workdir_path)
          if self.tree.root and self._workdir.startswith(self.tree.root):
              raise tmt.utils.GeneralError(
                  "Run workdir must not be inside fmf root.")
          self.config.last_run(self.workdir)
          # Show run id / workdir path
          self.info(self.workdir, color='magenta')
          self.debug(f"tmt version: {tmt.__version__}")
          # Attempt to load run data
          self.load()
          # Follow log instead of executing the run
          if self.opt('follow'):
              self.follow()
  
          # Propagate dry mode from provision to prepare, execute and finish
          # (basically nothing can be done if there is no guest provisioned)
          if tmt.steps.provision.Provision._opt("dry"):
              tmt.steps.prepare.Prepare._options["dry"] = True
              tmt.steps.execute.Execute._options["dry"] = True
              tmt.steps.finish.Finish._options["dry"] = True
  
          # Enable selected steps
          enabled_steps = self._context.obj.steps
          all_steps = self.opt('all') or not enabled_steps
          since = self.opt('since')
          until = self.opt('until')
          after = self.opt('after')
          before = self.opt('before')
          skip = self.opt('skip')
  
          if any([all_steps, since, until, after, before]):
              # Detect index of the first and last enabled step
              if since:
                  first = tmt.steps.STEPS.index(since)
              elif after:
                  first = tmt.steps.STEPS.index(after) + 1
              else:
                  first = tmt.steps.STEPS.index('discover')
              if until:
                  last = tmt.steps.STEPS.index(until)
              elif before:
                  last = tmt.steps.STEPS.index(before) - 1
              else:
                  last = tmt.steps.STEPS.index('finish')
              # Enable all steps between the first and last
              for index in range(first, last + 1):
                  step = tmt.steps.STEPS[index]
                  if step not in skip:
                      enabled_steps.add(step)
          self.debug(f"Enabled steps: {fmf.utils.listed(enabled_steps)}")
  
          # Show summary, store run data
          if not self.plans:
              raise tmt.utils.GeneralError("No plans found.")
          self.verbose('Found {0}.'.format(listed(self.plans, 'plan')))
          self.save()
  
          # Iterate over plans
          for plan in self.plans:
              plan.go()
  
          # Update the last run id at the very end
          # (override possible runs created during execution)
          self.config.last_run(self.workdir)
  
          # Give the final summary, remove workdir, handle exit codes
          self.finish()
  
  
  class Status(tmt.utils.Common):
      """ Status of tmt work directories. """
  
      LONGEST_STEP = max(tmt.steps.STEPS, key=lambda k: len(k))
      FIRST_COL_LEN = len(LONGEST_STEP) + 2
  
      @staticmethod
      def get_overall_plan_status(plan):
          """ Examines the plan status (find the last done step) """
          steps = list(plan.steps())
          step_names = list(plan.steps(names=True))
          for i in range(len(steps) - 1, -1, -1):
              if steps[i].status() == 'done':
                  if i + 1 == len(steps):
                      # Last enabled step, consider the whole plan done
                      return 'done'
                  else:
                      return step_names[i]
          return 'todo'
  
      def plan_matches_filters(self, plan):
          """ Check if the given plan matches filters from the command line """
          if self.opt('abandoned'):
              return (plan.provision.status() == 'done'
                      and plan.finish.status() == 'todo')
          if self.opt('active'):
              return any(step.status() == 'todo' for step in plan.steps())
          if self.opt('finished'):
              return all(step.status() == 'done' for step in plan.steps())
          return True
  
      @staticmethod
      def colorize_column(content):
          """ Add color to a status column """
          if 'done' in content:
              return style(content, fg='green')
          else:
              return style(content, fg='yellow')
  
      @classmethod
      def pad_with_spaces(cls, string):
          """ Append spaces to string to properly align the first column """
          return string + (cls.FIRST_COL_LEN - len(string)) * ' '
  
      def run_matches_filters(self, run):
          """ Check if the given run matches filters from the command line """
          if self.opt('abandoned') or self.opt('active'):
              # Any of the plans must be abandoned/active for the whole
              # run to be abandoned/active
              return any(self.plan_matches_filters(p) for p in run.plans)
          if self.opt('finished'):
              # All plans must be finished for the whole run to be finished
              return all(self.plan_matches_filters(p) for p in run.plans)
          return True
  
      def print_run_status(self, run):
          """ Display the overall status of the run """
          if not self.run_matches_filters(run):
              return
          # Find the earliest step in all plans' status
          earliest_step_index = len(tmt.steps.STEPS)
          for plan in run.plans:
              plan_status = self.get_overall_plan_status(plan)
              if plan_status == 'done':
                  continue
              elif plan_status == 'todo':
                  # If plan has no steps done, consider the whole run not done
                  earliest_step_index = -1
                  break
              plan_status_index = tmt.steps.STEPS.index(plan_status)
              if plan_status_index < earliest_step_index:
                  earliest_step_index = plan_status_index
  
          if earliest_step_index == len(tmt.steps.STEPS):
              run_status = 'done'
          elif earliest_step_index == -1:
              run_status = 'todo'
          else:
              run_status = tmt.steps.STEPS[earliest_step_index]
          run_status = self.colorize_column(self.pad_with_spaces(run_status))
          echo(run_status, nl=False)
          echo(run.workdir)
  
      def print_plans_status(self, run):
          """ Display the status of each plan of the given run """
          for plan in run.plans:
              if self.plan_matches_filters(plan):
                  plan_status = self.get_overall_plan_status(plan)
                  echo(self.colorize_column(self.pad_with_spaces(plan_status)),
                       nl=False)
                  echo(f'{run.workdir}  {plan.name}')
  
      def print_verbose_status(self, run):
          """ Display the status of each step of the given run """
          for plan in run.plans:
              if self.plan_matches_filters(plan):
                  for step in plan.steps(disabled=True):
                      column = (step.status() or '----') + ' '
                      echo(self.colorize_column(column), nl=False)
                  echo(f' {run.workdir}  {plan.name}')
  
      def process_run(self, run):
          """ Display the status of the given run based on verbosity """
          loaded, error = tmt.utils.load_run(run)
          if not loaded:
              self.warn(f"Failed to load run '{run.workdir}': {error}")
              return
          if self.opt('verbose') == 0:
              self.print_run_status(run)
          elif self.opt('verbose') == 1:
              self.print_plans_status(run)
          else:
              self.print_verbose_status(run)
  
      def print_header(self):
          """ Print the header of the status table based on verbosity """
          header = ''
          if self.opt('verbose') >= 2:
              for step in tmt.steps.STEPS:
                  header += (step[0:4] + ' ')
              header += ' '
          else:
              header = self.pad_with_spaces('status')
          header += 'id'
          echo(style(header, fg='blue'))
  
      def show(self):
          """ Display the current status """
          # Prepare absolute workdir path if --id was used
          id_ = self.opt('id')
          root_path = self.opt('workdir-root')
          self.print_header()
          for abs_path in tmt.utils.generate_runs(root_path, id_):
              run = Run(abs_path, self._context.obj.tree, self._context)
              self.process_run(run)
  
  
  class Clean(tmt.utils.Common):
      """ A class for cleaning up workdirs, guests or images """
  
      def __init__(self, parent=None, name=None, workdir=None, context=None):
          """
          Initialize name and relation with the parent object
  
          Always skip to initialize the work tree.
          """
          # Set the option to skip to initialize the work tree
          context.params[tmt.utils.PLAN_SKIP_WORKTREE_INIT] = True
          super().__init__(parent, name, workdir, context)
  
      def images(self):
          """ Clean images of provision plugins """
          self.info('images', color='blue')
          successful = True
          for method in tmt.steps.provision.ProvisionPlugin.methods():
              if not method.class_.clean_images(self, self.opt('dry')):
                  successful = False
          return successful
  
      def _matches_how(self, plan):
          """ Check if the given plan matches options """
          how = plan.provision.data[0].how
          target_how = self.opt('how')
          if target_how:
              return how == target_how
          # No target provision method, always matches
          return True
  
      def _stop_running_guests(self, run):
          """ Stop all running guests of a run """
          loaded, error = tmt.utils.load_run(run)
          if not loaded:
              self.warn(f"Failed to load run '{run.workdir}': {error}")
              return False
          # Clean guests if provision is done but finish is not done
          successful = True
          for plan in run.plans:
              if plan.provision.status() == 'done':
                  if plan.finish.status() != 'done':
                      # Wake up provision to load the active guests
                      plan.provision.wake()
                      if not self._matches_how(plan):
                          continue
                      if self.opt('dry'):
                          self.verbose(
                              f"Would stop guests in run '{run.workdir}'"
                              f" plan '{plan.name}'.", shift=1)
                      else:
                          self.verbose(f"Stopping guests in run '{run.workdir}' "
                                       f"plan '{plan.name}'.", shift=1)
                          # Set --quiet to avoid finish logging to terminal
                          quiet = self._context.params['quiet']
                          self._context.params['quiet'] = True
                          try:
                              plan.finish.go()
                          except tmt.utils.GeneralError as error:
                              self.warn(f"Could not stop guest in run "
                                        f"'{run.workdir}': {error}.", shift=1)
                              successful = False
                          finally:
                              self._context.params['quiet'] = quiet
          return successful
  
      def guests(self):
          """ Clean guests of runs """
          self.info('guests', color='blue')
          root_path = self.opt('workdir-root')
          id_ = self.opt('id_')
          if self.opt('last'):
              # Pass the context containing --last to Run to choose
              # the correct one.
              return self._stop_running_guests(Run(context=self._context))
          successful = True
          for abs_path in tmt.utils.generate_runs(root_path, id_):
              run = Run(abs_path, self._context.obj.tree, self._context)
              if not self._stop_running_guests(run):
                  successful = False
          return successful
  
      def _clean_workdir(self, path):
          """ Remove a workdir (unless in dry mode) """
          if self.opt('dry'):
              self.verbose(f"Would remove workdir '{path}'.", shift=1)
          else:
              self.verbose(f"Removing workdir '{path}'.", shift=1)
              try:
                  shutil.rmtree(path)
              except OSError as error:
                  self.warn(f"Failed to remove '{path}': {error}.", shift=1)
                  return False
          return True
  
      def runs(self):
          """ Clean workdirs of runs """
          self.info('runs', color='blue')
          root_path = self.opt('workdir-root')
          id_ = self.opt('id_')
          if self.opt('last'):
              # Pass the context containing --last to Run to choose
              # the correct one.
              last_run = Run(context=self._context)
              last_run._workdir_load(last_run._workdir_path)
              return self._clean_workdir(last_run.workdir)
          all_workdirs = [path for path in tmt.utils.generate_runs(root_path, id_)]
          keep = self.opt('keep')
          if keep is not None:
              # Sort by modify time of the workdirs and keep the newest workdirs
              all_workdirs.sort(key=lambda workdir: os.path.getmtime(
                  os.path.join(workdir, 'run.yaml')), reverse=True)
              all_workdirs = all_workdirs[keep:]
  
          successful = True
          for workdir in all_workdirs:
              if not self._clean_workdir(workdir):
                  successful = False
  
          return successful
  
  
  class Result:
      """
      Test result
  
      The following keys are expected in the 'data' dictionary::
  
          result ........... test execution result
          log .............. one or more log files
          note ............. additional result details
          duration ......... test execution time (hh:mm:ss)
  
      Required parameter 'test' or 'name' should contain a test reference.
      """
  
      _results = {
          'pass': 'green',
          'fail': 'red',
          'info': 'blue',
          'warn': 'yellow',
          'error': 'magenta',
          }
  
      def __init__(self, data, name=None, test=None):
          """ Initialize test result data """
  
          # Save the test name and optional note
          if not test and not name:
              raise tmt.utils.SpecificationError(
                  "Either name or test have to be specified")
          if test and not isinstance(test, Test):
              raise tmt.utils.SpecificationError(f"Invalid test '{test}'.")
          if name and not isinstance(name, str):
              raise tmt.utils.SpecificationError(f"Invalid test name '{name}'.")
          self.name = name or test.name
          self.note = data.get('note')
          self.duration = data.get('duration')
          if test:
              # Saving identifiable information for each test case so we can match them
              # to Polarion/Nitrate/other cases and report run results there
              self.ids = {tmt.identifier.ID_KEY: test.id}
              for key in EXTRA_RESULT_IDENTIFICATION_KEYS:
                  self.ids[key] = test.node.get(key)
              interpret = test.result or 'respect'
          else:
              try:
                  self.ids = data['ids']
              except KeyError:
                  self.ids = {}
              interpret = 'respect'
  
          # Check for valid results
          try:
              self.result = data['result']
          except KeyError:
              raise tmt.utils.SpecificationError("Missing test result.")
          if self.result not in self._results:
              raise tmt.utils.SpecificationError(
                  f"Invalid result '{self.result}'.")
  
          # Convert log into list if necessary
          try:
              self.log = tmt.utils.listify(data['log'])
          except KeyError:
              self.log = []
  
          # Handle alternative result interpretation
          if interpret not in ('respect', 'custom'):
              # Extend existing note or set a new one
              if self.note and isinstance(self.note, str):
                  self.note += f', original result: {self.result}'
              elif self.note is None:
                  self.note = f'original result: {self.result}'
              else:
                  raise tmt.utils.SpecificationError(
                      f"Test result note '{self.note}' must be a string.")
  
              if interpret == "xfail":
                  # Swap just fail<-->pass, keep the rest as is (info, warn,
                  # error)
                  self.result = {
                      'fail': 'pass', 'pass': 'fail'}.get(
                      self.result, self.result)
              elif interpret in self._results:
                  self.result = interpret
              else:
                  raise tmt.utils.SpecificationError(
                      f"Invalid result '{interpret}' in test '{self.name}'.")
  
      @staticmethod
      def total(results):
          """ Return dictionary with total stats for given results """
          stats = dict([(result, 0) for result in Result._results])
          for result in results:
              stats[result.result] += 1
          return stats
  
      @staticmethod
      def summary(results):
          """ Prepare a nice human summary of provided results """
          stats = Result.total(results)
          comments = []
          if stats.get('pass'):
              passed = ' ' + click.style('passed', fg='green')
              comments.append(fmf.utils.listed(stats['pass'], 'test') + passed)
          if stats.get('fail'):
              failed = ' ' + click.style('failed', fg='red')
              comments.append(fmf.utils.listed(stats['fail'], 'test') + failed)
          if stats.get('info'):
              count, comment = fmf.utils.listed(stats['info'], 'info').split()
              comments.append(count + ' ' + click.style(comment, fg='blue'))
          if stats.get('warn'):
              count, comment = fmf.utils.listed(stats['warn'], 'warn').split()
              comments.append(count + ' ' + click.style(comment, fg='yellow'))
          if stats.get('error'):
              count, comment = fmf.utils.listed(stats['error'], 'error').split()
              comments.append(count + ' ' + click.style(comment, fg='magenta'))
          return fmf.utils.listed(comments or ['no results found'])
  
      def show(self):
          """ Return a nicely colored result with test name (and note) """
          result = 'errr' if self.result == 'error' else self.result
          colored = style(result, fg=self._results[self.result])
          note = f" ({self.note})" if self.note else ''
          return f"{colored} {self.name}{note}"
  
      def export(self):
          """ Save result data for future wake-up """
          data = dict(result=self.result, log=self.log)
          if self.note:
              data['note'] = self.note
          if self.duration:
              data['duration'] = self.duration
          if self.ids:
              data['ids'] = self.ids
          return data
  
  
  @dataclasses.dataclass
  class LinkNeedle:
      """
      A container to use for searching links.
  
      ``relation`` and ``target`` fields hold regular expressions that
      are to be searched for in the corresponding fields of :py:class:`Link`
      instances.
      """
  
      relation: str = r'.*'
      target: str = r'.*'
  
      @classmethod
      def from_spec(cls, value: str) -> 'LinkNeedle':
          """
          Convert from a specification file or from a CLI option
  
          Specification is described in [1], this constructor takes care
          of parsing it into a corresponding ``LinkNeedle`` instance.
  
          [1] https://tmt.readthedocs.io/en/stable/spec/plans.html#fmf
          """
  
          parts = value.split(':', maxsplit=1)
  
          if len(parts) == 1:
              return LinkNeedle(target=parts[0])
  
          return LinkNeedle(relation=parts[0], target=parts[1])
  
      def __str__(self) -> str:
          return f'{self.relation}:{self.target}'
  
      def matches(self, link: 'Link') -> bool:
          """ Find out whether a given link matches this needle """
  
          # Rule out the simple case, mismatching relation.
          if not re.search(self.relation, link.relation):
              return False
  
          # If the target is a string, the test is trivial.
          if isinstance(link.target, str):
              return re.search(self.target, link.target) is not None
  
          # If the target is an fmf id, the current basic implementation will
          # check just the `name` key, if it's defined. More fields may come
          # later, pending support for more sophisticated parsing of link
          # needle on a command line.
          if link.target.name:
              return re.search(self.target, link.target.name) is not None
  
          return False
  
  
  @dataclasses.dataclass
  class Link(tmt.utils.SpecBasedContainer):
      """
      An internal "link" as defined by tmt specification.
  
      All links, after entering tmt internals, are converted from their raw
      representation into instances of this class.
  
      [1] https://tmt.readthedocs.io/en/stable/spec/core.html#link
      """
  
      DEFAULT_RELATIONSHIP: ClassVar[_RawLinkRelationName] = 'relates'
  
      relation: _RawLinkRelationName
      target: Union[str, FmfId]
      note: Optional[str] = None
  
      @classmethod
      def from_spec(cls, spec: _RawLink) -> 'Link':
          """
          Convert from a specification file or from a CLI option
  
          Specification is described in [1], this constructor takes care
          of parsing it into a corresponding ``Link`` instance.
  
          [1] https://tmt.readthedocs.io/en/stable/spec/core.html#link
          """
  
          # `spec` can be either a string, fmf id, or relation:target mapping with
          # a single key (modulo `note` key, of course).
  
          # String is simple: if `spec` is a string, it represents a target,
          # and we use the default relationship.
          if isinstance(spec, str):
              return Link(relation=Link.DEFAULT_RELATIONSHIP, target=spec)
  
          # From now on, `spec` is a mapping, and may contain the optional
          # `note` key. Extract the key for later.
          note = cast(Optional[str], spec.get('note', None))
  
          # Count how many relations are stored in spec.
          relations = [
              cast(_RawLinkRelationName, key) for key in spec if key not in (FmfId.keys + ['note'])
              ]
  
          # If there are no relations, spec must be an fmf id, representing
          # a target.
          if len(relations) == 0:
              return Link(
                  relation=Link.DEFAULT_RELATIONSHIP,
                  target=FmfId.from_spec(cast(_RawFmfId, spec)),
                  note=note)
  
          # More relations than 1 are a hard error, only 1 is allowed.
          if len(relations) > 1:
              raise tmt.utils.SpecificationError(
                  f"Multiple relations specified for the link "
                  f"({fmf.utils.listed(relations)}).")
  
          # At this point, we know there's just a single relation, its value is the target,
          # and note we already put aside.
          #
          # NOTE: as far as mypy knows, we did not narrow the type of `spec`, _RawFmfId
          # is still in play - but we do know it's no longer possible because such a
          # value we ruled out thanks to `"no relations" check above. At this point,
          # the right side of relation must be _RawLinkTarget and nothing else. Helping
          # mypy to realize that.
          relation = relations[0]
          raw_target = cast(_RawLinkTarget, spec[relation])
  
          # TODO: this should not happen with mandatory validation
          if relation not in Links._relations:
              raise tmt.utils.SpecificationError(
                  f"Invalid link relation '{relation}' (should be "
                  f"{fmf.utils.listed(Links._relations, join='or')}).")
  
          if isinstance(raw_target, str):
              return Link(relation=relation, target=raw_target, note=note)
  
          return Link(relation=relation, target=FmfId.from_spec(raw_target), note=note)
  
      def to_spec(self) -> _RawLinkRelation:
          """
          Convert to a form suitable for saving in a specification file
  
          No matter what the original specification was, every link will
          generate the very same type of specification, the ``relation: target``
          one.
  
          Output of this method is fully compatible with specification, and when
          given to :py:meth:`from_spec`, it shall create a ``Link`` instance
          with the same properties as the original one.
  
          [1] https://tmt.readthedocs.io/en/stable/spec/core.html#link
          """
  
          spec = {
              self.relation: self.target.to_spec() if isinstance(
                  self.target,
                  FmfId) else self.target}
  
          if self.note is not None:
              spec['note'] = self.note
  
          return spec
  
  
  class Links(tmt.utils.SpecBasedContainer):
      """
      Collection of links in tests, plans and stories.
  
      Provides abstraction over the whole collection of object's links.
  
      [1] https://tmt.readthedocs.io/en/stable/spec/core.html#link
      """
  
      # The list of all supported link relations
      _relations: List[_RawLinkRelationName] = [
          'verifies', 'verified-by',
          'implements', 'implemented-by',
          'documents', 'documented-by',
          'blocks', 'blocked-by',
          'duplicates', 'duplicated-by',
          'parent', 'child',
          'relates',
          ]
  
      _links: List[Link]
  
      def __init__(self, data: Optional[_RawLinks] = None):
          """ Create a collection from raw link data """
  
          # TODO: this should not happen with mandatory validation
          if data is not None and not isinstance(data, (str, dict, list)):
              raise tmt.utils.SpecificationError(
                  "Invalid link specification "
                  "(should be a string, fmf id or list of their combinations), "
                  f"got '{type(data).__name__}'.")
  
          # Nothing to do if no data provided
          if data is None:
              self._links = []
  
              return
  
          specs = data if isinstance(data, list) else [data]
  
          # Ensure that each link is in the canonical form
          self._links = [Link.from_spec(spec) for spec in specs]
  
      def to_spec(self) -> List[_RawLinkRelation]:
          """
          Convert to a form suitable for saving in a specification file
  
          No matter what the original specification was, every link will
          generate the very same type of specification, the ``relation: target``
          one.
  
          Output of this method is fully compatible with specification, and when
          used to instantiate :py:meth:`Link` object, it shall create a collection
          of links with the same properties as the original one.
  
          [1] https://tmt.readthedocs.io/en/stable/spec/core.html#link
          """
  
          return [
              link.to_spec()
              for link in self._links
              ]
  
      def get(self, relation: Optional[_RawLinkRelationName] = None) -> List[Link]:
          """ Get links with given relation, all by default """
          return [
              link for link in self._links
              if relation is None or link.relation == relation]
  
      def show(self) -> None:
          """ Format a list of links with their relations """
          for link in self._links:
              # TODO: needs a format for fmf id target
              echo(tmt.utils.format(link.relation.rstrip('-by'), f"{link.target}", key_color='cyan'))
  
      def has_link(self, needle: LinkNeedle) -> bool:
          """ Check whether this set of links contains a matching link """
  
          return any(needle.matches(link) for link in self._links)
  
files: tmt/base.py
files: 
  """ Base Metadata Classes """
  
  import copy
  import dataclasses
  import enum
  import functools
  import os
  import re
  import shutil
  import subprocess
  import sys
  import time
  from typing import Any, ClassVar, Dict, List, Optional, Tuple, Union, cast
  
  import click
  import fmf
  import fmf.base
  from click import confirm, echo, style
  from fmf.utils import listed
  from ruamel.yaml.error import MarkedYAMLError
  
  import tmt.export
  import tmt.identifier
  import tmt.steps
  import tmt.steps.discover
  import tmt.steps.execute
  import tmt.steps.finish
  import tmt.steps.prepare
  import tmt.steps.provision
  import tmt.steps.report
  import tmt.templates
  import tmt.utils
  from tmt.utils import FmfContextType, verdict
  
  if sys.version_info >= (3, 8):
      from typing import Literal, TypedDict
  else:
      from typing_extensions import Literal, TypedDict
  
  # Default test duration is 5m for individual tests discovered from L1
  # metadata and 1h for scripts defined directly in plans (L2 metadata).
  DEFAULT_TEST_DURATION_L1 = '5m'
  DEFAULT_TEST_DURATION_L2 = '1h'
  DEFAULT_ORDER = 50
  
  # How many already existing lines should tmt run --follow show
  FOLLOW_LINES = 10
  
  # Obsoleted test keys
  OBSOLETED_TEST_KEYS = "relevancy coverage".split()
  
  # Unofficial temporary test keys
  EXTRA_TEST_KEYS = (
      "extra-nitrate extra-hardware extra-pepa "
      "extra-summary extra-task id".split())
  
  # Unofficial temporary story keys
  EXTRA_STORY_KEYS = ("id".split())
  
  SECTIONS_HEADINGS = {
      'Setup': ['<h1>Setup</h1>'],
      'Test': ['<h1>Test</h1>',
               '<h1>Test .*</h1>'],
      'Step': ['<h2>Step</h2>',
               '<h2>Test Step</h2>'],
      'Expect': ['<h2>Expect</h2>',
                 '<h2>Result</h2>',
                 '<h2>Expected Result</h2>'],
      'Cleanup': ['<h1>Cleanup</h1>']
      }
  
  # Extra keys used for identification in Result class
  EXTRA_RESULT_IDENTIFICATION_KEYS = ['extra-nitrate', 'extra-task']
  
  
  #
  # fmf id types
  #
  # See https://fmf.readthedocs.io/en/latest/concept.html#identifiers for
  # formal specification.
  #
  
  # A "raw" fmf id as stored in fmf node data, i.e. as a mapping with various keys.
  # Used for a brief moment, to annotate raw fmf data before they are converted
  # into FmfId instances.
  _RawFmfId = TypedDict('_RawFmfId', {
      'url': Optional[str],
      'ref': Optional[str],
      'path': Optional[str],
      'name': Optional[str]
  })
  
  
  # An internal fmf id representation.
  @dataclasses.dataclass
  class FmfId(tmt.utils.SerializableContainer):
      # The list of valid fmf id keys
      keys: ClassVar[List[str]] = ['url', 'ref', 'path', 'name']
  
      url: Optional[str] = None
      ref: Optional[str] = None
      path: Optional[str] = None
      name: Optional[str] = None
  
      def to_dict(self) -> Dict[str, Any]:
          """ Return keys and values in the form of a dictionary """
  
          return dataclasses.asdict(self)
  
      def to_raw(self) -> Dict[str, Any]:
          """ Return keys and values as if they originated from fmf node """
  
          return self.to_dict()
  
      @classmethod
      def from_dict(cls, raw: _RawFmfId) -> 'FmfId':
          """ Construct an :py:class:`FmfId` from given input container """
  
          return FmfId(**{key: raw.get(key, None) for key in cls.keys})
  
      def validate(self) -> Tuple[bool, str]:
          """
          Validate fmf id and return a human readable error
  
          Return a tuple (boolean, message) as the result of validation.
          The boolean specifies the validation result and the message
          the validation error. In case the FMF id is valid, return an empty
          string as the message.
          """
          # Validate remote id and translate to human readable errors
          try:
              # Simple asdict() is not good enough, fmf does not like keys that exist but are `None`.
              # Don't include those.
              fmf.base.Tree.node({
                  key: value for key, value in self.to_dict().items()
                  if value is not None
                  })
          except fmf.utils.GeneralError as error:
              # Map fmf errors to more user friendly alternatives
              error_map: List[Tuple[str, str]] = [
                  ('git clone', f"repo '{self.url}' cannot be cloned"),
                  ('git checkout', f"git ref '{self.ref}' is invalid"),
                  ('directory path', f"path '{self.path}' is invalid"),
                  ('tree root', f"No tree found in repo '{self.url}', missing an '.fmf' directory?")
                  ]
  
              stringified_error = str(error)
  
              errors = [message for needle, message in error_map if needle in stringified_error]
              return (False, errors[0] if errors else stringified_error)
  
          return (True, '')
  
  #
  # Various types describing constructs as stored in "raw" fmf node data.
  # Used for a brief moment, to annotate raw fmf data before they are converted
  # into their internal representations.
  #
  
  #
  # A type describing the raw form of the core `link` attribute. See
  # https://tmt.readthedocs.io/en/stable/spec/core.html#link for its
  # formal specification. Internally, a link is represented by a `Link`
  # class instance, and types below describe the raw data coming from Fmf
  # nodes and CLI options.
  
  
  # Link relations.
  _RawLinkRelationName = Literal[
      'verifies', 'verified-by',
      'implements', 'implemented-by',
      'documents', 'documented-by',
      'blocks', 'blocked-by',
      'duplicates', 'duplicated-by',
      'parent', 'child',
      'relates',
      # Special case: not a relation, but it can appear where relations appear in
      # link data structures.
      'note'
  ]
  
  # Link target - can be either a string (like test case name or URL), or an fmf id.
  _RawLinkTarget = Union[str, _RawFmfId]
  
  # Basic "relation-aware" link - essentialy a mapping with one key/value pair.
  _RawLinkRelation = Dict[_RawLinkRelationName, _RawLinkTarget]
  
  # A single link can be represented as a string or FMF ID (meaning only target is specified),
  # or a "relation-aware" link aka mapping defined above.
  _RawLink = Union[
      str,
      _RawFmfId,
      _RawLinkRelation
  ]
  
  # Collection of links - can be either a single link, or a list of links, and all
  # link forms may be used together.
  _RawLinks = Union[
      _RawLink,
      List[_RawLink]
  ]
  
  
  # A type describing `adjust` content. See
  # https://tmt.readthedocs.io/en/stable/spec/core.html#adjust for its formal specification.
  #
  # The type is incomplete in the sense it does not describe all keys it may contain,
  # like `environment` or `provision`, and focuses only on the keys defined for `adjust`
  # itself.
  _RawAdjustRule = TypedDict(
      'AdjustRuleType',
      {
          'when': Optional[str],
          'continue': Optional[bool],
          'because': Optional[str]
          }
  )
  
  # A type describing content accepted by various require-like keys: - a string, fmf id,
  # or a list with a mixture of these two types.
  #
  # TODO: what other keys accept this kind of fmf data??
  _RawRequire = Union[
      str,
      _RawFmfId,
      List[Union[str, _RawFmfId]]
  ]
  
  
  class Core(tmt.utils.Common):
      """
      General node object
  
      Corresponds to given fmf.Tree node.
      Implements common Test, Plan and Story methods.
      Also defines L0 metadata and its manipulation.
      """
  
      # Core attributes (supported across all levels)
      summary: Optional[str] = None
      description: Optional[str] = None
      enabled: bool = True
      order: int = DEFAULT_ORDER
      link: Optional['Links'] = None
      id: Optional[str] = None
      tag: List[str] = []
      tier: Optional[str] = None
      adjust: Optional[List[_RawAdjustRule]] = None
  
      KEYS_SHOW_ORDER = [
          # Basic stuff
          'summary',
          'description',
          'enabled',
          'order',
          'id',
  
          # Filtering and more
          'tag',
          'tier',
          'link',
          'adjust',
          ]
  
      # Normalization methods
      _normalize_tag = tmt.utils.LoadFmfKeysMixin._normalize_string_list
  
      # TODO: remove when schema becomes mandatory, `order` shall never be `null`
      def _normalize_order(self, value: Optional[int]) -> int:
          if value is None:
              return DEFAULT_ORDER
          return int(value)
  
      def _normalize_link(self, value: _RawLinks) -> 'Links':
          return Links(data=value)
  
      def _normalize_adjust(
              self, value: Union[_RawAdjustRule, List[_RawAdjustRule]]) -> List[_RawAdjustRule]:
          if value is None:
              return []
          return [value] if not isinstance(value, list) else value
  
      def _normalize_tier(self, value: Optional[Union[int, str]]) -> Optional[str]:
          if value is None:
              return None
          return str(value)
  
      def __init__(self,
                   *,
                   node: fmf.Tree,
                   parent: Optional[tmt.utils.Common] = None,
                   **kwargs: Any) -> None:
          """ Initialize the node """
          kwargs.setdefault('logger', self)
          super().__init__(node=node, parent=parent, name=node.name, **kwargs)
  
          self.node = node
  
          # Verify id is not inherited from parent.
          if self.id and not tmt.identifier.key_defined_in_leaf(node, 'id'):
              raise tmt.utils.SpecificationError(
                  f"The 'id' key '{node.get('id')}' in '{self.name}' "
                  f"is inherited from parent, should be defined in a leaf.")
  
          # Store original metadata with applied defaults and including
          # keys which are not defined in the L1 metadata specification
          # Once the whole node has been initialized,
          # self._update_metadata() must be called to work correctly.
          self._metadata = self.node.data.copy()
  
      def __str__(self):
          """ Node name """
          return self.name
  
      def _update_metadata(self):
          """ Update the _metadata attribute """
          self._metadata.update(self.export(format_='dict'))
          self._metadata['name'] = self.name
  
      def _show_additional_keys(self):
          """ Show source files """
          if self.id is not None:
              echo(tmt.utils.format('id', self.id, key_color='magenta'))
          echo(tmt.utils.format(
              'sources', self.node.sources, key_color='magenta'))
  
      def _fmf_id(self):
          """ Show fmf identifier """
          echo(tmt.utils.format('fmf-id', self.fmf_id, key_color='magenta'))
  
      @property
      @functools.lru_cache(maxsize=None)
      def fmf_id(self):
          """ Return full fmf identifier of the node """
  
          def run(command):
              """ Run command, return output """
              cwd = fmf_root
              result = subprocess.run(
                  command.split(),
                  stdout=subprocess.PIPE,
                  stderr=subprocess.DEVNULL,
                  cwd=cwd)
              return result.stdout.strip().decode("utf-8")
  
          fmf_id = {'name': self.name}
  
          fmf_root = self.node.root
          # Prepare url (for now handle just the most common schemas)
          branch = run("git rev-parse --abbrev-ref --symbolic-full-name @{u}")
          try:
              remote_name = branch[:branch.index('/')]
          except ValueError:
              remote_name = 'origin'
          remote = run(f"git config --get remote.{remote_name}.url")
          fmf_id['url'] = tmt.utils.public_git_url(remote)
  
          # Get the ref (skip for master as it is the default)
          ref = run('git rev-parse --abbrev-ref HEAD')
          if ref != 'master':
              fmf_id['ref'] = ref
  
          # Construct path (if different from git root)
          git_root = run('git rev-parse --show-toplevel')
          if git_root != fmf_root:
              fmf_id['path'] = os.path.join(
                  '/', os.path.relpath(fmf_root, git_root))
  
          return fmf_id
  
      @classmethod
      def _save_context(cls, context):
          """ Save provided command line context for future use """
          super(Core, cls)._save_context(context)
  
          # Handle '.' as an alias for the current working directory
          names = cls._opt('names')
          if names is not None and '.' in names:
              root = context.obj.tree.root
              current = os.getcwd()
              # Handle special case when directly in the metadata root
              if current == root:
                  pattern = '/'
              # Prepare path from the tree root to the current directory
              else:
                  pattern = os.path.join('/', os.path.relpath(current, root))
                  # Prevent matching common prefix from other directories
                  pattern = f"{pattern}(/|$)"
              cls._context.params['names'] = tuple([
                  pattern if name == '.' else name for name in names])
  
      def name_and_summary(self):
          """ Node name and optional summary """
          if self.summary:
              return '{0} ({1})'.format(self.name, self.summary)
          else:
              return self.name
  
      def ls(self, summary=False):
          """ List node """
          echo(style(self.name, fg='red'))
          if summary and self.summary:
              echo(tmt.utils.format('summary', self.summary))
  
      def export(self, format_='yaml', keys=None):
          """ Export data into requested format (yaml or dict) """
          if keys is None:
              keys = self._keys()
  
          # Always include node name, add requested keys, ignore adjust
          data = dict(name=self.name)
          for key in keys:
              if key == 'adjust':
                  continue
  
              value = getattr(self, key)
  
              if key == 'link' and value:
                  # TODO: links must be saved in a form that can be than crunched by
                  # Links.__init__() method - it is tempting to use to_serialized()
                  # and from_unserialized(), but we don't use unserialization code
                  # when loading saved data back, so we can't go this way. Yet.
                  data[key] = cast('Links', value).to_raw()
  
              else:
                  data[key] = value
  
          # Choose proper format
          if format_ == 'dict':
              return data
          elif format_ == 'yaml':
              return tmt.utils.dict_to_yaml(data)
          else:
              raise tmt.utils.GeneralError(
                  f"Invalid export format '{format_}'.")
  
      def lint_keys(self, additional_keys):
          """ Return list of invalid keys used, empty when all good """
          known_keys = additional_keys + self._keys()
          return [key for key in self.node.get().keys() if key not in known_keys]
  
      def _lint_summary(self):
          """ Lint summary attribute """
          # Summary is advised with a resonable length
          if self.summary is None:
              verdict(None, "summary is very useful for quick inspection")
          elif len(self.summary) > 50:
              verdict(None, "summary should not exceed 50 characters")
          return True
  
      def has_link(self, needle: 'LinkNeedle') -> bool:
          """ Whether object contains specified link """
  
          if self.link is None:
              return False
  
          return self.link.has_link(needle)
  
  
  Node = Core
  
  
  class Test(tmt.utils.ValidateFmfMixin, tmt.utils.LoadFmfKeysMixin, Core):
      """ Test object (L1 Metadata) """
  
      # Basic test information
      contact: List[str] = []
      component: List[str] = []
  
      # Test execution data
      test: str
      path: Optional[str] = None
      framework: Optional[str] = None
      manual: bool = False
      require: List[Union[str, FmfId]] = []
      recommend: List[str] = []
      environment: Optional[tmt.utils.EnvironmentType] = {}
      duration: Optional[str] = DEFAULT_TEST_DURATION_L1
      result: str = 'respect'
  
      _normalize_contact = tmt.utils.LoadFmfKeysMixin._normalize_string_list
      _normalize_component = tmt.utils.LoadFmfKeysMixin._normalize_string_list
      _normalize_recommend = tmt.utils.LoadFmfKeysMixin._normalize_string_list
  
      def _normalize_require(
              self, value: Optional[_RawRequire]) -> List[Union[str, FmfId]]:
          if value is None:
              return []
          return [value] if isinstance(value, str) else value
  
      KEYS_SHOW_ORDER = [
          # Basic test information
          'summary',
          'description',
          'contact',
          'component',
          'id',
  
          # Test execution data
          'test',
          'path',
          'framework',
          'manual',
          'require',
          'recommend',
          'environment',
          'duration',
          'enabled',
          'order',
          'result',
  
          # Filtering attributes
          'tag',
          'tier',
          'link',
          ]
  
      @classmethod
      def from_dict(cls, mapping: Dict[str, Any], name: str, skip_validation: bool = False,
                    raise_on_validation_error: bool = False, **kwargs: Any) -> 'Test':
          """
          Initialize test data from a dictionary.
  
          Useful when data describing a test are stored in a mapping instead of an fmf node.
          """
  
          if not name.startswith('/'):
              raise tmt.utils.SpecificationError("Test name should start with a '/'.")
  
          node = fmf.Tree(mapping)
          node.name = name
  
          return cls(
              node=node,
              skip_validation=skip_validation,
              raise_on_validation_error=raise_on_validation_error,
              **kwargs)
  
      def __init__(
              self,
              *,
              node: fmf.Tree,
              skip_validation: bool = False,
              raise_on_validation_error: bool = False,
              **kwargs: Any) -> None:
          """
          Initialize test data from an fmf node or a dictionary
  
          The following two methods are supported:
  
              Test(node)
          """
  
          # Path defaults to the directory where metadata are stored or to
          # the root '/' if fmf metadata were not stored on the filesystem
          #
          # NOTE: default value of `path` attribute is not known when attribute
          # is declared, therefore we need to compute the default value and
          # assign it to attribute *before* calling superclass and its handy
          # node key extraction.
          try:
              directory = os.path.dirname(node.sources[-1])
              relative_path = os.path.relpath(directory, node.root)
              if relative_path == '.':
                  default_path = '/'
              else:
                  default_path = os.path.join('/', relative_path)
          except (AttributeError, IndexError):
              default_path = '/'
  
          self.path = default_path
  
          kwargs.setdefault('logger', self)
          super().__init__(
              node=node,
              skip_validation=skip_validation,
              raise_on_validation_error=raise_on_validation_error,
              **kwargs)
  
          # TODO: As long as validation is optional, a missing `test` key would be reported
          # as such but won't stop tmt from moving on.
          if self.test is None:
              raise tmt.utils.SpecificationError(
                  f"The 'test' attribute in '{self.name}' must be defined.")
  
          if self.framework == 'beakerlib':
              self.require.append('beakerlib')
  
          self._update_metadata()
  
      @staticmethod
      def overview(tree):
          """ Show overview of available tests """
          tests = [
              style(str(test), fg='red') for test in tree.tests()]
          echo(style(
              'Found {}{}{}.'.format(
                  listed(tests, 'test'),
                  ': ' if tests else '',
                  listed(tests, max=12)
                  ), fg='blue'))
  
      @staticmethod
      def create(name, template, path, force=False, dry=None):
          """ Create a new test """
          if dry is None:
              dry = Test._opt('dry')
  
          # Create directory
          if name == '.':
              directory_path = os.getcwd()
          else:
              directory_path = os.path.join(path, name.lstrip('/'))
              tmt.utils.create_directory(
                  directory_path, 'test directory', dry=dry)
  
          # Create metadata
          try:
              metadata_path = os.path.join(directory_path, 'main.fmf')
              tmt.utils.create_file(
                  path=metadata_path,
                  content=tmt.templates.TEST_METADATA[template],
                  name='test metadata',
                  dry=dry,
                  force=force)
          except KeyError:
              raise tmt.utils.GeneralError(f"Invalid template '{template}'.")
  
          # Create script
          script_path = os.path.join(directory_path, 'test.sh')
          try:
              content = tmt.templates.TEST[template]
          except KeyError:
              raise tmt.utils.GeneralError(f"Invalid template '{template}'.")
          tmt.utils.create_file(
              path=script_path, content=content,
              name='test script', dry=dry, force=force, mode=0o755)
  
      def show(self):
          """ Show test details """
          self.ls()
          for key in self.KEYS_SHOW_ORDER:
              value = getattr(self, key)
              if key == 'link':
                  value.show()
                  continue
              # No need to show the default order
              if key == 'order' and value == DEFAULT_ORDER:
                  continue
              if value not in [None, list(), dict()]:
                  echo(tmt.utils.format(key, value))
          if self.opt('verbose'):
              self._show_additional_keys()
              self._fmf_id()
          if self.opt('verbose', 0) >= 2:
              # Print non-empty unofficial attributes
              for key in sorted(self.node.get().keys()):
                  # Already asked to be printed
                  if key in self._keys():
                      continue
                  value = self.node.get(key)
                  if value not in [None, list(), dict()]:
                      echo(tmt.utils.format(key, value, key_color='blue'))
  
      def _lint_manual(self, test_path):
          """ Check that the manual instructions respect the specification """
          manual_test = os.path.join(test_path, self.test)
  
          # File does not exist
          if not os.path.exists(manual_test):
              return verdict(False, f"file '{self.test}' does not exist")
  
          # Check syntax for warnings
          warnings = tmt.export.check_md_file_respects_spec(manual_test)
          if warnings:
              for warning in warnings:
                  verdict(False, warning)
              return False
  
          # Everything looks ok
          return verdict(True, f"correct manual test syntax in '{self.test}'")
  
      def lint(self):
          """
          Check test against the L1 metadata specification.
  
          Return whether the test is valid.
          """
          self.ls()
          stripped_path = self.path.strip()
          test_path = self.node.root + stripped_path
  
          # Check test, path and summary (use bitwise '&' because 'and' is
          # lazy and would skip all verdicts following the first fail)
          valid = verdict(
              bool(self.test), 'test script must be defined')
          valid &= verdict(
              stripped_path.startswith('/'), 'directory path must be absolute')
          valid &= verdict(
              os.path.exists(test_path), 'directory path must exist')
          self._lint_summary()
  
          # Check for possible test case relevancy rules
          filename = self.node.sources[-1]
          metadata = tmt.utils.yaml_to_dict(self.read(filename))
          relevancy = metadata.pop('relevancy', None)
          if relevancy:
              # Convert into adjust rules if --fix enabled
              if self.opt('fix'):
                  metadata['adjust'] = tmt.convert.relevancy_to_adjust(relevancy)
                  self.write(filename, tmt.utils.dict_to_yaml(metadata))
                  verdict(None, 'relevancy converted into adjust')
              else:
                  valid = verdict(
                      False, 'relevancy has been obsoleted by adjust')
  
          # Check for possible coverage attribute
          coverage = metadata.pop('coverage', None)
          if coverage:
              valid = verdict(False, 'coverage has been obsoleted by link')
          # Check for unknown attributes
          # FIXME - Make additional attributes configurable
          # We don't want adjust in show/export so it is not yet in Test._keys
          invalid_keys = self.lint_keys(
              EXTRA_TEST_KEYS + OBSOLETED_TEST_KEYS + ['adjust'])
          if invalid_keys:
              valid = False
              for key in invalid_keys:
                  verdict(False, f"unknown attribute '{key}' is used")
          else:
              verdict(True, "correct attributes are used")
  
          # Check if the format of Markdown file respects the specification
          # https://tmt.readthedocs.io/en/latest/spec/tests.html#manual
          if self.manual:
              valid &= self._lint_manual(test_path)
  
          return valid
  
      def export(self, format_='yaml', keys=None):
          """
          Export test data into requested format
  
          In addition to 'yaml' and 'dict' it supports also a special
          format 'execute' used by the execute step which returns
          (test-name, test-data) tuples.
          """
  
          # Export to Nitrate test case management system
          if format_ == 'nitrate':
              tmt.export.export_to_nitrate(self)
  
          # Export to Polarion test case management system
          elif format_ == 'polarion':
              tmt.export.export_to_polarion(self)
  
          # Export the fmf identifier
          elif keys == 'fmf-id':
              if format_ == 'dict':
                  return self.fmf_id
              elif format_ == 'yaml':
                  return tmt.utils.dict_to_yaml(self.fmf_id)
              else:
                  raise tmt.utils.GeneralError(
                      f"Invalid test export format '{format_}'.")
  
          # Common node export otherwise
          else:
              return super().export(format_, keys)
  
  
  class Plan(tmt.utils.ValidateFmfMixin, tmt.utils.LoadFmfKeysMixin, Core):
      """ Plan object (L2 Metadata) """
  
      # `environment` and `environment-file` are NOT promoted to instance variables.
      context: FmfContextType = {}
      gate: List[str] = []
  
      _normalize_context = tmt.utils.LoadFmfKeysMixin._normalize_environment
      _normalize_gate = tmt.utils.LoadFmfKeysMixin._normalize_string_list
  
      extra_L2_keys = [
          'context',
          'environment',
          'environment-file',
          'gate',
          ]
  
      def __init__(
              self,
              *,
              node: fmf.Tree,
              run: Optional['Run'] = None,
              skip_validation: bool = False,
              raise_on_validation_error: bool = False,
              **kwargs: Any) -> None:
          """ Initialize the plan """
          kwargs.setdefault('logger', self)
          super().__init__(
              node=node,
              parent=run,
              skip_validation=skip_validation,
              raise_on_validation_error=raise_on_validation_error,
              **kwargs)
  
          # Save the run, prepare worktree and plan data directory
          self.my_run = run
          if self.my_run:
              # Skip to initialize the work tree if the corresponding option is
              # true. Note that 'tmt clean' consumes the option because it
              # should not initialize the work tree at all.
              if not run.opt(tmt.utils.PLAN_SKIP_WORKTREE_INIT):
                  self._initialize_worktree()
  
              self._initialize_data_directory()
  
          # Store 'environment' and 'environment-file' keys content
          self._environment = self._get_environment_vars(node)
          # Expand all environment variables in the node
          with tmt.utils.modify_environ(self.environment):
              self._expand_node_data(node.data)
  
          # Initialize test steps
          self.discover = tmt.steps.discover.Discover(
              plan=self, data=self.node.get('discover'))
          self.provision = tmt.steps.provision.Provision(
              plan=self, data=self.node.get('provision'))
          self.prepare = tmt.steps.prepare.Prepare(
              plan=self, data=self.node.get('prepare'))
          self.execute = tmt.steps.execute.Execute(
              plan=self, data=self.node.get('execute'))
          self.report = tmt.steps.report.Report(
              plan=self, data=self.node.get('report'))
          self.finish = tmt.steps.finish.Finish(
              plan=self, data=self.node.get('finish'))
  
          # Test execution context defined in the plan
          self._plan_context = self.node.get('context', dict())
  
          self._update_metadata()
  
      def _expand_node_data(self, data):
          """ Recursively expand variables in node data """
          if isinstance(data, str):
              return os.path.expandvars(data)
          elif isinstance(data, dict):
              for key, value in data.items():
                  data[key] = self._expand_node_data(value)
          elif isinstance(data, list):
              for i in range(len(data)):
                  data[i] = self._expand_node_data(data[i])
          return data
  
      @property
      def environment(self):
          """ Return combined environment from plan data and command line """
          if self.my_run:
              combined = self._environment.copy()
              # Command line variables take precedence
              combined.update(self.my_run.environment)
              # Include path to the plan data directory
              combined["TMT_PLAN_DATA"] = self.data_directory
              return combined
          else:
              return self._environment
  
      def _get_environment_vars(self, node):
          """ Get variables from 'environment' and 'environment-file' keys """
          # Environment variables from files
          environment_files = node.get("environment-file") or []
          if not isinstance(environment_files, list):
              raise tmt.utils.SpecificationError(
                  f"The 'environment-file' should be a list. "
                  f"Received '{type(environment_files).__name__}'.")
          combined = tmt.utils.environment_files_to_dict(
              environment_files, root=node.root)
  
          # Environment variables from key, make sure that values are string
          environment = dict([
              (key, str(value)) for key, value
              in node.get('environment', dict()).items()])
  
          # Combine both sources into one ('environment' key takes precendence)
          combined.update(environment)
          return combined
  
      def _initialize_worktree(self):
          """
          Prepare the worktree, a copy of the metadata tree root
  
          Used as cwd in prepare, execute and finish steps.
          """
          self.worktree = os.path.join(self.workdir, 'tree')
          tree_root = self.my_run.tree.root
  
          # Create an empty directory if there's no metadata tree
          if not tree_root:
              self.debug('Create an empty worktree (no metadata tree).', level=2)
              os.makedirs(self.worktree, exist_ok=True)
              return
  
          # Sync metadata root to the worktree
          self.debug(f"Sync the worktree to '{self.worktree}'.", level=2)
          self.run([
              "rsync", "-ar", "--exclude", ".git",
              f"{tree_root}/", self.worktree])
  
      def _initialize_data_directory(self):
          """
          Create the plan data directory
  
          This is used for storing logs and other artifacts created during
          prepare step, test execution or finish step and which are pulled
          from the guest for possible future inspection.
          """
          self.data_directory = os.path.join(self.workdir, "data")
          self.debug(
              f"Create the data directory '{self.data_directory}'.", level=2)
          os.makedirs(self.data_directory, exist_ok=True)
  
      def _fmf_context(self):
          """ Return combined context from plan data and command line """
          combined = self._plan_context.copy()
          combined.update(self._context.obj.fmf_context)
          return combined
  
      @staticmethod
      def edit_template(content):
          """ Edit the default template with custom values """
  
          content = tmt.utils.yaml_to_dict(content)
  
          # For each step check for possible command line data
          for step in tmt.steps.STEPS:
              options = Plan._opt(step)
              if not options:
                  continue
              step_data = []
  
              # For each option check for valid yaml and store
              for option in options:
                  try:
                      # FIXME: Ruamel.yaml "remembers" the used formatting when
                      #        using round-trip mode and since it comes from the
                      #        command-line, no formatting is applied resulting
                      #        in inconsistent formatting. Using a safe loader in
                      #        this case is a hack to make it forget, though
                      #        there may be a better way to do this.
                      try:
                          data = tmt.utils.yaml_to_dict(option, yaml_type='safe')
                          if not (data):
                              raise tmt.utils.GeneralError("Step data cannot be empty.")
                      except tmt.utils.GeneralError as error:
                          raise tmt.utils.GeneralError(
                              f"Invalid step data for {step}: '{option}'"
                              f"\n{error}")
                      step_data.append(data)
                  except MarkedYAMLError as error:
                      raise tmt.utils.GeneralError(
                          f"Invalid yaml data for {step}:\n{error}")
  
              # Use list only when multiple step data provided
              if len(step_data) == 1:
                  step_data = step_data[0]
              content[step] = step_data
  
          return tmt.utils.dict_to_yaml(content)
  
      @staticmethod
      def overview(tree):
          """ Show overview of available plans """
          plans = [
              style(str(plan), fg='red') for plan in tree.plans()]
          echo(style(
              'Found {}{}{}.'.format(
                  listed(plans, 'plan'),
                  ': ' if plans else '',
                  listed(plans, max=12)
                  ), fg='blue'))
  
      @staticmethod
      def create(name, template, path, force=False, dry=None):
          """ Create a new plan """
          # Prepare paths
          if dry is None:
              dry = Plan._opt('dry')
  
          (directory, plan) = os.path.split(name)
          directory_path = os.path.join(path, directory.lstrip('/'))
          has_fmf_ext = os.path.splitext(plan)[1] == '.fmf'
          plan_path = os.path.join(
              directory_path, plan + ('' if has_fmf_ext else '.fmf'))
  
          # Create directory & plan
          tmt.utils.create_directory(directory_path, 'plan directory', dry=dry)
          try:
              content = tmt.templates.PLAN[template]
          except KeyError:
              raise tmt.utils.GeneralError(
                  "Invalid template '{}'.".format(template))
  
          # Override template with data provided on command line
          content = Plan.edit_template(content)
  
          tmt.utils.create_file(
              path=plan_path, content=content,
              name='plan', dry=dry, force=force)
  
      def steps(self, enabled=True, disabled=False, names=False, skip=None):
          """
          Iterate over enabled / all steps
  
          Yields instances of all enabled steps by default. Use 'names' to
          yield step names only and 'disabled=True' to iterate over all.
          Use 'skip' to pass the list of steps to be skipped.
          """
          skip = skip or []
          for name in tmt.steps.STEPS:
              if name in skip:
                  continue
              step = getattr(self, name)
              if (enabled and step.enabled or disabled and not step.enabled):
                  yield name if names else step
  
      def show(self):
          """ Show plan details """
  
          # Summary and description first
          self.ls(summary=True)
          if self.description:
              echo(tmt.utils.format(
                  'description', self.description, key_color='green'))
  
          # Individual step details
          for step in self.steps(disabled=True):
              step.show()
  
          # Environment and context
          if self.environment:
              echo(tmt.utils.format(
                  'environment', self.environment, key_color='blue'))
          if self._fmf_context():
              echo(tmt.utils.format(
                  'context', self._fmf_context(), key_color='blue'))
  
          # The rest
          echo(tmt.utils.format('enabled', self.enabled, key_color='cyan'))
          if self.order != DEFAULT_ORDER:
              echo(tmt.utils.format('order', self.order, key_color='cyan'))
          if self.id:
              echo(tmt.utils.format('id', self.id, key_color='cyan'))
          if self.tag:
              echo(tmt.utils.format('tag', self.tag, key_color='cyan'))
          if self.tier:
              echo(tmt.utils.format('tier', self.tier, key_color='cyan'))
          self.link.show()
          if self.opt('verbose'):
              self._show_additional_keys()
  
      def _lint_execute(self):
          """ Lint execute step """
          execute = self.node.get('execute')
          if not execute:
              return verdict(False, "execute step must be defined with 'how'")
          if isinstance(execute, dict):
              execute = [execute]
  
          methods = [
              method.name
              for method in tmt.steps.execute.ExecutePlugin.methods()]
          correct = True
          for configuration in execute:
              how = configuration.get('how')
              if how not in methods:
                  name = configuration.get('name')
                  verdict(False,
                          f"unsupported execute method '{how}' in '{name}'")
                  correct = False
  
          return correct
  
      def _lint_discover(self):
          """ Lint discover step """
          # The discover step is optional
          discover = self.node.get('discover')
          if not discover:
              return True
          if isinstance(discover, dict):
              discover = [discover]
  
          methods = [
              method.name
              for method in tmt.steps.discover.DiscoverPlugin.methods()]
          correct = True
          for configuration in discover:
              how = configuration.get('how')
  
              if how not in methods:
                  verdict(False, f"unknown discover method '{how}'")
                  correct = False
                  continue
  
              # FIXME Add check for the shell discover method
              if how == 'shell':
                  continue
  
              if not self._lint_discover_fmf(configuration):
                  correct = False
          return correct
  
      @staticmethod
      def _lint_discover_fmf(discover):
          """ Lint fmf discover method """
          # Validate remote id and translate to human readable errors
          fmf_id_data = {
              key: value
              for key, value in discover.items()
              if key in ['url', 'ref', 'path']
              }
  
          valid, error = FmfId(**fmf_id_data).validate()
  
          if valid:
              name = discover.get('name')
              return verdict(True, f"fmf remote id in '{name}' is valid")
  
          return verdict(False, error)
  
      def lint(self):
          """
          Check plan against the L2 metadata specification
  
          Return whether the plan is valid.
          """
          self.ls()
  
          # Explore all available plugins
          tmt.plugins.explore()
  
          invalid_keys = self.lint_keys(
              list(self.steps(enabled=True, disabled=True, names=True)) +
              self.extra_L2_keys)
  
          if invalid_keys:
              for key in invalid_keys:
                  verdict(False, f"unknown attribute '{key}' is used")
          else:
              verdict(True, "correct attributes are used")
  
          return all([
              self._lint_summary(),
              self._lint_execute(),
              self._lint_discover(),
              len(invalid_keys) == 0])
  
      def go(self):
          """ Execute the plan """
          # Show plan name and summary (one blank line to separate plans)
          self.info('')
          self.info(self.name, color='red')
          if self.summary:
              self.verbose('summary', self.summary, 'green')
  
          # Additional debug info like plan environment
          self.debug('info', color='cyan', shift=0, level=3)
          self.debug('environment', self.environment, 'magenta', level=3)
          self.debug('context', self._fmf_context(), 'magenta', level=3)
  
          # Wake up all steps
          self.debug('wake', color='cyan', shift=0, level=2)
          for step in self.steps(disabled=True):
              self.debug(str(step), color='blue', level=2)
              try:
                  step.wake()
              except tmt.utils.SpecificationError as error:
                  # Re-raise the exception if the step is enabled (invalid
                  # step data), otherwise just warn the user and continue.
                  if step.enabled:
                      raise error
                  else:
                      step.warn(error)
  
          # Set up login and reboot plugins for all steps
          self.debug("action", color="blue", level=2)
          for step in self.steps(disabled=True):
              step.setup_actions()
  
          # Check if steps are not in stand-alone mode
          standalone = set()
          for step in self.steps():
              standalone_plugins = step.plugins_in_standalone_mode
              if standalone_plugins == 1:
                  standalone.add(step.name)
              elif standalone_plugins > 1:
                  raise tmt.utils.GeneralError(
                      f"Step '{step.name}' has multiple plugin configs which "
                      f"require running on their own. Combination of such "
                      f"configs is not possible.")
          if len(standalone) > 1:
              raise tmt.utils.GeneralError(
                  f'These steps require running on their own, their combination '
                  f'with the given options is not compatible: '
                  f'{fmf.utils.listed(standalone)}.')
          elif standalone:
              self._context.obj.steps = standalone
              self.debug(
                  f"Running the '{list(standalone)[0]}' step as standalone.")
  
          # Run enabled steps except 'finish'
          self.debug('go', color='cyan', shift=0, level=2)
          try:
              abort = False
              for step in self.steps(skip=['finish']):
                  step.go()
                  # Finish plan if no tests found (except dry mode)
                  if (step.name == 'discover' and not step.tests()
                          and not self.opt('dry')):
                      step.info(
                          'warning', 'No tests found, finishing plan.',
                          color='yellow', shift=1)
                      abort = True
                      return
          # Make sure we run 'finish' step always if enabled
          finally:
              if not abort and self.finish.enabled:
                  self.finish.go()
  
      def export(self, format_='yaml'):
          """
          Export plan data into requested format
  
          Supported formats are 'yaml' and 'dict'.
          """
          data = super().export(format_='dict')
  
          for key in self.extra_L2_keys:
              value = self.node.data.get(key)
              if value:
                  data[key] = value
  
          for step in tmt.steps.STEPS:
              value = self.node.data.get(step)
              if value:
                  data[step] = value
  
          # Choose proper format
          if format_ == 'dict':
              return data
          elif format_ == 'yaml':
              return tmt.utils.dict_to_yaml(data)
          else:
              raise tmt.utils.GeneralError(
                  f"Invalid plan export format '{format_}'.")
  
  
  class StoryPriority(enum.Enum):
      MUST_HAVE = 'must have'
      SHOULD_HAVE = 'should have'
      COULD_HAVE = 'could have'
      WILL_NOT_HAVE = 'will not have'
  
      # We need a custom "to string" conversion to support fmf's filter().
      def __str__(self) -> str:
          return self.value
  
  
  class Story(tmt.utils.ValidateFmfMixin, tmt.utils.LoadFmfKeysMixin, Core):
      """ User story object """
  
      example: List[str] = []
      story: str
      title: Optional[str] = None
      priority: Optional[StoryPriority] = None
  
      _normalize_example = tmt.utils.LoadFmfKeysMixin._normalize_string_list
  
      def _normalize_priority(self, value: Optional[str]) -> Optional[StoryPriority]:
          if value is None:
              return None
          return StoryPriority(value)
  
      KEYS_SHOW_ORDER = [
          'summary',
          'title',
          'story',
          'id',
          'priority',
          'description',
          'example',
          'enabled',
          'order',
          'tag',
          'tier',
          'link',
          ]
  
      def __init__(
              self,
              *,
              node: fmf.Tree,
              skip_validation: bool = False,
              raise_on_validation_error: bool = False,
              **kwargs: Any) -> None:
          """ Initialize the story """
          kwargs.setdefault('logger', self)
          super().__init__(node=node, skip_validation=skip_validation,
                           raise_on_validation_error=raise_on_validation_error, **kwargs)
          self._update_metadata()
  
      @property
      def documented(self):
          """ Return links to relevant documentation """
          return self.link.get('documented-by')
  
      @property
      def verified(self):
          """ Return links to relevant test coverage """
          return self.link.get('verified-by')
  
      @property
      def implemented(self):
          """ Return links to relevant source code """
          return self.link.get('implemented-by')
  
      def _match(
              self, implemented, verified, documented, covered,
              unimplemented, unverified, undocumented, uncovered):
          """ Return true if story matches given conditions """
          if implemented and not self.implemented:
              return False
          if verified and not self.verified:
              return False
          if documented and not self.documented:
              return False
          if unimplemented and self.implemented:
              return False
          if unverified and self.verified:
              return False
          if undocumented and self.documented:
              return False
          if uncovered and (
                  self.implemented and self.verified and self.documented):
              return False
          if covered and not (
                  self.implemented and self.verified and self.documented):
              return False
          return True
  
      @staticmethod
      def create(name, template, path, force=False, dry=None):
          """ Create a new story """
          if dry is None:
              dry = Story._opt('dry')
  
          # Prepare paths
          (directory, story) = os.path.split(name)
          directory_path = os.path.join(path, directory.lstrip('/'))
          has_fmf_ext = os.path.splitext(story)[1] == '.fmf'
          story_path = os.path.join(directory_path,
                                    story + ('' if has_fmf_ext else '.fmf'))
  
          # Create directory & story
          tmt.utils.create_directory(directory_path, 'story directory', dry=dry)
          try:
              content = tmt.templates.STORY[template]
          except KeyError:
              raise tmt.utils.GeneralError(
                  "Invalid template '{}'.".format(template))
  
          tmt.utils.create_file(
              path=story_path, content=content,
              name='story', dry=dry, force=force)
  
      @staticmethod
      def overview(tree):
          """ Show overview of available stories """
          stories = [
              style(str(story), fg='red') for story in tree.stories()]
          echo(style(
              'Found {}{}{}.'.format(
                  listed(stories, 'story'),
                  ': ' if stories else '',
                  listed(stories, max=12)
                  ), fg='blue'))
  
      def show(self):
          """ Show story details """
          self.ls()
          for key in self.KEYS_SHOW_ORDER:
              value = getattr(self, key)
              if key == 'link':
                  value.show()
                  continue
              if key == 'priority' and value is not None:
                  value = cast(StoryPriority, value).value
              if key == 'order' and value == DEFAULT_ORDER:
                  continue
              if value is not None and value != []:
                  wrap = False if key == 'example' else 'auto'
                  echo(tmt.utils.format(key, value, wrap=wrap))
          if self.opt('verbose'):
              self._show_additional_keys()
  
      def coverage(self, code, test, docs):
          """ Show story coverage """
          if code:
              code = bool(self.implemented)
              verdict(code, good='done ', bad='todo ', nl=False)
          if test:
              test = bool(self.verified)
              verdict(test, good='done ', bad='todo ', nl=False)
          if docs:
              docs = bool(self.documented)
              verdict(docs, good='done ', bad='todo ', nl=False)
          echo(self)
          return (code, test, docs)
  
      def export(self, format_='rst', title=True):
          """ Export story data into requested format """
  
          # Use common Core export unless 'rst' requested
          if format_ != 'rst':
              return super().export(format_=format_)
  
          output = ''
  
          # Title and its anchor
          if title:
              depth = len(re.findall('/', self.name)) - 1
              if self.title and self.title != self.node.parent.get('title'):
                  title = self.title
              else:
                  title = re.sub('.*/', '', self.name)
              output += f'\n.. _{self.name}:\n'
              output += '\n{}\n{}\n'.format(title, '=~^:-><'[depth] * len(title))
  
          # Summary, story and description
          if self.summary and self.summary != self.node.parent.get('summary'):
              output += '\n{}\n'.format(self.summary)
          if self.story != self.node.parent.get('story'):
              output += '\n*{}*\n'.format(self.story.strip())
          # Insert note about unimplemented feature (leaf nodes only)
          if not self.node.children and not self.implemented:
              output += '\n.. note:: This is a draft, '
              output += 'the story is not implemented yet.\n'
          if (self.description and
                  self.description != self.node.parent.get('description')):
              output += '\n{}\n'.format(self.description)
  
          # Examples
          if self.example and self.example != self.node.parent.get('example'):
              examples = tmt.utils.listify(self.example)
              first = True
              for example in examples:
                  if first:
                      output += '\nExamples::\n\n'
                      first = False
                  else:
                      output += '\n::\n\n'
                  output += tmt.utils.format(
                      '', example, wrap=False, indent=4,
                      key_color=None, value_color=None) + '\n'
  
          # Status
          if not self.node.children:
              status = []
              for coverage in ['implemented', 'verified', 'documented']:
                  if getattr(self, coverage):
                      status.append(coverage)
              output += "\nStatus: {}\n".format(
                  listed(status) if status else 'idea')
  
          return output
  
      def _lint_story(self):
          story = self.node.get('story')
          if not story:
              return verdict(False, "story is required")
          return True
  
      def lint(self):
          """
          Check story against the L3 metadata specification.
  
          Return whether the story is valid.
          """
          self.ls()
          invalid_keys = self.lint_keys(EXTRA_STORY_KEYS)
  
          if invalid_keys:
              for key in invalid_keys:
                  verdict(False, f"unknown attribute '{key}' is used")
          else:
              verdict(True, "correct attributes are used")
  
          return all([self._lint_summary(),
                      self._lint_story(),
                      len(invalid_keys) == 0])
  
  
  class Tree(tmt.utils.Common):
      """ Test Metadata Tree """
  
      def __init__(self, path='.', tree=None, context=None):
          """ Initialize tmt tree from directory path or given fmf tree """
          self._path = path
          self._tree = tree
          self._custom_context = context
  
      def _fmf_context(self):
          """ Use custom fmf context if provided, default otherwise """
          if self._custom_context is not None:
              return self._custom_context
          return super()._fmf_context()
  
      def _filters_conditions(self, nodes, filters, conditions, links: List['LinkNeedle'], excludes):
          """ Apply filters and conditions, return pruned nodes """
          result = []
          for node in nodes:
              filter_vars = copy.deepcopy(node._metadata)
              cond_vars = node._metadata
              # Add a lowercase version of bool variables for filtering
              bool_vars = {
                  key: [value, str(value).lower()]
                  for key, value in filter_vars.items()
                  if isinstance(value, bool)}
              filter_vars.update(bool_vars)
              # Conditions
              try:
                  if not all(fmf.utils.evaluate(condition, cond_vars, node)
                             for condition in conditions):
                      continue
              except fmf.utils.FilterError:
                  # Handle missing attributes as if condition failed
                  continue
              except Exception as error:
                  raise tmt.utils.GeneralError(
                      f"Invalid --condition raised exception: {error}")
              # Filters
              try:
                  if not all(fmf.utils.filter(filter_, filter_vars, regexp=True)
                             for filter_ in filters):
                      continue
              except fmf.utils.FilterError:
                  # Handle missing attributes as if filter failed
                  continue
              # Links
              try:
                  # Links are in OR relation
                  if links and all(not node.has_link(needle) for needle in links):
                      continue
              except BaseException:
                  # Handle broken link as not matching
                  continue
              # Exclude
              if any(node for expr in excludes if re.search(expr, node.name)):
                  continue
              result.append(node)
          return result
  
      @property
      def tree(self):
          """ Initialize tree only when accessed """
          if self._tree is None:
              try:
                  self._tree = fmf.Tree(self._path)
              except fmf.utils.RootError:
                  raise tmt.utils.MetadataError(
                      f"No metadata found in the '{self._path}' directory. "
                      f"Use 'tmt init' to get started.")
              except fmf.utils.FileError as error:
                  raise tmt.utils.GeneralError(f"Invalid yaml syntax: {error}")
              # Adjust metadata for current fmf context
              self._tree.adjust(fmf.context.Context(**self._fmf_context()))
          return self._tree
  
      @tree.setter
      def tree(self, new_tree):
          self._tree = new_tree
  
      @property
      def root(self):
          """ Metadata root """
          return self.tree.root
  
      def tests(self, keys=None, names=None, filters=None, conditions=None,
                unique=True, links: Optional[List['LinkNeedle']] = None, excludes=None):
          """ Search available tests """
          # Handle defaults, apply possible command line options
          keys = (keys or []) + ['test']
          names = names or []
          filters = (filters or []) + list(Test._opt('filters', []))
          conditions = (conditions or []) + list(Test._opt('conditions', []))
          links = (links or []) + [
              LinkNeedle.from_raw(value)
              for value in cast(List[str], Test._opt('links', []))
              ]
          excludes = (excludes or []) + list(Test._opt('exclude', []))
          # Used in: tmt run test --name NAME, tmt test ls NAME...
          cmd_line_names = list(Test._opt('names', []))
  
          def name_filter(nodes):
              """ Filter nodes based on names provided on the command line """
              if not cmd_line_names:
                  return nodes
              return [
                  node for node in nodes
                  if any(re.search(name, node.name) for name in cmd_line_names)]
  
          # Append post filter to support option --enabled or --disabled
          if Test._opt('enabled'):
              filters.append('enabled:true')
          if Test._opt('disabled'):
              filters.append('enabled:false')
  
          if Test._opt('source'):
              tests = [
                  Test(node=test) for test in self.tree.prune(
                      keys=keys, sources=cmd_line_names)]
          else:
              # First let's build the list of test objects based on keys & names.
              # If duplicate test names are allowed, match test name/regexp
              # one-by-one and preserve the order of tests within a plan.
              if not unique and names:
                  tests = []
                  for name in names:
                      selected_tests = [
                          Test(node=test) for test
                          in name_filter(self.tree.prune(keys=keys, names=[name]))]
                      tests.extend(
                          sorted(selected_tests, key=lambda test: test.order))
              # Otherwise just perform a regular key/name filtering
              else:
                  selected_tests = [
                      Test(node=test) for test
                      in name_filter(self.tree.prune(keys=keys, names=names))]
                  tests = sorted(selected_tests, key=lambda test: test.order)
  
          # Apply filters & conditions
          return self._filters_conditions(
              tests, filters, conditions, links, excludes)
  
      def plans(self, keys=None, names=None, filters=None, conditions=None,
                run=None, links: Optional[List['LinkNeedle']] = None, excludes=None):
          """ Search available plans """
          # Handle defaults, apply possible command line options
          keys = (keys or []) + ['execute']
          names = (names or []) + list(Plan._opt('names', []))
          filters = (filters or []) + list(Plan._opt('filters', []))
          conditions = (conditions or []) + list(Plan._opt('conditions', []))
          links = (links or []) + [
              LinkNeedle.from_raw(value)
              for value in cast(List[str], Plan._opt('links', []))
              ]
          excludes = (excludes or []) + list(Plan._opt('exclude', []))
  
          # Append post filter to support option --enabled or --disabled
          if Plan._opt('enabled'):
              filters.append('enabled:true')
          if Plan._opt('disabled'):
              filters.append('enabled:false')
  
          # For --source option use names as sources
          if Plan._opt('source'):
              sources = names
              names = None
          else:
              sources = None
  
          # Build the list, convert to objects, sort and filter
          plans = [
              Plan(node=plan, run=run) for plan
              in self.tree.prune(keys=keys, names=names, sources=sources)]
          return self._filters_conditions(
              sorted(plans, key=lambda plan: plan.order),
              filters, conditions, links, excludes)
  
      def stories(self, keys=None, names=None, filters=None, conditions=None,
                  whole=False, links: Optional[List['LinkNeedle']] = None, excludes=None):
          """ Search available stories """
          # Handle defaults, apply possible command line options
          keys = (keys or []) + ['story']
          names = (names or []) + list(Story._opt('names', []))
          filters = (filters or []) + list(Story._opt('filters', []))
          conditions = (conditions or []) + list(Story._opt('conditions', []))
          links = (links or []) + [
              LinkNeedle.from_raw(value)
              for value in cast(List[str], Story._opt('links', []))
              ]
          excludes = (excludes or []) + list(Story._opt('exclude', []))
  
          # Append post filter to support option --enabled or --disabled
          if Story._opt('enabled'):
              filters.append('enabled:true')
          if Story._opt('disabled'):
              filters.append('enabled:false')
  
          # For --source option use names as sources
          if Story._opt('source'):
              sources = names
              names = None
          else:
              sources = None
  
          # Build the list, convert to objects, sort and filter
          stories = [
              Story(node=story) for story
              in self.tree.prune(keys=keys, names=names, whole=whole, sources=sources)]
          return self._filters_conditions(
              sorted(stories, key=lambda story: story.order),
              filters, conditions, links, excludes)
  
      @staticmethod
      def init(path, template, force, **kwargs):
          """ Initialize a new tmt tree, optionally with a template """
          path = os.path.realpath(path)
          dry = Tree._opt('dry')
  
          # Check for existing tree
          try:
              tree = tmt.Tree(path)
              # Are we creating a new tree under the existing one?
              if path == tree.root:
                  echo(f"Tree '{tree.root}' already exists.")
              else:
                  # Are we creating a nested tree?
                  echo(f"Path '{path}' already has a parent tree root '{tree.root}'.")
                  if not force and not confirm("Do you really want to initialize a nested tree?"):
                      return
                  tree = None
          except tmt.utils.GeneralError:
              tree = None
  
          # Create a new tree
          if tree is None:
              if dry:
                  echo(f"Tree '{path}' would be initialized.")
              else:
                  try:
                      fmf.Tree.init(path)
                      tree = tmt.Tree(path)
                      path = tree.root
                  except fmf.utils.GeneralError as error:
                      raise tmt.utils.GeneralError(
                          f"Failed to initialize tree in '{path}': {error}")
                  echo(f"Tree '{tree.root}' initialized.")
  
          # Populate the tree with example objects if requested
          if template == 'empty':
              choices = listed(tmt.templates.INIT_TEMPLATES, join='or')
              echo(
                  f"To populate it with example content, "
                  f"use --template with {choices}.")
          else:
              echo(f"Applying template '{template}'.")
  
          if template == 'mini':
              tmt.Plan.create('/plans/example', 'mini', path, force, dry)
          elif template == 'base':
              tmt.Test.create('/tests/example', 'beakerlib', path, force, dry)
              tmt.Plan.create('/plans/example', 'base', path, force, dry)
          elif template == 'full':
              tmt.Test.create('/tests/example', 'shell', path, force, dry)
              tmt.Plan.create('/plans/example', 'full', path, force, dry)
              tmt.Story.create('/stories/example', 'full', path, force, dry)
  
  
  class Run(tmt.utils.Common):
      """ Test run, a container of plans """
  
      def __init__(self, id_=None, tree=None, context=None):
          """ Initialize tree, workdir and plans """
          # Use the last run id if requested
          self.config = tmt.utils.Config()
          if context.params.get('last'):
              id_ = self.config.last_run()
              if id_ is None:
                  raise tmt.utils.GeneralError(
                      "No last run id found. Have you executed any run?")
          if context.params.get('follow') and id_ is None:
              raise tmt.utils.GeneralError(
                  "Run id has to be specified in order to use --follow.")
          # Do not create workdir now, postpone it until later, as options
          # have not been processed yet and we do not want commands such as
          # tmt run discover --how fmf --help to create a new workdir.
          super().__init__(context=context)
          self._workdir_path = id_ or True
          self._tree = tree
          self._plans = None
          self._environment_from_workdir = dict()
          self._environment_from_options = None
          self.remove = self.opt('remove')
  
      def _use_default_plan(self):
          """ Prepare metadata tree with only the default plan """
          default_plan = tmt.utils.yaml_to_dict(tmt.templates.DEFAULT_PLAN)
          # The default discover method for this case is 'shell'
          default_plan['/plans/default']['discover']['how'] = 'shell'
          self.tree = tmt.Tree(tree=fmf.Tree(default_plan))
          self.debug("No metadata found, using the default plan.")
  
      def _save_tree(self, tree):
          """ Save metadata tree, handle the default plan """
          default_plan = tmt.utils.yaml_to_dict(tmt.templates.DEFAULT_PLAN)
          try:
              self.tree = tree if tree else tmt.Tree('.')
              self.debug(f"Using tree '{self.tree.root}'.")
              # Clear the tree and insert default plan if requested
              if Plan._opt("default"):
                  new_tree = fmf.Tree(default_plan)
                  new_tree.root = self.tree.root
                  self.tree.tree = new_tree
                  self.debug("Enforcing use of default plan")
              # Insert default plan if no plan detected
              if not list(self.tree.tree.prune(keys=['execute'])):
                  self.tree.tree.update(default_plan)
                  self.debug("No plan found, adding the default plan.")
          # Create an empty default plan if no fmf metadata found
          except tmt.utils.MetadataError:
              self._use_default_plan()
  
      @property
      def environment(self):
          """ Return environment combined from wake up and command line """
          # Gather environment variables from options only once
          if self._environment_from_options is None:
              self._environment_from_options = dict()
              # Variables gathered from 'environment-file' options
              self._environment_from_options.update(
                  tmt.utils.environment_files_to_dict(
                      (self.opt('environment-file') or []),
                      root=self.tree.root))
              # Variables from 'environment' options (highest priority)
              self._environment_from_options.update(
                  tmt.utils.environment_to_dict(self.opt('environment')))
  
          # Combine workdir and command line
          combined = self._environment_from_workdir.copy()
          combined.update(self._environment_from_options)
          return combined
  
      def save(self):
          """ Save list of selected plans and enabled steps """
          data = {
              'root': self.tree.root,
              'plans': [plan.name for plan in self._plans],
              'steps': list(self._context.obj.steps),
              'environment': self.environment,
              'remove': self.remove,
              }
          self.write('run.yaml', tmt.utils.dict_to_yaml(data))
  
      def load_from_workdir(self):
          """
          Load the run from its workdir, do not require the root in
          run.yaml to exist. Doest not load the fmf tree.
  
          Use only when the data in workdir is sufficient (e.g. tmt
          clean and status only require the steps to be loaded and
          their status).
          """
          self._save_tree(self._tree)
          self._workdir_load(self._workdir_path)
          try:
              data = tmt.utils.yaml_to_dict(self.read('run.yaml'))
          except tmt.utils.FileError:
              self.debug('Run data not found.')
              return
          self._environment_from_workdir = data.get('environment')
          self._context.obj.steps = set(data['steps'])
          plans = []
  
          # The root directory of the tree may not be available, create
          # an fmf node that only contains the necessary attributes
          # required for plan/step loading. We will also need a dummy
          # parent for these nodes, so we would correctly load each
          # plan's name.
          dummy_parent = fmf.Tree({'summary': 'unused'})
  
          for plan in data.get('plans'):
              node = fmf.Tree({'execute': None}, name=plan, parent=dummy_parent)
              plans.append(Plan(node=node, run=self, skip_validation=True))
  
          self._plans = plans
  
      def load(self):
          """ Load list of selected plans and enabled steps """
          try:
              data = tmt.utils.yaml_to_dict(self.read('run.yaml'))
          except tmt.utils.FileError:
              self.debug('Run data not found.')
              return
  
          # If run id was given and root was not explicitly specified,
          # create a new Tree from the root in run.yaml
          if self._workdir and 'root' in data and not self.opt('root'):
              if data['root']:
                  self._save_tree(tmt.Tree(data['root']))
              else:
                  # The run was used without any metadata, default plan
                  # was used, load it
                  self._use_default_plan()
  
          # Filter plans by name unless specified on the command line
          plan_options = ['names', 'filters', 'conditions', 'links', 'default']
          if not any(Plan._opt(option) for option in plan_options):
              self._plans = [
                  plan for plan in self.tree.plans(run=self)
                  if plan.name in data['plans']]
  
          # Initialize steps only if not selected on the command line
          step_options = 'all since until after before skip'.split()
          selected = any(self.opt(option) for option in step_options)
          if not selected and not self._context.obj.steps:
              self._context.obj.steps = set(data['steps'])
  
          # Store loaded environment
          self._environment_from_workdir = data.get('environment')
          self.debug(
              f"Loaded environment: '{self._environment_from_workdir}'.",
              level=3)
  
          # If the remove was enabled, restore it, option overrides
          self.remove = self.remove or data.get('remove', 'False')
          self.debug(f"Remove workdir when finished: {self.remove}", level=3)
  
      @property
      def plans(self):
          """ Test plans for execution """
          if self._plans is None:
              self._plans = self.tree.plans(run=self, filters=['enabled:true'])
          return self._plans
  
      def finish(self):
          """ Check overall results, return appropriate exit code """
          # We get interesting results only if execute or prepare step is enabled
          execute = self.plans[0].execute
          report = self.plans[0].report
          interesting_results = execute.enabled or report.enabled
  
          # Gather all results and give an overall summary
          results = [
              result
              for plan in self.plans
              for result in plan.execute.results()]
          if interesting_results:
              self.info('')
              self.info('total', Result.summary(results), color='cyan')
  
          # Remove the workdir if enabled
          if self.remove and self.plans[0].finish.enabled:
              self._workdir_cleanup(self.workdir)
  
          # Skip handling of the exit codes in dry mode and
          # when there are no interesting results available
          if self.opt('dry') or not interesting_results:
              return
  
          # Return 0 if test execution has been intentionally skipped
          if tmt.steps.execute.Execute._opt("dry"):
              raise SystemExit(0)
          # Return appropriate exit code based on the total stats
          stats = Result.total(results)
          if sum(stats.values()) == 0:
              raise SystemExit(3)
          if stats['error']:
              raise SystemExit(2)
          if stats['fail'] + stats['warn']:
              raise SystemExit(1)
          if stats['pass']:
              raise SystemExit(0)
          raise SystemExit(2)
  
      def follow(self):
          """ Periodically check for new lines in the log. """
          logfile = open(os.path.join(self.workdir, tmt.utils.LOG_FILENAME), 'r')
          # Move to the end of the file
          logfile.seek(0, os.SEEK_END)
          # Rewind some lines back to show more context
          location = logfile.tell()
          read_lines = 0
          while location >= 0:
              logfile.seek(location)
              location -= 1
              current_char = logfile.read(1)
              if current_char == '\n':
                  read_lines += 1
              if read_lines > FOLLOW_LINES:
                  break
  
          while True:
              line = logfile.readline()
              if line:
                  print(line, end='')
              else:
                  time.sleep(0.5)
  
      def go(self):
          """ Go and do test steps for selected plans """
          # Create the workdir and save last run
          self._save_tree(self._tree)
          self._workdir_load(self._workdir_path)
          if self.tree.root and self._workdir.startswith(self.tree.root):
              raise tmt.utils.GeneralError(
                  "Run workdir must not be inside fmf root.")
          self.config.last_run(self.workdir)
          # Show run id / workdir path
          self.info(self.workdir, color='magenta')
          self.debug(f"tmt version: {tmt.__version__}")
          # Attempt to load run data
          self.load()
          # Follow log instead of executing the run
          if self.opt('follow'):
              self.follow()
  
          # Propagate dry mode from provision to prepare, execute and finish
          # (basically nothing can be done if there is no guest provisioned)
          if tmt.steps.provision.Provision._opt("dry"):
              tmt.steps.prepare.Prepare._options["dry"] = True
              tmt.steps.execute.Execute._options["dry"] = True
              tmt.steps.finish.Finish._options["dry"] = True
  
          # Enable selected steps
          enabled_steps = self._context.obj.steps
          all_steps = self.opt('all') or not enabled_steps
          since = self.opt('since')
          until = self.opt('until')
          after = self.opt('after')
          before = self.opt('before')
          skip = self.opt('skip')
  
          if any([all_steps, since, until, after, before]):
              # Detect index of the first and last enabled step
              if since:
                  first = tmt.steps.STEPS.index(since)
              elif after:
                  first = tmt.steps.STEPS.index(after) + 1
              else:
                  first = tmt.steps.STEPS.index('discover')
              if until:
                  last = tmt.steps.STEPS.index(until)
              elif before:
                  last = tmt.steps.STEPS.index(before) - 1
              else:
                  last = tmt.steps.STEPS.index('finish')
              # Enable all steps between the first and last
              for index in range(first, last + 1):
                  step = tmt.steps.STEPS[index]
                  if step not in skip:
                      enabled_steps.add(step)
          self.debug(f"Enabled steps: {fmf.utils.listed(enabled_steps)}")
  
          # Show summary, store run data
          if not self.plans:
              raise tmt.utils.GeneralError("No plans found.")
          self.verbose('Found {0}.'.format(listed(self.plans, 'plan')))
          self.save()
  
          # Iterate over plans
          for plan in self.plans:
              plan.go()
  
          # Update the last run id at the very end
          # (override possible runs created during execution)
          self.config.last_run(self.workdir)
  
          # Give the final summary, remove workdir, handle exit codes
          self.finish()
  
  
  class Status(tmt.utils.Common):
      """ Status of tmt work directories. """
  
      LONGEST_STEP = max(tmt.steps.STEPS, key=lambda k: len(k))
      FIRST_COL_LEN = len(LONGEST_STEP) + 2
  
      @staticmethod
      def get_overall_plan_status(plan):
          """ Examines the plan status (find the last done step) """
          steps = list(plan.steps())
          step_names = list(plan.steps(names=True))
          for i in range(len(steps) - 1, -1, -1):
              if steps[i].status() == 'done':
                  if i + 1 == len(steps):
                      # Last enabled step, consider the whole plan done
                      return 'done'
                  else:
                      return step_names[i]
          return 'todo'
  
      def plan_matches_filters(self, plan):
          """ Check if the given plan matches filters from the command line """
          if self.opt('abandoned'):
              return (plan.provision.status() == 'done'
                      and plan.finish.status() == 'todo')
          if self.opt('active'):
              return any(step.status() == 'todo' for step in plan.steps())
          if self.opt('finished'):
              return all(step.status() == 'done' for step in plan.steps())
          return True
  
      @staticmethod
      def colorize_column(content):
          """ Add color to a status column """
          if 'done' in content:
              return style(content, fg='green')
          else:
              return style(content, fg='yellow')
  
      @classmethod
      def pad_with_spaces(cls, string):
          """ Append spaces to string to properly align the first column """
          return string + (cls.FIRST_COL_LEN - len(string)) * ' '
  
      def run_matches_filters(self, run):
          """ Check if the given run matches filters from the command line """
          if self.opt('abandoned') or self.opt('active'):
              # Any of the plans must be abandoned/active for the whole
              # run to be abandoned/active
              return any(self.plan_matches_filters(p) for p in run.plans)
          if self.opt('finished'):
              # All plans must be finished for the whole run to be finished
              return all(self.plan_matches_filters(p) for p in run.plans)
          return True
  
      def print_run_status(self, run):
          """ Display the overall status of the run """
          if not self.run_matches_filters(run):
              return
          # Find the earliest step in all plans' status
          earliest_step_index = len(tmt.steps.STEPS)
          for plan in run.plans:
              plan_status = self.get_overall_plan_status(plan)
              if plan_status == 'done':
                  continue
              elif plan_status == 'todo':
                  # If plan has no steps done, consider the whole run not done
                  earliest_step_index = -1
                  break
              plan_status_index = tmt.steps.STEPS.index(plan_status)
              if plan_status_index < earliest_step_index:
                  earliest_step_index = plan_status_index
  
          if earliest_step_index == len(tmt.steps.STEPS):
              run_status = 'done'
          elif earliest_step_index == -1:
              run_status = 'todo'
          else:
              run_status = tmt.steps.STEPS[earliest_step_index]
          run_status = self.colorize_column(self.pad_with_spaces(run_status))
          echo(run_status, nl=False)
          echo(run.workdir)
  
      def print_plans_status(self, run):
          """ Display the status of each plan of the given run """
          for plan in run.plans:
              if self.plan_matches_filters(plan):
                  plan_status = self.get_overall_plan_status(plan)
                  echo(self.colorize_column(self.pad_with_spaces(plan_status)),
                       nl=False)
                  echo(f'{run.workdir}  {plan.name}')
  
      def print_verbose_status(self, run):
          """ Display the status of each step of the given run """
          for plan in run.plans:
              if self.plan_matches_filters(plan):
                  for step in plan.steps(disabled=True):
                      column = (step.status() or '----') + ' '
                      echo(self.colorize_column(column), nl=False)
                  echo(f' {run.workdir}  {plan.name}')
  
      def process_run(self, run):
          """ Display the status of the given run based on verbosity """
          loaded, error = tmt.utils.load_run(run)
          if not loaded:
              self.warn(f"Failed to load run '{run.workdir}': {error}")
              return
          if self.opt('verbose') == 0:
              self.print_run_status(run)
          elif self.opt('verbose') == 1:
              self.print_plans_status(run)
          else:
              self.print_verbose_status(run)
  
      def print_header(self):
          """ Print the header of the status table based on verbosity """
          header = ''
          if self.opt('verbose') >= 2:
              for step in tmt.steps.STEPS:
                  header += (step[0:4] + ' ')
              header += ' '
          else:
              header = self.pad_with_spaces('status')
          header += 'id'
          echo(style(header, fg='blue'))
  
      def show(self):
          """ Display the current status """
          # Prepare absolute workdir path if --id was used
          id_ = self.opt('id')
          root_path = self.opt('workdir-root')
          self.print_header()
          for abs_path in tmt.utils.generate_runs(root_path, id_):
              run = Run(abs_path, self._context.obj.tree, self._context)
              self.process_run(run)
  
  
  class Clean(tmt.utils.Common):
      """ A class for cleaning up workdirs, guests or images """
  
      def __init__(self, parent=None, name=None, workdir=None, context=None):
          """
          Initialize name and relation with the parent object
  
          Always skip to initialize the work tree.
          """
          # Set the option to skip to initialize the work tree
          context.params[tmt.utils.PLAN_SKIP_WORKTREE_INIT] = True
          super().__init__(parent, name, workdir, context)
  
      def images(self):
          """ Clean images of provision plugins """
          self.info('images', color='blue')
          successful = True
          for method in tmt.steps.provision.ProvisionPlugin.methods():
              if not method.class_.clean_images(self, self.opt('dry')):
                  successful = False
          return successful
  
      def _matches_how(self, plan):
          """ Check if the given plan matches options """
          how = plan.provision.data[0].how
          target_how = self.opt('how')
          if target_how:
              return how == target_how
          # No target provision method, always matches
          return True
  
      def _stop_running_guests(self, run):
          """ Stop all running guests of a run """
          loaded, error = tmt.utils.load_run(run)
          if not loaded:
              self.warn(f"Failed to load run '{run.workdir}': {error}")
              return False
          # Clean guests if provision is done but finish is not done
          successful = True
          for plan in run.plans:
              if plan.provision.status() == 'done':
                  if plan.finish.status() != 'done':
                      # Wake up provision to load the active guests
                      plan.provision.wake()
                      if not self._matches_how(plan):
                          continue
                      if self.opt('dry'):
                          self.verbose(
                              f"Would stop guests in run '{run.workdir}'"
                              f" plan '{plan.name}'.", shift=1)
                      else:
                          self.verbose(f"Stopping guests in run '{run.workdir}' "
                                       f"plan '{plan.name}'.", shift=1)
                          # Set --quiet to avoid finish logging to terminal
                          quiet = self._context.params['quiet']
                          self._context.params['quiet'] = True
                          try:
                              plan.finish.go()
                          except tmt.utils.GeneralError as error:
                              self.warn(f"Could not stop guest in run "
                                        f"'{run.workdir}': {error}.", shift=1)
                              successful = False
                          finally:
                              self._context.params['quiet'] = quiet
          return successful
  
      def guests(self):
          """ Clean guests of runs """
          self.info('guests', color='blue')
          root_path = self.opt('workdir-root')
          id_ = self.opt('id_')
          if self.opt('last'):
              # Pass the context containing --last to Run to choose
              # the correct one.
              return self._stop_running_guests(Run(context=self._context))
          successful = True
          for abs_path in tmt.utils.generate_runs(root_path, id_):
              run = Run(abs_path, self._context.obj.tree, self._context)
              if not self._stop_running_guests(run):
                  successful = False
          return successful
  
      def _clean_workdir(self, path):
          """ Remove a workdir (unless in dry mode) """
          if self.opt('dry'):
              self.verbose(f"Would remove workdir '{path}'.", shift=1)
          else:
              self.verbose(f"Removing workdir '{path}'.", shift=1)
              try:
                  shutil.rmtree(path)
              except OSError as error:
                  self.warn(f"Failed to remove '{path}': {error}.", shift=1)
                  return False
          return True
  
      def runs(self):
          """ Clean workdirs of runs """
          self.info('runs', color='blue')
          root_path = self.opt('workdir-root')
          id_ = self.opt('id_')
          if self.opt('last'):
              # Pass the context containing --last to Run to choose
              # the correct one.
              last_run = Run(context=self._context)
              last_run._workdir_load(last_run._workdir_path)
              return self._clean_workdir(last_run.workdir)
          all_workdirs = [path for path in tmt.utils.generate_runs(root_path, id_)]
          keep = self.opt('keep')
          if keep is not None:
              # Sort by modify time of the workdirs and keep the newest workdirs
              all_workdirs.sort(key=lambda workdir: os.path.getmtime(
                  os.path.join(workdir, 'run.yaml')), reverse=True)
              all_workdirs = all_workdirs[keep:]
  
          successful = True
          for workdir in all_workdirs:
              if not self._clean_workdir(workdir):
                  successful = False
  
          return successful
  
  
  class Result:
      """
      Test result
  
      The following keys are expected in the 'data' dictionary::
  
          result ........... test execution result
          log .............. one or more log files
          note ............. additional result details
          duration ......... test execution time (hh:mm:ss)
  
      Required parameter 'test' or 'name' should contain a test reference.
      """
  
      _results = {
          'pass': 'green',
          'fail': 'red',
          'info': 'blue',
          'warn': 'yellow',
          'error': 'magenta',
          }
  
      def __init__(self, data, name=None, test=None):
          """ Initialize test result data """
  
          # Save the test name and optional note
          if not test and not name:
              raise tmt.utils.SpecificationError(
                  "Either name or test have to be specified")
          if test and not isinstance(test, Test):
              raise tmt.utils.SpecificationError(f"Invalid test '{test}'.")
          if name and not isinstance(name, str):
              raise tmt.utils.SpecificationError(f"Invalid test name '{name}'.")
          self.name = name or test.name
          self.note = data.get('note')
          self.duration = data.get('duration')
          if test:
              # Saving identifiable information for each test case so we can match them
              # to Polarion/Nitrate/other cases and report run results there
              self.ids = {tmt.identifier.ID_KEY: test.id}
              for key in EXTRA_RESULT_IDENTIFICATION_KEYS:
                  self.ids[key] = test.node.get(key)
              interpret = test.result or 'respect'
          else:
              try:
                  self.ids = data['ids']
              except KeyError:
                  self.ids = {}
              interpret = 'respect'
  
          # Check for valid results
          try:
              self.result = data['result']
          except KeyError:
              raise tmt.utils.SpecificationError("Missing test result.")
          if self.result not in self._results:
              raise tmt.utils.SpecificationError(
                  f"Invalid result '{self.result}'.")
  
          # Convert log into list if necessary
          try:
              self.log = tmt.utils.listify(data['log'])
          except KeyError:
              self.log = []
  
          # Handle alternative result interpretation
          if interpret not in ('respect', 'custom'):
              # Extend existing note or set a new one
              if self.note and isinstance(self.note, str):
                  self.note += f', original result: {self.result}'
              elif self.note is None:
                  self.note = f'original result: {self.result}'
              else:
                  raise tmt.utils.SpecificationError(
                      f"Test result note '{self.note}' must be a string.")
  
              if interpret == "xfail":
                  # Swap just fail<-->pass, keep the rest as is (info, warn,
                  # error)
                  self.result = {
                      'fail': 'pass', 'pass': 'fail'}.get(
                      self.result, self.result)
              elif interpret in self._results:
                  self.result = interpret
              else:
                  raise tmt.utils.SpecificationError(
                      f"Invalid result '{interpret}' in test '{self.name}'.")
  
      @staticmethod
      def total(results):
          """ Return dictionary with total stats for given results """
          stats = dict([(result, 0) for result in Result._results])
          for result in results:
              stats[result.result] += 1
          return stats
  
      @staticmethod
      def summary(results):
          """ Prepare a nice human summary of provided results """
          stats = Result.total(results)
          comments = []
          if stats.get('pass'):
              passed = ' ' + click.style('passed', fg='green')
              comments.append(fmf.utils.listed(stats['pass'], 'test') + passed)
          if stats.get('fail'):
              failed = ' ' + click.style('failed', fg='red')
              comments.append(fmf.utils.listed(stats['fail'], 'test') + failed)
          if stats.get('info'):
              count, comment = fmf.utils.listed(stats['info'], 'info').split()
              comments.append(count + ' ' + click.style(comment, fg='blue'))
          if stats.get('warn'):
              count, comment = fmf.utils.listed(stats['warn'], 'warn').split()
              comments.append(count + ' ' + click.style(comment, fg='yellow'))
          if stats.get('error'):
              count, comment = fmf.utils.listed(stats['error'], 'error').split()
              comments.append(count + ' ' + click.style(comment, fg='magenta'))
          return fmf.utils.listed(comments or ['no results found'])
  
      def show(self):
          """ Return a nicely colored result with test name (and note) """
          result = 'errr' if self.result == 'error' else self.result
          colored = style(result, fg=self._results[self.result])
          note = f" ({self.note})" if self.note else ''
          return f"{colored} {self.name}{note}"
  
      def export(self):
          """ Save result data for future wake-up """
          data = dict(result=self.result, log=self.log)
          if self.note:
              data['note'] = self.note
          if self.duration:
              data['duration'] = self.duration
          if self.ids:
              data['ids'] = self.ids
          return data
  
  
  @dataclasses.dataclass
  class LinkNeedle:
      """
      A container to use for searching links.
  
      ``relation`` and ``target`` fields hold regular expressions that
      are to be searched for in the corresponding fields of :py:class:`Link`
      instances.
      """
  
      relation: str = r'.*'
      target: str = r'.*'
  
      @classmethod
      def from_raw(cls, value: str) -> 'LinkNeedle':
          """
          Create a ``LinkNeedle`` instance from its specification.
  
          Specification is described in [1], this constructor takes care
          of parsing it into a corresponding ``LinkNeedle`` instance.
  
          [1] https://tmt.readthedocs.io/en/stable/spec/plans.html#fmf
          """
  
          parts = value.split(':', maxsplit=1)
  
          if len(parts) == 1:
              return LinkNeedle(target=parts[0])
  
          return LinkNeedle(relation=parts[0], target=parts[1])
  
      def __str__(self) -> str:
          return f'{self.relation}:{self.target}'
  
      def matches(self, link: 'Link') -> bool:
          """ Find out whether a given link matches this needle """
  
          # Rule out the simple case, mismatching relation.
          if not re.search(self.relation, link.relation):
              return False
  
          # If the target is a string, the test is trivial.
          if isinstance(link.target, str):
              return re.search(self.target, link.target) is not None
  
          # If the target is an fmf id, the current basic implementation will
          # check just the `name` key, if it's defined. More fields may come
          # later, pending support for more sophisticated parsing of link
          # needle on a command line.
          if link.target.name:
              return re.search(self.target, link.target.name) is not None
  
          return False
  
  
  @dataclasses.dataclass
  class Link:
      """
      An internal "link" as defined by tmt specification.
  
      All links, after entering tmt internals, are converted from their raw
      representation into instances of this class.
  
      [1] https://tmt.readthedocs.io/en/stable/spec/core.html#link
      """
  
      DEFAULT_RELATIONSHIP: ClassVar[_RawLinkRelationName] = 'relates'
  
      relation: _RawLinkRelationName
      target: Union[str, FmfId]
      note: Optional[str] = None
  
      @classmethod
      def from_raw(cls, spec: _RawLink) -> 'Link':
          """
          Create a ``Link`` instance from its specification.
  
          Specification is described in [1], this constructor takes care
          of parsing it into a corresponding ``Link`` instance.
  
          [1] https://tmt.readthedocs.io/en/stable/spec/core.html#link
          """
  
          # `spec` can be either a string, fmf id, or relation:target mapping with
          # a single key (modulo `note` key, of course).
  
          # String is simple: if `spec` is a string, it represents a target,
          # and we use the default relationship.
          if isinstance(spec, str):
              return Link(relation=Link.DEFAULT_RELATIONSHIP, target=spec)
  
          # From now on, `spec` is a mapping, and may contain the optional
          # `note` key. Extract the key for later.
          note = cast(Optional[str], spec.get('note', None))
  
          # Count how many relations are stored in spec.
          relations = [
              cast(_RawLinkRelationName, key) for key in spec if key not in (FmfId.keys + ['note'])
              ]
  
          # If there are no relations, spec must be an fmf id, representing
          # a target.
          if len(relations) == 0:
              return Link(
                  relation=Link.DEFAULT_RELATIONSHIP,
                  target=FmfId.from_dict(cast(_RawFmfId, spec)),
                  note=note)
  
          # More relations than 1 are a hard error, only 1 is allowed.
          if len(relations) > 1:
              raise tmt.utils.SpecificationError(
                  f"Multiple relations specified for the link "
                  f"({fmf.utils.listed(relations)}).")
  
          # At this point, we know there's just a single relation, its value is the target,
          # and note we already put aside.
          #
          # NOTE: as far as mypy knows, we did not narrow the type of `spec`, _RawFmfId
          # is still in play - but we do know it's no longer possible because such a
          # value we ruled out thanks to `"no relations" check above. At this point,
          # the right side of relation must be _RawLinkTarget and nothing else. Helping
          # mypy to realize that.
          relation = relations[0]
          raw_target = cast(_RawLinkTarget, spec[relation])
  
          # TODO: this should not happen with mandatory validation
          if relation not in Links._relations:
              raise tmt.utils.SpecificationError(
                  f"Invalid link relation '{relation}' (should be "
                  f"{fmf.utils.listed(Links._relations, join='or')}).")
  
          if isinstance(raw_target, str):
              return Link(relation=relation, target=raw_target, note=note)
  
          return Link(relation=relation, target=FmfId.from_dict(raw_target), note=note)
  
      def to_raw(self) -> _RawLinkRelation:
          """
          Convert this link into a corresponding link specification.
  
          No matter what the original specification was, every link will
          generate the very same type of specification, the ``relation: target``
          one.
  
          Output of this method is fully compatible with specification, and when
          given to :py:meth:`from_raw`, it shall create a ``Link`` instance
          with the same properties as the original one.
  
          [1] https://tmt.readthedocs.io/en/stable/spec/core.html#link
          """
  
          spec = {
              self.relation: self.target.to_dict() if isinstance(
                  self.target,
                  FmfId) else self.target}
  
          if self.note is not None:
              spec['note'] = self.note
  
          return spec
  
  
  class Links:
      """
      Collection of links in tests, plans and stories.
  
      Provides abstraction over the whole collection of object's links.
  
      [1] https://tmt.readthedocs.io/en/stable/spec/core.html#link
      """
  
      # The list of all supported link relations
      _relations: List[_RawLinkRelationName] = [
          'verifies', 'verified-by',
          'implements', 'implemented-by',
          'documents', 'documented-by',
          'blocks', 'blocked-by',
          'duplicates', 'duplicated-by',
          'parent', 'child',
          'relates',
          ]
  
      _links: List[Link]
  
      def __init__(self, data: Optional[_RawLinks] = None):
          """ Create a collection from raw link data """
  
          # TODO: this should not happen with mandatory validation
          if data is not None and not isinstance(data, (str, dict, list)):
              raise tmt.utils.SpecificationError(
                  "Invalid link specification "
                  "(should be a string, fmf id or list of their combinations), "
                  f"got '{type(data).__name__}'.")
  
          # Nothing to do if no data provided
          if data is None:
              self._links = []
  
              return
  
          specs = data if isinstance(data, list) else [data]
  
          # Ensure that each link is in the canonical form
          self._links = [Link.from_raw(spec) for spec in specs]
  
      def to_raw(self) -> List[_RawLinkRelation]:
          """
          Convert this collection of links into a corresponding specification.
  
          No matter what the original specification was, every link will
          generate the very same type of specification, the ``relation: target``
          one.
  
          Output of this method is fully compatible with specification, and when
          used to instantiate :py:meth:`Link` object, it shall create a collection
          of links with the same properties as the original one.
  
          [1] https://tmt.readthedocs.io/en/stable/spec/core.html#link
          """
  
          return [
              link.to_raw()
              for link in self._links
              ]
  
      def get(self, relation: Optional[_RawLinkRelationName] = None) -> List[Link]:
          """ Get links with given relation, all by default """
          return [
              link for link in self._links
              if relation is None or link.relation == relation]
  
      def show(self) -> None:
          """ Format a list of links with their relations """
          for link in self._links:
              # TODO: needs a format for fmf id target
              echo(tmt.utils.format(link.relation.rstrip('-by'), f"{link.target}", key_color='cyan'))
  
      def has_link(self, needle: LinkNeedle) -> bool:
          """ Check whether this set of links contains a matching link """
  
          return any(needle.matches(link) for link in self._links)
  
files: tmt/base.py
files: 
  """ Step Classes """
  
  import dataclasses
  import re
  import sys
  import textwrap
  import warnings
  from typing import (TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Type,
                      TypeVar, Union, cast, overload)
  
  if sys.version_info >= (3, 8):
      from typing import TypedDict
  else:
      from typing_extensions import TypedDict
  
  import click
  from click import echo
  
  import tmt.options
  import tmt.utils
  from tmt.options import show_step_method_hints
  
  if TYPE_CHECKING:
      import tmt.base
      from tmt.base import Plan
      from tmt.steps.provision import Guest
  
  
  DEFAULT_PLUGIN_METHOD_ORDER: int = 50
  
  
  # Supported steps and actions
  STEPS = ['discover', 'provision', 'prepare', 'execute', 'report', 'finish']
  ACTIONS = ['login', 'reboot']
  
  # Step phase order
  PHASE_START = 10
  PHASE_BASE = 50
  PHASE_END = 90
  
  
  class Phase(tmt.utils.Common):
      """ A phase of a step """
  
      def __init__(
              self,
              order: int = tmt.utils.DEFAULT_PLUGIN_ORDER,
              *args: Any,
              **kwargs: Any):
          super().__init__(*args, **kwargs)
          self.order: int = order
  
      def enabled_on_guest(self, guest: 'Guest') -> bool:
          """ Phases are enabled across all guests by default """
          return True
  
      @property
      def is_in_standalone_mode(self) -> bool:
          """
          True if the phase is in stand-alone mode.
  
          Stand-alone mode means that only this phase should be run as a part
          of the run (and not any other even if requested using --all).
          This is useful as some plugin options may completely change its
          behaviour from the regular behaviour based on options
          (e.g. listing images inside a provision plugin).
          """
          return False
  
  
  # A variable used to describe a generic type for all classes derived from Phase
  PhaseT = TypeVar('PhaseT', bound=Phase)
  
  
  _RawStepData = TypedDict('_RawStepData', {
      'how': str,
      'name': str
  }, total=False)
  
  RawStepDataArgument = Union[_RawStepData, List[_RawStepData]]
  
  
  T = TypeVar('T', bound='StepData')
  
  
  @dataclasses.dataclass
  class StepData(
          tmt.utils.SpecBasedContainer,
          tmt.utils.NormalizeKeysMixin,
          tmt.utils.SerializableContainer):
      """
      Keys necessary to describe, create, save and restore a step.
  
      Very basic set of keys shared across all steps.
  
      Provides basic functionality for transition between "raw" step data, which
      consists of fmf nodes and CLI options, and this container representation with
      keys and types more suitable for internal use.
  
      Implementation expects simple 1:1 relation between ``StepData`` attributes - keys -
      and their fmf/CLI sources, where keys replace options' dashes (``-``) with
      underscores (``_``). For example, to hold value of an fmf key ``foo-bar`` - or
      value of a corresponding CLI option, ``--foo-bar``, a step data class should
      declare key named ``foo_data``. All ``StepData`` methods would honor this mapping.
      """
  
      # TODO: we can easily add lists of keys for various verbosity levels...
      KEYS_SHOW_ORDER = ['name', 'how']
  
      name: str
      how: str
      order: int = tmt.utils.DEFAULT_PLUGIN_ORDER
      summary: Optional[str] = None
  
      def to_spec(self) -> _RawStepData:  # type: ignore[override]
          """ Convert to a form suitable for saving in a specification file """
  
          return cast(_RawStepData, {
              tmt.utils.key_to_option(key): value
              for key, value in self.items()
              })
  
      @classmethod
      def pre_normalization(cls, raw_data: _RawStepData, logger: tmt.utils.Common) -> None:
          """ Called before normalization, useful for tweaking raw data """
  
          logger.debug(f'{cls.__name__}: original raw data', str(raw_data), level=4)
  
      def post_normalization(self, raw_data: _RawStepData, logger: tmt.utils.Common) -> None:
          """ Called after normalization, useful for tweaking normalized data """
  
          pass
  
      @classmethod
      def from_spec(  # type: ignore[override]
              cls: Type[T],
              raw_data: _RawStepData,
              logger: tmt.utils.Common
              ) -> T:
          """ Convert from a specification file or from a CLI option """
  
          cls.pre_normalization(raw_data, logger)
  
          data = cls(name=raw_data['name'], how=raw_data['how'])
          data._load_keys(cast(Dict[str, Any], raw_data), cls.__name__, logger)
  
          data.post_normalization(raw_data, logger)
  
          return data
  
  
  @dataclasses.dataclass
  class WhereableStepData:
      """
      Keys shared by step data that may be limited to a particular guest.
  
      To be used as a mixin class, adds necessary keys.
  
      See [1] and [2] for specification.
  
      1. https://tmt.readthedocs.io/en/stable/spec/plans.html#where
      2. https://tmt.readthedocs.io/en/stable/spec/plans.html#spec-plans-prepare-where
      """
  
      where: Optional[str] = None
  
  
  class Step(tmt.utils.Common):
      """ Common parent of all test steps """
  
      # Default implementation for all steps is "shell", but some
      # steps like provision may have better defaults for their
      # area of expertise.
      DEFAULT_HOW: str = 'shell'
  
      # Refers to a base class for all plugins registered with this step.
      _plugin_base_class: Type['BasePlugin']
  
      #: Stores the normalized step data. Initialized first time step's `data`
      #: is accessed.
      #
      # The delayed initialization is necessary to support `how` changes via
      # command-line - code instantiating steps must be able to invalidate
      # and replace raw step data entries before they get normalized and become
      # the single source of information for plugins involved.
      _data: List[StepData]
  
      #: Stores the original raw step data. Initialized by :py:meth:`__init__`
      #: or :py:meth:`wake`, and serves as a source for normalization performed
      #: by :py:meth:`_normalize_data`.
      _raw_data: List[_RawStepData]
  
      def __init__(
              self,
              plan: 'Plan',
              data: Optional[RawStepDataArgument] = None,
              name: Optional[str] = None,
              workdir: tmt.utils.WorkdirArgumentType = None) -> None:
          """ Initialize and check the step data """
          super().__init__(name=name, parent=plan, workdir=workdir)
          # Initialize data
          self.plan: 'Plan' = plan
          self._status: Optional[str] = None
          self._phases: List[Phase] = []
  
          # Normalize raw data to be a list of step configuration data, one item per
          # distinct step configuration. Make sure all items have `name`` and `how` keys.
          #
          # NOTE: this is not a normalization step as performed by NormalizeKeysMixin.
          # Here we make sure the raw data can be consumed by the delegation code, we
          # do not modify any existing content of raw data items.
  
          # Create an empty step by default (can be updated from cli)
          if data is None:
              self._raw_data = [{}]
  
          # Convert to list if only a single config provided
          elif isinstance(data, dict):
              self._raw_data = [data]
  
          # List is as good as it gets
          elif isinstance(data, list):
              self._raw_data = data
  
          # Shout about invalid configuration
          else:
              raise tmt.utils.GeneralError(
                  f"Invalid '{self}' config in '{self.plan}'.")
  
          for i, raw_datum in enumerate(self._raw_data):
              # Add default unique names even to multiple configs so that the users
              # don't need to specify it if they don't care about the name
              if raw_datum.get('name', None) is None:
                  raw_datum['name'] = f'{tmt.utils.DEFAULT_NAME}-{i}'
  
              # Set 'how' to the default if not specified
              if raw_datum.get('how', None) is None:
                  raw_datum['how'] = self.DEFAULT_HOW
  
      def _normalize_data(self, raw_data: List[_RawStepData]) -> List[StepData]:
          """
          Normalize step data entries.
  
          Every entry of ``raw_data`` is converted into an instance of
          :py:class:`StepData` or one of its subclasses. Particular class
          is derived from a plugin identified by raw data's ``how`` field
          and step's plugin registry.
          """
  
          data: List[StepData] = []
  
          for raw_datum in raw_data:
              plugin = self._plugin_base_class.delegate(self, raw_data=raw_datum)
  
              data.append(plugin.data)
  
          return data
  
      @property
      def data(self) -> List[StepData]:
          if not hasattr(self, '_data'):
              self._data = self._normalize_data(self._raw_data)
  
          return self._data
  
      @data.setter
      def data(self, data: List[StepData]) -> None:
          self._data = data
  
      @property
      def enabled(self) -> Optional[bool]:
          """ True if the step is enabled """
          try:
              return self.name in self.plan.my_run._context.obj.steps
          except AttributeError:
              return None
  
      @property
      def plugins_in_standalone_mode(self) -> int:
          """
          The number of plugins in standalone mode.
  
          Stand-alone mode means that only this step should be run as a part
          of the run (and not any other even if requested using --all).
          This is useful as some step options may completely change its
          behaviour from the regular behaviour based on options
          (e.g. listing images inside provision).
          """
          return sum(phase.is_in_standalone_mode for phase in self.phases())
  
      @classmethod
      def usage(cls, method_overview: str) -> str:
          """ Prepare general usage message for the step """
          # Main description comes from the class docstring
          if cls.__name__ is None:
              raise tmt.utils.GeneralError("Missing name of the step.")
  
          if cls.__doc__ is None:
              raise tmt.utils.GeneralError(
                  f"Missing docstring of the step {cls.__name__.lower()}.")
  
          usage = textwrap.dedent(cls.__doc__)
          # Append the list of supported methods
          usage += '\n\n' + method_overview
          # Give a hint about detailed help
          name = cls.__name__.lower()
          usage += (
              f"\n\nUse 'tmt run {name} --help --how <method>' to learn more "
              f"about given {name} method and all its supported options.")
          return usage
  
      def status(self, status: Optional[str] = None) -> Optional[str]:
          """
          Get and set current step status
  
          The meaning of the status is as follows:
          todo ... config, data and command line processed (we know what to do)
          done ... the final result of the step stored to workdir (we are done)
          """
          # Update status
          if status is not None:
              # Check for valid values
              if status not in ['todo', 'done']:
                  raise tmt.utils.GeneralError(f"Invalid status '{status}'.")
              # Show status only if changed
              elif self._status != status:
                  self._status = status
                  self.debug('status', status, color='yellow', level=2)
          # Return status
          return self._status
  
      def load(self) -> None:
          """ Load status and step data from the workdir """
          try:
              raw_step_data: Dict[Any, Any] = tmt.utils.yaml_to_dict(self.read('step.yaml'))
              self.debug('Successfully loaded step data.', level=2)
  
              self.data = [
                  StepData.unserialize(raw_datum) for raw_datum in raw_step_data['data']
                  ]
              self.status(raw_step_data['status'])
          except tmt.utils.GeneralError:
              self.debug('Step data not found.', level=2)
  
      def save(self) -> None:
          """ Save status and step data to the workdir """
          content: Dict[str, Any] = {
              'status': self.status(),
              'data': [datum.to_serialized() for datum in self.data]
              }
          self.write('step.yaml', tmt.utils.dict_to_yaml(content))
  
      def wake(self) -> None:
          """ Wake up the step (process workdir and command line) """
          # Cleanup possible old workdir if called with --force
          if self.opt('force'):
              self._workdir_cleanup()
  
          # Load stored data
          self.load()
  
          # Status 'todo' means the step has not finished successfully.
          # Probably interrupted in the middle. Clean up the work
          # directory to give it another chance with a fresh start.
          if self.status() == 'todo':
              self.debug("Step has not finished. Let's try once more!", level=2)
              self._workdir_cleanup()
  
          # Importing here to avoid circular imports
          import tmt.steps.report
  
          # Special handling for the report step to always enable force mode in
          # order to cover a very frequent use case 'tmt run --last report'
          # FIXME find a better way how to enable always-force per plugin
          if (isinstance(self, tmt.steps.report.Report) and
                  self.data[0].how in ['display', 'html']):
              self.debug("Report step always force mode enabled.")
              self._workdir_cleanup()
              self.status('todo')
  
          # Nothing more to do when the step is already done
          if self.status() == 'done':
              self.debug('Step is done, not touching its data.')
              return
  
          # Override step data with command line options
          how: str = self.opt('how')
          if how is not None:
              # If 'how' has been given, when it comes to current entries in `self.data`,
              # there are two options:
              #
              # * entry's `how` is the same as the one given via command-line. Then we can
              # keep step data we already have.
              # * entry's `how` is different, and then we need to throw the entry away and
              # replace it with new `how`.
              #
              # To handle both variants, we replace `self.data` with new set of entries,
              # based on newly constructed set of raw data.
              self.debug(f'CLI-provided how={how} overrides all existing step data', level=4)
  
              _raw_data: List[_RawStepData] = []
  
              for datum in self.data:
                  # We can re-use this one - to make handling easier, just dump it to "raw"
                  # form for _normalize_data().
                  if datum.how == how:
                      self.debug(f'  compatible:   {datum}', level=4)
                      _raw_data.append(datum.to_spec())
  
                  # Mismatch, throwing away, replacing with new `how` - but we can keep the name.
                  else:
                      self.debug(f'  incompatible: {datum}', level=4)
                      _raw_data.append({
                          'name': datum.name,
                          'how': how
                          })
  
              self.data = self._normalize_data(_raw_data)
              self._raw_data = _raw_data
  
              self.debug('updated data', str(self.data), level=4)
  
          else:
              self.debug('CLI did not change existing step data', level=4)
  
      def setup_actions(self) -> None:
          """ Insert login and reboot plugins if requested """
          for login_plugin in Login.plugins(step=self):
              self.debug(
                  f"Insert a login plugin into the '{self}' step "
                  f"with order '{login_plugin.order}'.", level=2)
              self._phases.append(login_plugin)
  
          for reboot_plugin in Reboot.plugins(step=self):
              self.debug(
                  f"Insert a reboot plugin into the '{self}' step "
                  f"with order '{reboot_plugin.order}'.", level=2)
              self._phases.append(reboot_plugin)
  
      @overload
      def phases(self, classes: None = None) -> List[Phase]:
          pass
  
      @overload
      def phases(self, classes: Type[PhaseT]) -> List[PhaseT]:
          pass
  
      @overload
      def phases(self, classes: Tuple[Type[PhaseT], ...]) -> List[PhaseT]:
          pass
  
      def phases(self, classes: Optional[Union[Type[PhaseT],
                 Tuple[Type[PhaseT], ...]]] = None) -> List[PhaseT]:
          """
          Iterate over phases by their order
  
          By default iterates over all available phases. Optional filter
          'classes' can be used to iterate only over instances of given
          class (single class or tuple of classes).
          """
  
          if classes is None:
              _classes: Tuple[Union[Type[Phase], Type[PhaseT]], ...] = (Phase,)
  
          elif not isinstance(classes, tuple):
              _classes = (classes,)
  
          else:
              _classes = classes
  
          return sorted(
              [cast(PhaseT, phase) for phase in self._phases if isinstance(phase, _classes)],
              key=lambda phase: phase.order)
  
      def actions(self) -> None:
          """ Run all loaded Login or Reboot action instances of the step """
          for phase in self.phases(classes=Action):
              phase.go()
  
      def go(self) -> None:
          """ Execute the test step """
          # Show step header and how
          self.info(self.name, color='blue')
          # Show workdir in verbose mode
          self.debug('workdir', self.workdir, 'magenta')
  
  
  class Method:
      """ Step implementation method """
  
      def __init__(
              self,
              name: str,
              class_: Optional[Type['BasePlugin']] = None,
              doc: Optional[str] = None,
              order: int = DEFAULT_PLUGIN_METHOD_ORDER
              ) -> None:
          """ Store method data """
  
          doc = (doc or getattr(class_, '__doc__') or '').strip()
  
          if not doc:
              if class_:
                  raise tmt.utils.GeneralError(f"Plugin class '{class_}' provides no docstring.")
  
              raise tmt.utils.GeneralError(f"Plugin method '{name}' provides no docstring.")
  
          self.name = name
          self.class_ = class_
          self.doc = doc
          self.order = order
  
          # Parse summary and description from provided doc string
          lines: List[str] = [re.sub('^    ', '', line)
                              for line in self.doc.split('\n')]
          self.summary: str = lines[0].strip()
          self.description: str = '\n'.join(lines[1:]).lstrip()
  
      def describe(self) -> str:
          """ Format name and summary for a nice method overview """
          return f'{self.name} '.ljust(22, '.') + f' {self.summary}'
  
      def usage(self) -> str:
          """ Prepare a detailed usage from summary and description """
          if self.description:
              usage: str = self.summary + '\n\n' + self.description
          else:
              usage = self.summary
          # Disable wrapping for all paragraphs
          return re.sub('\n\n', '\n\n\b\n', usage)
  
  
  def provides_method(
          name: str,
          doc: Optional[str] = None,
          order: int = DEFAULT_PLUGIN_METHOD_ORDER) -> Any:
      """
      A plugin class decorator to register plugin's method with tmt steps.
  
      In the following example, developer marks ``SomePlugin`` as providing two discover methods,
      ``foo`` and ``bar``, with ``bar`` being sorted to later position among methods:
  
      .. code-block:: python
  
         @tmt.steps.provides_method('foo')
         @tmt.steps.provides_method('bar', order=80)
         class SomePlugin(tmt.steps.discover.DicoverPlugin):
           ...
  
      :param name: name of the method.
      :param doc: method documentation. If not specified, docstring of the decorated class is used.
      :param order: order of the method among other step methods.
      """
  
      def _method(cls: Type['BasePlugin']) -> Any:
          plugin_method = Method(name, class_=cls, doc=doc, order=order)
  
          cast('BasePlugin', cls.__bases__[0])._supported_methods.append(plugin_method)
  
          return cls
  
      return _method
  
  
  class PluginIndex(type):
      """
      Plugin metaclass used to register all available plugins.
  
      .. deprecated:: 1.17
         Instead of declaring `_methods` list, use :py:func:`provides_method` decorator
         to announce provided methods.
      """
  
      def __init__(
              cls,
              name: str,
              bases: List['BasePlugin'],
              attributes: Any) -> None:
          """ Store all defined methods in the parent class """
  
          plugin_methods = cast(Optional[List[Method]], getattr(cls, '_methods'))
  
          if not plugin_methods:
              return
  
          warnings.warn(
              'Use of _methods to declare plugin methods is deprecated.'
              'Use tmt.steps.provides_method decorator instead.',
              DeprecationWarning,
              stacklevel=2)
  
          for plugin_method in plugin_methods:
              plugin_method.class_ = cast(Type['BasePlugin'], cls)
              # Add to the list of supported methods in parent class
              bases[0]._supported_methods.append(plugin_method)
  
  
  class BasePlugin(Phase, metaclass=PluginIndex):
      """ Common parent of all step plugins """
  
      # Deprecated, use @provides_method(...) instead. left for backward
      # compatibility with out-of-tree plugins.
      _methods: List[Method] = []
  
      # Default implementation for all steps is shell
      # except for provision (virtual) and report (display)
      how: str = 'shell'
  
      # List of all supported methods aggregated from all plugins of the same step.
      _supported_methods: List[Method] = []
  
      _data_class: Type[StepData] = StepData
      data: StepData
  
      # TODO: do we need this list? Can whatever code is using it use _data_class directly?
      # List of supported keys
      # (used for import/export to/from attributes during load and save)
      @property
      def _keys(self) -> List[str]:
          return list(self._data_class.keys())
  
      def __init__(
              self,
              step: Step,
              data: StepData,
              workdir: tmt.utils.WorkdirArgumentType = None) -> None:
          """ Store plugin name, data and parent step """
          # Store name, data and parent step
          super().__init__(
              parent=step,
              name=data.name,
              workdir=workdir,
              order=data.order)
          # It is not possible to use TypedDict here because
          # all keys are not known at the time of the class definition
          self.data = data
          self.step = step
  
      @classmethod
      def base_command(
              cls,
              usage: str,
              method_class: Optional[Type[click.Command]] = None) -> click.Command:
          """ Create base click command (common for all step plugins) """
          raise NotImplementedError
  
      @classmethod
      def options(cls, how: Optional[str] = None) -> List[tmt.options.ClickOptionDecoratorType]:
          """ Prepare command line options for given method """
          # Include common options supported across all plugins
          return tmt.options.verbose_debug_quiet + tmt.options.force_dry
  
      @classmethod
      def command(cls) -> click.Command:
          """ Prepare click command for all supported methods """
          # Create one command for each supported method
          commands: Dict[str, click.Command] = {}
          method_overview: str = f'Supported methods ({cls.how} by default):\n\n\b'
          for method in cls.methods():
              assert method.class_ is not None
              method_overview += f'\n{method.describe()}'
              command: click.Command = cls.base_command(usage=method.usage())
              # Apply plugin specific options
              for option in method.class_.options(method.name):
                  command = option(command)
              commands[method.name] = command
  
          # Create base command with common options using method class
          method_class = tmt.options.create_method_class(commands)
          command = cls.base_command(usage=method_overview, method_class=method_class)
          # Apply common options
          for option in cls.options():
              command = option(command)
          return command
  
      @classmethod
      def methods(cls) -> List[Method]:
          """ Return all supported methods ordered by priority """
          return sorted(cls._supported_methods, key=lambda method: method.order)
  
      @classmethod
      def delegate(
              cls,
              step: Step,
              data: Optional[StepData] = None,
              raw_data: Optional[_RawStepData] = None) -> 'BasePlugin':
          """
          Return plugin instance implementing the data['how'] method
  
          Supports searching by method prefix as well (e.g. 'virtual').
          The first matching method with the lowest 'order' wins.
          """
  
          if data is not None:
              how = data.how
          elif raw_data is not None:
              how = raw_data['how']
          else:
              raise tmt.utils.GeneralError('Either data or raw data must be given.')
  
          step.debug(
              f'{cls.__name__}.delegate(step={step}, data={data}, raw_data={raw_data})',
              level=3)
  
          # Filter matching methods, pick the one with the lowest order
          for method in cls.methods():
              assert method.class_ is not None
              if method.name.startswith(how):
                  step.debug(
                      f"Using the '{method.class_.__name__}' plugin "
                      f"for the '{how}' method.", level=2)
  
                  plugin_class = method.class_
                  plugin_data_class = plugin_class._data_class
  
                  # If we're given raw data, construct a step data instance, applying
                  # normalization in the process.
                  if raw_data is not None:
                      try:
                          data = plugin_data_class.from_spec(raw_data, step)
  
                      except Exception as exc:
                          raise tmt.utils.GeneralError(
                              f'Failed to load step data for {plugin_data_class.__name__}: {exc}',
                              original=exc)
  
                  assert data is not None
                  assert data.__class__ is plugin_data_class, \
                      f'Data package is instance of {data.__class__.__name__}, ' \
                      f'plugin {plugin_class.__name__} ' \
                      f'expects {plugin_data_class.__name__}'
  
                  plugin = plugin_class(step, data)
                  assert isinstance(plugin, BasePlugin)
                  return plugin
  
          show_step_method_hints(step, step.name, how)
          # Report invalid method
          if step.plan is None:
              raise tmt.utils.GeneralError(f"Plan for {step.name} is not set.")
          raise tmt.utils.SpecificationError(
              f"Unsupported {step.name} method '{how}' "
              f"in the '{step.plan.name}' plan.")
  
      def default(self, option: str, default: Optional[Any] = None) -> Any:
          """ Return default data for given option """
  
          value = self._data_class.default(tmt.utils.option_to_key(option), default=default)
  
          if value is None:
              return default
  
          return value
  
      def get(self, option: str, default: Optional[Any] = None) -> Any:
          """ Get option from plugin data, user/system config or defaults """
  
          # Check plugin data first
          #
          # Since self.data is a dataclass instance, the option would probably exist.
          # As long as there's no typo in name, it would be defined. Which complicates
          # the handling of "default" as in "return *this* when attribute is unset".
          key = tmt.utils.option_to_key(option)
  
          try:
              value = getattr(self.data, key)
  
              # If the value is no longer the default one, return the value. If it
              # still matches the default value, instead of returning the default
              # value right away, call `self.default()` so the plugin has chance to
              # catch calls for computed or virtual keys, keys that don't exist as
              # atributes of our step data.
              #
              # One way would be to subclass step's base plugin class' step data class
              # (which is a subclass of `StepData` and `SerializedContainer`), and
              # override its `default()` method to handle these keys. But, plugins often
              # are pretty happy with the base data class, many don't need their own
              # step data class, and plugin developer might be forced to create a subclass
              # just for this single method override.
              #
              # Instead, keep plugin's `default()` around - everyone can use it to get
              # default value for a given option/key, and plugins can override it as needed
              # (they will always subclass step's base plugin class anyway!). Default
              # implementation would delegate to step data `default()`, and everyone's
              # happy.
  
              if value != self.data.default(key):
                  return value
  
          except AttributeError:
              pass
  
          return self.default(option, default)
  
      def show(self, keys: Optional[List[str]] = None) -> None:
          """ Show plugin details for given or all available keys """
          # Avoid circular imports
          import tmt.base
  
          # Show empty config with default method only in verbose mode
          if self.data.is_bare and not self.opt('verbose'):
              return
          # Step name (and optional summary)
          echo(tmt.utils.format(
              self.step.name, self.get('summary') or '',
              key_color='blue', value_color='blue'))
          # Show all or requested step attributes
          if keys is None:
              keys = list(set(self.data.keys()))
  
          def _emit_key(key: str) -> None:
              # Skip showing the default name
              if key == 'name' and self.name.startswith(tmt.utils.DEFAULT_NAME):
                  return
  
              # Skip showing summary again
              if key == 'summary':
                  return
  
              value = self.get(key)
  
              # No need to show the default order
              if key == 'order' and value == tmt.base.DEFAULT_ORDER:
                  return
  
              if value is None:
                  return
  
              # TODO: hides keys that were used to be in the output...
              # if value == self.data.default(key):
              #     return
  
              echo(tmt.utils.format(tmt.utils.key_to_option(key), value))
  
          # First, follow the order prefered by step data, but emit only the keys
          # that are allowed. Each emitted key would be removed so we wouldn't
          # emit it again when showing the unsorted rest of keys.
          for key in self.data.KEYS_SHOW_ORDER:
              if key not in keys:
                  continue
  
              _emit_key(key)
  
              keys.remove(key)
  
          # Show the rest
          for key in keys:
              _emit_key(key)
  
      def enabled_on_guest(self, guest: 'Guest') -> bool:
          """ Check if the plugin is enabled on the specific guest """
          where: str = self.get('where')
          if not where:
              return True
          return where in (guest.name, guest.role)
  
      def _update_data_from_options(self, keys: List[str]) -> None:
          for key in keys:
              value = self.opt(tmt.utils.key_to_option(key))
              if value:
                  setattr(self.data, key, value)
  
      def wake(self) -> None:
          """
          Wake up the plugin, process data, apply options
  
          Check command line options corresponding to plugin keys
          and store their value into the 'self.data' dictionary if
          their value is True or non-empty.
  
          By default, all supported options corresponding to common
          and plugin-specific keys are processed. List of key names
          in the 'keys' parameter can be used to override only
          selected ones.
          """
  
          assert self.data.__class__ is self._data_class, \
              f'Plugin {self.__class__.__name__} woken with incompatible ' \
              f'data {self.data}, ' \
              f'expects {self._data_class.__name__}'
  
          if self.step.status() == 'done':
              self.debug('step is done, not overwriting plugin data')
              return
  
          # TODO: conflicts with `upgrade` plugin which does this on purpose :/
          # if self.opt('how') is not None:
          #     assert self.opt('how') in [method.name for method in self.methods()], \
          #         f'Plugin {self.__class__.__name__} woken with unsupported ' \
          #         f'how "{self.opt("how")}", ' \
          #         f'supported methods {", ".join([method.name for method in self.methods()])}, ' \
          #         f'current data is {self.data}'
  
          self._update_data_from_options(keys=self._keys)
  
      # NOTE: it's tempting to rename this method to `go()` and use more natural
      # `super().go()` in child classes' `go()` methods. But, `go()` does not have
      # the same signature across all plugin types, therefore we cannot have shared
      # `go()` method in superclass - overriding it in (some) child classes would
      # raise a typing linter error reporting superclass signature differs from the
      # one in a subclass.
      #
      # Therefore we need a different name, and a way how not to forget to call this
      # method from child classes.
      def go_prolog(self) -> None:
          """ Perform actions shared among plugins when beginning their tasks """
          # Show the method
          self.info('how', self.get('how'), 'magenta')
          # Give summary if provided
          if self.get('summary'):
              self.info('summary', self.get('summary'), 'magenta')
          # Show name only if it's not the default one
          if not self.name.startswith(tmt.utils.DEFAULT_NAME):
              self.info('name', self.name, 'magenta')
          # Include order in verbose mode
          self.verbose('order', str(self.order), 'magenta', level=3)
  
      def requires(self) -> List[str]:
          """ List of packages required by the plugin on the guest """
          return []
  
  
  class GuestlessPlugin(BasePlugin):
      """ Common parent of all step plugins that do not work against a particular guest """
  
      def go(self) -> None:
          """ Perform actions shared among plugins when beginning their tasks """
  
          self.go_prolog()
  
  
  class Plugin(BasePlugin):
      """ Common parent of all step plugins that do work against a particular guest """
  
      def go(self, guest: 'Guest') -> None:
          """ Perform actions shared among plugins when beginning their tasks """
  
          self.go_prolog()
  
  
  class Action(Phase):
      """ A special action performed during a normal step. """
  
      # Dictionary containing list of requested phases for each enabled step
      _phases: Optional[Dict[str, List[int]]] = None
  
      @classmethod
      def phases(cls, step: Step) -> List[int]:
          """ Return list of phases enabled for given step """
          # Build the phase list unless done before
          if cls._phases is None:
              cls._phases = cls._parse_phases(step)
          # Return enabled phases, empty list if step not found
          try:
              return cls._phases[step.name]
          except KeyError:
              return []
  
      @classmethod
      def _parse_phases(cls, step: Step) -> Dict[str, List[int]]:
          """ Parse options and store phase order """
          phases = dict()
          options: List[str] = cls._opt('step', default=[])
  
          # Use the end of the last enabled step if no --step given
          if not options:
              login_during: Optional[Step] = None
              # The last run may have failed before all enabled steps were
              # completed, select the last step done
              if step.plan is None:
                  raise tmt.utils.GeneralError(
                      f"Plan for {step.name} is not set.")
              if step.plan.my_run.opt('last'):
                  steps: List[Step] = [
                      s for s in step.plan.steps() if s.status() == 'done']
                  login_during = steps[-1] if steps else None
              # Default to the last enabled step if no completed step found
              if login_during is None:
                  login_during = list(step.plan.steps())[-1]
              # Only login if the error occurred after provision
              if login_during != step.plan.discover:
                  phases[login_during.name] = [PHASE_END]
  
          # Process provided options
          for option in options:
              # Parse the step:phase format
              matched = re.match(r'(\w+)(:(\w+))?', option)
              if matched:
                  step_name, _, phase = matched.groups()
              if not matched or step_name not in STEPS:
                  raise tmt.utils.GeneralError(f"Invalid step '{option}'.")
              # Check phase format, convert into int, use end by default
              try:
                  phase = int(phase)
              except TypeError:
                  phase = PHASE_END
              except ValueError:
                  # Convert 'start' and 'end' aliases
                  try:
                      phase = cast(Dict[str, int],
                                   dict(start=PHASE_START, end=PHASE_END))[phase]
                  except KeyError:
                      raise tmt.utils.GeneralError(f"Invalid phase '{phase}'.")
              # Store the phase for given step
              try:
                  phases[step_name].append(phase)
              except KeyError:
                  phases[step_name] = [phase]
          return phases
  
      def go(self) -> None:
          raise NotImplementedError()
  
  
  class Reboot(Action):
      """ Reboot guest """
  
      # True if reboot enabled
      _enabled: bool = False
  
      def __init__(self, step: Step, order: int) -> None:
          """ Initialize relations, store the reboot order """
          super().__init__(parent=step, name='reboot', order=order)
  
      @classmethod
      def command(
              cls,
              method_class: Optional[Method] = None,
              usage: Optional[str] = None) -> click.Command:
          """ Create the reboot command """
          @click.command()
          @click.pass_context
          @click.option(
              '-s', '--step', metavar='STEP[:PHASE]', multiple=True,
              help='Reboot machine during given phase of selected step(s).')
          @click.option(
              '--hard', is_flag=True,
              help='Hard reboot of the machine. Unsaved data may be lost.')
          def reboot(context: Any, **kwargs: Any) -> None:
              """ Reboot the guest. """
              Reboot._save_context(context)
              Reboot._enabled = True
  
          return reboot
  
      @classmethod
      def plugins(cls, step: Step) -> List['Reboot']:
          """ Return list of reboot instances for given step """
          if not Reboot._enabled:
              return []
          return [Reboot(step, phase) for phase in cls.phases(step)]
  
      def go(self) -> None:
          """ Reboot the guest(s) """
          self.info('reboot', 'Rebooting guest', color='yellow')
          assert isinstance(self.parent, Step)
          assert hasattr(self.parent, 'plan') and self.parent.plan is not None
          for guest in self.parent.plan.provision.guests():
              guest.reboot(hard=self.opt('hard'))
          self.info('reboot', 'Reboot finished', color='yellow')
  
  
  class Login(Action):
      """ Log into the guest """
  
      # True if interactive login enabled
      _enabled: bool = False
  
      def __init__(self, step: Step, order: int):
          """ Initialize relations, store the login order """
          super().__init__(parent=step, name='login', order=order)
  
      @classmethod
      def command(
              cls,
              method_class: Optional[Method] = None,
              usage: Optional[str] = None) -> click.Command:
          """ Create the login command """
          @click.command()
          @click.pass_context
          @click.option(
              '-s', '--step', metavar='STEP[:PHASE]', multiple=True,
              help='Log in during given phase of selected step(s).')
          @click.option(
              '-w', '--when', metavar='RESULT', multiple=True,
              type=click.Choice(tmt.base.Result._results),
              help='Log in if a test finished with given result(s).')
          @click.option(
              '-c', '--command', metavar='COMMAND',
              multiple=True, default=['bash'],
              help="Run given command(s). Default is 'bash'.")
          def login(context: Any, **kwargs: Any) -> None:
              """
              Provide user with an interactive shell on the guest.
  
              By default the shell is provided at the end of the last
              enabled step. When used together with the --last option the
              last completed step is selected. Use one or more --step
              options to select a different step instead.
  
              Optional phase can be provided to specify the exact phase of
              the step when the shell should be provided. The following
              values are supported:
  
              \b
                  start ... beginning of the step (same as '10')
                  end ..... end of the step (default, same as '90')
                  00-99 ... integer order defining the exact phase
  
              Usually the main step execution happens with order 50.
              Consult individual step documentation for more details.
  
              For the execute step and following steps it is also possible
              to conditionally enable the login feature only if some of
              the tests finished with given result (pass, info, fail,
              warn, error).
              """
              Login._save_context(context)
              Login._enabled = True
  
          return login
  
      @classmethod
      def plugins(cls, step: Step) -> List['Login']:
          """ Return list of login instances for given step """
          if not Login._enabled:
              return []
          return [Login(step, phase) for phase in cls.phases(step)]
  
      def go(self) -> None:
          """ Login to the guest(s) """
          # Verify possible test result condition
          count: Dict[str, int] = dict()
          expected_results: Optional[List[str]] = self.opt('when')
  
          assert isinstance(self.parent, Step)
          assert hasattr(self.parent, 'plan') and self.parent.plan is not None
  
          if expected_results:
              for expected_result in expected_results:
                  count[expected_result] = len([
                      result for result in self.parent.plan.execute.results()
                      if result.result == expected_result])
              if not any(count.values()):
                  self.info('Skipping interactive shell', color='yellow')
                  return
  
          # Run the interactive command
          commands: List[str] = self.opt('command')
          self.info('login', 'Starting interactive shell', color='yellow')
          for guest in self.parent.plan.provision.guests():
              # Attempt to push the workdir to the guest
              try:
                  guest.push()
                  cwd: Optional[str] = self.parent.plan.worktree
              except tmt.utils.GeneralError:
                  self.warn("Failed to push workdir to the guest.")
                  cwd = None
              # Execute all requested commands
              for command in commands:
                  self.debug(f"Run '{command}' in interactive mode.")
                  guest.execute(command, interactive=True, cwd=cwd)
          self.info('login', 'Interactive shell finished', color='yellow')
  
files: tmt/steps/__init__.py
files: 
  """ Step Classes """
  
  import dataclasses
  import re
  import sys
  import textwrap
  import warnings
  from typing import (TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Type,
                      TypeVar, Union, cast, overload)
  
  if sys.version_info >= (3, 8):
      from typing import TypedDict
  else:
      from typing_extensions import TypedDict
  
  import click
  from click import echo
  
  import tmt.options
  import tmt.utils
  from tmt.options import show_step_method_hints
  
  if TYPE_CHECKING:
      import tmt.base
      from tmt.base import Plan
      from tmt.steps.provision import Guest
  
  
  DEFAULT_PLUGIN_METHOD_ORDER: int = 50
  
  
  # Supported steps and actions
  STEPS = ['discover', 'provision', 'prepare', 'execute', 'report', 'finish']
  ACTIONS = ['login', 'reboot']
  
  # Step phase order
  PHASE_START = 10
  PHASE_BASE = 50
  PHASE_END = 90
  
  
  class Phase(tmt.utils.Common):
      """ A phase of a step """
  
      def __init__(
              self,
              order: int = tmt.utils.DEFAULT_PLUGIN_ORDER,
              *args: Any,
              **kwargs: Any):
          super().__init__(*args, **kwargs)
          self.order: int = order
  
      def enabled_on_guest(self, guest: 'Guest') -> bool:
          """ Phases are enabled across all guests by default """
          return True
  
      @property
      def is_in_standalone_mode(self) -> bool:
          """
          True if the phase is in stand-alone mode.
  
          Stand-alone mode means that only this phase should be run as a part
          of the run (and not any other even if requested using --all).
          This is useful as some plugin options may completely change its
          behaviour from the regular behaviour based on options
          (e.g. listing images inside a provision plugin).
          """
          return False
  
  
  # A variable used to describe a generic type for all classes derived from Phase
  PhaseT = TypeVar('PhaseT', bound=Phase)
  
  
  _RawStepData = TypedDict('_RawStepData', {
      'how': str,
      'name': str
  }, total=False)
  
  RawStepDataArgument = Union[_RawStepData, List[_RawStepData]]
  
  
  T = TypeVar('T', bound='StepData')
  
  
  @dataclasses.dataclass
  class StepData(tmt.utils.NormalizeKeysMixin, tmt.utils.SerializableContainer):
      """
      Keys necessary to describe, create, save and restore a step.
  
      Very basic set of keys shared across all steps.
  
      Provides basic functionality for transition between "raw" step data, which
      consists of fmf nodes and CLI options, and this container representation with
      keys and types more suitable for internal use.
  
      Implementation expects simple 1:1 relation between ``StepData`` attributes - keys -
      and their fmf/CLI sources, where keys replace options' dashes (``-``) with
      underscores (``_``). For example, to hold value of an fmf key ``foo-bar`` - or
      value of a corresponding CLI option, ``--foo-bar``, a step data class should
      declare key named ``foo_data``. All ``StepData`` methods would honor this mapping.
      """
  
      # TODO: we can easily add lists of keys for various verbosity levels...
      KEYS_SHOW_ORDER = ['name', 'how']
  
      name: str
      how: str
      order: int = tmt.utils.DEFAULT_PLUGIN_ORDER
      summary: Optional[str] = None
  
      def to_raw(self) -> _RawStepData:
          """
          Serialize step data instance to a raw representation.
  
          The returned value can be used to recreate step data when given
          to :py:meth:`from_raw`.
          """
  
          return cast(_RawStepData, {
              tmt.utils.key_to_option(key): value
              for key, value in self.items()
              })
  
      @classmethod
      def pre_normalization(cls, raw_data: _RawStepData, logger: tmt.utils.Common) -> None:
          """ Called before normalization, useful for tweaking raw data """
  
          logger.debug(f'{cls.__name__}: original raw data', str(raw_data), level=4)
  
      def post_normalization(self, raw_data: _RawStepData, logger: tmt.utils.Common) -> None:
          """ Called after normalization, useful for tweaking normalized data """
  
          pass
  
      @classmethod
      def from_raw(cls: Type[T], raw_data: _RawStepData, logger: tmt.utils.Common) -> T:
          """
          Unserialize step data instance from its a raw representation.
          """
  
          cls.pre_normalization(raw_data, logger)
  
          data = cls(name=raw_data['name'], how=raw_data['how'])
          data._load_keys(cast(Dict[str, Any], raw_data), cls.__name__, logger)
  
          data.post_normalization(raw_data, logger)
  
          return data
  
  
  @dataclasses.dataclass
  class WhereableStepData:
      """
      Keys shared by step data that may be limited to a particular guest.
  
      To be used as a mixin class, adds necessary keys.
  
      See [1] and [2] for specification.
  
      1. https://tmt.readthedocs.io/en/stable/spec/plans.html#where
      2. https://tmt.readthedocs.io/en/stable/spec/plans.html#spec-plans-prepare-where
      """
  
      where: Optional[str] = None
  
  
  class Step(tmt.utils.Common):
      """ Common parent of all test steps """
  
      # Default implementation for all steps is "shell", but some
      # steps like provision may have better defaults for their
      # area of expertise.
      DEFAULT_HOW: str = 'shell'
  
      # Refers to a base class for all plugins registered with this step.
      _plugin_base_class: Type['BasePlugin']
  
      #: Stores the normalized step data. Initialized first time step's `data`
      #: is accessed.
      #
      # The delayed initialization is necessary to support `how` changes via
      # command-line - code instantiating steps must be able to invalidate
      # and replace raw step data entries before they get normalized and become
      # the single source of information for plugins involved.
      _data: List[StepData]
  
      #: Stores the original raw step data. Initialized by :py:meth:`__init__`
      #: or :py:meth:`wake`, and serves as a source for normalization performed
      #: by :py:meth:`_normalize_data`.
      _raw_data: List[_RawStepData]
  
      def __init__(
              self,
              plan: 'Plan',
              data: Optional[RawStepDataArgument] = None,
              name: Optional[str] = None,
              workdir: tmt.utils.WorkdirArgumentType = None) -> None:
          """ Initialize and check the step data """
          super().__init__(name=name, parent=plan, workdir=workdir)
          # Initialize data
          self.plan: 'Plan' = plan
          self._status: Optional[str] = None
          self._phases: List[Phase] = []
  
          # Normalize raw data to be a list of step configuration data, one item per
          # distinct step configuration. Make sure all items have `name`` and `how` keys.
          #
          # NOTE: this is not a normalization step as performed by NormalizeKeysMixin.
          # Here we make sure the raw data can be consumed by the delegation code, we
          # do not modify any existing content of raw data items.
  
          # Create an empty step by default (can be updated from cli)
          if data is None:
              self._raw_data = [{}]
  
          # Convert to list if only a single config provided
          elif isinstance(data, dict):
              self._raw_data = [data]
  
          # List is as good as it gets
          elif isinstance(data, list):
              self._raw_data = data
  
          # Shout about invalid configuration
          else:
              raise tmt.utils.GeneralError(
                  f"Invalid '{self}' config in '{self.plan}'.")
  
          for i, raw_datum in enumerate(self._raw_data):
              # Add default unique names even to multiple configs so that the users
              # don't need to specify it if they don't care about the name
              if raw_datum.get('name', None) is None:
                  raw_datum['name'] = f'{tmt.utils.DEFAULT_NAME}-{i}'
  
              # Set 'how' to the default if not specified
              if raw_datum.get('how', None) is None:
                  raw_datum['how'] = self.DEFAULT_HOW
  
      def _normalize_data(self, raw_data: List[_RawStepData]) -> List[StepData]:
          """
          Normalize step data entries.
  
          Every entry of ``raw_data`` is converted into an instance of
          :py:class:`StepData` or one of its subclasses. Particular class
          is derived from a plugin identified by raw data's ``how`` field
          and step's plugin registry.
          """
  
          data: List[StepData] = []
  
          for raw_datum in raw_data:
              plugin = self._plugin_base_class.delegate(self, raw_data=raw_datum)
  
              data.append(plugin.data)
  
          return data
  
      @property
      def data(self) -> List[StepData]:
          if not hasattr(self, '_data'):
              self._data = self._normalize_data(self._raw_data)
  
          return self._data
  
      @data.setter
      def data(self, data: List[StepData]) -> None:
          self._data = data
  
      @property
      def enabled(self) -> Optional[bool]:
          """ True if the step is enabled """
          try:
              return self.name in self.plan.my_run._context.obj.steps
          except AttributeError:
              return None
  
      @property
      def plugins_in_standalone_mode(self) -> int:
          """
          The number of plugins in standalone mode.
  
          Stand-alone mode means that only this step should be run as a part
          of the run (and not any other even if requested using --all).
          This is useful as some step options may completely change its
          behaviour from the regular behaviour based on options
          (e.g. listing images inside provision).
          """
          return sum(phase.is_in_standalone_mode for phase in self.phases())
  
      @classmethod
      def usage(cls, method_overview: str) -> str:
          """ Prepare general usage message for the step """
          # Main description comes from the class docstring
          if cls.__name__ is None:
              raise tmt.utils.GeneralError("Missing name of the step.")
  
          if cls.__doc__ is None:
              raise tmt.utils.GeneralError(
                  f"Missing docstring of the step {cls.__name__.lower()}.")
  
          usage = textwrap.dedent(cls.__doc__)
          # Append the list of supported methods
          usage += '\n\n' + method_overview
          # Give a hint about detailed help
          name = cls.__name__.lower()
          usage += (
              f"\n\nUse 'tmt run {name} --help --how <method>' to learn more "
              f"about given {name} method and all its supported options.")
          return usage
  
      def status(self, status: Optional[str] = None) -> Optional[str]:
          """
          Get and set current step status
  
          The meaning of the status is as follows:
          todo ... config, data and command line processed (we know what to do)
          done ... the final result of the step stored to workdir (we are done)
          """
          # Update status
          if status is not None:
              # Check for valid values
              if status not in ['todo', 'done']:
                  raise tmt.utils.GeneralError(f"Invalid status '{status}'.")
              # Show status only if changed
              elif self._status != status:
                  self._status = status
                  self.debug('status', status, color='yellow', level=2)
          # Return status
          return self._status
  
      def load(self) -> None:
          """ Load status and step data from the workdir """
          try:
              raw_step_data: Dict[Any, Any] = tmt.utils.yaml_to_dict(self.read('step.yaml'))
              self.debug('Successfully loaded step data.', level=2)
  
              self.data = [
                  StepData.unserialize(raw_datum) for raw_datum in raw_step_data['data']
                  ]
              self.status(raw_step_data['status'])
          except tmt.utils.GeneralError:
              self.debug('Step data not found.', level=2)
  
      def save(self) -> None:
          """ Save status and step data to the workdir """
          content: Dict[str, Any] = {
              'status': self.status(),
              'data': [datum.to_serialized() for datum in self.data]
              }
          self.write('step.yaml', tmt.utils.dict_to_yaml(content))
  
      def wake(self) -> None:
          """ Wake up the step (process workdir and command line) """
          # Cleanup possible old workdir if called with --force
          if self.opt('force'):
              self._workdir_cleanup()
  
          # Load stored data
          self.load()
  
          # Status 'todo' means the step has not finished successfully.
          # Probably interrupted in the middle. Clean up the work
          # directory to give it another chance with a fresh start.
          if self.status() == 'todo':
              self.debug("Step has not finished. Let's try once more!", level=2)
              self._workdir_cleanup()
  
          # Importing here to avoid circular imports
          import tmt.steps.report
  
          # Special handling for the report step to always enable force mode in
          # order to cover a very frequent use case 'tmt run --last report'
          # FIXME find a better way how to enable always-force per plugin
          if (isinstance(self, tmt.steps.report.Report) and
                  self.data[0].how in ['display', 'html']):
              self.debug("Report step always force mode enabled.")
              self._workdir_cleanup()
              self.status('todo')
  
          # Nothing more to do when the step is already done
          if self.status() == 'done':
              self.debug('Step is done, not touching its data.')
              return
  
          # Override step data with command line options
          how: str = self.opt('how')
          if how is not None:
              # If 'how' has been given, when it comes to current entries in `self.data`,
              # there are two options:
              #
              # * entry's `how` is the same as the one given via command-line. Then we can
              # keep step data we already have.
              # * entry's `how` is different, and then we need to throw the entry away and
              # replace it with new `how`.
              #
              # To handle both variants, we replace `self.data` with new set of entries,
              # based on newly constructed set of raw data.
              self.debug(f'CLI-provided how={how} overrides all existing step data', level=4)
  
              _raw_data: List[_RawStepData] = []
  
              for datum in self.data:
                  # We can re-use this one - to make handling easier, just dump it to "raw"
                  # form for _normalize_data().
                  if datum.how == how:
                      self.debug(f'  compatible:   {datum}', level=4)
                      _raw_data.append(datum.to_raw())
  
                  # Mismatch, throwing away, replacing with new `how` - but we can keep the name.
                  else:
                      self.debug(f'  incompatible: {datum}', level=4)
                      _raw_data.append({
                          'name': datum.name,
                          'how': how
                          })
  
              self.data = self._normalize_data(_raw_data)
              self._raw_data = _raw_data
  
              self.debug('updated data', str(self.data), level=4)
  
          else:
              self.debug('CLI did not change existing step data', level=4)
  
      def setup_actions(self) -> None:
          """ Insert login and reboot plugins if requested """
          for login_plugin in Login.plugins(step=self):
              self.debug(
                  f"Insert a login plugin into the '{self}' step "
                  f"with order '{login_plugin.order}'.", level=2)
              self._phases.append(login_plugin)
  
          for reboot_plugin in Reboot.plugins(step=self):
              self.debug(
                  f"Insert a reboot plugin into the '{self}' step "
                  f"with order '{reboot_plugin.order}'.", level=2)
              self._phases.append(reboot_plugin)
  
      @overload
      def phases(self, classes: None = None) -> List[Phase]:
          pass
  
      @overload
      def phases(self, classes: Type[PhaseT]) -> List[PhaseT]:
          pass
  
      @overload
      def phases(self, classes: Tuple[Type[PhaseT], ...]) -> List[PhaseT]:
          pass
  
      def phases(self, classes: Optional[Union[Type[PhaseT],
                 Tuple[Type[PhaseT], ...]]] = None) -> List[PhaseT]:
          """
          Iterate over phases by their order
  
          By default iterates over all available phases. Optional filter
          'classes' can be used to iterate only over instances of given
          class (single class or tuple of classes).
          """
  
          if classes is None:
              _classes: Tuple[Union[Type[Phase], Type[PhaseT]], ...] = (Phase,)
  
          elif not isinstance(classes, tuple):
              _classes = (classes,)
  
          else:
              _classes = classes
  
          return sorted(
              [cast(PhaseT, phase) for phase in self._phases if isinstance(phase, _classes)],
              key=lambda phase: phase.order)
  
      def actions(self) -> None:
          """ Run all loaded Login or Reboot action instances of the step """
          for phase in self.phases(classes=Action):
              phase.go()
  
      def go(self) -> None:
          """ Execute the test step """
          # Show step header and how
          self.info(self.name, color='blue')
          # Show workdir in verbose mode
          self.debug('workdir', self.workdir, 'magenta')
  
  
  class Method:
      """ Step implementation method """
  
      def __init__(
              self,
              name: str,
              class_: Optional[Type['BasePlugin']] = None,
              doc: Optional[str] = None,
              order: int = DEFAULT_PLUGIN_METHOD_ORDER
              ) -> None:
          """ Store method data """
  
          doc = (doc or getattr(class_, '__doc__') or '').strip()
  
          if not doc:
              if class_:
                  raise tmt.utils.GeneralError(f"Plugin class '{class_}' provides no docstring.")
  
              raise tmt.utils.GeneralError(f"Plugin method '{name}' provides no docstring.")
  
          self.name = name
          self.class_ = class_
          self.doc = doc
          self.order = order
  
          # Parse summary and description from provided doc string
          lines: List[str] = [re.sub('^    ', '', line)
                              for line in self.doc.split('\n')]
          self.summary: str = lines[0].strip()
          self.description: str = '\n'.join(lines[1:]).lstrip()
  
      def describe(self) -> str:
          """ Format name and summary for a nice method overview """
          return f'{self.name} '.ljust(22, '.') + f' {self.summary}'
  
      def usage(self) -> str:
          """ Prepare a detailed usage from summary and description """
          if self.description:
              usage: str = self.summary + '\n\n' + self.description
          else:
              usage = self.summary
          # Disable wrapping for all paragraphs
          return re.sub('\n\n', '\n\n\b\n', usage)
  
  
  def provides_method(
          name: str,
          doc: Optional[str] = None,
          order: int = DEFAULT_PLUGIN_METHOD_ORDER) -> Any:
      """
      A plugin class decorator to register plugin's method with tmt steps.
  
      In the following example, developer marks ``SomePlugin`` as providing two discover methods,
      ``foo`` and ``bar``, with ``bar`` being sorted to later position among methods:
  
      .. code-block:: python
  
         @tmt.steps.provides_method('foo')
         @tmt.steps.provides_method('bar', order=80)
         class SomePlugin(tmt.steps.discover.DicoverPlugin):
           ...
  
      :param name: name of the method.
      :param doc: method documentation. If not specified, docstring of the decorated class is used.
      :param order: order of the method among other step methods.
      """
  
      def _method(cls: Type['BasePlugin']) -> Any:
          plugin_method = Method(name, class_=cls, doc=doc, order=order)
  
          cast('BasePlugin', cls.__bases__[0])._supported_methods.append(plugin_method)
  
          return cls
  
      return _method
  
  
  class PluginIndex(type):
      """
      Plugin metaclass used to register all available plugins.
  
      .. deprecated:: 1.17
         Instead of declaring `_methods` list, use :py:func:`provides_method` decorator
         to announce provided methods.
      """
  
      def __init__(
              cls,
              name: str,
              bases: List['BasePlugin'],
              attributes: Any) -> None:
          """ Store all defined methods in the parent class """
  
          plugin_methods = cast(Optional[List[Method]], getattr(cls, '_methods'))
  
          if not plugin_methods:
              return
  
          warnings.warn(
              'Use of _methods to declare plugin methods is deprecated.'
              'Use tmt.steps.provides_method decorator instead.',
              DeprecationWarning,
              stacklevel=2)
  
          for plugin_method in plugin_methods:
              plugin_method.class_ = cast(Type['BasePlugin'], cls)
              # Add to the list of supported methods in parent class
              bases[0]._supported_methods.append(plugin_method)
  
  
  class BasePlugin(Phase, metaclass=PluginIndex):
      """ Common parent of all step plugins """
  
      # Deprecated, use @provides_method(...) instead. left for backward
      # compatibility with out-of-tree plugins.
      _methods: List[Method] = []
  
      # Default implementation for all steps is shell
      # except for provision (virtual) and report (display)
      how: str = 'shell'
  
      # List of all supported methods aggregated from all plugins of the same step.
      _supported_methods: List[Method] = []
  
      _data_class: Type[StepData] = StepData
      data: StepData
  
      # TODO: do we need this list? Can whatever code is using it use _data_class directly?
      # List of supported keys
      # (used for import/export to/from attributes during load and save)
      @property
      def _keys(self) -> List[str]:
          return list(self._data_class.keys())
  
      def __init__(
              self,
              step: Step,
              data: StepData,
              workdir: tmt.utils.WorkdirArgumentType = None) -> None:
          """ Store plugin name, data and parent step """
          # Store name, data and parent step
          super().__init__(
              parent=step,
              name=data.name,
              workdir=workdir,
              order=data.order)
          # It is not possible to use TypedDict here because
          # all keys are not known at the time of the class definition
          self.data = data
          self.step = step
  
      @classmethod
      def base_command(
              cls,
              usage: str,
              method_class: Optional[Type[click.Command]] = None) -> click.Command:
          """ Create base click command (common for all step plugins) """
          raise NotImplementedError
  
      @classmethod
      def options(cls, how: Optional[str] = None) -> List[tmt.options.ClickOptionDecoratorType]:
          """ Prepare command line options for given method """
          # Include common options supported across all plugins
          return tmt.options.verbose_debug_quiet + tmt.options.force_dry
  
      @classmethod
      def command(cls) -> click.Command:
          """ Prepare click command for all supported methods """
          # Create one command for each supported method
          commands: Dict[str, click.Command] = {}
          method_overview: str = f'Supported methods ({cls.how} by default):\n\n\b'
          for method in cls.methods():
              assert method.class_ is not None
              method_overview += f'\n{method.describe()}'
              command: click.Command = cls.base_command(usage=method.usage())
              # Apply plugin specific options
              for option in method.class_.options(method.name):
                  command = option(command)
              commands[method.name] = command
  
          # Create base command with common options using method class
          method_class = tmt.options.create_method_class(commands)
          command = cls.base_command(usage=method_overview, method_class=method_class)
          # Apply common options
          for option in cls.options():
              command = option(command)
          return command
  
      @classmethod
      def methods(cls) -> List[Method]:
          """ Return all supported methods ordered by priority """
          return sorted(cls._supported_methods, key=lambda method: method.order)
  
      @classmethod
      def delegate(
              cls,
              step: Step,
              data: Optional[StepData] = None,
              raw_data: Optional[_RawStepData] = None) -> 'BasePlugin':
          """
          Return plugin instance implementing the data['how'] method
  
          Supports searching by method prefix as well (e.g. 'virtual').
          The first matching method with the lowest 'order' wins.
          """
  
          if data is not None:
              how = data.how
          elif raw_data is not None:
              how = raw_data['how']
          else:
              raise tmt.utils.GeneralError('Either data or raw data must be given.')
  
          step.debug(
              f'{cls.__name__}.delegate(step={step}, data={data}, raw_data={raw_data})',
              level=3)
  
          # Filter matching methods, pick the one with the lowest order
          for method in cls.methods():
              assert method.class_ is not None
              if method.name.startswith(how):
                  step.debug(
                      f"Using the '{method.class_.__name__}' plugin "
                      f"for the '{how}' method.", level=2)
  
                  plugin_class = method.class_
                  plugin_data_class = plugin_class._data_class
  
                  # If we're given raw data, construct a step data instance, applying
                  # normalization in the process.
                  if raw_data is not None:
                      try:
                          data = plugin_data_class.from_raw(raw_data, step)
  
                      except Exception as exc:
                          raise tmt.utils.GeneralError(
                              f'Failed to load step data for {plugin_data_class.__name__}: {exc}',
                              original=exc)
  
                  assert data is not None
                  assert data.__class__ is plugin_data_class, \
                      f'Data package is instance of {data.__class__.__name__}, ' \
                      f'plugin {plugin_class.__name__} ' \
                      f'expects {plugin_data_class.__name__}'
  
                  plugin = plugin_class(step, data)
                  assert isinstance(plugin, BasePlugin)
                  return plugin
  
          show_step_method_hints(step, step.name, how)
          # Report invalid method
          if step.plan is None:
              raise tmt.utils.GeneralError(f"Plan for {step.name} is not set.")
          raise tmt.utils.SpecificationError(
              f"Unsupported {step.name} method '{how}' "
              f"in the '{step.plan.name}' plan.")
  
      def default(self, option: str, default: Optional[Any] = None) -> Any:
          """ Return default data for given option """
  
          value = self._data_class.default(tmt.utils.option_to_key(option), default=default)
  
          if value is None:
              return default
  
          return value
  
      def get(self, option: str, default: Optional[Any] = None) -> Any:
          """ Get option from plugin data, user/system config or defaults """
  
          # Check plugin data first
          #
          # Since self.data is a dataclass instance, the option would probably exist.
          # As long as there's no typo in name, it would be defined. Which complicates
          # the handling of "default" as in "return *this* when attribute is unset".
          key = tmt.utils.option_to_key(option)
  
          try:
              value = getattr(self.data, key)
  
              # If the value is no longer the default one, return the value. If it
              # still matches the default value, instead of returning the default
              # value right away, call `self.default()` so the plugin has chance to
              # catch calls for computed or virtual keys, keys that don't exist as
              # atributes of our step data.
              #
              # One way would be to subclass step's base plugin class' step data class
              # (which is a subclass of `StepData` and `SerializedContainer`), and
              # override its `default()` method to handle these keys. But, plugins often
              # are pretty happy with the base data class, many don't need their own
              # step data class, and plugin developer might be forced to create a subclass
              # just for this single method override.
              #
              # Instead, keep plugin's `default()` around - everyone can use it to get
              # default value for a given option/key, and plugins can override it as needed
              # (they will always subclass step's base plugin class anyway!). Default
              # implementation would delegate to step data `default()`, and everyone's
              # happy.
  
              if value != self.data.default(key):
                  return value
  
          except AttributeError:
              pass
  
          return self.default(option, default)
  
      def show(self, keys: Optional[List[str]] = None) -> None:
          """ Show plugin details for given or all available keys """
          # Avoid circular imports
          import tmt.base
  
          # Show empty config with default method only in verbose mode
          if self.data.is_bare and not self.opt('verbose'):
              return
          # Step name (and optional summary)
          echo(tmt.utils.format(
              self.step.name, self.get('summary') or '',
              key_color='blue', value_color='blue'))
          # Show all or requested step attributes
          if keys is None:
              keys = list(set(self.data.keys()))
  
          def _emit_key(key: str) -> None:
              # Skip showing the default name
              if key == 'name' and self.name.startswith(tmt.utils.DEFAULT_NAME):
                  return
  
              # Skip showing summary again
              if key == 'summary':
                  return
  
              value = self.get(key)
  
              # No need to show the default order
              if key == 'order' and value == tmt.base.DEFAULT_ORDER:
                  return
  
              if value is None:
                  return
  
              # TODO: hides keys that were used to be in the output...
              # if value == self.data.default(key):
              #     return
  
              echo(tmt.utils.format(tmt.utils.key_to_option(key), value))
  
          # First, follow the order prefered by step data, but emit only the keys
          # that are allowed. Each emitted key would be removed so we wouldn't
          # emit it again when showing the unsorted rest of keys.
          for key in self.data.KEYS_SHOW_ORDER:
              if key not in keys:
                  continue
  
              _emit_key(key)
  
              keys.remove(key)
  
          # Show the rest
          for key in keys:
              _emit_key(key)
  
      def enabled_on_guest(self, guest: 'Guest') -> bool:
          """ Check if the plugin is enabled on the specific guest """
          where: str = self.get('where')
          if not where:
              return True
          return where in (guest.name, guest.role)
  
      def _update_data_from_options(self, keys: List[str]) -> None:
          for key in keys:
              value = self.opt(tmt.utils.key_to_option(key))
              if value:
                  setattr(self.data, key, value)
  
      def wake(self) -> None:
          """
          Wake up the plugin, process data, apply options
  
          Check command line options corresponding to plugin keys
          and store their value into the 'self.data' dictionary if
          their value is True or non-empty.
  
          By default, all supported options corresponding to common
          and plugin-specific keys are processed. List of key names
          in the 'keys' parameter can be used to override only
          selected ones.
          """
  
          assert self.data.__class__ is self._data_class, \
              f'Plugin {self.__class__.__name__} woken with incompatible ' \
              f'data {self.data}, ' \
              f'expects {self._data_class.__name__}'
  
          if self.step.status() == 'done':
              self.debug('step is done, not overwriting plugin data')
              return
  
          # TODO: conflicts with `upgrade` plugin which does this on purpose :/
          # if self.opt('how') is not None:
          #     assert self.opt('how') in [method.name for method in self.methods()], \
          #         f'Plugin {self.__class__.__name__} woken with unsupported ' \
          #         f'how "{self.opt("how")}", ' \
          #         f'supported methods {", ".join([method.name for method in self.methods()])}, ' \
          #         f'current data is {self.data}'
  
          self._update_data_from_options(keys=self._keys)
  
      # NOTE: it's tempting to rename this method to `go()` and use more natural
      # `super().go()` in child classes' `go()` methods. But, `go()` does not have
      # the same signature across all plugin types, therefore we cannot have shared
      # `go()` method in superclass - overriding it in (some) child classes would
      # raise a typing linter error reporting superclass signature differs from the
      # one in a subclass.
      #
      # Therefore we need a different name, and a way how not to forget to call this
      # method from child classes.
      def go_prolog(self) -> None:
          """ Perform actions shared among plugins when beginning their tasks """
          # Show the method
          self.info('how', self.get('how'), 'magenta')
          # Give summary if provided
          if self.get('summary'):
              self.info('summary', self.get('summary'), 'magenta')
          # Show name only if it's not the default one
          if not self.name.startswith(tmt.utils.DEFAULT_NAME):
              self.info('name', self.name, 'magenta')
          # Include order in verbose mode
          self.verbose('order', str(self.order), 'magenta', level=3)
  
      def requires(self) -> List[str]:
          """ List of packages required by the plugin on the guest """
          return []
  
  
  class GuestlessPlugin(BasePlugin):
      """ Common parent of all step plugins that do not work against a particular guest """
  
      def go(self) -> None:
          """ Perform actions shared among plugins when beginning their tasks """
  
          self.go_prolog()
  
  
  class Plugin(BasePlugin):
      """ Common parent of all step plugins that do work against a particular guest """
  
      def go(self, guest: 'Guest') -> None:
          """ Perform actions shared among plugins when beginning their tasks """
  
          self.go_prolog()
  
  
  class Action(Phase):
      """ A special action performed during a normal step. """
  
      # Dictionary containing list of requested phases for each enabled step
      _phases: Optional[Dict[str, List[int]]] = None
  
      @classmethod
      def phases(cls, step: Step) -> List[int]:
          """ Return list of phases enabled for given step """
          # Build the phase list unless done before
          if cls._phases is None:
              cls._phases = cls._parse_phases(step)
          # Return enabled phases, empty list if step not found
          try:
              return cls._phases[step.name]
          except KeyError:
              return []
  
      @classmethod
      def _parse_phases(cls, step: Step) -> Dict[str, List[int]]:
          """ Parse options and store phase order """
          phases = dict()
          options: List[str] = cls._opt('step', default=[])
  
          # Use the end of the last enabled step if no --step given
          if not options:
              login_during: Optional[Step] = None
              # The last run may have failed before all enabled steps were
              # completed, select the last step done
              if step.plan is None:
                  raise tmt.utils.GeneralError(
                      f"Plan for {step.name} is not set.")
              if step.plan.my_run.opt('last'):
                  steps: List[Step] = [
                      s for s in step.plan.steps() if s.status() == 'done']
                  login_during = steps[-1] if steps else None
              # Default to the last enabled step if no completed step found
              if login_during is None:
                  login_during = list(step.plan.steps())[-1]
              # Only login if the error occurred after provision
              if login_during != step.plan.discover:
                  phases[login_during.name] = [PHASE_END]
  
          # Process provided options
          for option in options:
              # Parse the step:phase format
              matched = re.match(r'(\w+)(:(\w+))?', option)
              if matched:
                  step_name, _, phase = matched.groups()
              if not matched or step_name not in STEPS:
                  raise tmt.utils.GeneralError(f"Invalid step '{option}'.")
              # Check phase format, convert into int, use end by default
              try:
                  phase = int(phase)
              except TypeError:
                  phase = PHASE_END
              except ValueError:
                  # Convert 'start' and 'end' aliases
                  try:
                      phase = cast(Dict[str, int],
                                   dict(start=PHASE_START, end=PHASE_END))[phase]
                  except KeyError:
                      raise tmt.utils.GeneralError(f"Invalid phase '{phase}'.")
              # Store the phase for given step
              try:
                  phases[step_name].append(phase)
              except KeyError:
                  phases[step_name] = [phase]
          return phases
  
      def go(self) -> None:
          raise NotImplementedError()
  
  
  class Reboot(Action):
      """ Reboot guest """
  
      # True if reboot enabled
      _enabled: bool = False
  
      def __init__(self, step: Step, order: int) -> None:
          """ Initialize relations, store the reboot order """
          super().__init__(parent=step, name='reboot', order=order)
  
      @classmethod
      def command(
              cls,
              method_class: Optional[Method] = None,
              usage: Optional[str] = None) -> click.Command:
          """ Create the reboot command """
          @click.command()
          @click.pass_context
          @click.option(
              '-s', '--step', metavar='STEP[:PHASE]', multiple=True,
              help='Reboot machine during given phase of selected step(s).')
          @click.option(
              '--hard', is_flag=True,
              help='Hard reboot of the machine. Unsaved data may be lost.')
          def reboot(context: Any, **kwargs: Any) -> None:
              """ Reboot the guest. """
              Reboot._save_context(context)
              Reboot._enabled = True
  
          return reboot
  
      @classmethod
      def plugins(cls, step: Step) -> List['Reboot']:
          """ Return list of reboot instances for given step """
          if not Reboot._enabled:
              return []
          return [Reboot(step, phase) for phase in cls.phases(step)]
  
      def go(self) -> None:
          """ Reboot the guest(s) """
          self.info('reboot', 'Rebooting guest', color='yellow')
          assert isinstance(self.parent, Step)
          assert hasattr(self.parent, 'plan') and self.parent.plan is not None
          for guest in self.parent.plan.provision.guests():
              guest.reboot(hard=self.opt('hard'))
          self.info('reboot', 'Reboot finished', color='yellow')
  
  
  class Login(Action):
      """ Log into the guest """
  
      # True if interactive login enabled
      _enabled: bool = False
  
      def __init__(self, step: Step, order: int):
          """ Initialize relations, store the login order """
          super().__init__(parent=step, name='login', order=order)
  
      @classmethod
      def command(
              cls,
              method_class: Optional[Method] = None,
              usage: Optional[str] = None) -> click.Command:
          """ Create the login command """
          @click.command()
          @click.pass_context
          @click.option(
              '-s', '--step', metavar='STEP[:PHASE]', multiple=True,
              help='Log in during given phase of selected step(s).')
          @click.option(
              '-w', '--when', metavar='RESULT', multiple=True,
              type=click.Choice(tmt.base.Result._results),
              help='Log in if a test finished with given result(s).')
          @click.option(
              '-c', '--command', metavar='COMMAND',
              multiple=True, default=['bash'],
              help="Run given command(s). Default is 'bash'.")
          def login(context: Any, **kwargs: Any) -> None:
              """
              Provide user with an interactive shell on the guest.
  
              By default the shell is provided at the end of the last
              enabled step. When used together with the --last option the
              last completed step is selected. Use one or more --step
              options to select a different step instead.
  
              Optional phase can be provided to specify the exact phase of
              the step when the shell should be provided. The following
              values are supported:
  
              \b
                  start ... beginning of the step (same as '10')
                  end ..... end of the step (default, same as '90')
                  00-99 ... integer order defining the exact phase
  
              Usually the main step execution happens with order 50.
              Consult individual step documentation for more details.
  
              For the execute step and following steps it is also possible
              to conditionally enable the login feature only if some of
              the tests finished with given result (pass, info, fail,
              warn, error).
              """
              Login._save_context(context)
              Login._enabled = True
  
          return login
  
      @classmethod
      def plugins(cls, step: Step) -> List['Login']:
          """ Return list of login instances for given step """
          if not Login._enabled:
              return []
          return [Login(step, phase) for phase in cls.phases(step)]
  
      def go(self) -> None:
          """ Login to the guest(s) """
          # Verify possible test result condition
          count: Dict[str, int] = dict()
          expected_results: Optional[List[str]] = self.opt('when')
  
          assert isinstance(self.parent, Step)
          assert hasattr(self.parent, 'plan') and self.parent.plan is not None
  
          if expected_results:
              for expected_result in expected_results:
                  count[expected_result] = len([
                      result for result in self.parent.plan.execute.results()
                      if result.result == expected_result])
              if not any(count.values()):
                  self.info('Skipping interactive shell', color='yellow')
                  return
  
          # Run the interactive command
          commands: List[str] = self.opt('command')
          self.info('login', 'Starting interactive shell', color='yellow')
          for guest in self.parent.plan.provision.guests():
              # Attempt to push the workdir to the guest
              try:
                  guest.push()
                  cwd: Optional[str] = self.parent.plan.worktree
              except tmt.utils.GeneralError:
                  self.warn("Failed to push workdir to the guest.")
                  cwd = None
              # Execute all requested commands
              for command in commands:
                  self.debug(f"Run '{command}' in interactive mode.")
                  guest.execute(command, interactive=True, cwd=cwd)
          self.info('login', 'Interactive shell finished', color='yellow')
  
files: tmt/steps/__init__.py
files: import dataclasses
  import glob
  import os
  import re
  import shutil
  import subprocess
  from typing import List, Optional, cast
  
  import click
  import fmf
  
  import tmt
  import tmt.base
  import tmt.beakerlib
  import tmt.options
  import tmt.steps
  import tmt.steps.discover
  import tmt.utils
  
  
  @dataclasses.dataclass
  class DiscoverFmfStepData(tmt.steps.discover.DiscoverStepData):
      url: Optional[str] = None
      ref: Optional[str] = None
      path: Optional[str] = None
      test: List[str] = dataclasses.field(default_factory=list)
      link: List[str] = dataclasses.field(default_factory=list)
      filter: List[str] = dataclasses.field(default_factory=list)
      modified_only: bool = False
      modified_url: Optional[str] = None
      modified_ref: Optional[str] = None
      dist_git_init: bool = False
      dist_git_remove_fmf_root: bool = False
      dist_git_merge: bool = False
      dist_git_extract: Optional[str] = None
      fmf_id: bool = False
      exclude: List[str] = dataclasses.field(default_factory=list)
  
      # Legacy fields
      repository: Optional[str] = None
      revision: Optional[str] = None
  
      _normalize_test = tmt.utils.NormalizeKeysMixin._normalize_string_list
      _normalize_link = tmt.utils.NormalizeKeysMixin._normalize_string_list
      _normalize_filter = tmt.utils.NormalizeKeysMixin._normalize_string_list
      _normalize_exclude = tmt.utils.NormalizeKeysMixin._normalize_string_list
  
      def post_normalization(self, raw_data: tmt.steps._RawStepData,
                             logger: tmt.utils.Common) -> None:
          super().post_normalization(raw_data, logger)
  
          if self.repository:
              self.url = self.repository
  
          if self.revision:
              self.ref = self.revision
  
  
  @tmt.steps.provides_method('fmf')
  class DiscoverFmf(tmt.steps.discover.DiscoverPlugin):
      """
      Discover available tests from fmf metadata
  
      By default all available tests from the current repository are used
      so the minimal configuration looks like this:
  
          discover:
              how: fmf
  
      Full config example:
  
          discover:
              how: fmf
              url: https://github.com/teemtee/tmt
              ref: main
              path: /fmf/root
              test: /tests/basic
              filter: 'tier: 1'
  
      If no 'ref' is provided, the default branch from the origin is used.
  
      For DistGit repo one can extract source tarball first and discover
      tests from it by using 'distgit-source: true'. It can be used
      together with 'ref', 'path' and 'url', However 'ref' is not possible
      without using 'url'.
  
          discover:
              how: fmf
              dist-git-source: true
  
      Related config options (all optional):
      * dist-git-merge - set to True if you want to copy in extracted
        sources to the local repo
      * dist-git-init - set to True and 'fmf init' will be called inside
        extracted sources (at dist-git-extract or top directory)
      * dist-git-extract - directory (glob supported) to copy from
        extracted sources (defaults to inner fmf-root)
      * dist-git-remove-fmf-root - set to True to remove fmf root from
        extracted sources
  
      Selecting tests containing specified link is possible using 'link'
      option accepting RELATION:TARGET format of values. Regular
      expressions are supported for both relation and target part of the
      value. Relation can be omitted to target match any relation.
  
          discover:
              how: fmf
              link: verifies:.*issue/850$
  
      It is also possible to limit tests only to those that have changed
      in git since a given revision. This can be particularly useful when
      testing changes to tests themselves (e.g. in a pull request CI).
  
      Related config options (all optional):
      * modified-only - set to True if you want to filter modified tests
      * modified-url - fetched as "reference" remote in the test dir
      * modified-ref - the ref to compare against
  
      Example to compare local repo against upstream 'main' branch:
  
          discover:
              how: fmf
              modified-only: True
              modified-url: https://github.com/teemtee/tmt
              modified-ref: reference/main
  
      Note that internally the modified tests are appended to the list
      specified via 'test', so those tests will also be selected even if
      not modified.
      """
  
      _data_class = DiscoverFmfStepData
  
      REF_OPTION = click.option(
          '-r', '--ref', metavar='REVISION',
          help='Branch, tag or commit specifying the git revision.')
      TEST_OPTION = click.option(
          '-t', '--test', metavar='NAMES', multiple=True,
          help='Select tests by name.')
      FILTER_OPTION = click.option(
          '-F', '--filter', metavar='FILTERS', multiple=True,
          help='Include only tests matching the filter.')
      EXCLUDE_OPTION = click.option(
          '-x', '--exclude', metavar='[REGEXP]', multiple=True,
          help="Exclude a regular expression from search result.")
  
      # Options which require .git to be present for their functionality
      _REQUIRES_GIT = (
          "ref",
          "modified-url",
          "modified-only",
          "fmf-id",
          )
  
      @classmethod
      def options(cls, how: Optional[str] = None) -> List[tmt.options.ClickOptionDecoratorType]:
          """ Prepare command line options for given method """
          options = super().options(how)
          options[:0] = [
              click.option(
                  '-u', '--url', metavar='REPOSITORY',
                  help='URL of the git repository with fmf metadata.'),
              cls.REF_OPTION,
              click.option(
                  '--modified-url', metavar='REPOSITORY',
                  help='URL of the reference git repository with fmf metadata.'),
              click.option(
                  '--modified-ref', metavar='REVISION',
                  help='Branch, tag or commit specifying the reference git '
                  'revision (if not provided, the default branch is used).'),
              click.option(
                  '-m', '--modified-only', is_flag=True,
                  help='If set, select only tests modified '
                  'since reference revision.'),
              click.option(
                  '-p', '--path', metavar='ROOT',
                  help='Path to the metadata tree root.'),
              cls.TEST_OPTION,
              click.option(
                  '--link', metavar="RELATION:TARGET", multiple=True,
                  help="Filter by linked objects (regular expressions are "
                       "supported for both relation and target)."),
              cls.FILTER_OPTION,
              cls.EXCLUDE_OPTION,
              click.option(
                  '--fmf-id', default=False, is_flag=True,
                  help='Show fmf identifiers for tests discovered in plan.'),
              click.option(
                  '--sync-repo', default=False, is_flag=True,
                  help='Force the sync of the whole git repo. By default, the '
                       'repo is copied only if the used options require it.'),
              click.option(
                  '--dist-git-init', is_flag=True,
                  help='Initialize fmf root inside extracted sources '
                       '(at dist-git-extract or top directory).'),
              click.option(
                  '--dist-git-remove-fmf-root', is_flag=True,
                  help='Remove fmf root from extracted source '
                       '(top one or selected by copy-path, '
                       'happens before dist-git-extract.'),
              click.option(
                  '--dist-git-merge', is_flag=True,
                  help='Merge copied sources and plan fmf root.'),
              click.option(
                  '--dist-git-extract',
                  help='What to copy from extracted sources, globbing is '
                       'supported. Defaults to the top fmf root if it is '
                       'present, otherwise top directory (shortcut "/").'
                  ),
              ]
          return options
  
      @property
      def is_in_standalone_mode(self) -> bool:
          """ Enable standalone mode when listing fmf ids """
          if self.opt('fmf_id'):
              return True
          return super().is_in_standalone_mode
  
      def go(self) -> None:
          """ Discover available tests """
          super(DiscoverFmf, self).go()
  
          # Check url and path, prepare test directory
          url = self.get('url')
          path = self.get('path')
          # Save the test directory so that others can reference it
          ref = self.get('ref')
          assert self.workdir is not None
          self.testdir = os.path.join(self.workdir, 'tests')
          sourcedir = os.path.join(self.workdir, 'source')
          dist_git_source = self.get('dist-git-source', False)
          dist_git_merge = self.get('dist-git-merge', False)
          dist_git_init = self.get('dist-git-init', False)
          dist_git_extract = self.get('dist-git-extract', None)
          dist_git_remove_fmf_root = self.get('dist-git-remove-fmf-root', False)
  
          # Self checks
          if dist_git_source and not dist_git_merge and (ref or url):
              raise tmt.utils.DiscoverError(
                  "Cannot manipulate with dist-git without "
                  "the `--dist-git-merge` option.")
  
          def get_git_root(dir: str) -> str:
              stdout, _ = self.run(["git", "rev-parse", "--show-toplevel"], cwd=dir, dry=True)
              assert stdout is not None
              return stdout.strip("\n")
  
          # Raise an exception if --fmf-id uses w/o url and git root
          # doesn't exist for discovered plan
          if self.opt('fmf_id'):
              def assert_git_url(plan_name: Optional[str] = None) -> None:
                  try:
                      subprocess.run(
                          'git rev-parse --show-toplevel'.split(),
                          stdout=subprocess.PIPE,
                          stderr=subprocess.DEVNULL,
                          check=True)
                  except subprocess.CalledProcessError:
                      raise tmt.utils.DiscoverError(
                          f"`tmt run discover --fmf-id` without `url` option in "
                          f"plan `{plan_name}` can be used only within"
                          f" git repo.")
              # It covers only one case, when there is:
              # 1) no --url on CLI
              # 2) plan w/o url exists in test run
              if not self.opt('url'):
                  try:
                      fmf_tree = fmf.Tree(os.getcwd())
                  except fmf.utils.RootError:
                      raise tmt.utils.DiscoverError(
                          "No metadata found in the current directory. "
                          "Use 'tmt init' to get started.")
                  for i, attr in enumerate(fmf_tree.climb()):
                      try:
                          plan_url = attr.data.get('discover').get('url')
                          plan_name = attr.name
                          if not plan_url:
                              assert_git_url(plan_name)
                      except AttributeError:
                          pass
              # All other cases are covered by this condition
              if not url:
                  assert_git_url(self.step.plan.name)
  
          # Clone provided git repository (if url given) with disabled
          # prompt to ignore possibly missing or private repositories
          if url:
              self.info('url', url, 'green')
              self.debug(f"Clone '{url}' to '{self.testdir}'.")
              # Shallow clone to speed up testing and
              # minimize data transfers if ref is not provided
              tmt.utils.git_clone(
                  url, self.testdir, self, env={"GIT_ASKPASS": "echo"}, shallow=ref is None)
              git_root = self.testdir
          # Copy git repository root to workdir
          else:
              fmf_root = path or self.step.plan.my_run.tree.root
              requires_git = self.opt('sync-repo') or any(
                  self.get(opt) for opt in self._REQUIRES_GIT)
              # Path for distgit sources cannot be checked until the
              # tarball is extracted
              if path and not os.path.isdir(path) and not dist_git_source:
                  raise tmt.utils.DiscoverError(
                      f"Provided path '{path}' is not a directory.")
              if dist_git_source:
                  # Ensure we're in a git repo when extracting dist-git sources
                  try:
                      git_root = get_git_root(self.step.plan.my_run.tree.root)
                  except tmt.utils.RunError:
                      raise tmt.utils.DiscoverError(
                          f"{self.step.plan.my_run.tree.root} is not a git repo")
              else:
                  if fmf_root is None:
                      raise tmt.utils.DiscoverError(
                          "No metadata found in the current directory.")
                  # Check git repository root (use fmf root if not found)
                  try:
                      git_root = get_git_root(fmf_root)
                  except tmt.utils.RunError:
                      self.debug(f"Git root not found, using '{fmf_root}.'")
                      git_root = fmf_root
                  # Set path to relative path from the git root to fmf root
                  path = os.path.relpath(
                      fmf_root, git_root if requires_git else fmf_root)
  
              # And finally copy the git/fmf root directory to testdir
              # (for dist-git case only when merge explicitly requested)
              directory = git_root if requires_git else fmf_root
              self.info('directory', directory, 'green')
              if not dist_git_source or dist_git_merge:
                  self.debug(f"Copy '{directory}' to '{self.testdir}'.")
                  if not self.opt('dry'):
                      shutil.copytree(directory, self.testdir, symlinks=True)
  
          # Checkout revision if requested
          if ref:
              self.info('ref', ref, 'green')
              self.debug(f"Checkout ref '{ref}'.")
              self.run(
                  ['git', 'checkout', '-f', str(ref)],
                  cwd=self.testdir)
  
          # Show current commit hash if inside a git repository
          if os.path.isdir(self.testdir):
              try:
                  hash_, _ = self.run(["git", "rev-parse", "--short", "HEAD"],
                                      cwd=self.testdir)
                  if hash_ is not None:
                      self.verbose('hash', hash_.strip(), 'green')
              except (tmt.utils.RunError, AttributeError):
                  pass
  
          # Fetch and extract distgit sources
          if dist_git_source:
              try:
                  self.extract_distgit_source(
                      git_root, sourcedir, self.get('dist-git-type'))
              except Exception as error:
                  raise tmt.utils.DiscoverError(
                      "Failed to process 'dist-git-source'.", original=error)
  
              # Check what should be extracted from the sources
              if dist_git_extract:
                  if dist_git_extract == '/':
                      dist_git_extract = sourcedir
                  else:
                      try:
                          dist_git_extract = glob.glob(os.path.join(
                              sourcedir, dist_git_extract.lstrip('/')))[0]
                      except IndexError:
                          raise tmt.utils.DiscoverError(
                              f"Couldn't glob '{dist_git_extract}' "
                              f"within extracted sources.")
  
              # Check sources for the fmf root, copy git root if not found
              if not dist_git_init and not dist_git_extract:
                  try:
                      top_fmf_root = tmt.utils.find_fmf_root(sourcedir)[0]
                  except tmt.utils.MetadataError:
                      dist_git_extract = sourcedir
                      if not dist_git_merge:
                          self.warn(
                              "Extracted sources do not contain fmf root, "
                              "merging with plan data. Avoid this warning by "
                              "explicit use of the '--dist-git-merge' option.")
                          self.debug(f"Copy '{git_root}' to '{self.testdir}'.")
                          if not self.opt('dry'):
                              shutil.copytree(
                                  git_root, self.testdir, symlinks=True)
  
              # Initialize or remove fmf root
              if dist_git_init:
                  if not dist_git_extract:
                      dist_git_extract = sourcedir
                  if not self.opt('dry'):
                      fmf.Tree.init(dist_git_extract)
              elif dist_git_remove_fmf_root:
                  if not self.opt('dry'):
                      shutil.rmtree(os.path.join(
                          dist_git_extract or top_fmf_root, '.fmf'))
                  if not dist_git_extract:
                      dist_git_extract = sourcedir
  
              # Now can safely default to top_fmf_root
              if not dist_git_extract:
                  dist_git_extract = top_fmf_root
  
              # Now copy dist_git_extract into tests
              if not self.opt('dry'):
                  tmt.utils.copytree(
                      dist_git_extract,
                      self.testdir,
                      symlinks=True,
                      dirs_exist_ok=True)
  
          # Adjust path and optionally show
          if path is None or path == '.':
              path = ''
          else:
              self.info('path', path, 'green')
  
          # Prepare the whole tree path and test path prefix
          tree_path = os.path.join(self.testdir, path.lstrip('/'))
          if not os.path.isdir(tree_path) and not self.opt('dry'):
              raise tmt.utils.DiscoverError(
                  f"Metadata tree path '{path}' not found.")
          prefix_path = os.path.join('/tests', path.lstrip('/'))
  
          # Show filters and test names if provided
          # Check the 'test --filter' option first, then from discover
          filters = list(tmt.base.Test._opt('filters') or self.get('filter', []))
          for filter_ in filters:
              self.info('filter', filter_, 'green')
          # Names of tests selected by --test option
          names = self.get('test', [])
          if names:
              self.info('tests', fmf.utils.listed(names), 'green')
  
          # Check the 'test --link' option first, then from discover
          raw_link_needles = cast(List[str], tmt.Test._opt('links', []) or self.get('link', []))
          link_needles = [tmt.base.LinkNeedle.from_spec(
              raw_needle) for raw_needle in raw_link_needles]
  
          for link_needle in link_needles:
              self.info('link', link_needle, 'green')
  
          excludes = list(tmt.base.Test._opt('exclude')
                          or self.get('exclude', []))
  
          # Filter only modified tests if requested
          modified_only = self.get('modified-only')
          modified_url = self.get('modified-url')
          if modified_url:
              self.info('modified-url', modified_url, 'green')
              self.debug(f"Fetch also '{modified_url}' as 'reference'.")
              self.run(['git', 'remote', 'add', 'reference', modified_url],
                       cwd=self.testdir)
              self.run(['git', 'fetch', 'reference'], cwd=self.testdir)
          if modified_only:
              modified_ref = self.get(
                  'modified-ref', tmt.utils.default_branch(self.testdir))
              self.info('modified-ref', modified_ref, 'green')
              output = self.run(
                  ['git', 'log', '--format=', '--stat', '--name-only',
                   f"{modified_ref}..HEAD"], cwd=self.testdir)[0]
              if output:
                  directories = [os.path.dirname(name) for name in output.split('\n')]
                  modified = set(f"^/{re.escape(name)}" for name in directories if name)
                  self.debug(f"Limit to modified test dirs: {modified}", level=3)
                  names.extend(modified)
  
          # Initialize the metadata tree, search for available tests
          self.debug(f"Check metadata tree in '{tree_path}'.")
          if self.opt('dry'):
              self._tests = []
              return
          tree = tmt.Tree(path=tree_path, context=self.step.plan._fmf_context())
          self._tests = tree.tests(
              filters=filters,
              names=names,
              conditions=["manual is False"],
              unique=False,
              links=link_needles,
              excludes=excludes)
  
          # Prefix tests and handle library requires
          for test in self._tests:
              # Prefix test path with 'tests' and possible 'path' prefix
              test.path = os.path.join(prefix_path, test.path.lstrip('/'))
              # Check for possible required beakerlib libraries
              if test.require or test.recommend:
                  test.require, test.recommend, _ = tmt.beakerlib.dependencies(
                      test.require, test.recommend, parent=self)
  
          # Add TMT_SOURCE_DIR variable for each test
          if dist_git_source:
              for test in self._tests:
                  test.environment['TMT_SOURCE_DIR'] = sourcedir
  
      def tests(self) -> List[tmt.base.Test]:
          """ Return all discovered tests """
          return self._tests
  
files: tmt/steps/discover/fmf.py
files: import dataclasses
  import glob
  import os
  import re
  import shutil
  import subprocess
  from typing import List, Optional, cast
  
  import click
  import fmf
  
  import tmt
  import tmt.base
  import tmt.beakerlib
  import tmt.options
  import tmt.steps
  import tmt.steps.discover
  import tmt.utils
  
  
  @dataclasses.dataclass
  class DiscoverFmfStepData(tmt.steps.discover.DiscoverStepData):
      url: Optional[str] = None
      ref: Optional[str] = None
      path: Optional[str] = None
      test: List[str] = dataclasses.field(default_factory=list)
      link: List[str] = dataclasses.field(default_factory=list)
      filter: List[str] = dataclasses.field(default_factory=list)
      modified_only: bool = False
      modified_url: Optional[str] = None
      modified_ref: Optional[str] = None
      dist_git_init: bool = False
      dist_git_remove_fmf_root: bool = False
      dist_git_merge: bool = False
      dist_git_extract: Optional[str] = None
      fmf_id: bool = False
      exclude: List[str] = dataclasses.field(default_factory=list)
  
      # Legacy fields
      repository: Optional[str] = None
      revision: Optional[str] = None
  
      _normalize_test = tmt.utils.NormalizeKeysMixin._normalize_string_list
      _normalize_link = tmt.utils.NormalizeKeysMixin._normalize_string_list
      _normalize_filter = tmt.utils.NormalizeKeysMixin._normalize_string_list
      _normalize_exclude = tmt.utils.NormalizeKeysMixin._normalize_string_list
  
      def post_normalization(self, raw_data: tmt.steps._RawStepData,
                             logger: tmt.utils.Common) -> None:
          super().post_normalization(raw_data, logger)
  
          if self.repository:
              self.url = self.repository
  
          if self.revision:
              self.ref = self.revision
  
  
  @tmt.steps.provides_method('fmf')
  class DiscoverFmf(tmt.steps.discover.DiscoverPlugin):
      """
      Discover available tests from fmf metadata
  
      By default all available tests from the current repository are used
      so the minimal configuration looks like this:
  
          discover:
              how: fmf
  
      Full config example:
  
          discover:
              how: fmf
              url: https://github.com/teemtee/tmt
              ref: main
              path: /fmf/root
              test: /tests/basic
              filter: 'tier: 1'
  
      If no 'ref' is provided, the default branch from the origin is used.
  
      For DistGit repo one can extract source tarball first and discover
      tests from it by using 'distgit-source: true'. It can be used
      together with 'ref', 'path' and 'url', However 'ref' is not possible
      without using 'url'.
  
          discover:
              how: fmf
              dist-git-source: true
  
      Related config options (all optional):
      * dist-git-merge - set to True if you want to copy in extracted
        sources to the local repo
      * dist-git-init - set to True and 'fmf init' will be called inside
        extracted sources (at dist-git-extract or top directory)
      * dist-git-extract - directory (glob supported) to copy from
        extracted sources (defaults to inner fmf-root)
      * dist-git-remove-fmf-root - set to True to remove fmf root from
        extracted sources
  
      Selecting tests containing specified link is possible using 'link'
      option accepting RELATION:TARGET format of values. Regular
      expressions are supported for both relation and target part of the
      value. Relation can be omitted to target match any relation.
  
          discover:
              how: fmf
              link: verifies:.*issue/850$
  
      It is also possible to limit tests only to those that have changed
      in git since a given revision. This can be particularly useful when
      testing changes to tests themselves (e.g. in a pull request CI).
  
      Related config options (all optional):
      * modified-only - set to True if you want to filter modified tests
      * modified-url - fetched as "reference" remote in the test dir
      * modified-ref - the ref to compare against
  
      Example to compare local repo against upstream 'main' branch:
  
          discover:
              how: fmf
              modified-only: True
              modified-url: https://github.com/teemtee/tmt
              modified-ref: reference/main
  
      Note that internally the modified tests are appended to the list
      specified via 'test', so those tests will also be selected even if
      not modified.
      """
  
      _data_class = DiscoverFmfStepData
  
      REF_OPTION = click.option(
          '-r', '--ref', metavar='REVISION',
          help='Branch, tag or commit specifying the git revision.')
      TEST_OPTION = click.option(
          '-t', '--test', metavar='NAMES', multiple=True,
          help='Select tests by name.')
      FILTER_OPTION = click.option(
          '-F', '--filter', metavar='FILTERS', multiple=True,
          help='Include only tests matching the filter.')
      EXCLUDE_OPTION = click.option(
          '-x', '--exclude', metavar='[REGEXP]', multiple=True,
          help="Exclude a regular expression from search result.")
  
      # Options which require .git to be present for their functionality
      _REQUIRES_GIT = (
          "ref",
          "modified-url",
          "modified-only",
          "fmf-id",
          )
  
      @classmethod
      def options(cls, how: Optional[str] = None) -> List[tmt.options.ClickOptionDecoratorType]:
          """ Prepare command line options for given method """
          options = super().options(how)
          options[:0] = [
              click.option(
                  '-u', '--url', metavar='REPOSITORY',
                  help='URL of the git repository with fmf metadata.'),
              cls.REF_OPTION,
              click.option(
                  '--modified-url', metavar='REPOSITORY',
                  help='URL of the reference git repository with fmf metadata.'),
              click.option(
                  '--modified-ref', metavar='REVISION',
                  help='Branch, tag or commit specifying the reference git '
                  'revision (if not provided, the default branch is used).'),
              click.option(
                  '-m', '--modified-only', is_flag=True,
                  help='If set, select only tests modified '
                  'since reference revision.'),
              click.option(
                  '-p', '--path', metavar='ROOT',
                  help='Path to the metadata tree root.'),
              cls.TEST_OPTION,
              click.option(
                  '--link', metavar="RELATION:TARGET", multiple=True,
                  help="Filter by linked objects (regular expressions are "
                       "supported for both relation and target)."),
              cls.FILTER_OPTION,
              cls.EXCLUDE_OPTION,
              click.option(
                  '--fmf-id', default=False, is_flag=True,
                  help='Show fmf identifiers for tests discovered in plan.'),
              click.option(
                  '--sync-repo', default=False, is_flag=True,
                  help='Force the sync of the whole git repo. By default, the '
                       'repo is copied only if the used options require it.'),
              click.option(
                  '--dist-git-init', is_flag=True,
                  help='Initialize fmf root inside extracted sources '
                       '(at dist-git-extract or top directory).'),
              click.option(
                  '--dist-git-remove-fmf-root', is_flag=True,
                  help='Remove fmf root from extracted source '
                       '(top one or selected by copy-path, '
                       'happens before dist-git-extract.'),
              click.option(
                  '--dist-git-merge', is_flag=True,
                  help='Merge copied sources and plan fmf root.'),
              click.option(
                  '--dist-git-extract',
                  help='What to copy from extracted sources, globbing is '
                       'supported. Defaults to the top fmf root if it is '
                       'present, otherwise top directory (shortcut "/").'
                  ),
              ]
          return options
  
      @property
      def is_in_standalone_mode(self) -> bool:
          """ Enable standalone mode when listing fmf ids """
          if self.opt('fmf_id'):
              return True
          return super().is_in_standalone_mode
  
      def go(self) -> None:
          """ Discover available tests """
          super(DiscoverFmf, self).go()
  
          # Check url and path, prepare test directory
          url = self.get('url')
          path = self.get('path')
          # Save the test directory so that others can reference it
          ref = self.get('ref')
          assert self.workdir is not None
          self.testdir = os.path.join(self.workdir, 'tests')
          sourcedir = os.path.join(self.workdir, 'source')
          dist_git_source = self.get('dist-git-source', False)
          dist_git_merge = self.get('dist-git-merge', False)
          dist_git_init = self.get('dist-git-init', False)
          dist_git_extract = self.get('dist-git-extract', None)
          dist_git_remove_fmf_root = self.get('dist-git-remove-fmf-root', False)
  
          # Self checks
          if dist_git_source and not dist_git_merge and (ref or url):
              raise tmt.utils.DiscoverError(
                  "Cannot manipulate with dist-git without "
                  "the `--dist-git-merge` option.")
  
          def get_git_root(dir: str) -> str:
              stdout, _ = self.run(["git", "rev-parse", "--show-toplevel"], cwd=dir, dry=True)
              assert stdout is not None
              return stdout.strip("\n")
  
          # Raise an exception if --fmf-id uses w/o url and git root
          # doesn't exist for discovered plan
          if self.opt('fmf_id'):
              def assert_git_url(plan_name: Optional[str] = None) -> None:
                  try:
                      subprocess.run(
                          'git rev-parse --show-toplevel'.split(),
                          stdout=subprocess.PIPE,
                          stderr=subprocess.DEVNULL,
                          check=True)
                  except subprocess.CalledProcessError:
                      raise tmt.utils.DiscoverError(
                          f"`tmt run discover --fmf-id` without `url` option in "
                          f"plan `{plan_name}` can be used only within"
                          f" git repo.")
              # It covers only one case, when there is:
              # 1) no --url on CLI
              # 2) plan w/o url exists in test run
              if not self.opt('url'):
                  try:
                      fmf_tree = fmf.Tree(os.getcwd())
                  except fmf.utils.RootError:
                      raise tmt.utils.DiscoverError(
                          "No metadata found in the current directory. "
                          "Use 'tmt init' to get started.")
                  for i, attr in enumerate(fmf_tree.climb()):
                      try:
                          plan_url = attr.data.get('discover').get('url')
                          plan_name = attr.name
                          if not plan_url:
                              assert_git_url(plan_name)
                      except AttributeError:
                          pass
              # All other cases are covered by this condition
              if not url:
                  assert_git_url(self.step.plan.name)
  
          # Clone provided git repository (if url given) with disabled
          # prompt to ignore possibly missing or private repositories
          if url:
              self.info('url', url, 'green')
              self.debug(f"Clone '{url}' to '{self.testdir}'.")
              # Shallow clone to speed up testing and
              # minimize data transfers if ref is not provided
              tmt.utils.git_clone(
                  url, self.testdir, self, env={"GIT_ASKPASS": "echo"}, shallow=ref is None)
              git_root = self.testdir
          # Copy git repository root to workdir
          else:
              fmf_root = path or self.step.plan.my_run.tree.root
              requires_git = self.opt('sync-repo') or any(
                  self.get(opt) for opt in self._REQUIRES_GIT)
              # Path for distgit sources cannot be checked until the
              # tarball is extracted
              if path and not os.path.isdir(path) and not dist_git_source:
                  raise tmt.utils.DiscoverError(
                      f"Provided path '{path}' is not a directory.")
              if dist_git_source:
                  # Ensure we're in a git repo when extracting dist-git sources
                  try:
                      git_root = get_git_root(self.step.plan.my_run.tree.root)
                  except tmt.utils.RunError:
                      raise tmt.utils.DiscoverError(
                          f"{self.step.plan.my_run.tree.root} is not a git repo")
              else:
                  if fmf_root is None:
                      raise tmt.utils.DiscoverError(
                          "No metadata found in the current directory.")
                  # Check git repository root (use fmf root if not found)
                  try:
                      git_root = get_git_root(fmf_root)
                  except tmt.utils.RunError:
                      self.debug(f"Git root not found, using '{fmf_root}.'")
                      git_root = fmf_root
                  # Set path to relative path from the git root to fmf root
                  path = os.path.relpath(
                      fmf_root, git_root if requires_git else fmf_root)
  
              # And finally copy the git/fmf root directory to testdir
              # (for dist-git case only when merge explicitly requested)
              directory = git_root if requires_git else fmf_root
              self.info('directory', directory, 'green')
              if not dist_git_source or dist_git_merge:
                  self.debug(f"Copy '{directory}' to '{self.testdir}'.")
                  if not self.opt('dry'):
                      shutil.copytree(directory, self.testdir, symlinks=True)
  
          # Checkout revision if requested
          if ref:
              self.info('ref', ref, 'green')
              self.debug(f"Checkout ref '{ref}'.")
              self.run(
                  ['git', 'checkout', '-f', str(ref)],
                  cwd=self.testdir)
  
          # Show current commit hash if inside a git repository
          if os.path.isdir(self.testdir):
              try:
                  hash_, _ = self.run(["git", "rev-parse", "--short", "HEAD"],
                                      cwd=self.testdir)
                  if hash_ is not None:
                      self.verbose('hash', hash_.strip(), 'green')
              except (tmt.utils.RunError, AttributeError):
                  pass
  
          # Fetch and extract distgit sources
          if dist_git_source:
              try:
                  self.extract_distgit_source(
                      git_root, sourcedir, self.get('dist-git-type'))
              except Exception as error:
                  raise tmt.utils.DiscoverError(
                      "Failed to process 'dist-git-source'.", original=error)
  
              # Check what should be extracted from the sources
              if dist_git_extract:
                  if dist_git_extract == '/':
                      dist_git_extract = sourcedir
                  else:
                      try:
                          dist_git_extract = glob.glob(os.path.join(
                              sourcedir, dist_git_extract.lstrip('/')))[0]
                      except IndexError:
                          raise tmt.utils.DiscoverError(
                              f"Couldn't glob '{dist_git_extract}' "
                              f"within extracted sources.")
  
              # Check sources for the fmf root, copy git root if not found
              if not dist_git_init and not dist_git_extract:
                  try:
                      top_fmf_root = tmt.utils.find_fmf_root(sourcedir)[0]
                  except tmt.utils.MetadataError:
                      dist_git_extract = sourcedir
                      if not dist_git_merge:
                          self.warn(
                              "Extracted sources do not contain fmf root, "
                              "merging with plan data. Avoid this warning by "
                              "explicit use of the '--dist-git-merge' option.")
                          self.debug(f"Copy '{git_root}' to '{self.testdir}'.")
                          if not self.opt('dry'):
                              shutil.copytree(
                                  git_root, self.testdir, symlinks=True)
  
              # Initialize or remove fmf root
              if dist_git_init:
                  if not dist_git_extract:
                      dist_git_extract = sourcedir
                  if not self.opt('dry'):
                      fmf.Tree.init(dist_git_extract)
              elif dist_git_remove_fmf_root:
                  if not self.opt('dry'):
                      shutil.rmtree(os.path.join(
                          dist_git_extract or top_fmf_root, '.fmf'))
                  if not dist_git_extract:
                      dist_git_extract = sourcedir
  
              # Now can safely default to top_fmf_root
              if not dist_git_extract:
                  dist_git_extract = top_fmf_root
  
              # Now copy dist_git_extract into tests
              if not self.opt('dry'):
                  tmt.utils.copytree(
                      dist_git_extract,
                      self.testdir,
                      symlinks=True,
                      dirs_exist_ok=True)
  
          # Adjust path and optionally show
          if path is None or path == '.':
              path = ''
          else:
              self.info('path', path, 'green')
  
          # Prepare the whole tree path and test path prefix
          tree_path = os.path.join(self.testdir, path.lstrip('/'))
          if not os.path.isdir(tree_path) and not self.opt('dry'):
              raise tmt.utils.DiscoverError(
                  f"Metadata tree path '{path}' not found.")
          prefix_path = os.path.join('/tests', path.lstrip('/'))
  
          # Show filters and test names if provided
          # Check the 'test --filter' option first, then from discover
          filters = list(tmt.base.Test._opt('filters') or self.get('filter', []))
          for filter_ in filters:
              self.info('filter', filter_, 'green')
          # Names of tests selected by --test option
          names = self.get('test', [])
          if names:
              self.info('tests', fmf.utils.listed(names), 'green')
  
          # Check the 'test --link' option first, then from discover
          raw_link_needles = cast(List[str], tmt.Test._opt('links', []) or self.get('link', []))
          link_needles = [tmt.base.LinkNeedle.from_raw(
              raw_needle) for raw_needle in raw_link_needles]
  
          for link_needle in link_needles:
              self.info('link', link_needle, 'green')
  
          excludes = list(tmt.base.Test._opt('exclude')
                          or self.get('exclude', []))
  
          # Filter only modified tests if requested
          modified_only = self.get('modified-only')
          modified_url = self.get('modified-url')
          if modified_url:
              self.info('modified-url', modified_url, 'green')
              self.debug(f"Fetch also '{modified_url}' as 'reference'.")
              self.run(['git', 'remote', 'add', 'reference', modified_url],
                       cwd=self.testdir)
              self.run(['git', 'fetch', 'reference'], cwd=self.testdir)
          if modified_only:
              modified_ref = self.get(
                  'modified-ref', tmt.utils.default_branch(self.testdir))
              self.info('modified-ref', modified_ref, 'green')
              output = self.run(
                  ['git', 'log', '--format=', '--stat', '--name-only',
                   f"{modified_ref}..HEAD"], cwd=self.testdir)[0]
              if output:
                  directories = [os.path.dirname(name) for name in output.split('\n')]
                  modified = set(f"^/{re.escape(name)}" for name in directories if name)
                  self.debug(f"Limit to modified test dirs: {modified}", level=3)
                  names.extend(modified)
  
          # Initialize the metadata tree, search for available tests
          self.debug(f"Check metadata tree in '{tree_path}'.")
          if self.opt('dry'):
              self._tests = []
              return
          tree = tmt.Tree(path=tree_path, context=self.step.plan._fmf_context())
          self._tests = tree.tests(
              filters=filters,
              names=names,
              conditions=["manual is False"],
              unique=False,
              links=link_needles,
              excludes=excludes)
  
          # Prefix tests and handle library requires
          for test in self._tests:
              # Prefix test path with 'tests' and possible 'path' prefix
              test.path = os.path.join(prefix_path, test.path.lstrip('/'))
              # Check for possible required beakerlib libraries
              if test.require or test.recommend:
                  test.require, test.recommend, _ = tmt.beakerlib.dependencies(
                      test.require, test.recommend, parent=self)
  
          # Add TMT_SOURCE_DIR variable for each test
          if dist_git_source:
              for test in self._tests:
                  test.environment['TMT_SOURCE_DIR'] = sourcedir
  
      def tests(self) -> List[tmt.base.Test]:
          """ Return all discovered tests """
          return self._tests
  
files: tmt/steps/discover/fmf.py
files: import copy
  import dataclasses
  import os
  from typing import (TYPE_CHECKING, Any, Dict, List, Optional, Type, TypeVar,
                      Union, cast)
  
  import click
  import fmf
  
  import tmt
  import tmt.base
  import tmt.steps
  import tmt.steps.discover
  import tmt.utils
  
  if TYPE_CHECKING:
      import tmt.base
  
  
  T = TypeVar('T', bound='TestDescription')
  
  
  @dataclasses.dataclass
  class TestDescription(
          tmt.utils.SpecBasedContainer,
          tmt.utils.NormalizeKeysMixin,
          tmt.utils.SerializableContainer):
      """
      Keys necessary to describe a shell-based test.
  
      Provides basic functionality for tansition between "raw" step data representation,
      which consists of keys and values given by fmf tree and CLI options, and this
      container representation for internal use.
      """
  
      name: str
  
      # TODO: following keys are copy & pasted from base.Test. It would be much, much better
      # to re-use the definitions from base.Test instead copying them here, but base.Test
      # does not support save/load operations. This is a known issue, introduced by a patch
      # transitioning step data to data classes, it is temporary, and it will be fixed as
      # soon as possible - nobody want's to keep two very same lists of attributes.
      test: str
  
      # Core attributes (supported across all levels)
      summary: Optional[str] = None
      description: Optional[str] = None
      enabled: bool = True
      # TODO: ugly circular dependency (see tmt.base.DEFAULT_ORDER)
      order: int = 50
      link: Optional['tmt.base.Links'] = None
      id: Optional[str] = None
      tag: List[str] = dataclasses.field(default_factory=list)
      tier: Optional[str] = None
      adjust: Optional[List['tmt.base._RawAdjustRule']] = None
  
      # Basic test information
      contact: List[str] = dataclasses.field(default_factory=list)
      component: List[str] = dataclasses.field(default_factory=list)
  
      # Test execution data
      path: Optional[str] = None
      framework: Optional[str] = None
      manual: bool = False
      require: List[Union[str, 'tmt.base.FmfId']] = dataclasses.field(default_factory=list)
      recommend: List[str] = dataclasses.field(default_factory=list)
      environment: tmt.utils.EnvironmentType = dataclasses.field(default_factory=dict)
      duration: str = '1h'
      result: str = 'respect'
  
      # We can't even re-use normalization callbacks from tmt.base because of the cyclic import :/
      _normalize_tag = tmt.utils.LoadFmfKeysMixin._normalize_string_list
      _normalize_contact = tmt.utils.LoadFmfKeysMixin._normalize_string_list
      _normalize_component = tmt.utils.LoadFmfKeysMixin._normalize_string_list
      _normalize_recommend = tmt.utils.LoadFmfKeysMixin._normalize_string_list
  
      def _normalize_order(self, value: Optional[int]) -> int:
          if value is None:
              # TODO: ugly circular dependency (see tmt.base.DEFAULT_ORDER)
              return 50
          return int(value)
  
      def _normalize_link(self, value: 'tmt.base._RawLinks') -> 'tmt.base.Links':
          import tmt.base
  
          return tmt.base.Links(data=value)
  
      def _normalize_adjust(self,
                            value: Optional[Union[
                                'tmt.base._RawAdjustRule',
                                List['tmt.base._RawAdjustRule']]]
                            ) -> List['tmt.base._RawAdjustRule']:
          if value is None:
              return []
          return [value] if not isinstance(value, list) else value
  
      def _normalize_tier(self, value: Optional[Union[int, str]]) -> Optional[str]:
          if value is None:
              return None
          return str(value)
  
      def _normalize_require(
              self, value: Optional['tmt.base._RawRequire']) -> List[Union[str, 'tmt.base.FmfId']]:
          if value is None:
              return []
          # TODO: remove ignore when base.py becomes annotated
          return [value] if isinstance(value, str) else value  # type: ignore[no-any-return]
  
      # Our own implementation, parent uses `name` and `how`, and tests don't have any `how`.
      @classmethod
      def from_spec(  # type: ignore[override]
              cls: Type[T],
              raw_data: Dict[str, Any],
              logger: tmt.utils.Common
              ) -> T:
          """ Convert from a specification file or from a CLI option """
  
          data = cls(name=raw_data['name'], test=raw_data['test'])
          data._load_keys(raw_data, cls.__name__, logger)
  
          return data
  
      def to_spec(self) -> Dict[str, Any]:
          """ Convert to a form suitable for saving in a specification file """
  
          data = super().to_spec()
          data['link'] = self.link.to_spec() if self.link else None
          data['require'] = [
              require if isinstance(require, str) else require.to_spec()
              for require in self.require
              ]
  
          return data
  
      def to_serialized(self) -> Dict[str, Any]:
          """ Convert to a form suitable for saving in a file """
  
          data = super().to_serialized()
  
          # Using `to_spec()` on purpose: `Links` does not provide serialization
          # methods, because specification of links is already good enough. We
          # can use existing `to_spec()` method, and undo it with a simple
          # `Links(...)` call.
          data['link'] = self.link.to_spec() if self.link else None
          data['require'] = [
              require if isinstance(require, str) else require.to_serialized()
              for require in self.require
              ]
  
          return data
  
      @classmethod
      def from_serialized(cls, serialized: Dict[str, Any]) -> 'TestDescription':
          """ Convert from a serialized form loaded from a file """
  
          obj = super().from_serialized(serialized)
          obj.link = tmt.base.Links(serialized['link'])
          obj.require = [
              require if isinstance(require, str) else tmt.base.FmfId.from_serialized(require)
              for require in serialized['require']
              ]
  
          return obj
  
  
  @dataclasses.dataclass
  class DiscoverShellData(tmt.steps.discover.DiscoverStepData):
      tests: List[TestDescription] = dataclasses.field(default_factory=list)
  
      def _normalize_tests(self, value: List[Dict[str, Any]]
                           ) -> List[TestDescription]:
          return [TestDescription.from_spec(raw_datum, tmt.utils.Common()) for raw_datum in value]
  
      def to_serialized(self) -> Dict[str, Any]:
          """ Convert to a form suitable for saving in a file """
  
          serialized = super().to_serialized()
  
          serialized['tests'] = [test.to_serialized() for test in self.tests]
  
          return serialized
  
      @classmethod
      def from_serialized(cls, serialized: Dict[str, Any]) -> 'DiscoverShellData':
          """ Convert from a serialized form loaded from a file """
  
          obj = super().from_serialized(serialized)
  
          obj.tests = [TestDescription.from_serialized(
              serialized_test) for serialized_test in serialized['tests']]
  
          return obj
  
  
  @tmt.steps.provides_method('shell')
  class DiscoverShell(tmt.steps.discover.DiscoverPlugin):
      """
      Use provided list of shell script tests
  
      List of test cases to be executed can be defined manually directly
      in the plan as a list of dictionaries containing test name, actual
      test script and optionally a path to the test. Example config:
  
      discover:
          how: shell
          tests:
          - name: /help/main
            test: tmt --help
          - name: /help/test
            test: tmt test --help
          - name: /help/smoke
            test: ./smoke.sh
            path: /tests/shell
  
      For DistGit repo one can extract source tarball and use its code.
      It is extracted to TMT_SOURCE_DIR however no patches are applied
      (only source tarball is extracted).
  
      discover:
          how: shell
          dist-git-source: true
          tests:
          - name: /upstream
            test: cd $TMT_SOURCE_DIR/*/tests && make test
      """
  
      _data_class = DiscoverShellData
  
      _tests: List[tmt.base.Test] = []
  
      def show(self, keys: Optional[List[str]] = None) -> None:
          """ Show config details """
          super().show([])
          tests = cast(List[TestDescription], self.get('tests'))
          if tests:
              test_names = [test.name for test in tests]
              click.echo(tmt.utils.format('tests', test_names))
  
      def go(self) -> None:
          """ Discover available tests """
          super(DiscoverShell, self).go()
          tests = fmf.Tree(dict(summary='tests'))
  
          # dist-git related
          assert self.workdir is not None
          sourcedir = os.path.join(self.workdir, 'source')
          dist_git_source = self.get('dist-git-source', False)
  
          # Check and process each defined shell test
          for data in cast(DiscoverShellData, self.data).tests:
              # Create data copy (we want to keep original data for save()
              data = copy.deepcopy(data)
              # Extract name, make sure it is present
              # TODO: can this ever happen? With annotations, `name: str` and `test: str`, nothing
              # should ever assign `None` there and pass the test.
              if not data.name:
                  raise tmt.utils.SpecificationError(
                      f"Missing test name in '{self.step.plan.name}'.")
              # Make sure that the test script is defined
              if not data.test:
                  raise tmt.utils.SpecificationError(
                      f"Missing test script in '{self.step.plan.name}'.")
              # Prepare path to the test working directory (tree root by default)
              data.path = f"/tests{data.path}" if data.path else '/tests'
              # Apply default test duration unless provided
              if not data.duration:
                  data.duration = tmt.base.DEFAULT_TEST_DURATION_L2
              # Add source dir path variable
              if dist_git_source:
                  data.environment['TMT_SOURCE_DIR'] = sourcedir
              # Create a simple fmf node, adjust its name
              tests.child(data.name, data.to_spec())
  
          # Symlink tests directory to the plan work tree
          testdir = os.path.join(self.workdir, "tests")
          relative_path = os.path.relpath(self.step.plan.worktree, self.workdir)
          os.symlink(relative_path, testdir)
  
          if dist_git_source:
              try:
                  run_result = self.run(
                      ["git", "rev-parse", "--show-toplevel"],
                      cwd=self.step.plan.my_run.tree.root,
                      dry=True)[0]
                  assert run_result is not None
                  git_root = run_result.strip('\n')
              except tmt.utils.RunError:
                  raise tmt.utils.DiscoverError(
                      f"Directory '{self.step.plan.my_run.tree.root}' "
                      f"is not a git repository.")
              try:
                  self.extract_distgit_source(
                      git_root, sourcedir, self.get('dist-git-type'))
              except Exception as error:
                  raise tmt.utils.DiscoverError(
                      "Failed to process 'dist-git-source'.", original=error)
  
          # Use a tmt.Tree to apply possible command line filters
          tests = tmt.Tree(tree=tests).tests(conditions=["manual is False"])
          self._tests = tests
  
      def tests(self) -> List[tmt.base.Test]:
          return self._tests
  
files: tmt/steps/discover/shell.py
files: import copy
  import dataclasses
  import os
  from typing import (TYPE_CHECKING, Any, Dict, List, Optional, Type, TypeVar,
                      Union, cast)
  
  import click
  import fmf
  
  import tmt
  import tmt.base
  import tmt.steps
  import tmt.steps.discover
  import tmt.utils
  
  if TYPE_CHECKING:
      import tmt.base
  
  
  T = TypeVar('T', bound='TestDescription')
  
  
  @dataclasses.dataclass
  class TestDescription(tmt.utils.NormalizeKeysMixin, tmt.utils.SerializableContainer):
      """
      Keys necessary to describe a shell-based test.
  
      Provides basic functionality for tansition between "raw" step data representation,
      which consists of keys and values given by fmf tree and CLI options, and this
      container representation for internal use.
      """
  
      name: str
  
      # TODO: following keys are copy & pasted from base.Test. It would be much, much better
      # to re-use the definitions from base.Test instead copying them here, but base.Test
      # does not support save/load operations. This is a known issue, introduced by a patch
      # transitioning step data to data classes, it is temporary, and it will be fixed as
      # soon as possible - nobody want's to keep two very same lists of attributes.
      test: str
  
      # Core attributes (supported across all levels)
      summary: Optional[str] = None
      description: Optional[str] = None
      enabled: bool = True
      # TODO: ugly circular dependency (see tmt.base.DEFAULT_ORDER)
      order: int = 50
      link: Optional['tmt.base.Links'] = None
      id: Optional[str] = None
      tag: List[str] = dataclasses.field(default_factory=list)
      tier: Optional[str] = None
      adjust: Optional[List['tmt.base._RawAdjustRule']] = None
  
      # Basic test information
      contact: List[str] = dataclasses.field(default_factory=list)
      component: List[str] = dataclasses.field(default_factory=list)
  
      # Test execution data
      path: Optional[str] = None
      framework: Optional[str] = None
      manual: bool = False
      require: List[Union[str, 'tmt.base.FmfId']] = dataclasses.field(default_factory=list)
      recommend: List[str] = dataclasses.field(default_factory=list)
      environment: tmt.utils.EnvironmentType = dataclasses.field(default_factory=dict)
      duration: str = '1h'
      result: str = 'respect'
  
      # We can't even re-use normalization callbacks from tmt.base because of the cyclic import :/
      _normalize_tag = tmt.utils.LoadFmfKeysMixin._normalize_string_list
      _normalize_contact = tmt.utils.LoadFmfKeysMixin._normalize_string_list
      _normalize_component = tmt.utils.LoadFmfKeysMixin._normalize_string_list
      _normalize_recommend = tmt.utils.LoadFmfKeysMixin._normalize_string_list
  
      def _normalize_order(self, value: Optional[int]) -> int:
          if value is None:
              # TODO: ugly circular dependency (see tmt.base.DEFAULT_ORDER)
              return 50
          return int(value)
  
      def _normalize_link(self, value: 'tmt.base._RawLinks') -> 'tmt.base.Links':
          import tmt.base
  
          return tmt.base.Links(data=value)
  
      def _normalize_adjust(self,
                            value: Optional[Union[
                                'tmt.base._RawAdjustRule',
                                List['tmt.base._RawAdjustRule']]]
                            ) -> List['tmt.base._RawAdjustRule']:
          if value is None:
              return []
          return [value] if not isinstance(value, list) else value
  
      def _normalize_tier(self, value: Optional[Union[int, str]]) -> Optional[str]:
          if value is None:
              return None
          return str(value)
  
      def _normalize_require(
              self, value: Optional['tmt.base._RawRequire']) -> List[Union[str, 'tmt.base.FmfId']]:
          if value is None:
              return []
          # TODO: remove ignore when base.py becomes annotated
          return [value] if isinstance(value, str) else value  # type: ignore[no-any-return]
  
      # Our own implementation, parent uses `name` and `how`, and tests don't have any `how`.
      @classmethod
      def from_raw(cls: Type[T], raw_data: Dict[str, Any], logger: tmt.utils.Common) -> T:
          """
          Unserialize step data instance from its a raw representation.
          """
  
          data = cls(name=raw_data['name'], test=raw_data['test'])
          data._load_keys(raw_data, cls.__name__, logger)
  
          return data
  
      def to_raw(self) -> Dict[str, Any]:
          data = super().to_dict()
          data['link'] = self.link.to_raw() if self.link else None
          data['require'] = [
              require if isinstance(require, str) else require.to_raw()
              for require in self.require
              ]
  
          return data
  
      def to_serialized(self) -> Dict[str, Any]:
          data = super().to_serialized()
  
          data['link'] = self.link.to_raw() if self.link else None
          data['require'] = [
              require if isinstance(require, str) else require.to_serialized()
              for require in self.require
              ]
  
          return data
  
      @classmethod
      def from_serialized(cls, serialized: Dict[str, Any]) -> 'TestDescription':
          obj = super().from_serialized(serialized)
          obj.link = tmt.base.Links(serialized['link'])
          obj.require = [
              require if isinstance(require, str) else tmt.base.FmfId.from_serialized(require)
              for require in serialized['require']
              ]
  
          return obj
  
  
  @dataclasses.dataclass
  class DiscoverShellData(tmt.steps.discover.DiscoverStepData):
      tests: List[TestDescription] = dataclasses.field(default_factory=list)
  
      def _normalize_tests(self, value: List[Dict[str, Any]]
                           ) -> List[TestDescription]:
          return [TestDescription.from_raw(raw_datum, tmt.utils.Common()) for raw_datum in value]
  
      def to_serialized(self) -> Dict[str, Any]:
          serialized = super().to_serialized()
  
          serialized['tests'] = [test.to_serialized() for test in self.tests]
  
          return serialized
  
      @classmethod
      def from_serialized(cls, serialized: Dict[str, Any]) -> 'DiscoverShellData':
          obj = super().from_serialized(serialized)
  
          obj.tests = [TestDescription.from_serialized(
              serialized_test) for serialized_test in serialized['tests']]
  
          return obj
  
  
  @tmt.steps.provides_method('shell')
  class DiscoverShell(tmt.steps.discover.DiscoverPlugin):
      """
      Use provided list of shell script tests
  
      List of test cases to be executed can be defined manually directly
      in the plan as a list of dictionaries containing test name, actual
      test script and optionally a path to the test. Example config:
  
      discover:
          how: shell
          tests:
          - name: /help/main
            test: tmt --help
          - name: /help/test
            test: tmt test --help
          - name: /help/smoke
            test: ./smoke.sh
            path: /tests/shell
  
      For DistGit repo one can extract source tarball and use its code.
      It is extracted to TMT_SOURCE_DIR however no patches are applied
      (only source tarball is extracted).
  
      discover:
          how: shell
          dist-git-source: true
          tests:
          - name: /upstream
            test: cd $TMT_SOURCE_DIR/*/tests && make test
      """
  
      _data_class = DiscoverShellData
  
      _tests: List[tmt.base.Test] = []
  
      def show(self, keys: Optional[List[str]] = None) -> None:
          """ Show config details """
          super().show([])
          tests = cast(List[TestDescription], self.get('tests'))
          if tests:
              test_names = [test.name for test in tests]
              click.echo(tmt.utils.format('tests', test_names))
  
      def go(self) -> None:
          """ Discover available tests """
          super(DiscoverShell, self).go()
          tests = fmf.Tree(dict(summary='tests'))
  
          # dist-git related
          assert self.workdir is not None
          sourcedir = os.path.join(self.workdir, 'source')
          dist_git_source = self.get('dist-git-source', False)
  
          # Check and process each defined shell test
          for data in cast(DiscoverShellData, self.data).tests:
              # Create data copy (we want to keep original data for save()
              data = copy.deepcopy(data)
              # Extract name, make sure it is present
              # TODO: can this ever happen? With annotations, `name: str` and `test: str`, nothing
              # should ever assign `None` there and pass the test.
              if not data.name:
                  raise tmt.utils.SpecificationError(
                      f"Missing test name in '{self.step.plan.name}'.")
              # Make sure that the test script is defined
              if not data.test:
                  raise tmt.utils.SpecificationError(
                      f"Missing test script in '{self.step.plan.name}'.")
              # Prepare path to the test working directory (tree root by default)
              data.path = f"/tests{data.path}" if data.path else '/tests'
              # Apply default test duration unless provided
              if not data.duration:
                  data.duration = tmt.base.DEFAULT_TEST_DURATION_L2
              # Add source dir path variable
              if dist_git_source:
                  data.environment['TMT_SOURCE_DIR'] = sourcedir
              # Create a simple fmf node, adjust its name
              tests.child(data.name, data.to_raw())
  
          # Symlink tests directory to the plan work tree
          testdir = os.path.join(self.workdir, "tests")
          relative_path = os.path.relpath(self.step.plan.worktree, self.workdir)
          os.symlink(relative_path, testdir)
  
          if dist_git_source:
              try:
                  run_result = self.run(
                      ["git", "rev-parse", "--show-toplevel"],
                      cwd=self.step.plan.my_run.tree.root,
                      dry=True)[0]
                  assert run_result is not None
                  git_root = run_result.strip('\n')
              except tmt.utils.RunError:
                  raise tmt.utils.DiscoverError(
                      f"Directory '{self.step.plan.my_run.tree.root}' "
                      f"is not a git repository.")
              try:
                  self.extract_distgit_source(
                      git_root, sourcedir, self.get('dist-git-type'))
              except Exception as error:
                  raise tmt.utils.DiscoverError(
                      "Failed to process 'dist-git-source'.", original=error)
  
          # Use a tmt.Tree to apply possible command line filters
          tests = tmt.Tree(tree=tests).tests(conditions=["manual is False"])
          self._tests = tests
  
      def tests(self) -> List[tmt.base.Test]:
          return self._tests
  
files: tmt/steps/discover/shell.py
files: # coding: utf-8
  
  import dataclasses
  import datetime
  import os
  import platform
  import re
  import time
  import types
  from typing import Any, List, Optional, cast
  
  import click
  import fmf
  import requests
  
  import tmt
  import tmt.steps
  import tmt.steps.provision
  import tmt.utils
  from tmt.utils import WORKDIR_ROOT, ProvisionError, retry_session
  
  libvirt: Optional[types.ModuleType] = None
  testcloud: Optional[types.ModuleType] = None
  
  
  def import_testcloud() -> None:
      """
      Import testcloud module only when needed
  
      Until we have a separate package for each plugin.
      """
      global testcloud
      global libvirt
      try:
          import libvirt
          import testcloud.image
          import testcloud.instance
          import testcloud.util
      except ImportError:
          raise ProvisionError(
              "Install 'testcloud' to provision using this method.")
  
  
  # Testcloud cache to our tmt's workdir root
  TESTCLOUD_DATA = os.path.join(WORKDIR_ROOT, 'testcloud')
  TESTCLOUD_IMAGES = os.path.join(TESTCLOUD_DATA, 'images')
  
  # Userdata for cloud-init
  USER_DATA = """#cloud-config
  chpasswd:
    list: |
      {user_name}:%s
    expire: false
  users:
    - default
    - name: {user_name}
  ssh_authorized_keys:
    - {public_key}
  ssh_pwauth: true
  disable_root: false
  runcmd:
    - sed -i -e '/^.*PermitRootLogin/s/^.*$/PermitRootLogin yes/'
      -e '/^.*UseDNS/s/^.*$/UseDNS no/'
      -e '/^.*GSSAPIAuthentication/s/^.*$/GSSAPIAuthentication no/'
      /etc/ssh/sshd_config
    - systemctl reload sshd
    - [sh, -c, 'if [ ! -f /etc/systemd/network/20-tc-usernet.network ] &&
    systemctl status systemd-networkd | grep -q "enabled;\\svendor\\spreset:\\senabled";
    then mkdir -p /etc/systemd/network/ &&
    echo "[Match]" >> /etc/systemd/network/20-tc-usernet.network &&
    echo "Name=en*" >> /etc/systemd/network/20-tc-usernet.network &&
    echo "[Network]" >> /etc/systemd/network/20-tc-usernet.network &&
    echo "DHCP=yes" >> /etc/systemd/network/20-tc-usernet.network; fi']
    - [sh, -c, 'if systemctl status systemd-networkd |
    grep -q "enabled;\\svendor\\spreset:\\senabled"; then
    systemctl restart systemd-networkd; fi']
    - [sh, -c, 'if cat /etc/os-release |
    grep -q platform:el8; then systemctl restart sshd; fi']
  """
  
  COREOS_DATA = """variant: fcos
  version: 1.4.0
  passwd:
    users:
      - name: {user_name}
        ssh_authorized_keys:
          - {public_key}
  """
  
  # Libvirt domain XML template related variables
  DOMAIN_TEMPLATE_NAME = 'domain-template.jinja'
  DOMAIN_TEMPLATE_FILE = os.path.join(TESTCLOUD_DATA, DOMAIN_TEMPLATE_NAME)
  DOMAIN_TEMPLATE = """<domain type='{{ virt_type }}' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
    <name>{{ domain_name }}</name>
    <uuid>{{ uuid }}</uuid>
    <memory unit='KiB'>{{ memory }}</memory>
    <currentMemory unit='KiB'>{{ memory }}</currentMemory>
    <vcpu placement='static'>2</vcpu>
    <os>
      <type arch='{{ arch }}' machine='{{ model }}'>hvm</type>
      {{ uefi_loader }}
      <boot dev='hd'/>
    </os>
    {{ cpu }}
    {{ extra_specs }}
    <clock offset='utc'>
      <timer name='rtc' tickpolicy='catchup'/>
      <timer name='pit' tickpolicy='delay'/>
      <timer name='hpet' present='no'/>
    </clock>
    <on_poweroff>destroy</on_poweroff>
    <on_reboot>restart</on_reboot>
    <on_crash>restart</on_crash>
    <devices>
      <emulator>{{ emulator_path }}</emulator>
      <disk type='file' device='disk'>
        <driver name='qemu' type='qcow2' cache='unsafe'/>
        <source file="{{ disk }}"/>
        <target dev='vda' bus='virtio'/>
        {{ boot_drive_address }}
      </disk>
      <disk type='file' device='disk'>
        <driver name='qemu' type='raw'/>
        <source file="{{ seed }}"/>
        <target dev='vdb' bus='virtio'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
      </disk>
      <interface type='{{ network_type }}'>
        <mac address="{{ mac_address }}"/>
        {{ network_source }}
        {{ ip_setup }}
        <model type='virtio'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
      </interface>
      <serial type='pty'>
        <target port='0'/>
      </serial>
      <console type='pty'>
        <target type='serial' port='0'/>
      </console>
      <input type="keyboard" bus="virtio"/>
      <rng model='virtio'>
        <backend model='random'>/dev/urandom</backend>
      </rng>
    </devices>
    {{ qemu_args }}
  </domain>
  """  # noqa: E501
  
  # VM defaults
  DEFAULT_BOOT_TIMEOUT = 60      # seconds
  DEFAULT_CONNECT_TIMEOUT = 60   # seconds
  NON_KVM_ADDITIONAL_WAIT = 10       # seconds
  NON_KVM_TIMEOUT_COEF = 10          # times
  
  # SSH key type, set None for ssh-keygen default one
  SSH_KEYGEN_TYPE = "ecdsa"
  
  DEFAULT_USER = 'root'
  DEFAULT_MEMORY = 2048
  DEFAULT_DISK = 10
  DEFAULT_IMAGE = 'fedora'
  DEFAULT_CONNECTION = 'session'
  DEFAULT_ARCH = platform.machine()
  
  
  @dataclasses.dataclass
  class TestcloudGuestData(tmt.steps.provision.GuestSshData):
      # Override parent class with our defaults
      user: str = DEFAULT_USER
  
      image: str = DEFAULT_IMAGE
      memory: int = DEFAULT_MEMORY
      disk: int = DEFAULT_DISK
      connection: str = DEFAULT_CONNECTION
      arch: str = DEFAULT_ARCH
  
      image_url: Optional[str] = None
      instance_name: Optional[str] = None
  
  
  @dataclasses.dataclass
  class ProvisionTestcloudData(TestcloudGuestData, tmt.steps.StepData):
      pass
  
  
  @tmt.steps.provides_method('virtual.testcloud')
  class ProvisionTestcloud(tmt.steps.provision.ProvisionPlugin):
      """
      Local virtual machine using testcloud
  
      Minimal config which uses the latest fedora image:
  
          provision:
              how: virtual
  
      Here's a full config example:
  
          provision:
              how: virtual
              image: fedora
              user: root
              memory: 2048
  
      As the image use 'fedora' for the latest released Fedora compose,
      'rawhide' for the latest Rawhide compose, short aliases such as
      'fedora-32', 'f-32' or 'f32' for specific release or a full url to
      the qcow2 image for example from:
  
          https://kojipkgs.fedoraproject.org/compose/
  
      Short names are also provided for 'centos', 'centos-stream',
      'debian' and 'ubuntu' (e.g. 'centos-8' or 'c8').
  
      Supported Fedora CoreOS images are:
  
          fedora-coreos
          fedora-coreos-stable
          fedora-coreos-testing
          fedora-coreos-next
  
      Use the full path for images stored on local disk, for example:
  
          /var/tmp/images/Fedora-Cloud-Base-31-1.9.x86_64.qcow2
  
      In addition to the qcow2 format, vagrant boxes can be used as well,
      testcloud will take care of unpacking the image for you.
      """
  
      _data_class = ProvisionTestcloudData
  
      # Guest instance
      _guest = None
  
      @classmethod
      def options(cls, how: Optional[str] = None) -> List[tmt.options.ClickOptionDecoratorType]:
          """ Prepare command line options for testcloud """
          options = super().options(how)
          options[:0] = [
              click.option(
                  '-i', '--image', metavar='IMAGE',
                  help='Select image to be used. Provide a short name, '
                       'full path to a local file or a complete url.'),
              click.option(
                  '-m', '--memory', metavar='MEMORY', type=int,
                  help='Set available memory in MB, 2048 MB by default.'),
              click.option(
                  '-D', '--disk', metavar='MEMORY', type=int,
                  help='Specify disk size in GB, 10 GB by default.'),
              click.option(
                  '-u', '--user', metavar='USER',
                  help='Username to use for all guest operations.'),
              click.option(
                  '-c', '--connection',
                  type=click.Choice(['session', 'system']),
                  help="What session type to use, 'session' by default."),
              click.option(
                  '-a', '--arch',
                  type=click.Choice(['x86_64', 'aarch64', 's390x', 'ppc64le']),
                  help="What architecture to virtualize, host arch by default."),
              ]
          return options
  
      # TODO: Revisit this `type: ignore` once `Guest` becomes a generic type
      def wake(self, data: Optional[TestcloudGuestData] = None) -> None:  # type: ignore[override]
          """ Wake up the plugin, process data, apply options """
          super().wake(data=data)
  
          # Wake up testcloud instance
          if data:
              guest = GuestTestcloud(data, name=self.name, parent=self.step)
              guest.wake()
              self._guest = guest
  
      def go(self) -> None:
          """ Provision the testcloud instance """
          super().go()
  
          # Give info about provided data
          data = TestcloudGuestData(**{
              key: self.get(key)
              for key in TestcloudGuestData.keys()
              })
  
          # Once plan schema is enforced this won't be necessary
          # click enforces int for cmdline and schema validation
          # will make sure 'int' gets from plan data.
          # Another key is 'port' however that is not exposed to the cli
          for int_key in ["memory", "disk", "port"]:
              value = getattr(data, int_key)
              if value is not None:
                  try:
                      setattr(data, int_key, int(value))
                  except ValueError:
                      raise tmt.utils.SpecificationError(
                          f"Value '{value}' cannot be converted to int for '{int_key}' attribute.")
  
          for key, value in data.items():
              if key == 'memory':
                  self.info('memory', f"{value} MB", 'green')
              elif key == 'disk':
                  self.info('disk', f"{value} GB", 'green')
              elif key == 'connection':
                  self.verbose('connection', value, 'green')
              elif key == 'key':
                  if value:
                      self.info('key', fmf.utils.listed(value), 'green')
              elif value is not None:
                  self.info(key, value, 'green')
  
          # Create a new GuestTestcloud instance and start it
          self._guest = GuestTestcloud(data, name=self.name, parent=self.step)
          self._guest.start()
  
      def guest(self) -> Optional[tmt.Guest]:
          """ Return the provisioned guest """
          return self._guest
  
      @classmethod
      def clean_images(cls, clean: Any, dry: Any) -> bool:
          """ Remove the testcloud images """
          clean.info('testcloud', shift=1, color='green')
          if not os.path.exists(TESTCLOUD_IMAGES):
              clean.warn(
                  f"Directory '{TESTCLOUD_IMAGES}' does not exist.", shift=2)
              return True
          successful = True
          for image in os.listdir(TESTCLOUD_IMAGES):
              image = os.path.join(TESTCLOUD_IMAGES, image)
              if dry:
                  clean.verbose(f"Would remove '{image}'.", shift=2)
              else:
                  clean.verbose(f"Removing '{image}'.", shift=2)
                  try:
                      os.remove(image)
                  except OSError:
                      clean.error(f"Failed to remove '{image}'.", shift=2)
                      successful = False
          return successful
  
  
  class GuestTestcloud(tmt.GuestSsh):
      """
      Testcloud Instance
  
      The following keys are expected in the 'data' dictionary::
  
          image ...... qcov image name or url
          user ....... user name to log in
          memory ..... memory size for vm
          disk ....... disk size for vm
          connection . either session (default) or system, to be passed to qemu
          arch ....... architecture for the VM, host arch is the default
      """
  
      # TODO: Revisit this `type: ignore` once `Guest` becomes a generic type
      _data_class = TestcloudGuestData  # type: ignore[assignment]
  
      image: str
      image_url: Optional[str]
      instance_name: Optional[str]
      memory: int
      disk: str
      connection: str
      arch: str
  
      # Not to be saved, recreated from image_url/instance_name/... every
      # time guest is instantiated.
      _image: Optional['testcloud.image.Image'] = None  # type: ignore[name-defined]
      _instance: Optional['testcloud.instance.Instance'] = None  # type: ignore[name-defined]
  
      def _get_url(self, url: str, message: str) -> requests.Response:
          """ Get url, retry when fails, return response """
  
          def try_get_url() -> requests.Response:
              try:
                  with retry_session() as session:
                      response = session.get(url)
  
                  if response.ok:
                      return response
  
              except requests.RequestException:
                  pass
              finally:
                  raise tmt.utils.WaitingIncomplete()
  
          try:
              return tmt.utils.wait(
                  self, try_get_url, datetime.timedelta(
                      seconds=DEFAULT_CONNECT_TIMEOUT), tick=1)
  
          except tmt.utils.WaitingTimedOutError:
              raise ProvisionError(
                  f'Failed to {message} in {DEFAULT_CONNECT_TIMEOUT}s.')
  
      def _guess_image_url(self, name: str) -> str:
          """ Guess image url for given name """
  
          # Try to check if given url is a local file
          if os.path.isabs(name) and os.path.isfile(name):
              return f'file://{name}'
  
          name = name.lower().strip()
          url = None
  
          # Map fedora aliases (e.g. rawhide, fedora, fedora-32, f-32, f32)
          matched_fedora = re.match(r'^f(edora)?-?(\d+)$', name)
          # Map fedora coreos aliases (e.g. stable, next or testing)
          matched_fedora_coreos = re.match(r'^f(edora-coreos)?-?(stable|testing|next)$', name)
          # Map centos aliases (e.g. centos:X, centos, centos-stream:X)
          matched_centos = [re.match(r'^c(entos)?-?(\d+)$', name),
                            re.match(r'^c(entos-stream)?-?(\d+)$', name)]
          matched_ubuntu = re.match(r'^u(buntu)?-?(\w+)$', name)
          matched_debian = re.match(r'^d(ebian)?-?(\w+)$', name)
  
          assert testcloud is not None
          # Plain name match means we want the latest release
          if name == 'fedora':
              url = testcloud.util.get_fedora_image_url("latest", self.arch)
          elif name == 'fedora-coreos':
              url = testcloud.util.get_fedora_image_url("stable", self.arch)
          elif name == 'centos':
              url = testcloud.util.get_centos_image_url("latest", self.arch)
          elif name == 'centos-stream':
              url = testcloud.util.get_centos_image_url(
                  "latest", stream=True, arch=self.arch)
          elif name == 'ubuntu':
              url = testcloud.util.get_ubuntu_image_url("latest", self.arch)
          elif name == 'debian':
              url = testcloud.util.get_debian_image_url("latest", self.arch)
  
          elif matched_fedora:
              url = testcloud.util.get_fedora_image_url(
                  matched_fedora.group(2), self.arch)
          elif matched_fedora_coreos:
              url = testcloud.util.get_fedora_image_url(
                  matched_fedora_coreos.group(2), self.arch)
          elif matched_centos[0]:
              url = testcloud.util.get_centos_image_url(
                  matched_centos[0].group(2), self.arch)
          elif matched_centos[1]:
              url = testcloud.util.get_centos_image_url(
                  matched_centos[1].group(2), stream=True, arch=self.arch)
          elif matched_ubuntu:
              url = testcloud.util.get_ubuntu_image_url(
                  matched_ubuntu.group(2), self.arch)
          elif matched_debian:
              url = testcloud.util.get_debian_image_url(
                  matched_debian.group(2), self.arch)
          elif 'rawhide' in name:
              url = testcloud.util.get_fedora_image_url("rawhide", self.arch)
  
          if not url:
              raise ProvisionError(f"Could not map '{name}' to compose.")
          return cast(str, url)
  
      @staticmethod
      def _create_template() -> None:
          """ Create libvirt domain template """
          # Write always to ovewrite possible outdated version
          with open(DOMAIN_TEMPLATE_FILE, 'w') as template:
              template.write(DOMAIN_TEMPLATE)
  
      def wake(self) -> None:
          """ Wake up the guest """
          self.debug(
              f"Waking up testcloud instance '{self.instance_name}'.",
              level=2, shift=0)
          self.prepare_config()
          assert testcloud is not None
          self._image = testcloud.image.Image(self.image_url)
          self._instance = testcloud.instance.Instance(
              self.instance_name, image=self._image,
              connection=f"qemu:///{self.connection}", desired_arch=self.arch)
  
      def prepare_ssh_key(self, key_type: Optional[str] = None) -> None:
          """ Prepare ssh key for authentication """
          # Create ssh key paths
          key_name = "id_{}".format(key_type if key_type is not None else 'rsa')
          assert self.workdir is not None
          self.key = [os.path.join(self.workdir, key_name)]
          self.pubkey = os.path.join(self.workdir, f'{key_name}.pub')
  
          # Generate ssh key
          self.debug('Generating an ssh key.')
          command = ["ssh-keygen", "-f", self.key[0], "-N", ""]
          if key_type is not None:
              command.extend(["-t", key_type])
          self.run(command)
          self.verbose('key', self.key[0], 'green')
          with open(self.pubkey, 'r') as pubkey:
              self.config.USER_DATA = USER_DATA.format(
                  user_name=self.user, public_key=pubkey.read())
          with open(self.pubkey, 'r') as pubkey:
              self.config.COREOS_DATA = COREOS_DATA.format(
                  user_name=self.user, public_key=pubkey.read())
  
      def prepare_config(self) -> None:
          """ Prepare common configuration """
          import_testcloud()
  
          # Get configuration
          assert testcloud is not None
          self.config = testcloud.config.get_config()
  
          # Make sure download progress is disabled unless in debug mode,
          # so it does not spoil our logging
          self.config.DOWNLOAD_PROGRESS = self.opt('debug') > 2
          self.config.DOWNLOAD_PROGRESS_VERBOSE = False
  
          # Configure to tmt's storage directories
          self.config.DATA_DIR = TESTCLOUD_DATA
          self.config.STORE_DIR = TESTCLOUD_IMAGES
  
      def start(self) -> None:
          """ Start provisioned guest """
          if self.opt('dry'):
              return
          # Make sure required directories exist
          os.makedirs(TESTCLOUD_DATA, exist_ok=True)
          os.makedirs(TESTCLOUD_IMAGES, exist_ok=True)
  
          # Make sure libvirt domain template exists
          GuestTestcloud._create_template()
  
          # Prepare config
          self.prepare_config()
  
          # Kick off image URL with the given image
          self.image_url = self.image
  
          # If image does not start with http/https/file, consider it a
          # mapping value and try to guess the URL
          if not re.match(r'^(?:https?|file)://.*', self.image_url):
              self.image_url = self._guess_image_url(self.image_url)
              self.debug(f"Guessed image url: '{self.image_url}'", level=3)
  
          # Initialize and prepare testcloud image
          assert testcloud is not None
          self._image = testcloud.image.Image(self.image_url)
          self.verbose('qcow', self._image.name, 'green')
          if not os.path.exists(self._image.local_path):
              self.info('progress', 'downloading...', 'cyan')
          try:
              self._image.prepare()
          except FileNotFoundError as error:
              raise ProvisionError(
                  f"Image '{self._image.local_path}' not found.", original=error)
          except (testcloud.exceptions.TestcloudPermissionsError,
                  PermissionError) as error:
              raise ProvisionError(
                  f"Failed to prepare the image. Check the '{TESTCLOUD_IMAGES}' "
                  f"directory permissions.", original=error)
  
          # Create instance
          self.instance_name = self._tmt_name()
          self._instance = testcloud.instance.Instance(
              name=self.instance_name, image=self._image,
              connection=f"qemu:///{self.connection}", desired_arch=self.arch)
          self.verbose('name', self.instance_name, 'green')
  
          # Decide if we want to multiply timeouts when emulating an architecture
          time_coeff = NON_KVM_TIMEOUT_COEF if not self._instance.kvm else 1
  
          # Decide which networking setup to use
          # Autodetect works with libguestfs python bindings
          # We fall back to basic heuristics based on file name
          # without that installed (eg. from pypi).
          # https://bugzilla.redhat.com/show_bug.cgi?id=1075594
          try:
              import guestfs  # noqa: F401
          except ImportError:
              match_legacy = re.search(
                  r'(rhel|centos).*-7', self.image_url.lower())
              if match_legacy:
                  self._instance.pci_net = "e1000"
              else:
                  self._instance.pci_net = "virtio-net-pci"
  
          # Prepare ssh key
          # TODO: Maybe... some better way to do this?
          if "coreos" in self.image.lower():
              self._instance.coreos = True
              self._instance.ssh_path = self.key[0]
          self.prepare_ssh_key(SSH_KEYGEN_TYPE)
  
          # Boot the virtual machine
          self.info('progress', 'booting...', 'cyan')
          self._instance.ram = self.memory
          self._instance.disk_size = self.disk
          assert libvirt is not None
          try:
              self._instance.prepare()
              self._instance.spawn_vm()
              self._instance.start(DEFAULT_BOOT_TIMEOUT * time_coeff)
          except (testcloud.exceptions.TestcloudInstanceError,
                  libvirt.libvirtError) as error:
              raise ProvisionError(
                  f'Failed to boot testcloud instance ({error}).')
          self.guest = self._instance.get_ip()
          self.port = int(self._instance.get_instance_port())
          self.verbose('ip', self.guest, 'green')
          self.verbose('port', str(self.port), 'green')
          self._instance.create_ip_file(self.guest)
  
          # Wait a bit until the box is up
          if not self.reconnect(
                  timeout=DEFAULT_CONNECT_TIMEOUT *
                  time_coeff,
                  tick=1):
              raise ProvisionError(
                  f"Failed to connect in {DEFAULT_CONNECT_TIMEOUT * time_coeff}s.")
  
          if not self._instance.kvm:
              self.debug(
                  f"Waiting {NON_KVM_ADDITIONAL_WAIT} seconds "
                  f"for non-kvm instance...")
              time.sleep(NON_KVM_ADDITIONAL_WAIT)
  
      def stop(self) -> None:
          """ Stop provisioned guest """
          super().stop()
          # Stop only if the instance successfully booted
          if self._instance and self.guest:
              self.debug(f"Stopping testcloud instance '{self.instance_name}'.")
              assert testcloud is not None
              try:
                  self._instance.stop()
              except testcloud.exceptions.TestcloudInstanceError as error:
                  raise tmt.utils.ProvisionError(
                      f"Failed to stop testcloud instance: {error}")
              self.info('guest', 'stopped', 'green')
  
      def remove(self) -> None:
          """ Remove the guest (disk cleanup) """
          if self._instance:
              self.debug(f"Removing testcloud instance '{self.instance_name}'.")
              try:
                  self._instance.remove(autostop=True)
              except FileNotFoundError as error:
                  raise tmt.utils.ProvisionError(
                      f"Failed to remove testcloud instance: {error}")
              self.info('guest', 'removed', 'green')
  
      def reboot(self,
                 hard: bool = False,
                 command: Optional[str] = None,
                 timeout: Optional[int] = None,
                 tick: float = tmt.utils.DEFAULT_WAIT_TICK,
                 tick_increase: float = tmt.utils.DEFAULT_WAIT_TICK_INCREASE) -> Any:
          """ Reboot the guest, return True if successful """
          # Use custom reboot command if provided
          if command:
              return super().reboot(hard=hard, command=command)
          if not self._instance:
              raise tmt.utils.ProvisionError("No instance initialized.")
          self._instance.reboot(soft=not hard)
          return self.reconnect(timeout=timeout)
  
files: tmt/steps/provision/testcloud.py
files: # coding: utf-8
  
  import dataclasses
  import datetime
  import os
  import platform
  import re
  import time
  import types
  from typing import Any, List, Optional, cast
  
  import click
  import fmf
  import requests
  
  import tmt
  import tmt.steps
  import tmt.steps.provision
  import tmt.utils
  from tmt.utils import WORKDIR_ROOT, ProvisionError, retry_session
  
  libvirt: Optional[types.ModuleType] = None
  testcloud: Optional[types.ModuleType] = None
  
  
  def import_testcloud() -> None:
      """
      Import testcloud module only when needed
  
      Until we have a separate package for each plugin.
      """
      global testcloud
      global libvirt
      try:
          import libvirt
          import testcloud.image
          import testcloud.instance
          import testcloud.util
      except ImportError:
          raise ProvisionError(
              "Install 'testcloud' to provision using this method.")
  
  
  # Testcloud cache to our tmt's workdir root
  TESTCLOUD_DATA = os.path.join(WORKDIR_ROOT, 'testcloud')
  TESTCLOUD_IMAGES = os.path.join(TESTCLOUD_DATA, 'images')
  
  # Userdata for cloud-init
  USER_DATA = """#cloud-config
  chpasswd:
    list: |
      {user_name}:%s
    expire: false
  users:
    - default
    - name: {user_name}
  ssh_authorized_keys:
    - {public_key}
  ssh_pwauth: true
  disable_root: false
  runcmd:
    - sed -i -e '/^.*PermitRootLogin/s/^.*$/PermitRootLogin yes/'
      -e '/^.*UseDNS/s/^.*$/UseDNS no/'
      -e '/^.*GSSAPIAuthentication/s/^.*$/GSSAPIAuthentication no/'
      /etc/ssh/sshd_config
    - systemctl reload sshd
    - [sh, -c, 'if [ ! -f /etc/systemd/network/20-tc-usernet.network ] &&
    systemctl status systemd-networkd | grep -q "enabled;\\svendor\\spreset:\\senabled";
    then mkdir -p /etc/systemd/network/ &&
    echo "[Match]" >> /etc/systemd/network/20-tc-usernet.network &&
    echo "Name=en*" >> /etc/systemd/network/20-tc-usernet.network &&
    echo "[Network]" >> /etc/systemd/network/20-tc-usernet.network &&
    echo "DHCP=yes" >> /etc/systemd/network/20-tc-usernet.network; fi']
    - [sh, -c, 'if systemctl status systemd-networkd |
    grep -q "enabled;\\svendor\\spreset:\\senabled"; then
    systemctl restart systemd-networkd; fi']
    - [sh, -c, 'if cat /etc/os-release |
    grep -q platform:el8; then systemctl restart sshd; fi']
  """
  
  COREOS_DATA = """variant: fcos
  version: 1.4.0
  passwd:
    users:
      - name: {user_name}
        ssh_authorized_keys:
          - {public_key}
  """
  
  # Libvirt domain XML template related variables
  DOMAIN_TEMPLATE_NAME = 'domain-template.jinja'
  DOMAIN_TEMPLATE_FILE = os.path.join(TESTCLOUD_DATA, DOMAIN_TEMPLATE_NAME)
  DOMAIN_TEMPLATE = """<domain type='{{ virt_type }}' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
    <name>{{ domain_name }}</name>
    <uuid>{{ uuid }}</uuid>
    <memory unit='KiB'>{{ memory }}</memory>
    <currentMemory unit='KiB'>{{ memory }}</currentMemory>
    <vcpu placement='static'>2</vcpu>
    <os>
      <type arch='{{ arch }}' machine='{{ model }}'>hvm</type>
      {{ uefi_loader }}
      <boot dev='hd'/>
    </os>
    {{ cpu }}
    {{ extra_specs }}
    <clock offset='utc'>
      <timer name='rtc' tickpolicy='catchup'/>
      <timer name='pit' tickpolicy='delay'/>
      <timer name='hpet' present='no'/>
    </clock>
    <on_poweroff>destroy</on_poweroff>
    <on_reboot>restart</on_reboot>
    <on_crash>restart</on_crash>
    <devices>
      <emulator>{{ emulator_path }}</emulator>
      <disk type='file' device='disk'>
        <driver name='qemu' type='qcow2' cache='unsafe'/>
        <source file="{{ disk }}"/>
        <target dev='vda' bus='virtio'/>
        {{ boot_drive_address }}
      </disk>
      <disk type='file' device='disk'>
        <driver name='qemu' type='raw'/>
        <source file="{{ seed }}"/>
        <target dev='vdb' bus='virtio'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
      </disk>
      <interface type='{{ network_type }}'>
        <mac address="{{ mac_address }}"/>
        {{ network_source }}
        {{ ip_setup }}
        <model type='virtio'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
      </interface>
      <serial type='pty'>
        <target port='0'/>
      </serial>
      <console type='pty'>
        <target type='serial' port='0'/>
      </console>
      <input type="keyboard" bus="virtio"/>
      <rng model='virtio'>
        <backend model='random'>/dev/urandom</backend>
      </rng>
    </devices>
    {{ qemu_args }}
  </domain>
  """  # noqa: E501
  
  # VM defaults
  DEFAULT_BOOT_TIMEOUT = 60      # seconds
  DEFAULT_CONNECT_TIMEOUT = 60   # seconds
  NON_KVM_ADDITIONAL_WAIT = 10       # seconds
  NON_KVM_TIMEOUT_COEF = 10          # times
  
  # SSH key type, set None for ssh-keygen default one
  SSH_KEYGEN_TYPE = "ecdsa"
  
  DEFAULT_USER = 'root'
  DEFAULT_MEMORY = 2048
  DEFAULT_DISK = 10
  DEFAULT_IMAGE = 'fedora'
  DEFAULT_CONNECTION = 'session'
  DEFAULT_ARCH = platform.machine()
  
  
  @dataclasses.dataclass
  class TestcloudGuestData(tmt.steps.provision.GuestSshData):
      # Override parent class with our defaults
      user: str = DEFAULT_USER
  
      image: str = DEFAULT_IMAGE
      memory: int = DEFAULT_MEMORY
      disk: int = DEFAULT_DISK
      connection: str = DEFAULT_CONNECTION
      arch: str = DEFAULT_ARCH
  
      image_url: Optional[str] = None
      instance_name: Optional[str] = None
  
  
  @dataclasses.dataclass
  class ProvisionTestcloudData(TestcloudGuestData, tmt.steps.StepData):
      pass
  
  
  @tmt.steps.provides_method('virtual.testcloud')
  class ProvisionTestcloud(tmt.steps.provision.ProvisionPlugin):
      """
      Local virtual machine using testcloud
  
      Minimal config which uses the latest fedora image:
  
          provision:
              how: virtual
  
      Here's a full config example:
  
          provision:
              how: virtual
              image: fedora
              user: root
              memory: 2048
  
      As the image use 'fedora' for the latest released Fedora compose,
      'rawhide' for the latest Rawhide compose, short aliases such as
      'fedora-32', 'f-32' or 'f32' for specific release or a full url to
      the qcow2 image for example from:
  
          https://kojipkgs.fedoraproject.org/compose/
  
      Short names are also provided for 'centos', 'centos-stream',
      'debian' and 'ubuntu' (e.g. 'centos-8' or 'c8').
  
      Supported Fedora CoreOS images are:
  
          fedora-coreos
          fedora-coreos-stable
          fedora-coreos-testing
          fedora-coreos-next
  
      Use the full path for images stored on local disk, for example:
  
          /var/tmp/images/Fedora-Cloud-Base-31-1.9.x86_64.qcow2
  
      In addition to the qcow2 format, vagrant boxes can be used as well,
      testcloud will take care of unpacking the image for you.
      """
  
      _data_class = ProvisionTestcloudData
  
      # Guest instance
      _guest = None
  
      @classmethod
      def options(cls, how: Optional[str] = None) -> List[tmt.options.ClickOptionDecoratorType]:
          """ Prepare command line options for testcloud """
          options = super().options(how)
          options[:0] = [
              click.option(
                  '-i', '--image', metavar='IMAGE',
                  help='Select image to be used. Provide a short name, '
                       'full path to a local file or a complete url.'),
              click.option(
                  '-m', '--memory', metavar='MEMORY', type=int,
                  help='Set available memory in MB, 2048 MB by default.'),
              click.option(
                  '-D', '--disk', metavar='MEMORY', type=int,
                  help='Specify disk size in GB, 10 GB by default.'),
              click.option(
                  '-u', '--user', metavar='USER',
                  help='Username to use for all guest operations.'),
              click.option(
                  '-c', '--connection',
                  type=click.Choice(['session', 'system']),
                  help="What session type to use, 'session' by default."),
              click.option(
                  '-a', '--arch',
                  type=click.Choice(['x86_64', 'aarch64', 's390x', 'ppc64le']),
                  help="What architecture to virtualize, host arch by default."),
              ]
          return options
  
      # TODO: Revisit this `type: ignore` once `Guest` becomes a generic type
      def wake(self, data: Optional[TestcloudGuestData] = None) -> None:  # type: ignore[override]
          """ Wake up the plugin, process data, apply options """
          super().wake(data=data)
  
          # Wake up testcloud instance
          if data:
              guest = GuestTestcloud(data, name=self.name, parent=self.step)
              guest.wake()
              self._guest = guest
  
      def go(self) -> None:
          """ Provision the testcloud instance """
          super().go()
  
          # Give info about provided data
          data = TestcloudGuestData(**{
              key: self.get(key)
              for key in TestcloudGuestData.keys()
              })
  
          # Once plan schema is enforced this won't be necessary
          # click enforces int for cmdline and schema validation
          # will make sure 'int' gets from plan data.
          # Another key is 'port' however that is not exposed to the cli
          for int_key in ["memory", "disk", "port"]:
              value = getattr(data, int_key)
              if value is not None:
                  try:
                      setattr(data, int_key, int(value))
                  except ValueError:
                      raise tmt.utils.SpecificationError(
                          f"Value '{value}' cannot be converted to int for '{int_key}' attribute.")
  
          for key, value in data.to_dict().items():
              if key == 'memory':
                  self.info('memory', f"{value} MB", 'green')
              elif key == 'disk':
                  self.info('disk', f"{value} GB", 'green')
              elif key == 'connection':
                  self.verbose('connection', value, 'green')
              elif key == 'key':
                  if value:
                      self.info('key', fmf.utils.listed(value), 'green')
              elif value is not None:
                  self.info(key, value, 'green')
  
          # Create a new GuestTestcloud instance and start it
          self._guest = GuestTestcloud(data, name=self.name, parent=self.step)
          self._guest.start()
  
      def guest(self) -> Optional[tmt.Guest]:
          """ Return the provisioned guest """
          return self._guest
  
      @classmethod
      def clean_images(cls, clean: Any, dry: Any) -> bool:
          """ Remove the testcloud images """
          clean.info('testcloud', shift=1, color='green')
          if not os.path.exists(TESTCLOUD_IMAGES):
              clean.warn(
                  f"Directory '{TESTCLOUD_IMAGES}' does not exist.", shift=2)
              return True
          successful = True
          for image in os.listdir(TESTCLOUD_IMAGES):
              image = os.path.join(TESTCLOUD_IMAGES, image)
              if dry:
                  clean.verbose(f"Would remove '{image}'.", shift=2)
              else:
                  clean.verbose(f"Removing '{image}'.", shift=2)
                  try:
                      os.remove(image)
                  except OSError:
                      clean.error(f"Failed to remove '{image}'.", shift=2)
                      successful = False
          return successful
  
  
  class GuestTestcloud(tmt.GuestSsh):
      """
      Testcloud Instance
  
      The following keys are expected in the 'data' dictionary::
  
          image ...... qcov image name or url
          user ....... user name to log in
          memory ..... memory size for vm
          disk ....... disk size for vm
          connection . either session (default) or system, to be passed to qemu
          arch ....... architecture for the VM, host arch is the default
      """
  
      # TODO: Revisit this `type: ignore` once `Guest` becomes a generic type
      _data_class = TestcloudGuestData  # type: ignore[assignment]
  
      image: str
      image_url: Optional[str]
      instance_name: Optional[str]
      memory: int
      disk: str
      connection: str
      arch: str
  
      # Not to be saved, recreated from image_url/instance_name/... every
      # time guest is instantiated.
      _image: Optional['testcloud.image.Image'] = None  # type: ignore[name-defined]
      _instance: Optional['testcloud.instance.Instance'] = None  # type: ignore[name-defined]
  
      def _get_url(self, url: str, message: str) -> requests.Response:
          """ Get url, retry when fails, return response """
  
          def try_get_url() -> requests.Response:
              try:
                  with retry_session() as session:
                      response = session.get(url)
  
                  if response.ok:
                      return response
  
              except requests.RequestException:
                  pass
              finally:
                  raise tmt.utils.WaitingIncomplete()
  
          try:
              return tmt.utils.wait(
                  self, try_get_url, datetime.timedelta(
                      seconds=DEFAULT_CONNECT_TIMEOUT), tick=1)
  
          except tmt.utils.WaitingTimedOutError:
              raise ProvisionError(
                  f'Failed to {message} in {DEFAULT_CONNECT_TIMEOUT}s.')
  
      def _guess_image_url(self, name: str) -> str:
          """ Guess image url for given name """
  
          # Try to check if given url is a local file
          if os.path.isabs(name) and os.path.isfile(name):
              return f'file://{name}'
  
          name = name.lower().strip()
          url = None
  
          # Map fedora aliases (e.g. rawhide, fedora, fedora-32, f-32, f32)
          matched_fedora = re.match(r'^f(edora)?-?(\d+)$', name)
          # Map fedora coreos aliases (e.g. stable, next or testing)
          matched_fedora_coreos = re.match(r'^f(edora-coreos)?-?(stable|testing|next)$', name)
          # Map centos aliases (e.g. centos:X, centos, centos-stream:X)
          matched_centos = [re.match(r'^c(entos)?-?(\d+)$', name),
                            re.match(r'^c(entos-stream)?-?(\d+)$', name)]
          matched_ubuntu = re.match(r'^u(buntu)?-?(\w+)$', name)
          matched_debian = re.match(r'^d(ebian)?-?(\w+)$', name)
  
          assert testcloud is not None
          # Plain name match means we want the latest release
          if name == 'fedora':
              url = testcloud.util.get_fedora_image_url("latest", self.arch)
          elif name == 'fedora-coreos':
              url = testcloud.util.get_fedora_image_url("stable", self.arch)
          elif name == 'centos':
              url = testcloud.util.get_centos_image_url("latest", self.arch)
          elif name == 'centos-stream':
              url = testcloud.util.get_centos_image_url(
                  "latest", stream=True, arch=self.arch)
          elif name == 'ubuntu':
              url = testcloud.util.get_ubuntu_image_url("latest", self.arch)
          elif name == 'debian':
              url = testcloud.util.get_debian_image_url("latest", self.arch)
  
          elif matched_fedora:
              url = testcloud.util.get_fedora_image_url(
                  matched_fedora.group(2), self.arch)
          elif matched_fedora_coreos:
              url = testcloud.util.get_fedora_image_url(
                  matched_fedora_coreos.group(2), self.arch)
          elif matched_centos[0]:
              url = testcloud.util.get_centos_image_url(
                  matched_centos[0].group(2), self.arch)
          elif matched_centos[1]:
              url = testcloud.util.get_centos_image_url(
                  matched_centos[1].group(2), stream=True, arch=self.arch)
          elif matched_ubuntu:
              url = testcloud.util.get_ubuntu_image_url(
                  matched_ubuntu.group(2), self.arch)
          elif matched_debian:
              url = testcloud.util.get_debian_image_url(
                  matched_debian.group(2), self.arch)
          elif 'rawhide' in name:
              url = testcloud.util.get_fedora_image_url("rawhide", self.arch)
  
          if not url:
              raise ProvisionError(f"Could not map '{name}' to compose.")
          return cast(str, url)
  
      @staticmethod
      def _create_template() -> None:
          """ Create libvirt domain template """
          # Write always to ovewrite possible outdated version
          with open(DOMAIN_TEMPLATE_FILE, 'w') as template:
              template.write(DOMAIN_TEMPLATE)
  
      def wake(self) -> None:
          """ Wake up the guest """
          self.debug(
              f"Waking up testcloud instance '{self.instance_name}'.",
              level=2, shift=0)
          self.prepare_config()
          assert testcloud is not None
          self._image = testcloud.image.Image(self.image_url)
          self._instance = testcloud.instance.Instance(
              self.instance_name, image=self._image,
              connection=f"qemu:///{self.connection}", desired_arch=self.arch)
  
      def prepare_ssh_key(self, key_type: Optional[str] = None) -> None:
          """ Prepare ssh key for authentication """
          # Create ssh key paths
          key_name = "id_{}".format(key_type if key_type is not None else 'rsa')
          assert self.workdir is not None
          self.key = [os.path.join(self.workdir, key_name)]
          self.pubkey = os.path.join(self.workdir, f'{key_name}.pub')
  
          # Generate ssh key
          self.debug('Generating an ssh key.')
          command = ["ssh-keygen", "-f", self.key[0], "-N", ""]
          if key_type is not None:
              command.extend(["-t", key_type])
          self.run(command)
          self.verbose('key', self.key[0], 'green')
          with open(self.pubkey, 'r') as pubkey:
              self.config.USER_DATA = USER_DATA.format(
                  user_name=self.user, public_key=pubkey.read())
          with open(self.pubkey, 'r') as pubkey:
              self.config.COREOS_DATA = COREOS_DATA.format(
                  user_name=self.user, public_key=pubkey.read())
  
      def prepare_config(self) -> None:
          """ Prepare common configuration """
          import_testcloud()
  
          # Get configuration
          assert testcloud is not None
          self.config = testcloud.config.get_config()
  
          # Make sure download progress is disabled unless in debug mode,
          # so it does not spoil our logging
          self.config.DOWNLOAD_PROGRESS = self.opt('debug') > 2
          self.config.DOWNLOAD_PROGRESS_VERBOSE = False
  
          # Configure to tmt's storage directories
          self.config.DATA_DIR = TESTCLOUD_DATA
          self.config.STORE_DIR = TESTCLOUD_IMAGES
  
      def start(self) -> None:
          """ Start provisioned guest """
          if self.opt('dry'):
              return
          # Make sure required directories exist
          os.makedirs(TESTCLOUD_DATA, exist_ok=True)
          os.makedirs(TESTCLOUD_IMAGES, exist_ok=True)
  
          # Make sure libvirt domain template exists
          GuestTestcloud._create_template()
  
          # Prepare config
          self.prepare_config()
  
          # Kick off image URL with the given image
          self.image_url = self.image
  
          # If image does not start with http/https/file, consider it a
          # mapping value and try to guess the URL
          if not re.match(r'^(?:https?|file)://.*', self.image_url):
              self.image_url = self._guess_image_url(self.image_url)
              self.debug(f"Guessed image url: '{self.image_url}'", level=3)
  
          # Initialize and prepare testcloud image
          assert testcloud is not None
          self._image = testcloud.image.Image(self.image_url)
          self.verbose('qcow', self._image.name, 'green')
          if not os.path.exists(self._image.local_path):
              self.info('progress', 'downloading...', 'cyan')
          try:
              self._image.prepare()
          except FileNotFoundError as error:
              raise ProvisionError(
                  f"Image '{self._image.local_path}' not found.", original=error)
          except (testcloud.exceptions.TestcloudPermissionsError,
                  PermissionError) as error:
              raise ProvisionError(
                  f"Failed to prepare the image. Check the '{TESTCLOUD_IMAGES}' "
                  f"directory permissions.", original=error)
  
          # Create instance
          self.instance_name = self._tmt_name()
          self._instance = testcloud.instance.Instance(
              name=self.instance_name, image=self._image,
              connection=f"qemu:///{self.connection}", desired_arch=self.arch)
          self.verbose('name', self.instance_name, 'green')
  
          # Decide if we want to multiply timeouts when emulating an architecture
          time_coeff = NON_KVM_TIMEOUT_COEF if not self._instance.kvm else 1
  
          # Decide which networking setup to use
          # Autodetect works with libguestfs python bindings
          # We fall back to basic heuristics based on file name
          # without that installed (eg. from pypi).
          # https://bugzilla.redhat.com/show_bug.cgi?id=1075594
          try:
              import guestfs  # noqa: F401
          except ImportError:
              match_legacy = re.search(
                  r'(rhel|centos).*-7', self.image_url.lower())
              if match_legacy:
                  self._instance.pci_net = "e1000"
              else:
                  self._instance.pci_net = "virtio-net-pci"
  
          # Prepare ssh key
          # TODO: Maybe... some better way to do this?
          if "coreos" in self.image.lower():
              self._instance.coreos = True
              self._instance.ssh_path = self.key[0]
          self.prepare_ssh_key(SSH_KEYGEN_TYPE)
  
          # Boot the virtual machine
          self.info('progress', 'booting...', 'cyan')
          self._instance.ram = self.memory
          self._instance.disk_size = self.disk
          assert libvirt is not None
          try:
              self._instance.prepare()
              self._instance.spawn_vm()
              self._instance.start(DEFAULT_BOOT_TIMEOUT * time_coeff)
          except (testcloud.exceptions.TestcloudInstanceError,
                  libvirt.libvirtError) as error:
              raise ProvisionError(
                  f'Failed to boot testcloud instance ({error}).')
          self.guest = self._instance.get_ip()
          self.port = int(self._instance.get_instance_port())
          self.verbose('ip', self.guest, 'green')
          self.verbose('port', str(self.port), 'green')
          self._instance.create_ip_file(self.guest)
  
          # Wait a bit until the box is up
          if not self.reconnect(
                  timeout=DEFAULT_CONNECT_TIMEOUT *
                  time_coeff,
                  tick=1):
              raise ProvisionError(
                  f"Failed to connect in {DEFAULT_CONNECT_TIMEOUT * time_coeff}s.")
  
          if not self._instance.kvm:
              self.debug(
                  f"Waiting {NON_KVM_ADDITIONAL_WAIT} seconds "
                  f"for non-kvm instance...")
              time.sleep(NON_KVM_ADDITIONAL_WAIT)
  
      def stop(self) -> None:
          """ Stop provisioned guest """
          super().stop()
          # Stop only if the instance successfully booted
          if self._instance and self.guest:
              self.debug(f"Stopping testcloud instance '{self.instance_name}'.")
              assert testcloud is not None
              try:
                  self._instance.stop()
              except testcloud.exceptions.TestcloudInstanceError as error:
                  raise tmt.utils.ProvisionError(
                      f"Failed to stop testcloud instance: {error}")
              self.info('guest', 'stopped', 'green')
  
      def remove(self) -> None:
          """ Remove the guest (disk cleanup) """
          if self._instance:
              self.debug(f"Removing testcloud instance '{self.instance_name}'.")
              try:
                  self._instance.remove(autostop=True)
              except FileNotFoundError as error:
                  raise tmt.utils.ProvisionError(
                      f"Failed to remove testcloud instance: {error}")
              self.info('guest', 'removed', 'green')
  
      def reboot(self,
                 hard: bool = False,
                 command: Optional[str] = None,
                 timeout: Optional[int] = None,
                 tick: float = tmt.utils.DEFAULT_WAIT_TICK,
                 tick_increase: float = tmt.utils.DEFAULT_WAIT_TICK_INCREASE) -> Any:
          """ Reboot the guest, return True if successful """
          # Use custom reboot command if provided
          if command:
              return super().reboot(hard=hard, command=command)
          if not self._instance:
              raise tmt.utils.ProvisionError("No instance initialized.")
          self._instance.reboot(soft=not hard)
          return self.reconnect(timeout=timeout)
  
files: tmt/steps/provision/testcloud.py
files: 
  """ Test Metadata Utilities """
  
  import contextlib
  import copy
  import dataclasses
  import datetime
  import functools
  import glob
  import io
  import os
  import pprint
  import re
  import shlex
  import shutil
  import subprocess
  import sys
  import time
  import unicodedata
  from collections import OrderedDict
  from functools import lru_cache
  from pathlib import Path
  from threading import Thread
  from typing import (IO, TYPE_CHECKING, Any, Callable, Dict, Generator,
                      Iterable, List, NamedTuple, Optional, Pattern, Tuple, Type,
                      TypeVar, Union, cast, overload)
  
  import click
  import fmf
  import jsonschema
  import pkg_resources
  import requests
  import requests.adapters
  import requests.packages.urllib3.util.retry
  import urllib3.exceptions
  from click import echo, style, wrap_text
  from ruamel.yaml import YAML, scalarstring
  from ruamel.yaml.comments import CommentedMap
  from ruamel.yaml.parser import ParserError
  
  if sys.version_info >= (3, 8):
      from typing import Literal, Protocol
  else:
      from typing_extensions import Literal, Protocol
  
  
  if TYPE_CHECKING:
      import tmt.base
      import tmt.steps
  
  log = fmf.utils.Logging('tmt').logger
  
  # Default workdir root and max
  WORKDIR_ROOT = '/var/tmp/tmt'
  WORKDIR_MAX = 1000
  
  # Log in workdir
  LOG_FILENAME = 'log.txt'
  
  # Maximum number of lines of stdout/stderr to show upon errors
  OUTPUT_LINES = 100
  # Default output width
  OUTPUT_WIDTH = 79
  
  # Hierarchy indent
  INDENT = 4
  
  # Default name and order for step plugins
  DEFAULT_NAME = 'default'
  DEFAULT_PLUGIN_ORDER = 50
  DEFAULT_PLUGIN_ORDER_MULTIHOST = 10
  DEFAULT_PLUGIN_ORDER_REQUIRES = 70
  DEFAULT_PLUGIN_ORDER_RECOMMENDS = 75
  
  # Config directory
  CONFIG_PATH = '~/.config/tmt'
  
  # Special process return code
  PROCESS_TIMEOUT = 124
  
  # Default select.select(timeout) in seconds
  DEFAULT_SELECT_TIMEOUT = 5
  
  # Default shell and options to be set for all shell scripts
  DEFAULT_SHELL = "/bin/bash"
  SHELL_OPTIONS = 'set -eo pipefail'
  
  # Defaults for HTTP/HTTPS retries and timeouts (see `retry_session()`).
  DEFAULT_RETRY_SESSION_RETRIES: int = 3
  DEFAULT_RETRY_SESSION_BACKOFF_FACTOR: float = 0.1
  
  # Defaults for HTTP/HTTPS retries for getting environment file
  # Retry with exponential backoff, maximum duration ~511 seconds
  ENVFILE_RETRY_SESSION_RETRIES: int = 10
  ENVFILE_RETRY_SESSION_BACKOFF_FACTOR: float = 1
  
  # Default for wait()-related options
  DEFAULT_WAIT_TICK: float = 30.0
  DEFAULT_WAIT_TICK_INCREASE: float = 1.0
  
  # A stand-in variable for generic use.
  T = TypeVar('T')
  
  # A FMF context type, representing name/values context.
  FmfContextType = Dict[str, List[str]]
  
  # A "environment" type, representing name/value environment variables.
  EnvironmentType = Dict[str, str]
  
  # Workdir argument type, can be True, a string, a path or None
  WorkdirArgumentType = Union[Literal[True], str, None]
  
  # Workdir type, can be None or a string
  WorkdirType = Optional[str]
  
  # Option to skip to initialize work tree in plan
  PLAN_SKIP_WORKTREE_INIT = 'plan_skip_worktree_init'
  
  
  class BaseLoggerFnType(Protocol):
      def __call__(
              self,
              key: str,
              value: Optional[str] = None,
              color: Optional[str] = None,
              shift: int = 0,
              level: int = 1,
              err: bool = False) -> None:
          pass
  
  
  class LevelessLoggerFnType(Protocol):
      def __call__(
              self,
              key: str,
              value: Optional[str] = None,
              color: Optional[str] = None,
              shift: int = 0,
              err: bool = False) -> None:
          pass
  
  
  class SemanticLoggerFnType(Protocol):
      def __call__(self, message: str, shift: int = 0) -> None:
          pass
  
  
  LoggerFnType = Union[
      BaseLoggerFnType,
      LevelessLoggerFnType,
      SemanticLoggerFnType]
  
  
  def indent(
          key: str,
          value: Optional[str] = None,
          color: Optional[str] = None,
          level: int = 0) -> str:
      """
      Indent a key/value message.
  
      If both ``key`` and ``value`` are specified, ``{key}: {value}``
      message is rendered. Otherwise, just ``key`` is used alone. If
      ``value`` contains multiple lines, each but the very first line is
      indented by one extra level.
  
      :param value: optional value to print at right side of ``key``.
      :param color: optional color to apply on ``key``.
      :param level: number of indentation levels. Each level is indented
                    by :py:data:`INDENT` spaces.
      """
  
      indent = ' ' * INDENT * level
      deeper = ' ' * INDENT * (level + 1)
  
      # Colorize
      if color is not None:
          key = style(key, fg=color)
  
      # Handle key only
      if value is None:
          message = key
  
      # Handle key + value
      else:
          # Multiline content indented deeper
          if isinstance(value, str):
              lines = value.splitlines()
              if len(lines) > 1:
                  value = ''.join([f"\n{deeper}{line}" for line in lines])
  
          message = f'{key}: {value}'
  
      return indent + message
  
  
  class Config:
      """ User configuration """
  
      def __init__(self) -> None:
          """ Initialize config directory path """
          self.path = os.path.expanduser(CONFIG_PATH)
          if not os.path.exists(self.path):
              try:
                  os.makedirs(self.path)
              except OSError as error:
                  raise GeneralError(
                      f"Failed to create config '{self.path}'.\n{error}")
  
      def last_run(self, run_id: Optional[str] = None) -> Optional[str]:
          """ Get and set last run id """
          symlink = os.path.join(self.path, 'last-run')
          if run_id:
              try:
                  os.remove(symlink)
              except OSError:
                  pass
              try:
                  os.symlink(run_id, symlink)
              except FileExistsError:
                  # Race when tmt runs in parallel
                  log.warning(f"Race condition, unable to save last run '{run_id}'.")
              except OSError as error:
                  raise GeneralError(
                      f"Unable to save last run '{self.path}'.\n{error}")
              return run_id
          if os.path.islink(symlink):
              return os.path.realpath(symlink)
          return None
  
  
  class StreamLogger(Thread):
      """
      Reading pipes of running process in threads.
  
      Code based on:
      https://github.com/packit/packit/blob/main/packit/utils/logging.py#L10
      """
  
      def __init__(self,
                   stream: Optional[IO[bytes]],
                   log_header: str,
                   logger: BaseLoggerFnType) -> None:
          super().__init__(daemon=True)
          self.stream = stream
          self.output: List[str] = []
          self.log_header = log_header
          self.logger = logger
  
      def run(self) -> None:
          if self.stream is None:
              return
  
          for _line in self.stream:
              line = _line.decode('utf-8', errors='replace')
              if line != '':
                  self.logger(
                      self.log_header,
                      line.rstrip('\n'),
                      'yellow',
                      level=3)
              self.output.append(line)
  
      def get_output(self) -> str:
          return "".join(self.output)
  
  
  # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  #  Common
  # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  
  CommonDerivedType = TypeVar('CommonDerivedType', bound='Common')
  
  
  class CommandOutput(NamedTuple):
      stdout: Optional[str]
      stderr: Optional[str]
  
  
  class Common:
      """
      Common shared stuff
  
      Takes care of command line context, options and workdir handling.
      Provides logging functions info(), verbose() and debug().
      Implements read() and write() for comfortable file access.
      Provides the run() method for easy command execution.
      """
  
      # Command line context, options and workdir
      _context: Optional[click.Context] = None
      # When set to true, _opt will be ignored (default will be returned)
      ignore_class_options: bool = False
      _options: Dict[str, Any] = dict()
      _workdir: WorkdirType = None
  
      # TODO: must be declared outside of __init__(), because it must exist before
      # __init__() gets called to allow logging helpers work correctly when used
      # from mixins. But that's not very clean, is it? :( Maybe decoupling logging
      # from Common class would help, such a class would be able to initialize
      # itself without involving the rest of Common code. On the other hand,
      # Common owns workdir, for example, whose value affects logging too, so no
      # clear solution so far.
      #
      # Note: cannot use CommonDerivedType - it's a TypeVar filled in by the type
      # given to __init__() and therefore the type it's representing *now* is
      # unknown. but we know `parent` will be derived from `Common` class, so it's
      # mostly fine.
      parent: Optional['Common'] = None
  
      def __init__(
              self,
              parent: Optional[CommonDerivedType] = None,
              name: Optional[str] = None,
              workdir: WorkdirArgumentType = None,
              context: Optional[click.Context] = None,
              relative_indent: int = 1,
              **kwargs: Any) -> None:
          """
          Initialize name and relation with the parent object
  
          Prepare the workdir for provided id / directory path
          or generate a new workdir name if workdir=True given.
          Store command line context and options for future use
          if context is provided.
          """
          # Use lowercase class name as the default name
          self.name = name or self.__class__.__name__.lower()
          self.parent = parent
  
          # Prepare a safe variant of the name which does not contain
          # spaces or other special characters to prevent problems with
          # tools which do not expect them (e.g. in directory names).
          self.safe_name = re.sub(r"[^\w/-]+", "-", self.name).strip("-")
  
          # Relative log indent level shift against the parent
          self._relative_indent = relative_indent
  
          # Store command line context
          if context:
              self._save_context_to_instance(context)
  
          # Initialize the workdir if requested
          self._workdir_load(workdir)
  
      def __str__(self) -> str:
          """ Name is the default string representation """
          return self.name
  
      @classmethod
      def _save_context(cls, context: click.Context) -> None:
          """ Save provided command line context and options for future use """
          cls._context = context
          cls._options = context.params
  
      def _save_context_to_instance(self, context: click.Context) -> None:
          """ Save provided command line context and options to the instance """
          self._context = context
          self._options = context.params
  
      @overload
      @classmethod
      def _opt(cls, option: str) -> Any:
          pass
  
      @overload
      @classmethod
      def _opt(cls, option: str, default: T) -> T:
          pass
  
      @classmethod
      def _opt(cls, option: str, default: Any = None) -> Any:
          """ Get an option from the command line context (class version) """
          if cls.ignore_class_options:
              return default
          return cls._options.get(option, default)
  
      def _fmf_context(self) -> FmfContextType:
          """ Return the current fmf context """
          if self._context is None:
              return dict()
  
          try:
              return cast(FmfContextType, self._context.obj.fmf_context)
          except AttributeError:
              return dict()
  
      def opt(self, option: str, default: Optional[Any] = None) -> Any:
          """
          Get an option from the command line options
  
          Checks also parent options. For flags (boolean values) parent's
          True wins over child's False (e.g. run --quiet enables quiet
          mode for all included plans and steps).
  
          For options that can be used multiple times, the child overrides
          the parent if it was defined (e.g. run -av provision -vvv runs
          all steps except for provision in mildly verbose mode, provision
          is run with the most verbosity).
  
          Environment variables override command line options.
          """
          # Translate dashes to underscores to match click's conversion
          option = option.replace('-', '_')
          # Check the environment first
          if option == 'debug':
              try:
                  debug = os.environ['TMT_DEBUG']
                  return int(debug)
              except ValueError:
                  raise GeneralError(
                      f"Invalid debug level '{debug}', use an integer.")
              except KeyError:
                  pass
  
          # Get local option
          local = self._options.get(option, default)
          # Check parent option
          parent = None
          if self.parent:
              parent = self.parent.opt(option)
          # Special handling for special flags (parent's yes always wins)
          if option in ['quiet', 'force', 'dry']:
              return parent if parent else local
          # Special handling for counting options (child overrides the
          # parent if it was defined)
          elif option in ['debug', 'verbose']:
              winner = local if local else parent
              if winner is None:
                  winner = 0
              return winner
          else:
              return parent if parent is not None else local
  
      def _level(self) -> int:
          """ Hierarchy level """
          if self.parent is None:
              return -1
          else:
              return self.parent._level() + self._relative_indent
  
      def _indent(
              self,
              key: str,
              value: Optional[str] = None,
              color: Optional[str] = None,
              shift: int = 0) -> str:
          """ Indent message according to the object hierarchy """
  
          return indent(
              key,
              value=value,
              color=color,
              level=self._level() + shift)
  
      def _log(self, message: str) -> None:
          """ Append provided message to the current log """
          # Nothing to do if there is no workdir
          if self.workdir is None:
              return
  
          # Store log only in the top parent
          if self.parent:
              self.parent._log(message)
          else:
              with open(os.path.join(self.workdir, LOG_FILENAME), 'a') as log:
                  log.write(datetime.datetime.utcnow().strftime('%H:%M:%S') + ' '
                            + remove_color(message) + '\n')
  
      def print(
              self,
              key: str,
              value: Optional[str] = None,
              color: Optional[str] = None,
              shift: int = 0,
              err: bool = False) -> None:
          """ Print a message regardless the quiet mode """
          self._log(self._indent(key, value, color=None, shift=shift))
          echo(self._indent(key, value, color, shift), err=err)
  
      def info(
              self,
              key: str,
              value: Optional[str] = None,
              color: Optional[str] = None,
              shift: int = 0,
              err: bool = False) -> None:
          """ Show a message unless in quiet mode """
          self._log(self._indent(key, value, color=None, shift=shift))
          if not self.opt('quiet'):
              echo(self._indent(key, value, color, shift), err=err)
  
      def verbose(
              self,
              key: str,
              value: Optional[str] = None,
              color: Optional[str] = None,
              shift: int = 0,
              level: int = 1,
              err: bool = False) -> None:
          """
          Show message if in requested verbose mode level
  
          In quiet mode verbose messages are not displayed.
          """
          self._log(self._indent(key, value, color=None, shift=shift))
          if not self.opt('quiet') and self.opt('verbose') >= level:
              echo(self._indent(key, value, color, shift), err=err)
  
      def debug(
              self,
              key: str,
              value: Optional[str] = None,
              color: Optional[str] = None,
              shift: int = 0,
              level: int = 1,
              err: bool = False) -> None:
          """
          Show message if in requested debug mode level
  
          In quiet mode debug messages are not displayed.
          """
          self._log(self._indent(key, value, color=None, shift=shift))
          if not self.opt('quiet') and self.opt('debug') >= level:
              echo(self._indent(key, value, color, shift), err=err)
  
      def warn(self, message: str, shift: int = 0) -> None:
          """ Show a yellow warning message on info level, send to stderr """
          self.info('warn', message, color='yellow', shift=shift, err=True)
  
      def fail(self, message: str, shift: int = 0) -> None:
          """ Show a red failure message on info level, send to stderr """
          self.info('fail', message, color='red', shift=shift, err=True)
  
      def _run(self,
               command: Union[str, List[str]],
               cwd: Optional[str],
               shell: bool,
               env: Optional[EnvironmentType],
               log: Optional[BaseLoggerFnType],
               join: bool = False,
               interactive: bool = False,
               timeout: Optional[int] = None) -> CommandOutput:
          """
          Run command, capture the output
  
          By default stdout and stderr are captured separately.
          Use join=True to merge stderr into stdout.
          Use timeout=<seconds> to finish process after given time
          """
          # By default command ouput is logged using debug
          if not log:
              log = self.debug
          # Prepare the environment
          if env:
              if not isinstance(env, dict):
                  raise GeneralError(f"Invalid environment '{env}'.")
              # Do not modify current process environment
              environment = os.environ.copy()
              environment.update(env)
          else:
              environment = None
          self.debug('environment', pprint.pformat(environment), level=4)
  
          # Set only for shell=True as it would affect command
          executable = DEFAULT_SHELL if shell else None
  
          # Run the command in interactive mode if requested
          if interactive:
              try:
                  subprocess.run(
                      command, cwd=cwd, shell=shell, env=environment, check=True,
                      executable=executable)
              except subprocess.CalledProcessError:
                  # Interactive mode can return non-zero if the last command
                  # failed, ignore errors here
                  pass
              finally:
                  return CommandOutput(None, None)
  
          # Create the process
          try:
              process = subprocess.Popen(
                  command, cwd=cwd, shell=shell, env=environment,
                  stdin=subprocess.DEVNULL, stdout=subprocess.PIPE,
                  stderr=subprocess.STDOUT if join else subprocess.PIPE,
                  executable=executable)
          except FileNotFoundError as error:
              raise RunError(
                  f"File '{error.filename}' not found.", command, 127)
  
          stdout_thread = StreamLogger(
              process.stdout, log_header='out', logger=log)
          stderr_thread = stdout_thread
          if not join:
              stderr_thread = StreamLogger(
                  process.stderr, log_header='err', logger=log)
          stdout_thread.start()
          if not join:
              stderr_thread.start()
          try:
              process.wait(timeout=timeout)
          except subprocess.TimeoutExpired:
              process.kill()
              process.returncode = PROCESS_TIMEOUT
          stdout_thread.join()
          if not join:
              stderr_thread.join()
  
          # Handle the exit code, return output
          if process.returncode != 0:
              raise RunError(
                  message=f"Command returned '{process.returncode}'.",
                  command=command,
                  returncode=process.returncode,
                  stdout=stdout_thread.get_output(),
                  stderr=stderr_thread.get_output())
          if join:
              return CommandOutput(
                  stdout_thread.get_output(), None)
          else:
              return CommandOutput(
                  stdout_thread.get_output(), stderr_thread.get_output())
  
      def run(self,
              command: Union[str, List[str]],
              message: Optional[str] = None,
              cwd: Optional[str] = None,
              dry: bool = False,
              shell: bool = False,
              env: Optional[EnvironmentType] = None,
              interactive: bool = False,
              join: bool = False,
              log: Optional[BaseLoggerFnType] = None,
              timeout: Optional[int] = None) -> CommandOutput:
          """
          Run command, give message, handle errors
  
          Command is run in the workdir be default.
          In dry mode commands are not executed unless dry=True.
          Environment is updated with variables from the 'env' dictionary.
          Output is logged using self.debug() or custom 'log' function.
          Returns stdout if join=True, (stdout, stderr) tuple otherwise.
          """
  
          # A bit of logging - command, default message, error message for later...
          if isinstance(command, (list, tuple)):
              printable_command = ' '.join(shlex.quote(s) for s in command)
          else:
              printable_command = command
  
          if message:
              self.debug(message, level=2)
  
          self.debug(f'Run command: {printable_command}', level=2)
  
          # Nothing more to do in dry mode (unless requested)
          if self.opt('dry') and not dry:
              return CommandOutput(None, None)
  
          # Run the command, handle the exit code
          cwd = cwd or self.workdir
  
          # Fail nicely if the working directory does not exist
          if cwd and not os.path.exists(cwd):
              raise GeneralError(
                  f"The working directory '{cwd}' does not exist.")
  
          try:
              return self._run(
                  command, cwd, shell, env, log, join, interactive, timeout)
          except RunError as error:
              self.debug(error.message, level=3)
              message = f"Failed to run command: {printable_command} Reason: {error.message}"
              raise RunError(
                  message, error.command, error.returncode,
                  error.stdout, error.stderr)
  
      def read(self, path: str, level: int = 2) -> str:
          """ Read a file from the workdir """
          if self.workdir:
              path = os.path.join(self.workdir, path)
          self.debug(f"Read file '{path}'.", level=level)
          try:
              with open(path, encoding='utf-8', errors='replace') as data:
                  return data.read()
          except OSError as error:
              raise FileError(f"Failed to read '{path}'.\n{error}")
  
      def write(
              self,
              path: str,
              data: Any,
              mode: str = 'w',
              level: int = 2) -> None:
          """ Write a file to the workdir """
          if self.workdir:
              path = os.path.join(self.workdir, path)
          action = 'Append to' if mode == 'a' else 'Write'
          self.debug(f"{action} file '{path}'.", level=level)
          # Dry mode
          if self.opt('dry'):
              return
          try:
              with open(path, mode, encoding='utf-8', errors='replace') as file:
                  file.write(data)
          except OSError as error:
              raise FileError(f"Failed to write '{path}'.\n{error}")
  
      def _workdir_init(self, id_: WorkdirArgumentType = None) -> None:
          """
          Initialize the work directory
  
          Workdir under WORKDIR_ROOT is used/created if 'id' is provided.
          If 'id' is a path, that directory is used instead. Otherwise a
          new workdir is created under the WORKDIR_ROOT directory.
          """
          # Prepare the workdir name from given id or path
          if isinstance(id_, str):
              # Use provided directory if full path given
              if '/' in id_:
                  workdir = id_
              # Construct directory name under workdir root
              else:
                  workdir = os.path.join(WORKDIR_ROOT, id_)
          # Weird workdir id
          elif id_ is not None:
              raise GeneralError(
                  f"Invalid workdir '{id_}', expected a string or None.")
  
          def _check_or_create_workdir_root_with_perms() -> None:
              """ If created WORKDIR_ROOT has to be 1777 for multi-user"""
              if not os.path.isdir(WORKDIR_ROOT):
                  try:
                      os.makedirs(WORKDIR_ROOT, exist_ok=True)
                      os.chmod(WORKDIR_ROOT, 0o1777)
                  except OSError as error:
                      raise FileError(f"Failed to prepare workdir '{WORKDIR_ROOT}': {error}")
  
          if id_ is None:
              # Prepare WORKDIR_ROOT first
              _check_or_create_workdir_root_with_perms()
  
              # Generated unique id or fail, has to be atomic call
              for id_bit in range(1, WORKDIR_MAX + 1):
                  directory = 'run-{}'.format(str(id_bit).rjust(3, '0'))
                  workdir = os.path.join(WORKDIR_ROOT, directory)
                  try:
                      # Call is atomic, no race possible
                      os.makedirs(workdir)
                      break
                  except FileExistsError:
                      pass
              else:
                  raise GeneralError(
                      f"Workdir full. Cleanup the '{WORKDIR_ROOT}' directory.")
          else:
              # Cleanup possible old workdir if called with --scratch
              if self.opt('scratch'):
                  self._workdir_cleanup(workdir)
  
              if workdir.startswith(WORKDIR_ROOT):
                  _check_or_create_workdir_root_with_perms()
  
              # Create the workdir
              create_directory(workdir, 'workdir', quiet=True)
          self._workdir = workdir
  
      def _workdir_name(self) -> Optional[str]:
          """ Construct work directory name from parent workdir """
          # Need the parent workdir
          if self.parent is None or self.parent.workdir is None:
              return None
          # Join parent name with self
          return os.path.join(self.parent.workdir, self.safe_name.lstrip("/"))
  
      def _workdir_load(self, workdir: WorkdirArgumentType) -> None:
          """
          Create the given workdir if it is not None
  
          If workdir=True, the directory name is automatically generated.
          """
          if workdir is True:
              self._workdir_init()
          elif workdir is not None:
              self._workdir_init(workdir)
  
      def _workdir_cleanup(self, path: Optional[str] = None) -> None:
          """ Clean up the work directory """
          directory = path or self._workdir_name()
          if directory is not None:
              if os.path.isdir(directory):
                  self.debug(f"Clean up workdir '{directory}'.", level=2)
                  shutil.rmtree(directory)
          self._workdir = None
  
      @property
      def workdir(self) -> Optional[str]:
          """ Get the workdir, create if does not exist """
          if self._workdir is None:
              self._workdir = self._workdir_name()
              # Workdir not enabled, even parent does not have one
              if self._workdir is None:
                  return None
              # Create a child workdir under the parent workdir
              create_directory(self._workdir, 'workdir', quiet=True)
          return self._workdir
  
  # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  #  Exceptions
  # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  
  
  class GeneralError(Exception):
      """ General error """
  
      def __init__(self, *args: Any, **kwargs: Any) -> None:
          # Store the original exception for future use
          self.original = kwargs.get('original')
  
  
  class GitUrlError(GeneralError):
      """ Remote git url is not reachable """
  
  
  class FileError(GeneralError):
      """ File operation error """
  
  
  class RunError(GeneralError):
      """ Command execution error """
  
      def __init__(
              self,
              message: str,
              command: Union[str, List[str]],
              returncode: int,
              stdout: Optional[str] = None,
              stderr: Optional[str] = None,
              *args: Any,
              **kwargs: Any) -> None:
          super().__init__(*args, **kwargs)
          self.message = message
          self.command = command
          self.returncode = returncode
          self.stdout = stdout
          self.stderr = stderr
  
  
  class MetadataError(GeneralError):
      """ General metadata error """
  
  
  class SpecificationError(MetadataError):
      """ Metadata specification error """
  
      def __init__(
              self,
              message: str,
              validation_errors: Optional[List[Tuple[jsonschema.ValidationError, str]]] = None,
              *args: Any,
              **kwargs: Any) -> None:
          super().__init__(*args, **kwargs)
  
          self.message = message
          self.validation_errors = validation_errors
  
  
  class ConvertError(MetadataError):
      """ Metadata conversion error """
  
  
  class StructuredFieldError(GeneralError):
      """ StructuredField parsing error """
  
  
  class WaitingIncomplete(GeneralError):
      """ Waiting incomplete """
  
  
  class WaitingTimedOutError(GeneralError):
      """ Waiting ran out of time """
  
  
  # Step exceptions
  
  
  class DiscoverError(GeneralError):
      """ Discover step error """
  
  
  class ProvisionError(GeneralError):
      """ Provision step error """
  
  
  class PrepareError(GeneralError):
      """ Prepare step error """
  
  
  class ExecuteError(GeneralError):
      """ Execute step error """
  
  
  class RebootTimeoutError(ExecuteError):
      """ Reboot failed due to a timeout """
  
  
  class ReportError(GeneralError):
      """ Report step error """
  
  
  class FinishError(GeneralError):
      """ Finish step error """
  
  
  # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  #  Utilities
  # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  
  def quote(string: str) -> str:
      """ Surround a string with double quotes """
      return f'"{string}"'
  
  
  def ascii(text: Any) -> bytes:
      """ Transliterate special unicode characters into pure ascii """
      if not isinstance(text, str):
          text = str(text)
      return unicodedata.normalize('NFKD', text).encode('ascii', 'ignore')
  
  
  def listify(
          data: Union[Tuple[Any, ...], List[Any], str, Dict[Any, Any]],
          split: bool = False,
          keys: Optional[List[str]] = None) -> Union[List[Any], Dict[Any, Any]]:
      """
      Ensure that variable is a list, convert if necessary
      For dictionaries check all items or only those with provided keys.
      Also split strings on white-space/comma if split=True.
      """
      separator = re.compile(r'[\s,]+')
      if isinstance(data, tuple):
          data = list(data)
      if isinstance(data, list):
          return fmf.utils.split(data, separator) if split else data
      if isinstance(data, str):
          return fmf.utils.split(data, separator) if split else [data]
      if isinstance(data, dict):
          for key in keys or data:
              if key in data:
                  data[key] = listify(data[key], split=split)
          return data
      return [data]
  
  
  def copytree(
          src: str,
          dst: str,
          symlinks: bool = False,
          dirs_exist_ok: bool = False,
          ) -> Any:
      """ Similar to shutil.copytree but with dirs_exist_ok for Python < 3.8 """
      # No need to reimplement for newer python or if argument is not requested
      if not dirs_exist_ok or sys.version_info >= (3, 8):
          return shutil.copytree(
              src=src, dst=dst, symlinks=symlinks, dirs_exist_ok=dirs_exist_ok)
      # Choice was to either copy python implementation and change ONE line
      # or use rsync (or cp with shell)
      # We need to copy CONTENT of src into dst
      # so src has to end with / and dst cannot
      if src[-1] != '/':
          src += '/'
      if dst[-1] == '/':
          dst = dst[:-1]
  
      command = ["rsync", "-r"]
      if symlinks:
          command.append('-l')
      command.extend([src, dst])
  
      log.debug(f"Calling command '{command}'.")
      outcome = subprocess.run(
          command,
          stdin=subprocess.DEVNULL, stdout=subprocess.PIPE,
          stderr=subprocess.STDOUT, universal_newlines=True)
  
      if outcome.returncode != 0:
          raise shutil.Error(
              [f"Unable to copy '{src}' into '{dst}' using rsync.",
               outcome.returncode, outcome.stdout])
      return dst
  
  
  # These two are helpers for shell_to_dict and environment_to_dict -
  # there is some overlap of their functionality.
  def _add_simple_var(result: EnvironmentType, var: str) -> None:
      """
      Add a single NAME=VALUE pair into result dictionary
  
      Parse given string VAR to its constituents, NAME and VALUE, and add
      them to the provided dict.
      """
  
      matched = re.match("([^=]+)=(.*)", var)
      if not matched:
          raise GeneralError(f"Invalid variable specification '{var}'.")
      name, value = matched.groups()
      result[name] = value
  
  
  def _add_file_vars(result: EnvironmentType, filepath: str) -> None:
      """
      Add variables loaded from file into the result dictionary
  
      Load mapping from a YAML file 'filepath', and add its content -
      "name: value" entries - to the provided dict.
      """
  
      if not filepath[1:]:
          raise GeneralError(
              f"Invalid variable file specification '{filepath}'.")
  
      try:
          with open(filepath[1:], 'r') as file:
              # Handle empty file as an empty environment
              content = file.read()
              if not content:
                  log.warn(f"Empty environment file '{filepath}'.")
                  return
              file_vars = yaml_to_dict(content)
      except Exception as exception:
          raise GeneralError(
              f"Failed to load variables from '{filepath}': {exception}")
  
      for name, value in file_vars.items():
          result[name] = str(value)
  
  
  def shell_to_dict(variables: Union[str, List[str]]) -> EnvironmentType:
      """
      Convert shell-like variables into a dictionary
  
      Accepts single string or list of strings. Allowed forms are:
      'X=1'
      'X=1 Y=2 Z=3'
      ['X=1', 'Y=2', 'Z=3']
      ['X=1 Y=2 Z=3', 'A=1 B=2 C=3']
      'TXT="Some text with spaces in it"'
      """
      if not isinstance(variables, (list, tuple)):
          variables = [variables]
      result: EnvironmentType = dict()
      for variable in variables:
          if variable is None:
              continue
          for var in shlex.split(variable):
              _add_simple_var(result, var)
  
      return result
  
  
  def environment_to_dict(variables: Union[str, List[str]]) -> EnvironmentType:
      """
      Convert environment variables into a dictionary
  
      Variables may be specified in the following two ways:
  
      * NAME=VALUE pairs
      * @foo.yaml
  
      If "variable" starts with "@" character, it is treated as a path to
      a YAML file that contains "key: value" pairs which are then
      transparently loaded and added to the final dictionary.
  
      In general, allowed inputs are the same as in "shell_to_dict"
      function, with the addition of "@foo.yaml" form:
      'X=1'
      'X=1 Y=2 Z=3'
      ['X=1', 'Y=2', 'Z=3']
      ['X=1 Y=2 Z=3', 'A=1 B=2 C=3']
      'TXT="Some text with spaces in it"'
      @foo.yaml
      @../../bar.yaml
      """
  
      if not isinstance(variables, (list, tuple)):
          variables = [variables]
      result: EnvironmentType = dict()
  
      for variable in variables:
          if variable is None:
              continue
          for var in shlex.split(variable):
              if var.startswith('@'):
                  _add_file_vars(result, var)
              else:
                  _add_simple_var(result, var)
  
      return result
  
  
  @lru_cache(maxsize=None)
  def environment_file_to_dict(env_file: str, root: str = ".") -> EnvironmentType:
      """
      Read environment variables from the given file.
  
      File should be in YAML format (``.yaml`` or ``.yml`` suffixes), or in dotenv format.
  
      .. code-block:: bash
         :caption: dotenv file example
  
         A=B
         C=D
  
      .. code-block:: yaml
         :caption: YAML file example
  
         A: B
         C: D
  
      Path to each file should be relative to the metadata tree root.
  
      .. note::
  
         For loading environment variables from multiple files, see
         :py:func:`environment_files_to_dict`.
      """
  
      env_file = env_file.strip()
  
      # Fetch a remote file
      if env_file.startswith("http"):
          # Create retry session for longer retries, see #1229
          session = retry_session.create(
              retries=ENVFILE_RETRY_SESSION_RETRIES,
              backoff_factor=ENVFILE_RETRY_SESSION_BACKOFF_FACTOR,
              allowed_methods=('GET',),
              status_forcelist=(
                  429,  # Too Many Requests
                  500,  # Internal Server Error
                  502,  # Bad Gateway
                  503,  # Service Unavailable
                  504   # Gateway Timeout
                  ),
              )
          try:
              response = session.get(env_file)
              response.raise_for_status()
              content = response.text
          except requests.RequestException as error:
              raise GeneralError(
                  f"Failed to fetch the environment file from '{env_file}'. "
                  f"The problem was: '{error}'")
  
      # Read a local file
      else:
          # Ensure we don't escape from the metadata tree root
          try:
              root_path = Path(root).resolve()
              full_path = (Path(root_path) / Path(env_file)).resolve()
              full_path.relative_to(root_path)
          except ValueError:
              raise GeneralError(
                  f"The 'environment-file' path '{full_path}' is outside "
                  f"of the metadata tree root '{root}'.")
          if not Path(full_path).is_file():
              raise GeneralError(f"File '{full_path}' doesn't exist.")
  
          content = Path(full_path).read_text()
  
      # Parse yaml file
      if os.path.splitext(env_file)[1].lower() in ('.yaml', '.yml'):
          environment = parse_yaml(content)
  
      else:
          try:
              environment = parse_dotenv(content)
  
          except ValueError:
              raise GeneralError(
                  f"Failed to extract variables from environment file "
                  f"'{full_path}'. Ensure it has the proper format "
                  f"(i.e. A=B).")
  
      if not environment:
          log.warn(f"Empty environment file '{env_file}'.")
  
          return {}
  
      return environment
  
  
  def environment_files_to_dict(env_files: Iterable[str], root: str = ".") -> EnvironmentType:
      """
      Read environment variables from the given list of files.
  
      Files should be in YAML format (``.yaml`` or ``.yml`` suffixes), or in dotenv format.
  
      .. code-block:: bash
         :caption: dotenv file example
  
         A=B
         C=D
  
      .. code-block:: yaml
         :caption: YAML file example
  
         A: B
         C: D
  
      Path to each file should be relative to the metadata tree root.
  
      .. note::
  
         For loading environment variables from a single file, see
         :py:func:`environment_file_to_dict`, which is a function
         ``environment_files_to_dict()`` calls for each file,
         accumulating data from all input files.
      """
  
      result: EnvironmentType = {}
  
      for env_file in env_files:
          result.update(environment_file_to_dict(env_file, root=root))
  
      return result
  
  
  @contextlib.contextmanager
  def modify_environ(
          new_elements: EnvironmentType) -> Generator[None, None, None]:
      """ A context manager for os.environ that restores the initial state """
      environ_backup = os.environ.copy()
      os.environ.clear()
      os.environ.update(new_elements)
      try:
          yield
      finally:
          os.environ.clear()
          os.environ.update(environ_backup)
  
  
  def context_to_dict(context: List[str]) -> FmfContextType:
      """
      Convert command line context definition into a dictionary
  
      Does the same as environment_to_dict() plus separates possible
      comma-separated values into lists. Here's a couple of examples:
  
      distro=fedora-33 ---> {'distro': ['fedora']}
      arch=x86_64,ppc64 ---> {'arch': ['x86_64', 'ppc64']}
      """
      return {
          key: value.split(',')
          for key, value in environment_to_dict(context).items()}
  
  
  def dict_to_yaml(
          data: Union[Dict[str, Any], List[Any]],
          width: Optional[int] = None,
          sort: bool = False,
          start: bool = False) -> str:
      """ Convert dictionary into yaml """
      output = io.StringIO()
      yaml = YAML()
      yaml.indent(mapping=4, sequence=4, offset=2)
      yaml.default_flow_style = False
      yaml.allow_unicode = True
      yaml.encoding = 'utf-8'
      yaml.width = width
      yaml.explicit_start = start
      # Convert multiline strings
      scalarstring.walk_tree(data)
      if sort:
          # Sort the data https://stackoverflow.com/a/40227545
          sorted_data = CommentedMap()
          for key in sorted(data):
              sorted_data[key] = data[key]
          data = sorted_data
      yaml.dump(data, output)
      return output.getvalue()
  
  
  YamlTypType = Literal['rt', 'safe', 'unsafe', 'base']
  
  
  def yaml_to_dict(data: Any,
                   yaml_type: Optional[YamlTypType] = None) -> Dict[Any, Any]:
      """ Convert yaml into dictionary """
      yaml = YAML(typ=yaml_type)
      loaded_data = yaml.load(data)
      if loaded_data is None:
          return dict()
      if not isinstance(loaded_data, dict):
          raise GeneralError(
              f"Expected dictionary in yaml data, "
              f"got '{type(loaded_data).__name__}'.")
      return loaded_data
  
  
  def yaml_to_list(data: Any,
                   yaml_type: Optional[YamlTypType] = 'safe') -> List[Any]:
      """ Convert yaml into list """
      yaml = YAML(typ=yaml_type)
      try:
          loaded_data = yaml.load(data)
      except ParserError as error:
          raise GeneralError(f"Invalid yaml syntax: {error}")
  
      if loaded_data is None:
          return list()
      if not isinstance(loaded_data, list):
          raise GeneralError(
              f"Expected list in yaml data, "
              f"got '{type(loaded_data).__name__}'.")
      return list(loaded_data)
  
  
  def key_to_option(key: str) -> str:
      """ Convert a key name to corresponding option name """
  
      return key.replace('_', '-')
  
  
  def option_to_key(option: str) -> str:
      """ Convert an option name to corresponding key name """
  
      return option.replace('-', '_')
  
  
  @dataclasses.dataclass
  class DataContainer:
      """ A base class for objects that have keys and values """
  
      def to_dict(self) -> Dict[str, Any]:
          """
          Convert to a mapping.
  
          See :ref:`classes.rst` for more details.
          """
  
          return dataclasses.asdict(self)
  
      # This method should remain a class-method: 1. list of keys is known
      # already, therefore it's not necessary to create an instance, and
      # 2. some functionality makes use of this knowledge.
      @classmethod
      def keys(cls) -> Generator[str, None, None]:
          """ Iterate over key names """
  
          for field in dataclasses.fields(cls):
              yield field.name
  
      def values(self) -> Generator[Any, None, None]:
          """ Iterate over key values """
  
          yield from self.to_dict().values()
  
      def items(self) -> Generator[Tuple[str, Any], None, None]:
          """ Iterate over key/value pairs """
  
          yield from self.to_dict().items()
  
      @classmethod
      def _default(cls, key: str, default: Any = None) -> Any:
          """
          Return a default value for a given key.
  
          Keys may have a default value, or a default *factory* has been specified.
  
          :param key: key to look for.
          :param default: when key has no default value, ``default`` is returned.
          :returns: a default value defined for the key, or its ``default_factory``'s
              return value of ``default_factory``, or ``default`` when key has no
              default value.
          """
  
          for field in dataclasses.fields(cls):
              if key != field.name:
                  continue
  
              if not isinstance(field.default_factory, dataclasses._MISSING_TYPE):
                  return field.default_factory()
  
              if not isinstance(field.default, dataclasses._MISSING_TYPE):
                  return field.default
  
          else:
              return default
  
      @property
      def is_bare(self) -> bool:
          """
          Check whether all keys are either unset or have their default value.
  
          :returns: ``True`` if all keys either hold their default value
              or are not set at all, ``False`` otherwise.
          """
  
          for field in dataclasses.fields(self):
              value = getattr(self, field.name)
  
              if not isinstance(field.default_factory, dataclasses._MISSING_TYPE):
                  if value != field.default_factory():
                      return False
  
              elif not isinstance(field.default, dataclasses._MISSING_TYPE):
                  if value != field.default:
                      return False
  
              else:
                  pass
  
          return True
  
  
  SpecBasedContainerT = TypeVar('SpecBasedContainerT', bound='SpecBasedContainer')
  
  
  class SpecBasedContainer(DataContainer):
      @classmethod
      def from_spec(cls: Type[SpecBasedContainerT], spec: Any) -> SpecBasedContainerT:
          """
          Convert from a specification file or from a CLI option
  
          See :ref:`classes.rst` for more details.
  
          See :py:meth:`to_spec` for its counterpart.
          """
  
          raise NotImplementedError()
  
      def to_spec(self) -> Dict[str, Any]:
          """
          Convert to a form suitable for saving in a specification file
  
          See :ref:`classes.rst` for more details.
  
          See :py:meth:`from_spec` for its counterpart.
          """
  
          return self.to_dict()
  
  
  SerializableContainerDerivedType = TypeVar(
      'SerializableContainerDerivedType',
      bound='SerializableContainer')
  
  
  @dataclasses.dataclass
  class SerializableContainer(DataContainer):
      """ A mixin class for saving and loading objects """
  
      @classmethod
      def default(cls, key: str, default: Any = None) -> Any:
          return cls._default(key, default=default)
  
      #
      # Moving data between containers and objects owning them
      #
  
      def inject_to(self, obj: Any) -> None:
          """
          Inject keys from this container into attributes of a given object
          """
  
          for name, value in self.items():
              setattr(obj, name, value)
  
      @classmethod
      def extract_from(cls: Type[SerializableContainerDerivedType],
                       obj: Any) -> SerializableContainerDerivedType:
          """ Extract keys from given object, and save them in a container """
  
          data = cls()
  
          for key in cls.keys():
              value = getattr(obj, key)
              if value is not None:
                  setattr(data, key, value)
  
          return data
  
      #
      # Serialization - writing containers into YAML files, and restoring
      # them later.
      #
  
      def to_serialized(self) -> Dict[str, Any]:
          """
          Convert to a form suitable for saving in a file.
  
          See :ref:`classes.rst` for more details.
  
          See :py:meth:`from_serialized` for its counterpart.
          """
  
          fields = self.to_dict()
  
          # Add a special field tracking what class we just shattered to pieces.
          fields.update({
              '__class__': {
                  'module': self.__class__.__module__,
                  'name': self.__class__.__name__
                  }
              })
  
          return fields
  
      @classmethod
      def from_serialized(
              cls: Type[SerializableContainerDerivedType],
              serialized: Dict[str, Any]) -> SerializableContainerDerivedType:
          """
          Convert from a serialized form loaded from a file.
  
          See :ref:`classes.rst` for more details.
  
          See :py:meth:`to_serialized` for its counterpart.
          """
  
          # Our special key may or may not be present, depending on who
          # calls this method.  In any case, it is not needed, because we
          # already know what class to restore: this one.
          serialized.pop('__class__', None)
  
          return cls(**serialized)
  
      @staticmethod
      def unserialize(serialized: Dict[str, Any]
                      ) -> SerializableContainerDerivedType:
          """
          Convert from a serialized form loaded from a file.
  
          Similar to :py:meth:`from_serialized`, but this method knows
          nothing about container's class, and will locate the correct
          module and class by inspecting serialized data. Discovered
          class' :py:meth:`from_serialized` is then used to create the
          container.
  
          Used to transform data read from a YAML file into original
          containers when their classes are not know to the code.
          Restoring such containers requires inspection of serialized data
          and dynamic imports of modules as needed.
  
          See :ref:`classes.rst` for more details.
  
          See :py:meth:`to_serialized` for its counterpart.
          """
  
          from tmt.plugins import import_member
  
          # Unpack class info, to get nicer variable names
          if "__class__" not in serialized:
              raise GeneralError(
                  "Failed to load saved state, probably because of old data format.\n"
                  "Use 'tmt clean runs' to clean up old runs.")
  
          klass_info = serialized.pop('__class__')
          klass = import_member(klass_info['module'], klass_info['name'])
  
          # Stay away from classes that are not derived from this one, to
          # honor promise given by return value annotation.
          assert issubclass(klass, SerializableContainer)
  
          # Apparently, the issubclass() check above is not good enough for mypy.
          return cast(
              SerializableContainerDerivedType,
              klass.from_serialized(serialized))
  
  
  def markdown_to_html(filename: str) -> str:
      """
      Convert markdown to html
  
      Expects: Markdown document as a file.
      Returns: An HTML document as a string.
      """
      try:
          import markdown
      except ImportError:
          raise ConvertError("Install tmt-test-convert to export tests.")
  
      try:
          with open(filename, 'r') as file:
              try:
                  text = file.read()
              except UnicodeError:
                  raise MetadataError(f"Unable to read '{filename}'.")
              return markdown.markdown(text)
      except IOError:
          raise ConvertError(f"Unable to open '{filename}'.")
  
  
  def shell_variables(
          data: Union[List[str], Tuple[str, ...], Dict[str, Any]]) -> List[str]:
      """
      Prepare variables to be consumed by shell
  
      Convert dictionary or list/tuple of key=value pairs to list of
      key=value pairs where value is quoted with shlex.quote().
      """
  
      # Convert from list/tuple
      if isinstance(data, list) or isinstance(data, tuple):
          converted_data = []
          for item in data:
              splitted_item = item.split('=')
              key = splitted_item[0]
              value = shlex.quote('='.join(splitted_item[1:]))
              converted_data.append(f'{key}={value}')
          return converted_data
  
      # Convert from dictionary
      return [f"{key}={shlex.quote(str(value))}" for key, value in data.items()]
  
  
  def duration_to_seconds(duration: str) -> int:
      """ Convert sleep time format into seconds """
      units = {
          's': 1,
          'm': 60,
          'h': 60 * 60,
          'd': 60 * 60 * 24,
          }
      try:
          match = re.match(r'^(\d+)([smhd]?)$', str(duration))
          if match is None:
              raise SpecificationError(f"Invalid duration '{duration}'.")
          number, suffix = match.groups()
          return int(number) * units.get(suffix, 1)
      except (ValueError, AttributeError):
          raise SpecificationError(f"Invalid duration '{duration}'.")
  
  
  def verdict(
          decision: Optional[bool],
          comment: Optional[str] = None,
          good: str = 'pass',
          bad: str = 'fail',
          problem: str = 'warn',
          **kwargs: Any) -> Optional[bool]:
      """
      Print verdict in green, red or yellow based on the decision
  
      The supported decision values are:
  
          True .... good (green)
          False ... bad (red)
          None .... problem (yellow)
  
      Anything else raises an exception. Additional arguments
      are passed to the `echo` function. Returns back the decision.
      """
  
      if decision is False:
          text = style(bad, fg='red')
      elif decision is True:
          text = style(good, fg='green')
      elif decision is None:
          text = style(problem, fg='yellow')
      else:
          raise GeneralError(
              "Invalid decision value, must be 'True', 'False' or 'None'.")
      if comment:
          text = text + ' ' + comment
      echo(text, **kwargs)
      return decision
  
  
  def format(
          key: str,
          value: Union[None, str, List[Any], Dict[Any, Any]] = None,
          indent: int = 12,
          width: int = 72,
          wrap: Literal[True, False, 'auto'] = 'auto',
          key_color: str = 'green',
          value_color: str = 'black') -> str:
      """
      Nicely format and indent a key-value pair
  
      The following values for 'wrap' are supported:
  
          True .... always reformat text and wrap long lines
          False ... preserve text, no new line changes
          auto .... wrap only if text contains a long line
      """
      indent_string = (indent + 1) * ' '
      # Key
      output = '{} '.format(str(key).rjust(indent, ' '))
      if key_color is not None:
          output = style(output, fg=key_color)
      # Bool
      if isinstance(value, bool):
          output += ('true' if value else 'false')
      # List
      elif isinstance(value, list):
          # Make sure everything is string, prepare list, check for spaces
          value = [str(item) for item in value]
          listed_text = fmf.utils.listed(value)
          has_spaces = any(item.find(' ') > -1 for item in value)
          # Use listed output only for short lists without spaces
          if len(listed_text) < width - indent and not has_spaces:
              output += listed_text
          # Otherwise just place each item on a new line
          else:
              output += ('\n' + indent_string).join(value)
      # Dictionary
      elif isinstance(value, dict):
          # Place each key value pair on a separate line
          output += ('\n' + indent_string).join(
              f'{item[0]}: {item[1]}' for item in value.items())
      # Text
      elif isinstance(value, str):
          # In 'auto' mode enable wrapping when long lines present
          if wrap == 'auto':
              wrap = any(len(line) + indent - 7 > width for line in value.split('\n'))
          if wrap:
              output += (wrap_text(
                  value, width=width,
                  preserve_paragraphs=True,
                  initial_indent=indent_string,
                  subsequent_indent=indent_string).lstrip())
          else:
              output += (('\n' + indent_string).join(
                  value.rstrip().split('\n')))
      else:
          output += str(value)
      return output
  
  
  def create_directory(
          path: str,
          name: str,
          dry: bool = False,
          quiet: bool = False) -> None:
      """ Create a new directory, handle errors """
      say = log.debug if quiet else echo
      if os.path.isdir(path):
          say("Directory '{}' already exists.".format(path))
          return
      if dry:
          say("Directory '{}' would be created.".format(path))
          return
      try:
          os.makedirs(path, exist_ok=True)
          say("Directory '{}' created.".format(path))
      except OSError as error:
          raise FileError("Failed to create {} '{}' ({})".format(
              name, path, error))
  
  
  def create_file(
          path: str,
          content: str,
          name: str,
          dry: bool = False,
          force: bool = False,
          mode: int = 0o664,
          quiet: bool = False) -> None:
      """ Create a new file, handle errors """
      say = log.debug if quiet else echo
      action = 'would be created' if dry else 'created'
      if os.path.exists(path):
          if force:
              action = 'would be overwritten' if dry else 'overwritten'
          else:
              raise FileError("File '{}' already exists.".format(path))
  
      if dry:
          say("{} '{}' {}.".format(name.capitalize(), path, action))
          return
  
      try:
          with open(path, 'w') as file_:
              file_.write(content)
          say("{} '{}' {}.".format(name.capitalize(), path, action))
          os.chmod(path, mode)
      except OSError as error:
          raise FileError("Failed to create {} '{}' ({})".format(
              name, path, error))
  
  
  # Avoid multiple subprocess calls for the same url
  @lru_cache(maxsize=None)
  def check_git_url(url: str) -> str:
      """ Check that a remote git url is accessible """
      try:
          log.debug(f"Check git url '{url}'.")
          subprocess.check_call(
              ["git", "ls-remote", "--heads", url],
              stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
              env={"GIT_ASKPASS": "echo", "GIT_TERMINAL_PROMPT": "0"})
          return url
      except subprocess.CalledProcessError:
          raise GitUrlError(f"Unable to contact remote git via '{url}'.")
  
  
  def public_git_url(url: str) -> str:
      """
      Convert a git url into a public format
  
      Return url in the format which can be accessed without
      authentication. For now just cover the most common services.
      """
  
      # GitHub, GitLab
      # old: git@github.com:teemtee/tmt.git
      # new: https://github.com/teemtee/tmt.git
      matched = re.match('git@(.*):(.*)', url)
      if matched:
          host, project = matched.groups()
          return f'https://{host}/{project}'
  
      # RHEL packages
      # old: git+ssh://psplicha@pkgs.devel.redhat.com/tests/bash
      # old: ssh://psplicha@pkgs.devel.redhat.com/tests/bash
      # old: ssh://pkgs.devel.redhat.com/tests/bash
      # new: git://pkgs.devel.redhat.com/tests/bash
      matched = re.match(
          r'(git\+)?ssh://(\w+@)?(pkgs\.devel\.redhat\.com)/(.*)', url)
      if matched:
          _, _, host, project = matched.groups()
          return f'git://{host}/{project}'
  
      # Fedora packages, Pagure
      # old: git+ssh://psss@pkgs.fedoraproject.org/tests/shell
      # old: ssh://psss@pkgs.fedoraproject.org/tests/shell
      # new: https://pkgs.fedoraproject.org/tests/shell
      matched = re.match(r'(git\+)?ssh://(\w+@)?([^/]*)/(.*)', url)
      if matched:
          _, _, host, project = matched.groups()
          return f'https://{host}/{project}'
  
      # Otherwise return unmodified
      return url
  
  
  class TimeoutHTTPAdapter(requests.adapters.HTTPAdapter):
      """
      Spice up request's session with custom timeout.
      """
  
      def __init__(self, *args: Any, **kwargs: Any) -> None:
          self.timeout = kwargs.pop('timeout', None)
  
          super().__init__(*args, **kwargs)
  
      def send(  # type: ignore[override] # does not match superclass type on purpose
              self,
              request: requests.PreparedRequest,
              **kwargs: Any) -> requests.Response:
          kwargs.setdefault('timeout', self.timeout)
  
          return super().send(request, **kwargs)
  
  
  class RetryStrategy(requests.packages.urllib3.util.retry.Retry):  # type: ignore[misc]
      def increment(
              self,
              *args: Any,
              **kwargs: Any
              ) -> requests.packages.urllib3.util.retry.Retry:
          error = cast(Optional[Exception], kwargs.get('error', None))
  
          # Detect a subset of exception we do not want to follow with a retry.
          if error is not None:
              # Failed certificate verification - this issue will probably not get any better
              # should we try again.
              if isinstance(error, urllib3.exceptions.SSLError) \
                      and 'certificate verify failed' in str(error):
  
                  # [mpr] I'm not sure how stable this *iternal* API is, but pool seems to be the
                  # only place aware of the remote hostname. Try our best to get the hostname for
                  # a better error message, but don't crash because of a missing attribute or
                  # something as dumb.
  
                  connection_pool = kwargs.get('_pool', None)
  
                  if connection_pool is not None and hasattr(connection_pool, 'host'):
                      message = f"Certificate verify failed for '{connection_pool.host}'."
                  else:
                      message = 'Certificate verify failed.'
  
                  raise GeneralError(message, original=error) from error
  
          return super().increment(*args, **kwargs)
  
  
  class retry_session(contextlib.AbstractContextManager):  # type: ignore[type-arg]
      """
      Context manager for requests.Session() with retries and timeout
      """
      @staticmethod
      def create(
              retries: int = DEFAULT_RETRY_SESSION_RETRIES,
              backoff_factor: float = DEFAULT_RETRY_SESSION_BACKOFF_FACTOR,
              allowed_methods: Optional[Tuple[str, ...]] = None,
              status_forcelist: Optional[Tuple[int, ...]] = None,
              timeout: Optional[int] = None
              ) -> requests.Session:
          retry_strategy = RetryStrategy(
              total=retries,
              status_forcelist=status_forcelist,
              # `method_whitelist`` has been renamed to `allowed_methods` since
              # urllib3 1.26, and it will be removed in urllib3 2.0.
              # `allowed_methods` is therefore the future-proof name, but for the
              # sake of backward compatibility, internally we need to use the
              # deprecated parameter for now. Or request newer urllib3, but that
              # might a problem because of RPM availability.
              method_whitelist=allowed_methods,
              backoff_factor=backoff_factor)
  
          if timeout is not None:
              http_adapter: requests.adapters.HTTPAdapter = TimeoutHTTPAdapter(
                  timeout=timeout, max_retries=retry_strategy)
          else:
              http_adapter = requests.adapters.HTTPAdapter(
                  max_retries=retry_strategy)
  
          session = requests.Session()
          session.mount('http://', http_adapter)
          session.mount('https://', http_adapter)
  
          return session
  
      def __init__(
              self,
              retries: int = DEFAULT_RETRY_SESSION_RETRIES,
              backoff_factor: float = DEFAULT_RETRY_SESSION_BACKOFF_FACTOR,
              allowed_methods: Optional[Tuple[str, ...]] = None,
              status_forcelist: Optional[Tuple[int, ...]] = None,
              timeout: Optional[int] = None
              ) -> None:
          self.retries = retries
          self.backoff_factor = backoff_factor
          self.allowed_methods = allowed_methods
          self.status_forcelist = status_forcelist
          self.timeout = timeout
  
      def __enter__(self) -> requests.Session:
          return self.create(
              retries=self.retries,
              backoff_factor=self.backoff_factor,
              allowed_methods=self.allowed_methods,
              status_forcelist=self.status_forcelist,
              timeout=self.timeout)
  
      def __exit__(self, *args: Any) -> None:
          pass
  
  
  def remove_color(text: str) -> str:
      """ Remove ansi color sequences from the string """
      return re.sub(r'\x1B(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~])', '', text)
  
  
  def default_branch(repository: str, remote: str = 'origin') -> str:
      """ Detect default branch from given local git repository """
      head = os.path.join(repository, f'.git/refs/remotes/{remote}/HEAD')
      # Make sure the HEAD reference is available
      if not os.path.exists(head):
          subprocess.run(
              f'git remote set-head {remote} --auto'.split(), cwd=repository)
      # The ref format is 'ref: refs/remotes/origin/main'
      with open(head) as ref:
          return ref.read().strip().split('/')[-1]
  
  
  def parse_dotenv(content: str) -> EnvironmentType:
      """ Parse dotenv (shell) format of variables """
      return dict([line.split("=", maxsplit=1)
                  for line in shlex.split(content, comments=True)])
  
  
  def parse_yaml(content: str) -> EnvironmentType:
      """ Parse variables from yaml, ensure flat dictionary format """
      yaml_as_dict = YAML(typ="safe").load(content)
      # Handle empty file as an empty environment
      if yaml_as_dict is None:
          return dict()
      if any(isinstance(val, dict) for val in yaml_as_dict.values()):
          raise GeneralError(
              "Can't set the environment from the nested yaml config. The "
              "config should be just key, value pairs.")
      return {key: str(value) for key, value in yaml_as_dict.items()}
  
  
  def validate_git_status(test: 'tmt.base.Test') -> Tuple[bool, str]:
      """
      Validate that test has current metadata on fmf_id
  
      Return a tuple (boolean, message) as the result of validation.
  
      Checks that sources:
      - all local changes are committed
      - up to date on remote repository
      - .fmf/version marking fmf root is committed as well
  
      When all checks pass returns (True, '').
      """
      sources = test.node.sources + \
          [os.path.join(test.node.root, '.fmf', 'version')]
  
      # Use tmt's run instead of subprocess.run
      run = Common().run
  
      # Check for not committed metadata changes
      cmd = ['git', 'status', '--porcelain', '--'] + sources
      try:
          result = run(cmd, cwd=test.node.root, join=True)
      except RunError as error:
          return (
              False,
              f"Failed to run git status: {error.stdout}"
              )
  
      not_committed = []
      assert result.stdout is not None
      for line in result.stdout.split('\n'):
          if line:
              # XY PATH or XY ORIG -> PATH. XY and PATH are separated by space
              not_committed.append(line[3:])
  
      if not_committed:
          return (False, "Uncommitted changes in " + " ".join(not_committed))
  
      # Check for not pushed changes
      cmd = ["git", "rev-parse", "--abbrev-ref", "--symbolic-full-name", "@{u}"]
      try:
          result = run(cmd, cwd=test.node.root)
      except RunError as error:
          return (
              False,
              f'Failed to get remote branch, error raised: "{error.stderr}"'
              )
  
      assert result.stdout is not None
      remote_ref = result.stdout.strip()
  
      cmd = [
          'git',
          'diff',
          f'HEAD..{remote_ref}',
          '--name-status',
          '--'] + sources
      try:
          result = run(cmd, cwd=test.node.root)
      except RunError as error:
          return (
              False,
              f'Failed to diff against remote branch, error raised: "{error.stderr}"')
  
      not_pushed = []
      assert result.stdout is not None
      for line in result.stdout.split('\n'):
          if line:
              _, path = line.strip().split('\t', maxsplit=2)
              not_pushed.append(path)
      if not_pushed:
          return (False, "Not pushed changes in " + " ".join(not_pushed))
  
      return (True, '')
  
  
  def generate_runs(
          path: str, id_: Optional[str] = None) -> Generator[str, None, None]:
      """ Generate absolute paths to runs from path """
      # Prepare absolute workdir path if --id was used
      if id_:
          if '/' not in id_:
              id_ = os.path.join(path, id_)
          if os.path.isabs(id_):
              if os.path.exists(id_):
                  yield id_
              return
      if not os.path.exists(path):
          return
      for filename in os.listdir(path):
          abs_path = os.path.join(path, filename)
          # If id_ is None, the abs_path is considered valid (no filtering
          # is being applied). If it is defined, it has been transformed
          # to absolute path and must be equal to abs_path for the run
          # in abs_path to be generated.
          invalid_id = id_ and abs_path != id_
          invalid_run = not os.path.exists(
              os.path.join(abs_path, 'run.yaml'))
          if not os.path.isdir(abs_path) or invalid_id or invalid_run:
              continue
          yield abs_path
  
  
  def load_run(run: 'tmt.base.Run') -> Tuple[bool, Optional[Exception]]:
      """ Load a run and its steps from the workdir """
      try:
          run.load_from_workdir()
      except GeneralError as error:
          return False, error
      for plan in run.plans:
          for step in plan.steps(disabled=True):
              step.load()
      return True, None
  
  
  # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  #  StructuredField
  # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  
  SFSectionValueType = Union[str, List[str]]
  
  
  class StructuredField:
      """
      Handling multiple text data in a single text field
  
      The StructuredField allows you to easily store and extract several
      sections of text data to/from a single text field. The sections are
      separated by section names in square brackets and can be hosted in
      other text as well.
  
      The section names have to be provided on a separate line and there
      must be no leading/trailing white space before/after the brackets.
      The StructuredField supports two versions of the format:
  
      Version 0: Simple, concise, useful when neither the surrounding text
      or the section data can contain lines which could resemble section
      names. Here's an example of a simple StructuredField::
  
          Note written by human.
  
          [section-one]
          Section one content.
  
          [section-two]
          Section two content.
  
          [section-three]
          Section three content.
  
          [end]
  
          Another note written by human.
  
      Version 1: Includes unique header to prevent collisions with the
      surrounding text and escapes any section-like lines in the content::
  
          Note written by human.
  
          [structured-field-start]
          This is StructuredField version 1. Please, edit with care.
  
          [section-one]
          Section one content.
  
          [section-two]
          Section two content.
          [structured-field-escape][something-resembling-section-name]
  
          [section-three]
          Section three content.
  
          [structured-field-end]
  
          Another note written by human.
  
      Note that an additional empty line is added at the end of each
      section to improve the readability. This line is not considered
      to be part of the section content.
  
      Besides handling the whole section content it's also possible to
      store several key-value pairs in a single section, similarly as in
      the ini config format::
  
          [section]
          key1 = value1
          key2 = value2
          key3 = value3
  
      Provide the key name as the optional argument 'item' when accessing
      these single-line items. Note that the section cannot contain both
      plain text data and key-value pairs.
  
      Example::
  
          field = qe.StructuredField()
          field.set("project", "Project Name")
          field.set("details", "somebody", "owner")
          field.set("details", "2013-05-27", "started")
          field.set("description", "This is a description.\\n"
                  "It spans across multiple lines.\\n")
          print field.save()
  
              [structured-field-start]
              This is StructuredField version 1. Please, edit with care.
  
              [project]
              Project Name
  
              [details]
              owner = somebody
              started = 2013-05-27
  
              [description]
              This is a description.
              It spans across multiple lines.
  
              [structured-field-end]
  
          field.version(0)
          print field.save()
  
              [project]
              Project Name
  
              [details]
              owner = somebody
              started = 2013-05-27
  
              [description]
              This is a description.
              It spans across multiple lines.
  
              [end]
  
      Multiple values for the same key are supported as well. Enable this
      feature with 'multi=True' when initializing the structured field.
      If multiple values are present their list will be returned instead
      of a single string. Similarly use list for setting multiple values::
  
          field = qe.StructuredField(multi=True)
          requirements = ['hypervisor=', 'labcontroller=lab.example.com']
          field.set("hardware", requirements, "hostrequire")
          print field.save()
  
              [structured-field-start]
              This is StructuredField version 1. Please, edit with care.
  
              [hardware]
              hostrequire = hypervisor=
              hostrequire = labcontroller=lab.example.com
  
              [structured-field-end]
  
          print field.get("hardware", "hostrequire")
  
              ['hypervisor=', 'labcontroller=lab.example.com']
      """
  
      # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      #  StructuredField Special
      # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  
      def __init__(
              self,
              text: Optional[str] = None,
              version: int = 1,
              multi: bool = False) -> None:
          """ Initialize the structured field """
          self.version(version)
          self._header: str = ""
          self._footer: str = ""
          # Sections are internally stored in their serialized form, i.e. as
          # strings.
          self._sections: Dict[str, str] = {}
          self._order: List[str] = []
          self._multi = multi
          if text is not None:
              self.load(text)
  
      def __iter__(self) -> Generator[str, None, None]:
          """ By default iterate through all available section names """
          for section in self._order:
              yield section
  
      def __nonzero__(self) -> bool:
          """ True when any section is defined """
          return len(self._order) > 0
  
      __bool__ = __nonzero__
  
      # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      #  StructuredField Private
      # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  
      def _load_version_zero(self, text: str) -> None:
          """ Load version 0 format """
          # Attempt to split the text according to the section tag
          section = re.compile(r"\n?^\[([^\]]+)\]\n", re.MULTILINE)
          parts = section.split(text)
          # If just one part ---> no sections present, just plain text
          if len(parts) == 1:
              self._header = parts[0]
              return
          # Pick header & footer, make sure [end] tag is present
          self._header = parts[0]
          self._footer = re.sub("^\n", "", parts[-1])
          if parts[-2] != "end":
              raise StructuredFieldError("No [end] section tag found")
          # Convert to dictionary and save the order
          keys = parts[1:-2:2]
          values = parts[2:-2:2]
          for key, value in zip(keys, values):
              self.set(key, value)
  
      def _load(self, text: str) -> None:
          """ Load version 1+ format """
          # The text must exactly match the format
          format = re.compile(
              r"(.*)^\[structured-field-start\][ \t]*\n"
              r"(.*)\n\[structured-field-end\][ \t]*\n(.*)",
              re.DOTALL + re.MULTILINE)
          # No match ---> plain text or broken structured field
          matched = format.search(text)
          if not matched:
              if "[structured-field" in text:
                  raise StructuredFieldError("StructuredField parse error")
              self._header = text
              log.debug("StructuredField not found, treating as a plain text")
              return
          # Save header & footer (remove trailing new lines)
          self._header = re.sub("\n\n$", "\n", matched.groups()[0])
          if self._header:
              log.debug(u"Parsed header:\n{0}".format(self._header))
          self._footer = re.sub("^\n", "", matched.groups()[2])
          if self._footer:
              log.debug(u"Parsed footer:\n{0}".format(self._footer))
          # Split the content on the section names
          section = re.compile(r"\n\[([^\]]+)\][ \t]*\n", re.MULTILINE)
          parts = section.split(matched.groups()[1])
          # Detect the version
          version_match = re.search(r"version (\d+)", parts[0])
          if not version_match:
              log.error(parts[0])
              raise StructuredFieldError(
                  "Unable to detect StructuredField version")
          self.version(int(version_match.groups()[0]))
          log.debug(
              "Detected StructuredField version {0}".format(
                  self.version()))
          # Convert to dictionary, remove escapes and save the order
          keys = parts[1::2]
          escape = re.compile(r"^\[structured-field-escape\]", re.MULTILINE)
          values = [escape.sub("", value) for value in parts[2::2]]
          for key, value in zip(keys, values):
              self.set(key, value)
          log.debug(u"Parsed sections:\n{0}".format(
              pprint.pformat(self._sections)))
  
      def _save_version_zero(self) -> str:
          """ Save version 0 format """
          result = []
          if self._header:
              result.append(self._header)
          for section, content in self.iterate():
              result.append(u"[{0}]\n{1}".format(section, content))
          if self:
              result.append(u"[end]\n")
          if self._footer:
              result.append(self._footer)
          return "\n".join(result)
  
      def _save(self) -> str:
          """ Save version 1+ format """
          result = []
          # Regular expression for escaping section-like lines
          escape = re.compile(r"^(\[.+\])$", re.MULTILINE)
          # Header
          if self._header:
              result.append(self._header)
          # Sections
          if self:
              result.append(
                  u"[structured-field-start]\n"
                  u"This is StructuredField version {0}. "
                  u"Please, edit with care.\n".format(self._version))
              for section, content in self.iterate():
                  result.append(u"[{0}]\n{1}".format(section, escape.sub(
                      "[structured-field-escape]\\1", content)))
              result.append(u"[structured-field-end]\n")
          # Footer
          if self._footer:
              result.append(self._footer)
          return "\n".join(result)
  
      def _read_section(self, content: str) -> Dict[str, SFSectionValueType]:
          """ Parse config section and return ordered dictionary """
          dictionary: Dict[str, SFSectionValueType] = OrderedDict()
          for line in content.split("\n"):
              # Remove comments and skip empty lines
              line = re.sub("#.*", "", line)
              if re.match(r"^\s*$", line):
                  continue
              # Parse key and value
              matched = re.search("([^=]+)=(.*)", line)
              if not matched:
                  raise StructuredFieldError(
                      "Invalid key/value line: {0}".format(line))
              key = matched.groups()[0].strip()
              value = matched.groups()[1].strip()
              # Handle multiple values if enabled
              if key in dictionary and self._multi:
                  stored_value = dictionary[key]
                  if isinstance(stored_value, list):
                      stored_value.append(value)
                  else:
                      dictionary[key] = [stored_value, value]
              else:
                  dictionary[key] = value
          return dictionary
  
      def _write_section(self, dictionary: Dict[str, SFSectionValueType]) -> str:
          """ Convert dictionary into a config section format """
          section = ""
          for key in dictionary:
              if isinstance(dictionary[key], list):
                  for value in dictionary[key]:
                      section += "{0} = {1}\n".format(key, value)
              else:
                  section += "{0} = {1}\n".format(key, dictionary[key])
          return section
  
      # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      #  StructuredField Methods
      # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  
      def iterate(self) -> Generator[Tuple[str, str], None, None]:
          """ Return (section, content) tuples for all sections """
          for section in self:
              yield section, self._sections[section]
  
      def version(self, version: Optional[int] = None) -> int:
          """ Get or set the StructuredField version """
          if version is not None:
              if version in [0, 1]:
                  self._version = version
              else:
                  raise StructuredFieldError(
                      "Bad StructuredField version: {0}".format(version))
          return self._version
  
      def load(self, text: str, version: Optional[int] = None) -> None:
          """ Load the StructuredField from a string """
          if version is not None:
              self.version(version)
          # Make sure we got a text, convert from bytes if necessary
          if isinstance(text, bytes):
              text = text.decode("utf8")
          if not isinstance(text, str):
              raise StructuredFieldError(
                  "Invalid StructuredField, expecting string")
          # Remove possible carriage returns
          text = re.sub("\r\n", "\n", text)
          # Make sure the text has a new line at the end
          if text and text[-1] != "\n":
              text += "\n"
          log.debug(u"Parsing StructuredField\n{0}".format(text))
          # Parse respective format version
          if self._version == 0:
              self._load_version_zero(text)
          else:
              self._load(text)
  
      def save(self) -> str:
          """ Convert the StructuredField into a string """
          if self.version() == 0:
              return self._save_version_zero()
          else:
              return self._save()
  
      def header(self, content: Optional[str] = None) -> str:
          """ Get or set the header content """
          if content is not None:
              self._header = content
          return self._header
  
      def footer(self, content: Optional[str] = None) -> str:
          """ Get or set the footer content """
          if content is not None:
              self._footer = content
          return self._footer
  
      def sections(self) -> List[str]:
          """ Get the list of available sections """
          return self._order
  
      def get(
              self,
              section: str,
              item: Optional[str] = None) -> SFSectionValueType:
          """ Return content of given section or section item """
          try:
              content = self._sections[section]
          except KeyError:
              raise StructuredFieldError(
                  "Section [{0!r}] not found".format(ascii(section)))
          # Return the whole section content
          if item is None:
              return content
          # Return only selected item from the section
          try:
              return self._read_section(content)[item]
          except KeyError:
              raise StructuredFieldError(
                  "Unable to read '{0!r}' from section '{1!r}'".format(
                      ascii(item), ascii(section)))
  
      def set(self, section: str, content: Any,
              item: Optional[str] = None) -> None:
          """ Update content of given section or section item """
          # Convert to string if necessary, keep lists untouched
          if isinstance(content, list):
              pass
          elif isinstance(content, bytes):
              content = content.decode("utf8")
          elif not isinstance(content, str):
              content = str(content)
          # Set the whole section content
          if item is None:
              # Add new line if missing
              if content and content[-1] != "\n":
                  content += "\n"
              self._sections[section] = content
          # Set only selected item from the section
          else:
              try:
                  current = self._sections[section]
              except KeyError:
                  current = ""
              dictionary = self._read_section(current)
              dictionary[item] = content
              self._sections[section] = self._write_section(dictionary)
          # Remember the order when adding a new section
          if section not in self._order:
              self._order.append(section)
  
      def remove(self, section: str, item: Optional[str] = None) -> None:
          """ Remove given section or section item """
          # Remove the whole section
          if item is None:
              try:
                  del self._sections[section]
                  del self._order[self._order.index(section)]
              except KeyError:
                  raise StructuredFieldError(
                      "Section [{0!r}] not found".format(ascii(section)))
          # Remove only selected item from the section
          else:
              try:
                  dictionary = self._read_section(self._sections[section])
                  del(dictionary[item])
              except KeyError:
                  raise StructuredFieldError(
                      "Unable to remove '{0!r}' from section '{1!r}'".format(
                          ascii(item), ascii(section)))
              self._sections[section] = self._write_section(dictionary)
  
  
  class DistGitHandler:
      """ Common functionality for DistGit handlers """
      sources_file_name = 'sources'
      uri = "/rpms/{name}/{filename}/{hashtype}/{hash}/{filename}"
  
      usage_name: str  # Name to use for dist-git-type
      re_source: Pattern[str]
      re_ignore_extensions: Pattern[str] = re.compile(r'\.(sign|asc|key)$')
      lookaside_server: str
      remote_substring: Pattern[str]
  
      def url_and_name(self, cwd: str = '.') -> List[Tuple[str, str]]:
          """
          Return list of urls and basenames of the used source
  
          The 'cwd' parameter has to be a DistGit directory.
          """
          # Assumes <package>.spec
          globbed = glob.glob(os.path.join(cwd, '*.spec'))
          if len(globbed) != 1:
              raise GeneralError(f"No .spec file is present in '{cwd}'.")
          package = os.path.basename(globbed[0])[:-len('.spec')]
          ret_values = []
          try:
              with open(os.path.join(cwd, self.sources_file_name)) as f:
                  for line in f.readlines():
                      match = self.re_source.match(line)
                      if match is None:
                          raise GeneralError(
                              f"Couldn't match '{self.sources_file_name}' "
                              f"content with '{self.re_source.pattern}'.")
                      used_hash, source_name, hash_value = match.groups()
                      ret_values.append((self.lookaside_server + self.uri.format(
                          name=package,
                          filename=source_name,
                          hash=hash_value,
                          hashtype=used_hash.lower()
                          ), source_name))
          except Exception as error:
              raise GeneralError(
                  f"Couldn't read '{self.sources_file_name}' file.",
                  original=error)
          if not ret_values:
              raise GeneralError(
                  "No sources found in '{self.sources_file_name}' file.")
          return ret_values
  
      def its_me(self, remotes: List[str]) -> bool:
          """ True if self can work with remotes """
          return any(self.remote_substring.search(item) for item in remotes)
  
  
  class FedoraDistGit(DistGitHandler):
      """ Fedora Handler """
      usage_name = "fedora"
      re_source = re.compile(r"^(\w+) \(([^)]+)\) = ([0-9a-fA-F]+)$")
      lookaside_server = "https://src.fedoraproject.org/repo/pkgs"
      remote_substring = re.compile(r'fedoraproject\.org')
  
  
  class CentOSDistGit(DistGitHandler):
      """ CentOS Handler """
      usage_name = "centos"
      re_source = re.compile(r"^(\w+) \(([^)]+)\) = ([0-9a-fA-F]+)$")
      lookaside_server = "https://sources.stream.centos.org/sources"
      remote_substring = re.compile(r'redhat/centos')
  
  
  def get_distgit_handler(
          remotes: Optional[List[str]] = None,
          usage_name: Optional[str] = None) -> DistGitHandler:
      """
      Return the right DistGitHandler
  
      Pick the DistGitHandler class which understands specified
      remotes or by usage_name.
      """
      for candidate_class in DistGitHandler.__subclasses__():
          if usage_name is not None and usage_name == candidate_class.usage_name:
              return candidate_class()
          if remotes is not None:
              ret_val = candidate_class()
              if ret_val.its_me(remotes):
                  return ret_val
      raise GeneralError(f"No known remote in '{remotes}'.")
  
  
  def get_distgit_handler_names() -> List[str]:
      """ All known distgit handlers """
      return [i.usage_name for i in DistGitHandler.__subclasses__()]
  
  
  def git_clone(
          url: str,
          destination: str,
          common: Common,
          env: Optional[EnvironmentType] = None,
          shallow: bool = False
          ) -> CommandOutput:
      """
      Git clone url to destination, retry without shallow if necessary
  
      For shallow=True attempt to clone repository using --depth=1 option first.
      If not successful attempt to clone whole repo.
  
      Common instance is used to run the command for appropriate logging.
      Environment is updated by 'env' dictionary.
      """
      depth = ['--depth=1'] if shallow else []
      command = ['git', 'clone'] + depth + [url, destination]
      try:
          return common.run(command, env=env)
      except RunError:
          if not shallow:
              # Do not retry if shallow was not used
              raise
          # Git server might not support shallow cloning, try again
          return git_clone(url, destination, common, env, shallow=False)
  
  
  class updatable_message(contextlib.AbstractContextManager):  # type: ignore[type-arg]
      """ Updatable message suitable for progress-bar-like reporting """
  
      def __init__(
              self,
              key: str,
              enabled: bool = True,
              indent_level: int = 0,
              key_color: Optional[str] = None,
              default_value_color: Optional[str] = None
              ) -> None:
          """
          Updatable message suitable for progress-bar-like reporting.
  
          .. code:block:: python3
  
             with updatable_message('foo') as message:
                 while ...:
                     ...
  
                     # check state of remote request, and update message
                     state = remote_api.check()
                     message.update(state)
  
          :param key: a string to use as the left-hand part of logged message.
          :param enabled: if unset, no output would be performed.
          :param indent_level: desired indentation level.
          :param key_color: optional color to apply to ``key``.
          :param default_color: optional color to apply to value when
              :py:meth:`update` is called with ``color`` left out.
          """
  
          self.key = key
          self.enabled = enabled
          self.indent_level = indent_level
          self.key_color = key_color
          self.default_value_color = default_value_color
  
          # No progress if terminal not attached
          if not sys.stdout.isatty():
              self.enabled = False
  
          self._previous_line: Optional[str] = None
  
      def __enter__(self) -> 'updatable_message':
          return self
  
      def __exit__(self, *args: Any) -> None:
          sys.stdout.write('\n')
          sys.stdout.flush()
  
      def update(self, value: str, color: Optional[str] = None) -> None:
          if not self.enabled:
              return
  
          if self._previous_line is not None:
              message = value.ljust(len(self._previous_line))
  
          else:
              message = value
  
          self._previous_line = value
  
          message = indent(
              self.key,
              value=style(
                  message,
                  fg=color or self.default_value_color),
              color=self.key_color,
              level=self.indent_level)
  
          sys.stdout.write(f"\r{message}")
          sys.stdout.flush()
  
  
  def find_fmf_root(path: str) -> List[str]:
      """
      Search trough path and return all fmf roots that exist there
  
      Returned list is ordered by path length, shortest one first.
  
      Raise `MetadataError` if no fmf root is found.
      """
      fmf_roots = []
      for root, _, files in os.walk(path):
          if not os.path.basename(root) == '.fmf':
              continue
          if 'version' in files:
              fmf_roots.append(os.path.dirname(root))
      if not fmf_roots:
          raise MetadataError(f"No fmf root present inside '{path}'.")
      fmf_roots.sort(key=lambda path: len(path))
      return fmf_roots
  
  
  #
  # JSON schema-based validation helpers
  #
  # Aims at FMF data consumed by tmt, but can be used for any structure.
  #
  
  # `Schema` represents a loaded JSON schema structure. It may be fairly complex,
  # but it's not needed to provide the exhaustive and fully detailed type since
  # tmt code is not actually "reading" it. Loaded schema is passed down to
  # jsonschema library, and while `Any` would be perfectly valid, let's use an
  # alias to make schema easier to track in our code.
  Schema = Dict[str, Any]
  SchemaStore = Dict[str, Schema]
  
  
  def _patch_plan_schema(schema: Schema, store: SchemaStore) -> None:
      """
      Resolve references to per-plugin schema known to steps. All schemas have
      been loaded into store, all that's left is to update each step in plan
      schema with the list of schemas allowed for that particular step.
  
      For each step, we create the following schema (see also plan.yaml for the
      rest of plan schema):
  
      .. code-block:: yaml
  
         <step name>:
           oneOf:
             - $ref: "/schemas/<step name>/plugin1"
             - $ref: "/schemas/<step name>/plugin2"
             ...
             - $ref: "/schemas/<step name>/pluginN"
             - type: array
               items:
                 anyOf:
                   - $ref: "/schemas/<step name>/plugin1"
                   - $ref: "/schemas/<step name>/plugin2"
                   ...
                   - $ref: "/schemas/<step name>/pluginN"
      """
  
      for step in ('discover', 'execute', 'finish', 'prepare', 'provision', 'report'):
          step_schema_prefix = f'/schemas/{step}/'
  
          step_plugin_schema_ids = [schema_id for schema_id in store.keys() if schema_id.startswith(
              step_schema_prefix) and schema_id != '/schemas/provision/hardware']
  
          refs: List[Schema] = [
              {'$ref': schema_id} for schema_id in step_plugin_schema_ids
              ]
  
          schema['properties'][step] = {
              'oneOf': refs + [
                  {
                      'type': 'array',
                      'items': {
                          'anyOf': refs
                          }
                      }
                  ]
              }
  
  
  def _load_schema(schema_filepath: str) -> Schema:
      """
      Load a JSON schema from a given filepath.
  
      A helper returning the raw loaded schema.
      """
  
      if not os.path.isabs(schema_filepath):
          schema_filepath = os.path.join(
              pkg_resources.resource_filename(
                  'tmt', 'schemas'), schema_filepath)
  
      try:
          with open(schema_filepath, 'r', encoding='utf-8') as f:
              return cast(Schema, yaml_to_dict(f.read()))
  
      except Exception as exc:
          raise FileError(f"Failed to load schema file {schema_filepath}\n{exc}")
  
  
  @functools.lru_cache(maxsize=None)
  def load_schema(schema_filepath: str) -> Schema:
      """
      Load a JSON schema from a given filepath.
  
      Recommended for general use, the method may apply some post-loading touches
      to the given schema, and unless caller is interested in the raw content of
      the file, this functions should be used instead of the real workhorse of
      schema loading, :py:func:`_load_schema`.
      """
  
      schema = _load_schema(schema_filepath)
  
      if schema.get('$id') == '/schemas/plan':
          _patch_plan_schema(schema, load_schema_store())
  
      return schema
  
  
  @functools.lru_cache(maxsize=None)
  def load_schema_store() -> SchemaStore:
      """
      Load all available JSON schemas, and put them into a "store".
  
      Schema store is a simple mapping between schema IDs and schemas.
      """
  
      store: SchemaStore = {}
  
      schema_dirpath = pkg_resources.resource_filename('tmt', 'schemas')
  
      try:
          for dirpath, dirnames, filenames in os.walk(
                  schema_dirpath, followlinks=True):
              for filename in filenames:
                  # Ignore all files but YAML files.
                  if os.path.splitext(filename)[1].lower() not in ('.yaml', '.yml'):
                      continue
  
                  schema = _load_schema(os.path.join(dirpath, filename))
  
                  store[schema['$id']] = schema
  
      except Exception as exc:
          raise FileError(f"Failed to discover schema files\n{exc}")
  
      if '/schemas/plan' not in store:
          raise FileError('Failed to discover schema for plans')
  
      _patch_plan_schema(store['/schemas/plan'], store)
  
      return store
  
  
  def _prenormalize_fmf_node(node: fmf.Tree, schema_name: str) -> fmf.Tree:
      """
      Apply the minimal possible normalization steps to nodes before validating them with schemas.
  
      tmt allows some fields to have default values, and at least ``how`` field is necessary for
      schema-based validation to work reliably. Based on ``how`` field, plan schema identifies
      the correct *plugin* schema for step validation. Without ``how``, it's hard to pick the
      correct schema.
  
      This function tries to do minimal number of changes to a given fmf node to honor the promise
      of ``how`` being optional, with known defaults for each step. It might be possible to resolve
      this purely with schemas, but since we don't know how (yet?), a Python implementation has been
      chosen to unblock schema-based validation while keeping things easier for users. This may
      change in the future, dropping the need for this pre-validation step.
  
      .. note::
  
         This function is not part of the normalization process that happens after validation. The
         purpose of this function is to make the world nice and shiny for tmt users while avoiding
         the possibility of schema becoming way too complicated, especially when we would need
         non-trivial amount fo time for experiments.
  
         The real normalization process takes place after validation, and is responsible for
         converting raw fmf data to data types and structures more suited for tmt internal
         implementation.
      """
  
      # As of now, only `how` field in plan steps seems to be required for schema-based validation
      # to work correctly, therefore ignore any other node.
      if schema_name != 'plan.yaml':
          return node
  
      # Perform the very crude and careful semi-validation. We need to set the `how` key to a default
      # value - but it's not our job to validate the general structure of node data. Walk the "happy"
      # path, touch the node only when it matches the specification of being a mapping of steps and
      # these being either mappings or lists of mappings. Whenever we notice some value does not
      # match this basic structure, ignore the step completely - its issues will be cought by schema
      # later, don't waste time on steps that do not follow specification.
  
      # Fmf data describing a plan shall be a mapping (with keys like `discover` or `adjust`).
      if not isinstance(node.data, dict):
          return node
  
      # Avoid possible circular imports
      import tmt.steps
  
      def _process_step(step_name: str, step: Dict[Any, Any]) -> None:
          """
          Process a single step configuration.
          """
  
          # If `how` is set, don't touch it, and there's nothing to do.
          if 'how' in step:
              return
  
          # Magic!
          # No, seriously: step is implemented in `tmt.steps.$step_name` package,
          # by a class `tmt.steps.$step_name.$step_name_with_capitalized_first_letter`.
          # Instead of having a set of if-elif tests, we can reach the default `how`
          # dynamically.
  
          from tmt.plugins import import_member
  
          step_module_name = f'tmt.steps.{step_name}'
          step_class_name = step_name.capitalize()
  
          step_class = import_member(step_module_name, step_class_name)
  
          if not issubclass(step_class, tmt.steps.Step):
              raise GeneralError(
                  'Possible step {step_name} implementation '
                  f'{step_module_name}.{step_class_name} is not a subclass '
                  'of tmt.steps.Step class.')
  
          step['how'] = step_class.DEFAULT_HOW
  
      def _process_step_collection(step_name: str, step_collection: Any) -> None:
          """
          Process a collection of step configurations.
          """
  
          # Ignore anything that is not a step.
          if step_name not in tmt.steps.STEPS:
              return
  
          # A single step configuration, represented as a mapping.
          if isinstance(step_collection, dict):
              _process_step(step_name, step_collection)
  
              return
  
          # Multiple step configurations, as mappings in a list
          if isinstance(step_collection, list):
              for step_config in step_collection:
                  # Unexpected, maybe instead of a mapping describing a step someone put
                  # in an integer... Ignore, schema will report it.
                  if not isinstance(step_config, dict):
                      continue
  
                  _process_step(step_name, step_config)
  
      for step_name, step_config in node.data.items():
          _process_step_collection(step_name, step_config)
  
      return node
  
  
  def validate_fmf_node(
          node: fmf.Tree, schema_name: str) -> List[Tuple[jsonschema.ValidationError, str]]:
      """ Validate a given fmf node """
  
      node = _prenormalize_fmf_node(node, schema_name)
  
      result = node.validate(load_schema(schema_name), schema_store=load_schema_store())
  
      if result.result is True:
          return []
  
      # A bit of error formatting. It is possible to use str(error), but the result
      # is a bit too JSON-ish. Let's present an error message in a way that helps
      # users to point finger on each and every issue. But don't throw the original
      # errors away!
  
      errors: List[Tuple[jsonschema.ValidationError, str]] = []
  
      for error in result.errors:
          path = f'{node.name}:{".".join(error.path)}'
  
          errors.append((error, f'{path} - {error.message}'))
  
      return errors
  
  
  # A type for callbacks given to wait()
  WaitCheckType = Callable[[], T]
  
  
  def wait(
      parent: Common,
      check: WaitCheckType[T],
      timeout: datetime.timedelta,
      tick: float = DEFAULT_WAIT_TICK,
      tick_increase: float = DEFAULT_WAIT_TICK_INCREASE
  ) -> T:
      """
      Wait for a condition to become true.
  
      To test the condition state, a ``check`` callback is called every ``tick``
      seconds until ``check`` reports a success. The callback may:
  
      * decide the condition has been fulfilled. This is a successfull outcome,
        ``check`` shall then simply return, and waiting ends. Or,
      * decide more time is needed. This is not a successfull outcome, ``check``
        shall then raise :py:clas:`WaitingIncomplete` exception, and ``wait()``
        will try again later.
  
      :param parent: "owner" of the wait process. Used for its logging capability.
      :param check: a callable responsible for testing the condition. Accepts no
          arguments. To indicate more time and attempts are needed, the callable
          shall raise :py:class:`WaitingIncomplete`, otherwise it shall return
          without exception. Its return value will be propagated by ``wait()`` up
          to ``wait()``'s. All other exceptions raised by ``check`` will propagate
          to ``wait()``'s caller as well, terminating the wait.
      :param timeout: amount of time ``wait()`` is alowed to spend waiting for
          successfull outcome of ``check`` call.
      :param tick: how many seconds to wait between two consecutive calls of
          ``check``.
      :param tick_increase: a multiplier applied to ``tick`` after every attempt.
      :returns: value returned by ``check`` reporting success.
      :raises GeneralError: when ``tick`` is not a positive integer.
      :raises WaitingTimedOutError: when time quota has been consumed.
      """
  
      if tick <= 0:
          raise GeneralError('Tick must be a positive integer')
  
      NOW = time.monotonic
  
      deadline = NOW() + timeout.total_seconds()
  
      parent.debug(
          'wait',
          f"waiting for condition '{check.__name__}' with timeout {timeout},"
          f"deadline in {timeout.total_seconds()}, checking every {tick} seconds")
  
      while True:
          now = NOW()
  
          if now > deadline:
              raise WaitingTimedOutError()
  
          try:
              ret = check()
  
              # Perform one extra check: if `check()` succeeded, but took more time than
              # allowed, it should be recognized as a failed waiting too.
              now = NOW()
  
              if now > deadline:
                  parent.debug(
                      'wait',
                      f"'{check.__name__}' finished successfully but took too much time,"
                      f"{now - deadline} over quota")
  
                  raise WaitingTimedOutError()
  
              parent.debug(
                  'wait',
                  f"'{check.__name__}' finished successfully, {deadline - now} seconds left")
  
              return ret
  
          except WaitingIncomplete:
              parent.debug(
                  'wait',
                  f"'{check.__name__}' still pending, {deadline - now} seconds left")
  
              time.sleep(tick)
  
              tick *= tick_increase
  
              continue
  
  
  class ValidateFmfMixin:
      """
      Mixin adding validation of an fmf node.
  
      Loads a schema whose name is derived from class name, and uses fmf's validate()
      method to perform the validation.
      """
  
      def _validate_fmf_node(self, node: fmf.Tree, logger: Common,
                             raise_on_validation_error: bool) -> None:
          """ Validate a given fmf node """
  
          errors = validate_fmf_node(
              node, f'{self.__class__.__name__.lower()}.yaml')
  
          if errors:
              if raise_on_validation_error:
                  raise SpecificationError(
                      f'fmf node {node.name} failed validation',
                      validation_errors=errors)
  
              for _, error_message in errors:
                  logger.warn(error_message, shift=1)
  
      def __init__(
              self,
              *,
              node: fmf.Tree,
              logger: Common,
              skip_validation: bool = False,
              raise_on_validation_error: bool = False,
              **kwargs: Any) -> None:
          # Validate *before* letting next class in line touch the data.
          if not skip_validation:
              self._validate_fmf_node(node, logger, raise_on_validation_error)
  
          kwargs.setdefault('logger', self)
          super().__init__(node=node, **kwargs)  # type: ignore[call-arg]
  
  
  # A type representing compatible sources of keys and values.
  KeySource = Union[Dict[str, Any], fmf.Tree]
  
  
  class NormalizeKeysMixin:
      """
      Mixin adding support for loading fmf keys into object attributes.
  
      When invoked, annotated class-level variables are searched for in a given source
      container - a mapping, an fmf node, etc. - and if the key of the same name as the
      variable exists, its value is "promoted" to instance variable.
  
      If a method named ``_normalize_<variable name>`` exists, it is called with the fmf
      key value as its single argument, and its return value is assigned to instance
      variable. This gives class chance to modify or transform the original value when
      needed, e.g. to convert the original value to a type more suitable for internal
      processing.
      """
  
      # If specified, keys would be iterated over in the order as listed here.
      KEYS_SHOW_ORDER: List[str] = []
  
      # NOTE: these could be static methods, self is probably useless, but that would
      # cause complications when classes assign these to their members. That makes them
      # no longer static as far as class is concerned, which means they get called with
      # `self` as the first argument. A workaround would be to assign staticmethod()-ized
      # version of them, but that's too much repetition.
      #
      # TODO: wouldn't it be nice if these could be mention in dataclass.field()?
      # It would require a clone of dataclass.field() though.
      def _normalize_string_list(self, value: Union[None, str, List[str]]) -> List[str]:
          if value is None:
              return []
  
          return [value] if isinstance(value, str) else value
  
      def _normalize_environment(self, value: Optional[Dict[str, Any]]) -> EnvironmentType:
          if value is None:
              return {}
  
          return {
              name: str(value) for name, value in value.items()
              }
  
      @classmethod
      def _iter_key_annotations(cls) -> Generator[Tuple[str, Any], None, None]:
          """
          Iterate over keys' type annotations.
  
          Keys are yielded in the order: keys declared by parent classes first, then
          keys declared by the class itself, all following the order in which keys
          were defined in their respective classes.
  
          Yields:
              pairs of key name and its annotations.
          """
  
          def _iter_class_annotations(klass: type) -> Generator[Tuple[str, Any], None, None]:
              # Skip, needs fixes to become compatible
              if klass is Common:
                  return
  
              for name, value in klass.__dict__.get('__annotations__', {}).items():
                  # Skip special fields that are not keys.
                  if name == 'KEYS_SHOW_ORDER':
                      continue
  
                  yield (name, value)
  
          # Reverse MRO to start with the most base classes first, to iterate over keys
          # in the order they are defined.
          for klass in reversed(cls.__mro__):
              yield from _iter_class_annotations(klass)
  
      @classmethod
      def keys(cls) -> Generator[str, None, None]:
          """
          Iterate over key names.
  
          Keys are yielded in the order: keys declared by parent classes first, then
          keys declared by the class itself, all following the order in which keys
          were defined in their respective classes.
  
          Yields:
              key names.
          """
  
          for keyname, _ in cls._iter_key_annotations():
              yield keyname
  
      def items(self) -> Generator[Tuple[str, Any], None, None]:
          """
          Iterate over keys and their values.
  
          Keys are yielded in the order: keys declared by parent classes first, then
          keys declared by the class itself, all following the order in which keys
          were defined in their respective classes.
  
          Yields:
              pairs of key name and its value.
          """
  
          for keyname in self.keys():
              yield (keyname, getattr(self, keyname))
  
      # TODO: exists for backward compatibility for the transition period. Once full
      # type annotations land, there should be no need for extra _keys attribute.
      @classmethod
      def _keys(cls) -> List[str]:
          """ Return a list of names of object's keys. """
  
          return list(cls.keys())
  
      def _load_keys(
              self,
              key_source: Dict[str, Any],
              key_source_name: str,
              logger: Common) -> None:
          """ Extract values for class-level attributes, and verify they match declared types. """
  
          LOG_SHIFT, LOG_LEVEL = 2, 4
  
          debug_intro = functools.partial(logger.debug, shift=LOG_SHIFT - 1, level=LOG_LEVEL)
          debug = functools.partial(logger.debug, shift=LOG_SHIFT, level=LOG_LEVEL)
  
          debug_intro('key source')
          for k, v in key_source.items():
              debug(f'{k}: {v} ({type(v)})')
  
          debug('')
  
          for keyname, keytype in self._iter_key_annotations():
              key_address = f'{key_source_name}:{keyname}'
              source_keyname = key_to_option(keyname)
  
              # Do not indent this particular entry like the rest, so it could serve
              # as a "header" for a single key processing.
              debug_intro('key', key_address)
              debug('field', source_keyname)
  
              debug('desired type', str(keytype))
  
              value: Any = None
  
              # Verbose, let's hide it a bit deeper.
              debug('dict', self.__dict__, level=LOG_LEVEL + 1)
  
              if hasattr(self, keyname):
                  # If the key exists as instance's attribute already, it is because it's been
                  # declared with a default value, and the attribute now holds said default value.
                  default_value = getattr(self, keyname)
  
                  # If the default value is a mutable container, we cannot use it directly.
                  # Should we do so, the very same default value would be assigned to multiple
                  # instances/attributes instead of each instance having its own distinct container.
                  if isinstance(default_value, (list, dict)):
                      debug('detected mutable default')
                      default_value = copy.copy(default_value)
  
                  debug('default value', str(default_value))
                  debug('default value type', str(type(default_value)))
  
                  # try+except seems to work better than get(), especially when
                  # semantic of fmf.Tree.get() is slightly different than that
                  # of dict().get().
                  try:
                      value = key_source[source_keyname]
  
                  except KeyError:
                      value = default_value
  
                  debug('raw value', str(value))
                  debug('raw value type', str(type(value)))
  
              else:
                  value = key_source.get(source_keyname)
  
                  debug('raw value', str(value))
                  debug('raw value type', str(type(value)))
  
              normalize_callback = getattr(self, f'_normalize_{keyname}', None)
  
              if normalize_callback:
                  value = normalize_callback(value)
  
                  debug('normalized value', str(value))
                  debug('normalized value type', str(type(value)))
  
              debug('final value', str(value))
              debug('final value type', str(type(value)))
  
              # Set attribute by adding it to __dict__ directly. Messing with setattr()
              # might cause re-use of mutable values by other instances.
              self.__dict__[keyname] = value
  
              # Apparently pointless, but makes the debugging output more readable.
              # There may be plenty of tests and plans and keys, a bit of spacing
              # can't hurt.
              debug('')
  
          debug_intro('normalized fields')
          for k, v in self.__dict__.items():
              debug(f'{k}: {v} ({type(v)})')
  
          debug('')
  
      def __init__(self, **kwargs: Any) -> None:
          super().__init__(**kwargs)
  
  
  class LoadFmfKeysMixin(NormalizeKeysMixin):
      def __init__(
              self,
              *,
              node: fmf.Tree,
              logger: Common,
              **kwargs: Any) -> None:
          self._load_keys(node.get(), node.name, logger)
  
          kwargs.setdefault('logger', logger)
          super().__init__(node=node, **kwargs)
  
files: tmt/utils.py
files: 
  """ Test Metadata Utilities """
  
  import contextlib
  import copy
  import dataclasses
  import datetime
  import functools
  import glob
  import io
  import os
  import pprint
  import re
  import shlex
  import shutil
  import subprocess
  import sys
  import time
  import unicodedata
  from collections import OrderedDict
  from functools import lru_cache
  from pathlib import Path
  from threading import Thread
  from typing import (IO, TYPE_CHECKING, Any, Callable, Dict, Generator,
                      Iterable, List, NamedTuple, Optional, Pattern, Tuple, Type,
                      TypeVar, Union, cast, overload)
  
  import click
  import fmf
  import jsonschema
  import pkg_resources
  import requests
  import requests.adapters
  import requests.packages.urllib3.util.retry
  import urllib3.exceptions
  from click import echo, style, wrap_text
  from ruamel.yaml import YAML, scalarstring
  from ruamel.yaml.comments import CommentedMap
  from ruamel.yaml.parser import ParserError
  
  if sys.version_info >= (3, 8):
      from typing import Literal, Protocol
  else:
      from typing_extensions import Literal, Protocol
  
  
  if TYPE_CHECKING:
      import tmt.base
      import tmt.steps
  
  log = fmf.utils.Logging('tmt').logger
  
  # Default workdir root and max
  WORKDIR_ROOT = '/var/tmp/tmt'
  WORKDIR_MAX = 1000
  
  # Log in workdir
  LOG_FILENAME = 'log.txt'
  
  # Maximum number of lines of stdout/stderr to show upon errors
  OUTPUT_LINES = 100
  # Default output width
  OUTPUT_WIDTH = 79
  
  # Hierarchy indent
  INDENT = 4
  
  # Default name and order for step plugins
  DEFAULT_NAME = 'default'
  DEFAULT_PLUGIN_ORDER = 50
  DEFAULT_PLUGIN_ORDER_MULTIHOST = 10
  DEFAULT_PLUGIN_ORDER_REQUIRES = 70
  DEFAULT_PLUGIN_ORDER_RECOMMENDS = 75
  
  # Config directory
  CONFIG_PATH = '~/.config/tmt'
  
  # Special process return code
  PROCESS_TIMEOUT = 124
  
  # Default select.select(timeout) in seconds
  DEFAULT_SELECT_TIMEOUT = 5
  
  # Default shell and options to be set for all shell scripts
  DEFAULT_SHELL = "/bin/bash"
  SHELL_OPTIONS = 'set -eo pipefail'
  
  # Defaults for HTTP/HTTPS retries and timeouts (see `retry_session()`).
  DEFAULT_RETRY_SESSION_RETRIES: int = 3
  DEFAULT_RETRY_SESSION_BACKOFF_FACTOR: float = 0.1
  
  # Defaults for HTTP/HTTPS retries for getting environment file
  # Retry with exponential backoff, maximum duration ~511 seconds
  ENVFILE_RETRY_SESSION_RETRIES: int = 10
  ENVFILE_RETRY_SESSION_BACKOFF_FACTOR: float = 1
  
  # Default for wait()-related options
  DEFAULT_WAIT_TICK: float = 30.0
  DEFAULT_WAIT_TICK_INCREASE: float = 1.0
  
  # A stand-in variable for generic use.
  T = TypeVar('T')
  
  # A FMF context type, representing name/values context.
  FmfContextType = Dict[str, List[str]]
  
  # A "environment" type, representing name/value environment variables.
  EnvironmentType = Dict[str, str]
  
  # Workdir argument type, can be True, a string, a path or None
  WorkdirArgumentType = Union[Literal[True], str, None]
  
  # Workdir type, can be None or a string
  WorkdirType = Optional[str]
  
  # Option to skip to initialize work tree in plan
  PLAN_SKIP_WORKTREE_INIT = 'plan_skip_worktree_init'
  
  
  class BaseLoggerFnType(Protocol):
      def __call__(
              self,
              key: str,
              value: Optional[str] = None,
              color: Optional[str] = None,
              shift: int = 0,
              level: int = 1,
              err: bool = False) -> None:
          pass
  
  
  class LevelessLoggerFnType(Protocol):
      def __call__(
              self,
              key: str,
              value: Optional[str] = None,
              color: Optional[str] = None,
              shift: int = 0,
              err: bool = False) -> None:
          pass
  
  
  class SemanticLoggerFnType(Protocol):
      def __call__(self, message: str, shift: int = 0) -> None:
          pass
  
  
  LoggerFnType = Union[
      BaseLoggerFnType,
      LevelessLoggerFnType,
      SemanticLoggerFnType]
  
  
  def indent(
          key: str,
          value: Optional[str] = None,
          color: Optional[str] = None,
          level: int = 0) -> str:
      """
      Indent a key/value message.
  
      If both ``key`` and ``value`` are specified, ``{key}: {value}``
      message is rendered. Otherwise, just ``key`` is used alone. If
      ``value`` contains multiple lines, each but the very first line is
      indented by one extra level.
  
      :param value: optional value to print at right side of ``key``.
      :param color: optional color to apply on ``key``.
      :param level: number of indentation levels. Each level is indented
                    by :py:data:`INDENT` spaces.
      """
  
      indent = ' ' * INDENT * level
      deeper = ' ' * INDENT * (level + 1)
  
      # Colorize
      if color is not None:
          key = style(key, fg=color)
  
      # Handle key only
      if value is None:
          message = key
  
      # Handle key + value
      else:
          # Multiline content indented deeper
          if isinstance(value, str):
              lines = value.splitlines()
              if len(lines) > 1:
                  value = ''.join([f"\n{deeper}{line}" for line in lines])
  
          message = f'{key}: {value}'
  
      return indent + message
  
  
  class Config:
      """ User configuration """
  
      def __init__(self) -> None:
          """ Initialize config directory path """
          self.path = os.path.expanduser(CONFIG_PATH)
          if not os.path.exists(self.path):
              try:
                  os.makedirs(self.path)
              except OSError as error:
                  raise GeneralError(
                      f"Failed to create config '{self.path}'.\n{error}")
  
      def last_run(self, run_id: Optional[str] = None) -> Optional[str]:
          """ Get and set last run id """
          symlink = os.path.join(self.path, 'last-run')
          if run_id:
              try:
                  os.remove(symlink)
              except OSError:
                  pass
              try:
                  os.symlink(run_id, symlink)
              except FileExistsError:
                  # Race when tmt runs in parallel
                  log.warning(f"Race condition, unable to save last run '{run_id}'.")
              except OSError as error:
                  raise GeneralError(
                      f"Unable to save last run '{self.path}'.\n{error}")
              return run_id
          if os.path.islink(symlink):
              return os.path.realpath(symlink)
          return None
  
  
  class StreamLogger(Thread):
      """
      Reading pipes of running process in threads.
  
      Code based on:
      https://github.com/packit/packit/blob/main/packit/utils/logging.py#L10
      """
  
      def __init__(self,
                   stream: Optional[IO[bytes]],
                   log_header: str,
                   logger: BaseLoggerFnType) -> None:
          super().__init__(daemon=True)
          self.stream = stream
          self.output: List[str] = []
          self.log_header = log_header
          self.logger = logger
  
      def run(self) -> None:
          if self.stream is None:
              return
  
          for _line in self.stream:
              line = _line.decode('utf-8', errors='replace')
              if line != '':
                  self.logger(
                      self.log_header,
                      line.rstrip('\n'),
                      'yellow',
                      level=3)
              self.output.append(line)
  
      def get_output(self) -> str:
          return "".join(self.output)
  
  
  # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  #  Common
  # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  
  CommonDerivedType = TypeVar('CommonDerivedType', bound='Common')
  
  
  class CommandOutput(NamedTuple):
      stdout: Optional[str]
      stderr: Optional[str]
  
  
  class Common:
      """
      Common shared stuff
  
      Takes care of command line context, options and workdir handling.
      Provides logging functions info(), verbose() and debug().
      Implements read() and write() for comfortable file access.
      Provides the run() method for easy command execution.
      """
  
      # Command line context, options and workdir
      _context: Optional[click.Context] = None
      # When set to true, _opt will be ignored (default will be returned)
      ignore_class_options: bool = False
      _options: Dict[str, Any] = dict()
      _workdir: WorkdirType = None
  
      # TODO: must be declared outside of __init__(), because it must exist before
      # __init__() gets called to allow logging helpers work correctly when used
      # from mixins. But that's not very clean, is it? :( Maybe decoupling logging
      # from Common class would help, such a class would be able to initialize
      # itself without involving the rest of Common code. On the other hand,
      # Common owns workdir, for example, whose value affects logging too, so no
      # clear solution so far.
      #
      # Note: cannot use CommonDerivedType - it's a TypeVar filled in by the type
      # given to __init__() and therefore the type it's representing *now* is
      # unknown. but we know `parent` will be derived from `Common` class, so it's
      # mostly fine.
      parent: Optional['Common'] = None
  
      def __init__(
              self,
              parent: Optional[CommonDerivedType] = None,
              name: Optional[str] = None,
              workdir: WorkdirArgumentType = None,
              context: Optional[click.Context] = None,
              relative_indent: int = 1,
              **kwargs: Any) -> None:
          """
          Initialize name and relation with the parent object
  
          Prepare the workdir for provided id / directory path
          or generate a new workdir name if workdir=True given.
          Store command line context and options for future use
          if context is provided.
          """
          # Use lowercase class name as the default name
          self.name = name or self.__class__.__name__.lower()
          self.parent = parent
  
          # Prepare a safe variant of the name which does not contain
          # spaces or other special characters to prevent problems with
          # tools which do not expect them (e.g. in directory names).
          self.safe_name = re.sub(r"[^\w/-]+", "-", self.name).strip("-")
  
          # Relative log indent level shift against the parent
          self._relative_indent = relative_indent
  
          # Store command line context
          if context:
              self._save_context_to_instance(context)
  
          # Initialize the workdir if requested
          self._workdir_load(workdir)
  
      def __str__(self) -> str:
          """ Name is the default string representation """
          return self.name
  
      @classmethod
      def _save_context(cls, context: click.Context) -> None:
          """ Save provided command line context and options for future use """
          cls._context = context
          cls._options = context.params
  
      def _save_context_to_instance(self, context: click.Context) -> None:
          """ Save provided command line context and options to the instance """
          self._context = context
          self._options = context.params
  
      @overload
      @classmethod
      def _opt(cls, option: str) -> Any:
          pass
  
      @overload
      @classmethod
      def _opt(cls, option: str, default: T) -> T:
          pass
  
      @classmethod
      def _opt(cls, option: str, default: Any = None) -> Any:
          """ Get an option from the command line context (class version) """
          if cls.ignore_class_options:
              return default
          return cls._options.get(option, default)
  
      def _fmf_context(self) -> FmfContextType:
          """ Return the current fmf context """
          if self._context is None:
              return dict()
  
          try:
              return cast(FmfContextType, self._context.obj.fmf_context)
          except AttributeError:
              return dict()
  
      def opt(self, option: str, default: Optional[Any] = None) -> Any:
          """
          Get an option from the command line options
  
          Checks also parent options. For flags (boolean values) parent's
          True wins over child's False (e.g. run --quiet enables quiet
          mode for all included plans and steps).
  
          For options that can be used multiple times, the child overrides
          the parent if it was defined (e.g. run -av provision -vvv runs
          all steps except for provision in mildly verbose mode, provision
          is run with the most verbosity).
  
          Environment variables override command line options.
          """
          # Translate dashes to underscores to match click's conversion
          option = option.replace('-', '_')
          # Check the environment first
          if option == 'debug':
              try:
                  debug = os.environ['TMT_DEBUG']
                  return int(debug)
              except ValueError:
                  raise GeneralError(
                      f"Invalid debug level '{debug}', use an integer.")
              except KeyError:
                  pass
  
          # Get local option
          local = self._options.get(option, default)
          # Check parent option
          parent = None
          if self.parent:
              parent = self.parent.opt(option)
          # Special handling for special flags (parent's yes always wins)
          if option in ['quiet', 'force', 'dry']:
              return parent if parent else local
          # Special handling for counting options (child overrides the
          # parent if it was defined)
          elif option in ['debug', 'verbose']:
              winner = local if local else parent
              if winner is None:
                  winner = 0
              return winner
          else:
              return parent if parent is not None else local
  
      def _level(self) -> int:
          """ Hierarchy level """
          if self.parent is None:
              return -1
          else:
              return self.parent._level() + self._relative_indent
  
      def _indent(
              self,
              key: str,
              value: Optional[str] = None,
              color: Optional[str] = None,
              shift: int = 0) -> str:
          """ Indent message according to the object hierarchy """
  
          return indent(
              key,
              value=value,
              color=color,
              level=self._level() + shift)
  
      def _log(self, message: str) -> None:
          """ Append provided message to the current log """
          # Nothing to do if there is no workdir
          if self.workdir is None:
              return
  
          # Store log only in the top parent
          if self.parent:
              self.parent._log(message)
          else:
              with open(os.path.join(self.workdir, LOG_FILENAME), 'a') as log:
                  log.write(datetime.datetime.utcnow().strftime('%H:%M:%S') + ' '
                            + remove_color(message) + '\n')
  
      def print(
              self,
              key: str,
              value: Optional[str] = None,
              color: Optional[str] = None,
              shift: int = 0,
              err: bool = False) -> None:
          """ Print a message regardless the quiet mode """
          self._log(self._indent(key, value, color=None, shift=shift))
          echo(self._indent(key, value, color, shift), err=err)
  
      def info(
              self,
              key: str,
              value: Optional[str] = None,
              color: Optional[str] = None,
              shift: int = 0,
              err: bool = False) -> None:
          """ Show a message unless in quiet mode """
          self._log(self._indent(key, value, color=None, shift=shift))
          if not self.opt('quiet'):
              echo(self._indent(key, value, color, shift), err=err)
  
      def verbose(
              self,
              key: str,
              value: Optional[str] = None,
              color: Optional[str] = None,
              shift: int = 0,
              level: int = 1,
              err: bool = False) -> None:
          """
          Show message if in requested verbose mode level
  
          In quiet mode verbose messages are not displayed.
          """
          self._log(self._indent(key, value, color=None, shift=shift))
          if not self.opt('quiet') and self.opt('verbose') >= level:
              echo(self._indent(key, value, color, shift), err=err)
  
      def debug(
              self,
              key: str,
              value: Optional[str] = None,
              color: Optional[str] = None,
              shift: int = 0,
              level: int = 1,
              err: bool = False) -> None:
          """
          Show message if in requested debug mode level
  
          In quiet mode debug messages are not displayed.
          """
          self._log(self._indent(key, value, color=None, shift=shift))
          if not self.opt('quiet') and self.opt('debug') >= level:
              echo(self._indent(key, value, color, shift), err=err)
  
      def warn(self, message: str, shift: int = 0) -> None:
          """ Show a yellow warning message on info level, send to stderr """
          self.info('warn', message, color='yellow', shift=shift, err=True)
  
      def fail(self, message: str, shift: int = 0) -> None:
          """ Show a red failure message on info level, send to stderr """
          self.info('fail', message, color='red', shift=shift, err=True)
  
      def _run(self,
               command: Union[str, List[str]],
               cwd: Optional[str],
               shell: bool,
               env: Optional[EnvironmentType],
               log: Optional[BaseLoggerFnType],
               join: bool = False,
               interactive: bool = False,
               timeout: Optional[int] = None) -> CommandOutput:
          """
          Run command, capture the output
  
          By default stdout and stderr are captured separately.
          Use join=True to merge stderr into stdout.
          Use timeout=<seconds> to finish process after given time
          """
          # By default command ouput is logged using debug
          if not log:
              log = self.debug
          # Prepare the environment
          if env:
              if not isinstance(env, dict):
                  raise GeneralError(f"Invalid environment '{env}'.")
              # Do not modify current process environment
              environment = os.environ.copy()
              environment.update(env)
          else:
              environment = None
          self.debug('environment', pprint.pformat(environment), level=4)
  
          # Set only for shell=True as it would affect command
          executable = DEFAULT_SHELL if shell else None
  
          # Run the command in interactive mode if requested
          if interactive:
              try:
                  subprocess.run(
                      command, cwd=cwd, shell=shell, env=environment, check=True,
                      executable=executable)
              except subprocess.CalledProcessError:
                  # Interactive mode can return non-zero if the last command
                  # failed, ignore errors here
                  pass
              finally:
                  return CommandOutput(None, None)
  
          # Create the process
          try:
              process = subprocess.Popen(
                  command, cwd=cwd, shell=shell, env=environment,
                  stdin=subprocess.DEVNULL, stdout=subprocess.PIPE,
                  stderr=subprocess.STDOUT if join else subprocess.PIPE,
                  executable=executable)
          except FileNotFoundError as error:
              raise RunError(
                  f"File '{error.filename}' not found.", command, 127)
  
          stdout_thread = StreamLogger(
              process.stdout, log_header='out', logger=log)
          stderr_thread = stdout_thread
          if not join:
              stderr_thread = StreamLogger(
                  process.stderr, log_header='err', logger=log)
          stdout_thread.start()
          if not join:
              stderr_thread.start()
          try:
              process.wait(timeout=timeout)
          except subprocess.TimeoutExpired:
              process.kill()
              process.returncode = PROCESS_TIMEOUT
          stdout_thread.join()
          if not join:
              stderr_thread.join()
  
          # Handle the exit code, return output
          if process.returncode != 0:
              raise RunError(
                  message=f"Command returned '{process.returncode}'.",
                  command=command,
                  returncode=process.returncode,
                  stdout=stdout_thread.get_output(),
                  stderr=stderr_thread.get_output())
          if join:
              return CommandOutput(
                  stdout_thread.get_output(), None)
          else:
              return CommandOutput(
                  stdout_thread.get_output(), stderr_thread.get_output())
  
      def run(self,
              command: Union[str, List[str]],
              message: Optional[str] = None,
              cwd: Optional[str] = None,
              dry: bool = False,
              shell: bool = False,
              env: Optional[EnvironmentType] = None,
              interactive: bool = False,
              join: bool = False,
              log: Optional[BaseLoggerFnType] = None,
              timeout: Optional[int] = None) -> CommandOutput:
          """
          Run command, give message, handle errors
  
          Command is run in the workdir be default.
          In dry mode commands are not executed unless dry=True.
          Environment is updated with variables from the 'env' dictionary.
          Output is logged using self.debug() or custom 'log' function.
          Returns stdout if join=True, (stdout, stderr) tuple otherwise.
          """
  
          # A bit of logging - command, default message, error message for later...
          if isinstance(command, (list, tuple)):
              printable_command = ' '.join(shlex.quote(s) for s in command)
          else:
              printable_command = command
  
          if message:
              self.debug(message, level=2)
  
          self.debug(f'Run command: {printable_command}', level=2)
  
          # Nothing more to do in dry mode (unless requested)
          if self.opt('dry') and not dry:
              return CommandOutput(None, None)
  
          # Run the command, handle the exit code
          cwd = cwd or self.workdir
  
          # Fail nicely if the working directory does not exist
          if cwd and not os.path.exists(cwd):
              raise GeneralError(
                  f"The working directory '{cwd}' does not exist.")
  
          try:
              return self._run(
                  command, cwd, shell, env, log, join, interactive, timeout)
          except RunError as error:
              self.debug(error.message, level=3)
              message = f"Failed to run command: {printable_command} Reason: {error.message}"
              raise RunError(
                  message, error.command, error.returncode,
                  error.stdout, error.stderr)
  
      def read(self, path: str, level: int = 2) -> str:
          """ Read a file from the workdir """
          if self.workdir:
              path = os.path.join(self.workdir, path)
          self.debug(f"Read file '{path}'.", level=level)
          try:
              with open(path, encoding='utf-8', errors='replace') as data:
                  return data.read()
          except OSError as error:
              raise FileError(f"Failed to read '{path}'.\n{error}")
  
      def write(
              self,
              path: str,
              data: Any,
              mode: str = 'w',
              level: int = 2) -> None:
          """ Write a file to the workdir """
          if self.workdir:
              path = os.path.join(self.workdir, path)
          action = 'Append to' if mode == 'a' else 'Write'
          self.debug(f"{action} file '{path}'.", level=level)
          # Dry mode
          if self.opt('dry'):
              return
          try:
              with open(path, mode, encoding='utf-8', errors='replace') as file:
                  file.write(data)
          except OSError as error:
              raise FileError(f"Failed to write '{path}'.\n{error}")
  
      def _workdir_init(self, id_: WorkdirArgumentType = None) -> None:
          """
          Initialize the work directory
  
          Workdir under WORKDIR_ROOT is used/created if 'id' is provided.
          If 'id' is a path, that directory is used instead. Otherwise a
          new workdir is created under the WORKDIR_ROOT directory.
          """
          # Prepare the workdir name from given id or path
          if isinstance(id_, str):
              # Use provided directory if full path given
              if '/' in id_:
                  workdir = id_
              # Construct directory name under workdir root
              else:
                  workdir = os.path.join(WORKDIR_ROOT, id_)
          # Weird workdir id
          elif id_ is not None:
              raise GeneralError(
                  f"Invalid workdir '{id_}', expected a string or None.")
  
          def _check_or_create_workdir_root_with_perms() -> None:
              """ If created WORKDIR_ROOT has to be 1777 for multi-user"""
              if not os.path.isdir(WORKDIR_ROOT):
                  try:
                      os.makedirs(WORKDIR_ROOT, exist_ok=True)
                      os.chmod(WORKDIR_ROOT, 0o1777)
                  except OSError as error:
                      raise FileError(f"Failed to prepare workdir '{WORKDIR_ROOT}': {error}")
  
          if id_ is None:
              # Prepare WORKDIR_ROOT first
              _check_or_create_workdir_root_with_perms()
  
              # Generated unique id or fail, has to be atomic call
              for id_bit in range(1, WORKDIR_MAX + 1):
                  directory = 'run-{}'.format(str(id_bit).rjust(3, '0'))
                  workdir = os.path.join(WORKDIR_ROOT, directory)
                  try:
                      # Call is atomic, no race possible
                      os.makedirs(workdir)
                      break
                  except FileExistsError:
                      pass
              else:
                  raise GeneralError(
                      f"Workdir full. Cleanup the '{WORKDIR_ROOT}' directory.")
          else:
              # Cleanup possible old workdir if called with --scratch
              if self.opt('scratch'):
                  self._workdir_cleanup(workdir)
  
              if workdir.startswith(WORKDIR_ROOT):
                  _check_or_create_workdir_root_with_perms()
  
              # Create the workdir
              create_directory(workdir, 'workdir', quiet=True)
          self._workdir = workdir
  
      def _workdir_name(self) -> Optional[str]:
          """ Construct work directory name from parent workdir """
          # Need the parent workdir
          if self.parent is None or self.parent.workdir is None:
              return None
          # Join parent name with self
          return os.path.join(self.parent.workdir, self.safe_name.lstrip("/"))
  
      def _workdir_load(self, workdir: WorkdirArgumentType) -> None:
          """
          Create the given workdir if it is not None
  
          If workdir=True, the directory name is automatically generated.
          """
          if workdir is True:
              self._workdir_init()
          elif workdir is not None:
              self._workdir_init(workdir)
  
      def _workdir_cleanup(self, path: Optional[str] = None) -> None:
          """ Clean up the work directory """
          directory = path or self._workdir_name()
          if directory is not None:
              if os.path.isdir(directory):
                  self.debug(f"Clean up workdir '{directory}'.", level=2)
                  shutil.rmtree(directory)
          self._workdir = None
  
      @property
      def workdir(self) -> Optional[str]:
          """ Get the workdir, create if does not exist """
          if self._workdir is None:
              self._workdir = self._workdir_name()
              # Workdir not enabled, even parent does not have one
              if self._workdir is None:
                  return None
              # Create a child workdir under the parent workdir
              create_directory(self._workdir, 'workdir', quiet=True)
          return self._workdir
  
  # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  #  Exceptions
  # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  
  
  class GeneralError(Exception):
      """ General error """
  
      def __init__(self, *args: Any, **kwargs: Any) -> None:
          # Store the original exception for future use
          self.original = kwargs.get('original')
  
  
  class GitUrlError(GeneralError):
      """ Remote git url is not reachable """
  
  
  class FileError(GeneralError):
      """ File operation error """
  
  
  class RunError(GeneralError):
      """ Command execution error """
  
      def __init__(
              self,
              message: str,
              command: Union[str, List[str]],
              returncode: int,
              stdout: Optional[str] = None,
              stderr: Optional[str] = None,
              *args: Any,
              **kwargs: Any) -> None:
          super().__init__(*args, **kwargs)
          self.message = message
          self.command = command
          self.returncode = returncode
          self.stdout = stdout
          self.stderr = stderr
  
  
  class MetadataError(GeneralError):
      """ General metadata error """
  
  
  class SpecificationError(MetadataError):
      """ Metadata specification error """
  
      def __init__(
              self,
              message: str,
              validation_errors: Optional[List[Tuple[jsonschema.ValidationError, str]]] = None,
              *args: Any,
              **kwargs: Any) -> None:
          super().__init__(*args, **kwargs)
  
          self.message = message
          self.validation_errors = validation_errors
  
  
  class ConvertError(MetadataError):
      """ Metadata conversion error """
  
  
  class StructuredFieldError(GeneralError):
      """ StructuredField parsing error """
  
  
  class WaitingIncomplete(GeneralError):
      """ Waiting incomplete """
  
  
  class WaitingTimedOutError(GeneralError):
      """ Waiting ran out of time """
  
  
  # Step exceptions
  
  
  class DiscoverError(GeneralError):
      """ Discover step error """
  
  
  class ProvisionError(GeneralError):
      """ Provision step error """
  
  
  class PrepareError(GeneralError):
      """ Prepare step error """
  
  
  class ExecuteError(GeneralError):
      """ Execute step error """
  
  
  class RebootTimeoutError(ExecuteError):
      """ Reboot failed due to a timeout """
  
  
  class ReportError(GeneralError):
      """ Report step error """
  
  
  class FinishError(GeneralError):
      """ Finish step error """
  
  
  # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  #  Utilities
  # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  
  def quote(string: str) -> str:
      """ Surround a string with double quotes """
      return f'"{string}"'
  
  
  def ascii(text: Any) -> bytes:
      """ Transliterate special unicode characters into pure ascii """
      if not isinstance(text, str):
          text = str(text)
      return unicodedata.normalize('NFKD', text).encode('ascii', 'ignore')
  
  
  def listify(
          data: Union[Tuple[Any, ...], List[Any], str, Dict[Any, Any]],
          split: bool = False,
          keys: Optional[List[str]] = None) -> Union[List[Any], Dict[Any, Any]]:
      """
      Ensure that variable is a list, convert if necessary
      For dictionaries check all items or only those with provided keys.
      Also split strings on white-space/comma if split=True.
      """
      separator = re.compile(r'[\s,]+')
      if isinstance(data, tuple):
          data = list(data)
      if isinstance(data, list):
          return fmf.utils.split(data, separator) if split else data
      if isinstance(data, str):
          return fmf.utils.split(data, separator) if split else [data]
      if isinstance(data, dict):
          for key in keys or data:
              if key in data:
                  data[key] = listify(data[key], split=split)
          return data
      return [data]
  
  
  def copytree(
          src: str,
          dst: str,
          symlinks: bool = False,
          dirs_exist_ok: bool = False,
          ) -> Any:
      """ Similar to shutil.copytree but with dirs_exist_ok for Python < 3.8 """
      # No need to reimplement for newer python or if argument is not requested
      if not dirs_exist_ok or sys.version_info >= (3, 8):
          return shutil.copytree(
              src=src, dst=dst, symlinks=symlinks, dirs_exist_ok=dirs_exist_ok)
      # Choice was to either copy python implementation and change ONE line
      # or use rsync (or cp with shell)
      # We need to copy CONTENT of src into dst
      # so src has to end with / and dst cannot
      if src[-1] != '/':
          src += '/'
      if dst[-1] == '/':
          dst = dst[:-1]
  
      command = ["rsync", "-r"]
      if symlinks:
          command.append('-l')
      command.extend([src, dst])
  
      log.debug(f"Calling command '{command}'.")
      outcome = subprocess.run(
          command,
          stdin=subprocess.DEVNULL, stdout=subprocess.PIPE,
          stderr=subprocess.STDOUT, universal_newlines=True)
  
      if outcome.returncode != 0:
          raise shutil.Error(
              [f"Unable to copy '{src}' into '{dst}' using rsync.",
               outcome.returncode, outcome.stdout])
      return dst
  
  
  # These two are helpers for shell_to_dict and environment_to_dict -
  # there is some overlap of their functionality.
  def _add_simple_var(result: EnvironmentType, var: str) -> None:
      """
      Add a single NAME=VALUE pair into result dictionary
  
      Parse given string VAR to its constituents, NAME and VALUE, and add
      them to the provided dict.
      """
  
      matched = re.match("([^=]+)=(.*)", var)
      if not matched:
          raise GeneralError(f"Invalid variable specification '{var}'.")
      name, value = matched.groups()
      result[name] = value
  
  
  def _add_file_vars(result: EnvironmentType, filepath: str) -> None:
      """
      Add variables loaded from file into the result dictionary
  
      Load mapping from a YAML file 'filepath', and add its content -
      "name: value" entries - to the provided dict.
      """
  
      if not filepath[1:]:
          raise GeneralError(
              f"Invalid variable file specification '{filepath}'.")
  
      try:
          with open(filepath[1:], 'r') as file:
              # Handle empty file as an empty environment
              content = file.read()
              if not content:
                  log.warn(f"Empty environment file '{filepath}'.")
                  return
              file_vars = yaml_to_dict(content)
      except Exception as exception:
          raise GeneralError(
              f"Failed to load variables from '{filepath}': {exception}")
  
      for name, value in file_vars.items():
          result[name] = str(value)
  
  
  def shell_to_dict(variables: Union[str, List[str]]) -> EnvironmentType:
      """
      Convert shell-like variables into a dictionary
  
      Accepts single string or list of strings. Allowed forms are:
      'X=1'
      'X=1 Y=2 Z=3'
      ['X=1', 'Y=2', 'Z=3']
      ['X=1 Y=2 Z=3', 'A=1 B=2 C=3']
      'TXT="Some text with spaces in it"'
      """
      if not isinstance(variables, (list, tuple)):
          variables = [variables]
      result: EnvironmentType = dict()
      for variable in variables:
          if variable is None:
              continue
          for var in shlex.split(variable):
              _add_simple_var(result, var)
  
      return result
  
  
  def environment_to_dict(variables: Union[str, List[str]]) -> EnvironmentType:
      """
      Convert environment variables into a dictionary
  
      Variables may be specified in the following two ways:
  
      * NAME=VALUE pairs
      * @foo.yaml
  
      If "variable" starts with "@" character, it is treated as a path to
      a YAML file that contains "key: value" pairs which are then
      transparently loaded and added to the final dictionary.
  
      In general, allowed inputs are the same as in "shell_to_dict"
      function, with the addition of "@foo.yaml" form:
      'X=1'
      'X=1 Y=2 Z=3'
      ['X=1', 'Y=2', 'Z=3']
      ['X=1 Y=2 Z=3', 'A=1 B=2 C=3']
      'TXT="Some text with spaces in it"'
      @foo.yaml
      @../../bar.yaml
      """
  
      if not isinstance(variables, (list, tuple)):
          variables = [variables]
      result: EnvironmentType = dict()
  
      for variable in variables:
          if variable is None:
              continue
          for var in shlex.split(variable):
              if var.startswith('@'):
                  _add_file_vars(result, var)
              else:
                  _add_simple_var(result, var)
  
      return result
  
  
  @lru_cache(maxsize=None)
  def environment_file_to_dict(env_file: str, root: str = ".") -> EnvironmentType:
      """
      Read environment variables from the given file.
  
      File should be in YAML format (``.yaml`` or ``.yml`` suffixes), or in dotenv format.
  
      .. code-block:: bash
         :caption: dotenv file example
  
         A=B
         C=D
  
      .. code-block:: yaml
         :caption: YAML file example
  
         A: B
         C: D
  
      Path to each file should be relative to the metadata tree root.
  
      .. note::
  
         For loading environment variables from multiple files, see
         :py:func:`environment_files_to_dict`.
      """
  
      env_file = env_file.strip()
  
      # Fetch a remote file
      if env_file.startswith("http"):
          # Create retry session for longer retries, see #1229
          session = retry_session.create(
              retries=ENVFILE_RETRY_SESSION_RETRIES,
              backoff_factor=ENVFILE_RETRY_SESSION_BACKOFF_FACTOR,
              allowed_methods=('GET',),
              status_forcelist=(
                  429,  # Too Many Requests
                  500,  # Internal Server Error
                  502,  # Bad Gateway
                  503,  # Service Unavailable
                  504   # Gateway Timeout
                  ),
              )
          try:
              response = session.get(env_file)
              response.raise_for_status()
              content = response.text
          except requests.RequestException as error:
              raise GeneralError(
                  f"Failed to fetch the environment file from '{env_file}'. "
                  f"The problem was: '{error}'")
  
      # Read a local file
      else:
          # Ensure we don't escape from the metadata tree root
          try:
              root_path = Path(root).resolve()
              full_path = (Path(root_path) / Path(env_file)).resolve()
              full_path.relative_to(root_path)
          except ValueError:
              raise GeneralError(
                  f"The 'environment-file' path '{full_path}' is outside "
                  f"of the metadata tree root '{root}'.")
          if not Path(full_path).is_file():
              raise GeneralError(f"File '{full_path}' doesn't exist.")
  
          content = Path(full_path).read_text()
  
      # Parse yaml file
      if os.path.splitext(env_file)[1].lower() in ('.yaml', '.yml'):
          environment = parse_yaml(content)
  
      else:
          try:
              environment = parse_dotenv(content)
  
          except ValueError:
              raise GeneralError(
                  f"Failed to extract variables from environment file "
                  f"'{full_path}'. Ensure it has the proper format "
                  f"(i.e. A=B).")
  
      if not environment:
          log.warn(f"Empty environment file '{env_file}'.")
  
          return {}
  
      return environment
  
  
  def environment_files_to_dict(env_files: Iterable[str], root: str = ".") -> EnvironmentType:
      """
      Read environment variables from the given list of files.
  
      Files should be in YAML format (``.yaml`` or ``.yml`` suffixes), or in dotenv format.
  
      .. code-block:: bash
         :caption: dotenv file example
  
         A=B
         C=D
  
      .. code-block:: yaml
         :caption: YAML file example
  
         A: B
         C: D
  
      Path to each file should be relative to the metadata tree root.
  
      .. note::
  
         For loading environment variables from a single file, see
         :py:func:`environment_file_to_dict`, which is a function
         ``environment_files_to_dict()`` calls for each file,
         accumulating data from all input files.
      """
  
      result: EnvironmentType = {}
  
      for env_file in env_files:
          result.update(environment_file_to_dict(env_file, root=root))
  
      return result
  
  
  @contextlib.contextmanager
  def modify_environ(
          new_elements: EnvironmentType) -> Generator[None, None, None]:
      """ A context manager for os.environ that restores the initial state """
      environ_backup = os.environ.copy()
      os.environ.clear()
      os.environ.update(new_elements)
      try:
          yield
      finally:
          os.environ.clear()
          os.environ.update(environ_backup)
  
  
  def context_to_dict(context: List[str]) -> FmfContextType:
      """
      Convert command line context definition into a dictionary
  
      Does the same as environment_to_dict() plus separates possible
      comma-separated values into lists. Here's a couple of examples:
  
      distro=fedora-33 ---> {'distro': ['fedora']}
      arch=x86_64,ppc64 ---> {'arch': ['x86_64', 'ppc64']}
      """
      return {
          key: value.split(',')
          for key, value in environment_to_dict(context).items()}
  
  
  def dict_to_yaml(
          data: Union[Dict[str, Any], List[Any]],
          width: Optional[int] = None,
          sort: bool = False,
          start: bool = False) -> str:
      """ Convert dictionary into yaml """
      output = io.StringIO()
      yaml = YAML()
      yaml.indent(mapping=4, sequence=4, offset=2)
      yaml.default_flow_style = False
      yaml.allow_unicode = True
      yaml.encoding = 'utf-8'
      yaml.width = width
      yaml.explicit_start = start
      # Convert multiline strings
      scalarstring.walk_tree(data)
      if sort:
          # Sort the data https://stackoverflow.com/a/40227545
          sorted_data = CommentedMap()
          for key in sorted(data):
              sorted_data[key] = data[key]
          data = sorted_data
      yaml.dump(data, output)
      return output.getvalue()
  
  
  YamlTypType = Literal['rt', 'safe', 'unsafe', 'base']
  
  
  def yaml_to_dict(data: Any,
                   yaml_type: Optional[YamlTypType] = None) -> Dict[Any, Any]:
      """ Convert yaml into dictionary """
      yaml = YAML(typ=yaml_type)
      loaded_data = yaml.load(data)
      if loaded_data is None:
          return dict()
      if not isinstance(loaded_data, dict):
          raise GeneralError(
              f"Expected dictionary in yaml data, "
              f"got '{type(loaded_data).__name__}'.")
      return loaded_data
  
  
  def yaml_to_list(data: Any,
                   yaml_type: Optional[YamlTypType] = 'safe') -> List[Any]:
      """ Convert yaml into list """
      yaml = YAML(typ=yaml_type)
      try:
          loaded_data = yaml.load(data)
      except ParserError as error:
          raise GeneralError(f"Invalid yaml syntax: {error}")
  
      if loaded_data is None:
          return list()
      if not isinstance(loaded_data, list):
          raise GeneralError(
              f"Expected list in yaml data, "
              f"got '{type(loaded_data).__name__}'.")
      return list(loaded_data)
  
  
  def key_to_option(key: str) -> str:
      """ Convert a key name to corresponding option name """
  
      return key.replace('_', '-')
  
  
  def option_to_key(option: str) -> str:
      """ Convert an option name to corresponding key name """
  
      return option.replace('-', '_')
  
  
  SerializableContainerDerivedType = TypeVar(
      'SerializableContainerDerivedType',
      bound='SerializableContainer')
  
  
  @dataclasses.dataclass
  class SerializableContainer:
      """
      A mixin class for objects that may be saved in files and restored later
      """
  
      def to_dict(self) -> Dict[str, Any]:
          """ Return keys and values in the form of a dictionary """
  
          return dataclasses.asdict(self)
  
      # This method should remain a class-method: 1. list of keys is known
      # already, therefore it's not necessary to create an instance, and
      # 2. some functionality makes use of this knowledge.
      @classmethod
      def keys(cls) -> Generator[str, None, None]:
          """ Iterate over key names """
  
          for field in dataclasses.fields(cls):
              yield field.name
  
      def values(self) -> Generator[Any, None, None]:
          """ Iterate over key values """
  
          yield from self.to_dict().values()
  
      def items(self) -> Generator[Tuple[str, Any], None, None]:
          """ Iterate over key/value pairs """
  
          yield from self.to_dict().items()
  
      @classmethod
      def default(cls, key: str, default: Any = None) -> Any:
          """
          Return a default value for a given key.
  
          Keys may have a default value, or a default *factory* has been specified.
  
          :param key: key to look for.
          :param default: when key has no default value, ``default`` is returned.
          :returns: a default value defined for the key, or its ``default_factory``'s
              return value of ``default_factory``, or ``default`` when key has no
              default value.
          """
  
          for field in dataclasses.fields(cls):
              if key != field.name:
                  continue
  
              if not isinstance(field.default_factory, dataclasses._MISSING_TYPE):
                  return field.default_factory()
  
              if not isinstance(field.default, dataclasses._MISSING_TYPE):
                  return field.default
  
          else:
              return default
  
      @property
      def is_bare(self) -> bool:
          """
          Check whether all keys are either unset or have their default value.
  
          :returns: ``True`` if all keys either hold their default value
              or are not set at all, ``False`` otherwise.
          """
  
          for field in dataclasses.fields(self):
              value = getattr(self, field.name)
  
              if not isinstance(field.default_factory, dataclasses._MISSING_TYPE):
                  if value != field.default_factory():
                      return False
  
              elif not isinstance(field.default, dataclasses._MISSING_TYPE):
                  if value != field.default:
                      return False
  
              else:
                  pass
  
          return True
  
      #
      # Moving data between containers and objects owning them
      #
  
      def inject_to(self, obj: Any) -> None:
          """
          Inject keys from this container into attributes of a given object
          """
  
          for name, value in self.items():
              setattr(obj, name, value)
  
      @classmethod
      def extract_from(cls: Type[SerializableContainerDerivedType],
                       obj: Any) -> SerializableContainerDerivedType:
          """ Extract keys from given object, and save them in a container """
  
          data = cls()
  
          for key in cls.keys():
              value = getattr(obj, key)
              if value is not None:
                  setattr(data, key, value)
  
          return data
  
      #
      # Serialization - writing containers into YAML files, and restoring
      # them later.
      #
  
      def to_serialized(self) -> Dict[str, Any]:
          """
          Return keys and values in the form allowing later reconstruction.
  
          Used to transform container into a structure one can save in a
          YAML file, and restore it later.
  
          See :py:meth:`from_serialized` for its counterpart.
          """
  
          fields = self.to_dict()
  
          # Add a special field tracking what class we just shattered to pieces.
          fields.update({
              '__class__': {
                  'module': self.__class__.__module__,
                  'name': self.__class__.__name__
                  }
              })
  
          return fields
  
      @classmethod
      def from_serialized(
              cls: Type[SerializableContainerDerivedType],
              serialized: Dict[str, Any]) -> SerializableContainerDerivedType:
          """
          Recreate container from its serialized form.
  
          Used to transform data read from a YAML file into the original
          container.
  
          See :py:meth:`to_serialized` for its counterpart.
          """
  
          # Our special key may or may not be present, depending on who
          # calls this method.  In any case, it is not needed, because we
          # already know what class to restore: this one.
          serialized.pop('__class__', None)
  
          return cls(**serialized)
  
      @staticmethod
      def unserialize(serialized: Dict[str, Any]
                      ) -> SerializableContainerDerivedType:
          """
          Recreate container from its serialized form.
  
          Similar to :py:meth:`from_serialized`, but this method knows
          nothing about container's class, and will locate the correct
          module and class by inspecting serialized data. Discovered
          class' :py:meth:`from_serialized` is then used to create the
          container.
  
          Used to transform data read from a YAML file into original
          containers when their classes are not know to the code.
          Restoring such containers requires inspection of serialized data
          and dynamic imports of modules as needed.
          """
  
          from tmt.plugins import import_member
  
          # Unpack class info, to get nicer variable names
          if "__class__" not in serialized:
              raise GeneralError(
                  "Failed to load saved state, probably because of old data format.\n"
                  "Use 'tmt clean runs' to clean up old runs.")
  
          klass_info = serialized.pop('__class__')
          klass = import_member(klass_info['module'], klass_info['name'])
  
          # Stay away from classes that are not derived from this one, to
          # honor promise given by return value annotation.
          assert issubclass(klass, SerializableContainer)
  
          # Apparently, the issubclass() check above is not good enough for mypy.
          return cast(
              SerializableContainerDerivedType,
              klass.from_serialized(serialized))
  
  
  def markdown_to_html(filename: str) -> str:
      """
      Convert markdown to html
  
      Expects: Markdown document as a file.
      Returns: An HTML document as a string.
      """
      try:
          import markdown
      except ImportError:
          raise ConvertError("Install tmt-test-convert to export tests.")
  
      try:
          with open(filename, 'r') as file:
              try:
                  text = file.read()
              except UnicodeError:
                  raise MetadataError(f"Unable to read '{filename}'.")
              return markdown.markdown(text)
      except IOError:
          raise ConvertError(f"Unable to open '{filename}'.")
  
  
  def shell_variables(
          data: Union[List[str], Tuple[str, ...], Dict[str, Any]]) -> List[str]:
      """
      Prepare variables to be consumed by shell
  
      Convert dictionary or list/tuple of key=value pairs to list of
      key=value pairs where value is quoted with shlex.quote().
      """
  
      # Convert from list/tuple
      if isinstance(data, list) or isinstance(data, tuple):
          converted_data = []
          for item in data:
              splitted_item = item.split('=')
              key = splitted_item[0]
              value = shlex.quote('='.join(splitted_item[1:]))
              converted_data.append(f'{key}={value}')
          return converted_data
  
      # Convert from dictionary
      return [f"{key}={shlex.quote(str(value))}" for key, value in data.items()]
  
  
  def duration_to_seconds(duration: str) -> int:
      """ Convert sleep time format into seconds """
      units = {
          's': 1,
          'm': 60,
          'h': 60 * 60,
          'd': 60 * 60 * 24,
          }
      try:
          match = re.match(r'^(\d+)([smhd]?)$', str(duration))
          if match is None:
              raise SpecificationError(f"Invalid duration '{duration}'.")
          number, suffix = match.groups()
          return int(number) * units.get(suffix, 1)
      except (ValueError, AttributeError):
          raise SpecificationError(f"Invalid duration '{duration}'.")
  
  
  def verdict(
          decision: Optional[bool],
          comment: Optional[str] = None,
          good: str = 'pass',
          bad: str = 'fail',
          problem: str = 'warn',
          **kwargs: Any) -> Optional[bool]:
      """
      Print verdict in green, red or yellow based on the decision
  
      The supported decision values are:
  
          True .... good (green)
          False ... bad (red)
          None .... problem (yellow)
  
      Anything else raises an exception. Additional arguments
      are passed to the `echo` function. Returns back the decision.
      """
  
      if decision is False:
          text = style(bad, fg='red')
      elif decision is True:
          text = style(good, fg='green')
      elif decision is None:
          text = style(problem, fg='yellow')
      else:
          raise GeneralError(
              "Invalid decision value, must be 'True', 'False' or 'None'.")
      if comment:
          text = text + ' ' + comment
      echo(text, **kwargs)
      return decision
  
  
  def format(
          key: str,
          value: Union[None, str, List[Any], Dict[Any, Any]] = None,
          indent: int = 12,
          width: int = 72,
          wrap: Literal[True, False, 'auto'] = 'auto',
          key_color: str = 'green',
          value_color: str = 'black') -> str:
      """
      Nicely format and indent a key-value pair
  
      The following values for 'wrap' are supported:
  
          True .... always reformat text and wrap long lines
          False ... preserve text, no new line changes
          auto .... wrap only if text contains a long line
      """
      indent_string = (indent + 1) * ' '
      # Key
      output = '{} '.format(str(key).rjust(indent, ' '))
      if key_color is not None:
          output = style(output, fg=key_color)
      # Bool
      if isinstance(value, bool):
          output += ('true' if value else 'false')
      # List
      elif isinstance(value, list):
          # Make sure everything is string, prepare list, check for spaces
          value = [str(item) for item in value]
          listed_text = fmf.utils.listed(value)
          has_spaces = any(item.find(' ') > -1 for item in value)
          # Use listed output only for short lists without spaces
          if len(listed_text) < width - indent and not has_spaces:
              output += listed_text
          # Otherwise just place each item on a new line
          else:
              output += ('\n' + indent_string).join(value)
      # Dictionary
      elif isinstance(value, dict):
          # Place each key value pair on a separate line
          output += ('\n' + indent_string).join(
              f'{item[0]}: {item[1]}' for item in value.items())
      # Text
      elif isinstance(value, str):
          # In 'auto' mode enable wrapping when long lines present
          if wrap == 'auto':
              wrap = any(len(line) + indent - 7 > width for line in value.split('\n'))
          if wrap:
              output += (wrap_text(
                  value, width=width,
                  preserve_paragraphs=True,
                  initial_indent=indent_string,
                  subsequent_indent=indent_string).lstrip())
          else:
              output += (('\n' + indent_string).join(
                  value.rstrip().split('\n')))
      else:
          output += str(value)
      return output
  
  
  def create_directory(
          path: str,
          name: str,
          dry: bool = False,
          quiet: bool = False) -> None:
      """ Create a new directory, handle errors """
      say = log.debug if quiet else echo
      if os.path.isdir(path):
          say("Directory '{}' already exists.".format(path))
          return
      if dry:
          say("Directory '{}' would be created.".format(path))
          return
      try:
          os.makedirs(path, exist_ok=True)
          say("Directory '{}' created.".format(path))
      except OSError as error:
          raise FileError("Failed to create {} '{}' ({})".format(
              name, path, error))
  
  
  def create_file(
          path: str,
          content: str,
          name: str,
          dry: bool = False,
          force: bool = False,
          mode: int = 0o664,
          quiet: bool = False) -> None:
      """ Create a new file, handle errors """
      say = log.debug if quiet else echo
      action = 'would be created' if dry else 'created'
      if os.path.exists(path):
          if force:
              action = 'would be overwritten' if dry else 'overwritten'
          else:
              raise FileError("File '{}' already exists.".format(path))
  
      if dry:
          say("{} '{}' {}.".format(name.capitalize(), path, action))
          return
  
      try:
          with open(path, 'w') as file_:
              file_.write(content)
          say("{} '{}' {}.".format(name.capitalize(), path, action))
          os.chmod(path, mode)
      except OSError as error:
          raise FileError("Failed to create {} '{}' ({})".format(
              name, path, error))
  
  
  # Avoid multiple subprocess calls for the same url
  @lru_cache(maxsize=None)
  def check_git_url(url: str) -> str:
      """ Check that a remote git url is accessible """
      try:
          log.debug(f"Check git url '{url}'.")
          subprocess.check_call(
              ["git", "ls-remote", "--heads", url],
              stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
              env={"GIT_ASKPASS": "echo", "GIT_TERMINAL_PROMPT": "0"})
          return url
      except subprocess.CalledProcessError:
          raise GitUrlError(f"Unable to contact remote git via '{url}'.")
  
  
  def public_git_url(url: str) -> str:
      """
      Convert a git url into a public format
  
      Return url in the format which can be accessed without
      authentication. For now just cover the most common services.
      """
  
      # GitHub, GitLab
      # old: git@github.com:teemtee/tmt.git
      # new: https://github.com/teemtee/tmt.git
      matched = re.match('git@(.*):(.*)', url)
      if matched:
          host, project = matched.groups()
          return f'https://{host}/{project}'
  
      # RHEL packages
      # old: git+ssh://psplicha@pkgs.devel.redhat.com/tests/bash
      # old: ssh://psplicha@pkgs.devel.redhat.com/tests/bash
      # old: ssh://pkgs.devel.redhat.com/tests/bash
      # new: git://pkgs.devel.redhat.com/tests/bash
      matched = re.match(
          r'(git\+)?ssh://(\w+@)?(pkgs\.devel\.redhat\.com)/(.*)', url)
      if matched:
          _, _, host, project = matched.groups()
          return f'git://{host}/{project}'
  
      # Fedora packages, Pagure
      # old: git+ssh://psss@pkgs.fedoraproject.org/tests/shell
      # old: ssh://psss@pkgs.fedoraproject.org/tests/shell
      # new: https://pkgs.fedoraproject.org/tests/shell
      matched = re.match(r'(git\+)?ssh://(\w+@)?([^/]*)/(.*)', url)
      if matched:
          _, _, host, project = matched.groups()
          return f'https://{host}/{project}'
  
      # Otherwise return unmodified
      return url
  
  
  class TimeoutHTTPAdapter(requests.adapters.HTTPAdapter):
      """
      Spice up request's session with custom timeout.
      """
  
      def __init__(self, *args: Any, **kwargs: Any) -> None:
          self.timeout = kwargs.pop('timeout', None)
  
          super().__init__(*args, **kwargs)
  
      def send(  # type: ignore[override] # does not match superclass type on purpose
              self,
              request: requests.PreparedRequest,
              **kwargs: Any) -> requests.Response:
          kwargs.setdefault('timeout', self.timeout)
  
          return super().send(request, **kwargs)
  
  
  class RetryStrategy(requests.packages.urllib3.util.retry.Retry):  # type: ignore[misc]
      def increment(
              self,
              *args: Any,
              **kwargs: Any
              ) -> requests.packages.urllib3.util.retry.Retry:
          error = cast(Optional[Exception], kwargs.get('error', None))
  
          # Detect a subset of exception we do not want to follow with a retry.
          if error is not None:
              # Failed certificate verification - this issue will probably not get any better
              # should we try again.
              if isinstance(error, urllib3.exceptions.SSLError) \
                      and 'certificate verify failed' in str(error):
  
                  # [mpr] I'm not sure how stable this *iternal* API is, but pool seems to be the
                  # only place aware of the remote hostname. Try our best to get the hostname for
                  # a better error message, but don't crash because of a missing attribute or
                  # something as dumb.
  
                  connection_pool = kwargs.get('_pool', None)
  
                  if connection_pool is not None and hasattr(connection_pool, 'host'):
                      message = f"Certificate verify failed for '{connection_pool.host}'."
                  else:
                      message = 'Certificate verify failed.'
  
                  raise GeneralError(message, original=error) from error
  
          return super().increment(*args, **kwargs)
  
  
  class retry_session(contextlib.AbstractContextManager):  # type: ignore[type-arg]
      """
      Context manager for requests.Session() with retries and timeout
      """
      @staticmethod
      def create(
              retries: int = DEFAULT_RETRY_SESSION_RETRIES,
              backoff_factor: float = DEFAULT_RETRY_SESSION_BACKOFF_FACTOR,
              allowed_methods: Optional[Tuple[str, ...]] = None,
              status_forcelist: Optional[Tuple[int, ...]] = None,
              timeout: Optional[int] = None
              ) -> requests.Session:
          retry_strategy = RetryStrategy(
              total=retries,
              status_forcelist=status_forcelist,
              # `method_whitelist`` has been renamed to `allowed_methods` since
              # urllib3 1.26, and it will be removed in urllib3 2.0.
              # `allowed_methods` is therefore the future-proof name, but for the
              # sake of backward compatibility, internally we need to use the
              # deprecated parameter for now. Or request newer urllib3, but that
              # might a problem because of RPM availability.
              method_whitelist=allowed_methods,
              backoff_factor=backoff_factor)
  
          if timeout is not None:
              http_adapter: requests.adapters.HTTPAdapter = TimeoutHTTPAdapter(
                  timeout=timeout, max_retries=retry_strategy)
          else:
              http_adapter = requests.adapters.HTTPAdapter(
                  max_retries=retry_strategy)
  
          session = requests.Session()
          session.mount('http://', http_adapter)
          session.mount('https://', http_adapter)
  
          return session
  
      def __init__(
              self,
              retries: int = DEFAULT_RETRY_SESSION_RETRIES,
              backoff_factor: float = DEFAULT_RETRY_SESSION_BACKOFF_FACTOR,
              allowed_methods: Optional[Tuple[str, ...]] = None,
              status_forcelist: Optional[Tuple[int, ...]] = None,
              timeout: Optional[int] = None
              ) -> None:
          self.retries = retries
          self.backoff_factor = backoff_factor
          self.allowed_methods = allowed_methods
          self.status_forcelist = status_forcelist
          self.timeout = timeout
  
      def __enter__(self) -> requests.Session:
          return self.create(
              retries=self.retries,
              backoff_factor=self.backoff_factor,
              allowed_methods=self.allowed_methods,
              status_forcelist=self.status_forcelist,
              timeout=self.timeout)
  
      def __exit__(self, *args: Any) -> None:
          pass
  
  
  def remove_color(text: str) -> str:
      """ Remove ansi color sequences from the string """
      return re.sub(r'\x1B(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~])', '', text)
  
  
  def default_branch(repository: str, remote: str = 'origin') -> str:
      """ Detect default branch from given local git repository """
      head = os.path.join(repository, f'.git/refs/remotes/{remote}/HEAD')
      # Make sure the HEAD reference is available
      if not os.path.exists(head):
          subprocess.run(
              f'git remote set-head {remote} --auto'.split(), cwd=repository)
      # The ref format is 'ref: refs/remotes/origin/main'
      with open(head) as ref:
          return ref.read().strip().split('/')[-1]
  
  
  def parse_dotenv(content: str) -> EnvironmentType:
      """ Parse dotenv (shell) format of variables """
      return dict([line.split("=", maxsplit=1)
                  for line in shlex.split(content, comments=True)])
  
  
  def parse_yaml(content: str) -> EnvironmentType:
      """ Parse variables from yaml, ensure flat dictionary format """
      yaml_as_dict = YAML(typ="safe").load(content)
      # Handle empty file as an empty environment
      if yaml_as_dict is None:
          return dict()
      if any(isinstance(val, dict) for val in yaml_as_dict.values()):
          raise GeneralError(
              "Can't set the environment from the nested yaml config. The "
              "config should be just key, value pairs.")
      return {key: str(value) for key, value in yaml_as_dict.items()}
  
  
  def validate_git_status(test: 'tmt.base.Test') -> Tuple[bool, str]:
      """
      Validate that test has current metadata on fmf_id
  
      Return a tuple (boolean, message) as the result of validation.
  
      Checks that sources:
      - all local changes are committed
      - up to date on remote repository
      - .fmf/version marking fmf root is committed as well
  
      When all checks pass returns (True, '').
      """
      sources = test.node.sources + \
          [os.path.join(test.node.root, '.fmf', 'version')]
  
      # Use tmt's run instead of subprocess.run
      run = Common().run
  
      # Check for not committed metadata changes
      cmd = ['git', 'status', '--porcelain', '--'] + sources
      try:
          result = run(cmd, cwd=test.node.root, join=True)
      except RunError as error:
          return (
              False,
              f"Failed to run git status: {error.stdout}"
              )
  
      not_committed = []
      assert result.stdout is not None
      for line in result.stdout.split('\n'):
          if line:
              # XY PATH or XY ORIG -> PATH. XY and PATH are separated by space
              not_committed.append(line[3:])
  
      if not_committed:
          return (False, "Uncommitted changes in " + " ".join(not_committed))
  
      # Check for not pushed changes
      cmd = ["git", "rev-parse", "--abbrev-ref", "--symbolic-full-name", "@{u}"]
      try:
          result = run(cmd, cwd=test.node.root)
      except RunError as error:
          return (
              False,
              f'Failed to get remote branch, error raised: "{error.stderr}"'
              )
  
      assert result.stdout is not None
      remote_ref = result.stdout.strip()
  
      cmd = [
          'git',
          'diff',
          f'HEAD..{remote_ref}',
          '--name-status',
          '--'] + sources
      try:
          result = run(cmd, cwd=test.node.root)
      except RunError as error:
          return (
              False,
              f'Failed to diff against remote branch, error raised: "{error.stderr}"')
  
      not_pushed = []
      assert result.stdout is not None
      for line in result.stdout.split('\n'):
          if line:
              _, path = line.strip().split('\t', maxsplit=2)
              not_pushed.append(path)
      if not_pushed:
          return (False, "Not pushed changes in " + " ".join(not_pushed))
  
      return (True, '')
  
  
  def generate_runs(
          path: str, id_: Optional[str] = None) -> Generator[str, None, None]:
      """ Generate absolute paths to runs from path """
      # Prepare absolute workdir path if --id was used
      if id_:
          if '/' not in id_:
              id_ = os.path.join(path, id_)
          if os.path.isabs(id_):
              if os.path.exists(id_):
                  yield id_
              return
      if not os.path.exists(path):
          return
      for filename in os.listdir(path):
          abs_path = os.path.join(path, filename)
          # If id_ is None, the abs_path is considered valid (no filtering
          # is being applied). If it is defined, it has been transformed
          # to absolute path and must be equal to abs_path for the run
          # in abs_path to be generated.
          invalid_id = id_ and abs_path != id_
          invalid_run = not os.path.exists(
              os.path.join(abs_path, 'run.yaml'))
          if not os.path.isdir(abs_path) or invalid_id or invalid_run:
              continue
          yield abs_path
  
  
  def load_run(run: 'tmt.base.Run') -> Tuple[bool, Optional[Exception]]:
      """ Load a run and its steps from the workdir """
      try:
          run.load_from_workdir()
      except GeneralError as error:
          return False, error
      for plan in run.plans:
          for step in plan.steps(disabled=True):
              step.load()
      return True, None
  
  
  # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  #  StructuredField
  # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  
  SFSectionValueType = Union[str, List[str]]
  
  
  class StructuredField:
      """
      Handling multiple text data in a single text field
  
      The StructuredField allows you to easily store and extract several
      sections of text data to/from a single text field. The sections are
      separated by section names in square brackets and can be hosted in
      other text as well.
  
      The section names have to be provided on a separate line and there
      must be no leading/trailing white space before/after the brackets.
      The StructuredField supports two versions of the format:
  
      Version 0: Simple, concise, useful when neither the surrounding text
      or the section data can contain lines which could resemble section
      names. Here's an example of a simple StructuredField::
  
          Note written by human.
  
          [section-one]
          Section one content.
  
          [section-two]
          Section two content.
  
          [section-three]
          Section three content.
  
          [end]
  
          Another note written by human.
  
      Version 1: Includes unique header to prevent collisions with the
      surrounding text and escapes any section-like lines in the content::
  
          Note written by human.
  
          [structured-field-start]
          This is StructuredField version 1. Please, edit with care.
  
          [section-one]
          Section one content.
  
          [section-two]
          Section two content.
          [structured-field-escape][something-resembling-section-name]
  
          [section-three]
          Section three content.
  
          [structured-field-end]
  
          Another note written by human.
  
      Note that an additional empty line is added at the end of each
      section to improve the readability. This line is not considered
      to be part of the section content.
  
      Besides handling the whole section content it's also possible to
      store several key-value pairs in a single section, similarly as in
      the ini config format::
  
          [section]
          key1 = value1
          key2 = value2
          key3 = value3
  
      Provide the key name as the optional argument 'item' when accessing
      these single-line items. Note that the section cannot contain both
      plain text data and key-value pairs.
  
      Example::
  
          field = qe.StructuredField()
          field.set("project", "Project Name")
          field.set("details", "somebody", "owner")
          field.set("details", "2013-05-27", "started")
          field.set("description", "This is a description.\\n"
                  "It spans across multiple lines.\\n")
          print field.save()
  
              [structured-field-start]
              This is StructuredField version 1. Please, edit with care.
  
              [project]
              Project Name
  
              [details]
              owner = somebody
              started = 2013-05-27
  
              [description]
              This is a description.
              It spans across multiple lines.
  
              [structured-field-end]
  
          field.version(0)
          print field.save()
  
              [project]
              Project Name
  
              [details]
              owner = somebody
              started = 2013-05-27
  
              [description]
              This is a description.
              It spans across multiple lines.
  
              [end]
  
      Multiple values for the same key are supported as well. Enable this
      feature with 'multi=True' when initializing the structured field.
      If multiple values are present their list will be returned instead
      of a single string. Similarly use list for setting multiple values::
  
          field = qe.StructuredField(multi=True)
          requirements = ['hypervisor=', 'labcontroller=lab.example.com']
          field.set("hardware", requirements, "hostrequire")
          print field.save()
  
              [structured-field-start]
              This is StructuredField version 1. Please, edit with care.
  
              [hardware]
              hostrequire = hypervisor=
              hostrequire = labcontroller=lab.example.com
  
              [structured-field-end]
  
          print field.get("hardware", "hostrequire")
  
              ['hypervisor=', 'labcontroller=lab.example.com']
      """
  
      # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      #  StructuredField Special
      # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  
      def __init__(
              self,
              text: Optional[str] = None,
              version: int = 1,
              multi: bool = False) -> None:
          """ Initialize the structured field """
          self.version(version)
          self._header: str = ""
          self._footer: str = ""
          # Sections are internally stored in their serialized form, i.e. as
          # strings.
          self._sections: Dict[str, str] = {}
          self._order: List[str] = []
          self._multi = multi
          if text is not None:
              self.load(text)
  
      def __iter__(self) -> Generator[str, None, None]:
          """ By default iterate through all available section names """
          for section in self._order:
              yield section
  
      def __nonzero__(self) -> bool:
          """ True when any section is defined """
          return len(self._order) > 0
  
      __bool__ = __nonzero__
  
      # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      #  StructuredField Private
      # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  
      def _load_version_zero(self, text: str) -> None:
          """ Load version 0 format """
          # Attempt to split the text according to the section tag
          section = re.compile(r"\n?^\[([^\]]+)\]\n", re.MULTILINE)
          parts = section.split(text)
          # If just one part ---> no sections present, just plain text
          if len(parts) == 1:
              self._header = parts[0]
              return
          # Pick header & footer, make sure [end] tag is present
          self._header = parts[0]
          self._footer = re.sub("^\n", "", parts[-1])
          if parts[-2] != "end":
              raise StructuredFieldError("No [end] section tag found")
          # Convert to dictionary and save the order
          keys = parts[1:-2:2]
          values = parts[2:-2:2]
          for key, value in zip(keys, values):
              self.set(key, value)
  
      def _load(self, text: str) -> None:
          """ Load version 1+ format """
          # The text must exactly match the format
          format = re.compile(
              r"(.*)^\[structured-field-start\][ \t]*\n"
              r"(.*)\n\[structured-field-end\][ \t]*\n(.*)",
              re.DOTALL + re.MULTILINE)
          # No match ---> plain text or broken structured field
          matched = format.search(text)
          if not matched:
              if "[structured-field" in text:
                  raise StructuredFieldError("StructuredField parse error")
              self._header = text
              log.debug("StructuredField not found, treating as a plain text")
              return
          # Save header & footer (remove trailing new lines)
          self._header = re.sub("\n\n$", "\n", matched.groups()[0])
          if self._header:
              log.debug(u"Parsed header:\n{0}".format(self._header))
          self._footer = re.sub("^\n", "", matched.groups()[2])
          if self._footer:
              log.debug(u"Parsed footer:\n{0}".format(self._footer))
          # Split the content on the section names
          section = re.compile(r"\n\[([^\]]+)\][ \t]*\n", re.MULTILINE)
          parts = section.split(matched.groups()[1])
          # Detect the version
          version_match = re.search(r"version (\d+)", parts[0])
          if not version_match:
              log.error(parts[0])
              raise StructuredFieldError(
                  "Unable to detect StructuredField version")
          self.version(int(version_match.groups()[0]))
          log.debug(
              "Detected StructuredField version {0}".format(
                  self.version()))
          # Convert to dictionary, remove escapes and save the order
          keys = parts[1::2]
          escape = re.compile(r"^\[structured-field-escape\]", re.MULTILINE)
          values = [escape.sub("", value) for value in parts[2::2]]
          for key, value in zip(keys, values):
              self.set(key, value)
          log.debug(u"Parsed sections:\n{0}".format(
              pprint.pformat(self._sections)))
  
      def _save_version_zero(self) -> str:
          """ Save version 0 format """
          result = []
          if self._header:
              result.append(self._header)
          for section, content in self.iterate():
              result.append(u"[{0}]\n{1}".format(section, content))
          if self:
              result.append(u"[end]\n")
          if self._footer:
              result.append(self._footer)
          return "\n".join(result)
  
      def _save(self) -> str:
          """ Save version 1+ format """
          result = []
          # Regular expression for escaping section-like lines
          escape = re.compile(r"^(\[.+\])$", re.MULTILINE)
          # Header
          if self._header:
              result.append(self._header)
          # Sections
          if self:
              result.append(
                  u"[structured-field-start]\n"
                  u"This is StructuredField version {0}. "
                  u"Please, edit with care.\n".format(self._version))
              for section, content in self.iterate():
                  result.append(u"[{0}]\n{1}".format(section, escape.sub(
                      "[structured-field-escape]\\1", content)))
              result.append(u"[structured-field-end]\n")
          # Footer
          if self._footer:
              result.append(self._footer)
          return "\n".join(result)
  
      def _read_section(self, content: str) -> Dict[str, SFSectionValueType]:
          """ Parse config section and return ordered dictionary """
          dictionary: Dict[str, SFSectionValueType] = OrderedDict()
          for line in content.split("\n"):
              # Remove comments and skip empty lines
              line = re.sub("#.*", "", line)
              if re.match(r"^\s*$", line):
                  continue
              # Parse key and value
              matched = re.search("([^=]+)=(.*)", line)
              if not matched:
                  raise StructuredFieldError(
                      "Invalid key/value line: {0}".format(line))
              key = matched.groups()[0].strip()
              value = matched.groups()[1].strip()
              # Handle multiple values if enabled
              if key in dictionary and self._multi:
                  stored_value = dictionary[key]
                  if isinstance(stored_value, list):
                      stored_value.append(value)
                  else:
                      dictionary[key] = [stored_value, value]
              else:
                  dictionary[key] = value
          return dictionary
  
      def _write_section(self, dictionary: Dict[str, SFSectionValueType]) -> str:
          """ Convert dictionary into a config section format """
          section = ""
          for key in dictionary:
              if isinstance(dictionary[key], list):
                  for value in dictionary[key]:
                      section += "{0} = {1}\n".format(key, value)
              else:
                  section += "{0} = {1}\n".format(key, dictionary[key])
          return section
  
      # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      #  StructuredField Methods
      # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  
      def iterate(self) -> Generator[Tuple[str, str], None, None]:
          """ Return (section, content) tuples for all sections """
          for section in self:
              yield section, self._sections[section]
  
      def version(self, version: Optional[int] = None) -> int:
          """ Get or set the StructuredField version """
          if version is not None:
              if version in [0, 1]:
                  self._version = version
              else:
                  raise StructuredFieldError(
                      "Bad StructuredField version: {0}".format(version))
          return self._version
  
      def load(self, text: str, version: Optional[int] = None) -> None:
          """ Load the StructuredField from a string """
          if version is not None:
              self.version(version)
          # Make sure we got a text, convert from bytes if necessary
          if isinstance(text, bytes):
              text = text.decode("utf8")
          if not isinstance(text, str):
              raise StructuredFieldError(
                  "Invalid StructuredField, expecting string")
          # Remove possible carriage returns
          text = re.sub("\r\n", "\n", text)
          # Make sure the text has a new line at the end
          if text and text[-1] != "\n":
              text += "\n"
          log.debug(u"Parsing StructuredField\n{0}".format(text))
          # Parse respective format version
          if self._version == 0:
              self._load_version_zero(text)
          else:
              self._load(text)
  
      def save(self) -> str:
          """ Convert the StructuredField into a string """
          if self.version() == 0:
              return self._save_version_zero()
          else:
              return self._save()
  
      def header(self, content: Optional[str] = None) -> str:
          """ Get or set the header content """
          if content is not None:
              self._header = content
          return self._header
  
      def footer(self, content: Optional[str] = None) -> str:
          """ Get or set the footer content """
          if content is not None:
              self._footer = content
          return self._footer
  
      def sections(self) -> List[str]:
          """ Get the list of available sections """
          return self._order
  
      def get(
              self,
              section: str,
              item: Optional[str] = None) -> SFSectionValueType:
          """ Return content of given section or section item """
          try:
              content = self._sections[section]
          except KeyError:
              raise StructuredFieldError(
                  "Section [{0!r}] not found".format(ascii(section)))
          # Return the whole section content
          if item is None:
              return content
          # Return only selected item from the section
          try:
              return self._read_section(content)[item]
          except KeyError:
              raise StructuredFieldError(
                  "Unable to read '{0!r}' from section '{1!r}'".format(
                      ascii(item), ascii(section)))
  
      def set(self, section: str, content: Any,
              item: Optional[str] = None) -> None:
          """ Update content of given section or section item """
          # Convert to string if necessary, keep lists untouched
          if isinstance(content, list):
              pass
          elif isinstance(content, bytes):
              content = content.decode("utf8")
          elif not isinstance(content, str):
              content = str(content)
          # Set the whole section content
          if item is None:
              # Add new line if missing
              if content and content[-1] != "\n":
                  content += "\n"
              self._sections[section] = content
          # Set only selected item from the section
          else:
              try:
                  current = self._sections[section]
              except KeyError:
                  current = ""
              dictionary = self._read_section(current)
              dictionary[item] = content
              self._sections[section] = self._write_section(dictionary)
          # Remember the order when adding a new section
          if section not in self._order:
              self._order.append(section)
  
      def remove(self, section: str, item: Optional[str] = None) -> None:
          """ Remove given section or section item """
          # Remove the whole section
          if item is None:
              try:
                  del self._sections[section]
                  del self._order[self._order.index(section)]
              except KeyError:
                  raise StructuredFieldError(
                      "Section [{0!r}] not found".format(ascii(section)))
          # Remove only selected item from the section
          else:
              try:
                  dictionary = self._read_section(self._sections[section])
                  del(dictionary[item])
              except KeyError:
                  raise StructuredFieldError(
                      "Unable to remove '{0!r}' from section '{1!r}'".format(
                          ascii(item), ascii(section)))
              self._sections[section] = self._write_section(dictionary)
  
  
  class DistGitHandler:
      """ Common functionality for DistGit handlers """
      sources_file_name = 'sources'
      uri = "/rpms/{name}/{filename}/{hashtype}/{hash}/{filename}"
  
      usage_name: str  # Name to use for dist-git-type
      re_source: Pattern[str]
      re_ignore_extensions: Pattern[str] = re.compile(r'\.(sign|asc|key)$')
      lookaside_server: str
      remote_substring: Pattern[str]
  
      def url_and_name(self, cwd: str = '.') -> List[Tuple[str, str]]:
          """
          Return list of urls and basenames of the used source
  
          The 'cwd' parameter has to be a DistGit directory.
          """
          # Assumes <package>.spec
          globbed = glob.glob(os.path.join(cwd, '*.spec'))
          if len(globbed) != 1:
              raise GeneralError(f"No .spec file is present in '{cwd}'.")
          package = os.path.basename(globbed[0])[:-len('.spec')]
          ret_values = []
          try:
              with open(os.path.join(cwd, self.sources_file_name)) as f:
                  for line in f.readlines():
                      match = self.re_source.match(line)
                      if match is None:
                          raise GeneralError(
                              f"Couldn't match '{self.sources_file_name}' "
                              f"content with '{self.re_source.pattern}'.")
                      used_hash, source_name, hash_value = match.groups()
                      ret_values.append((self.lookaside_server + self.uri.format(
                          name=package,
                          filename=source_name,
                          hash=hash_value,
                          hashtype=used_hash.lower()
                          ), source_name))
          except Exception as error:
              raise GeneralError(
                  f"Couldn't read '{self.sources_file_name}' file.",
                  original=error)
          if not ret_values:
              raise GeneralError(
                  "No sources found in '{self.sources_file_name}' file.")
          return ret_values
  
      def its_me(self, remotes: List[str]) -> bool:
          """ True if self can work with remotes """
          return any(self.remote_substring.search(item) for item in remotes)
  
  
  class FedoraDistGit(DistGitHandler):
      """ Fedora Handler """
      usage_name = "fedora"
      re_source = re.compile(r"^(\w+) \(([^)]+)\) = ([0-9a-fA-F]+)$")
      lookaside_server = "https://src.fedoraproject.org/repo/pkgs"
      remote_substring = re.compile(r'fedoraproject\.org')
  
  
  class CentOSDistGit(DistGitHandler):
      """ CentOS Handler """
      usage_name = "centos"
      re_source = re.compile(r"^(\w+) \(([^)]+)\) = ([0-9a-fA-F]+)$")
      lookaside_server = "https://sources.stream.centos.org/sources"
      remote_substring = re.compile(r'redhat/centos')
  
  
  def get_distgit_handler(
          remotes: Optional[List[str]] = None,
          usage_name: Optional[str] = None) -> DistGitHandler:
      """
      Return the right DistGitHandler
  
      Pick the DistGitHandler class which understands specified
      remotes or by usage_name.
      """
      for candidate_class in DistGitHandler.__subclasses__():
          if usage_name is not None and usage_name == candidate_class.usage_name:
              return candidate_class()
          if remotes is not None:
              ret_val = candidate_class()
              if ret_val.its_me(remotes):
                  return ret_val
      raise GeneralError(f"No known remote in '{remotes}'.")
  
  
  def get_distgit_handler_names() -> List[str]:
      """ All known distgit handlers """
      return [i.usage_name for i in DistGitHandler.__subclasses__()]
  
  
  def git_clone(
          url: str,
          destination: str,
          common: Common,
          env: Optional[EnvironmentType] = None,
          shallow: bool = False
          ) -> CommandOutput:
      """
      Git clone url to destination, retry without shallow if necessary
  
      For shallow=True attempt to clone repository using --depth=1 option first.
      If not successful attempt to clone whole repo.
  
      Common instance is used to run the command for appropriate logging.
      Environment is updated by 'env' dictionary.
      """
      depth = ['--depth=1'] if shallow else []
      command = ['git', 'clone'] + depth + [url, destination]
      try:
          return common.run(command, env=env)
      except RunError:
          if not shallow:
              # Do not retry if shallow was not used
              raise
          # Git server might not support shallow cloning, try again
          return git_clone(url, destination, common, env, shallow=False)
  
  
  class updatable_message(contextlib.AbstractContextManager):  # type: ignore[type-arg]
      """ Updatable message suitable for progress-bar-like reporting """
  
      def __init__(
              self,
              key: str,
              enabled: bool = True,
              indent_level: int = 0,
              key_color: Optional[str] = None,
              default_value_color: Optional[str] = None
              ) -> None:
          """
          Updatable message suitable for progress-bar-like reporting.
  
          .. code:block:: python3
  
             with updatable_message('foo') as message:
                 while ...:
                     ...
  
                     # check state of remote request, and update message
                     state = remote_api.check()
                     message.update(state)
  
          :param key: a string to use as the left-hand part of logged message.
          :param enabled: if unset, no output would be performed.
          :param indent_level: desired indentation level.
          :param key_color: optional color to apply to ``key``.
          :param default_color: optional color to apply to value when
              :py:meth:`update` is called with ``color`` left out.
          """
  
          self.key = key
          self.enabled = enabled
          self.indent_level = indent_level
          self.key_color = key_color
          self.default_value_color = default_value_color
  
          # No progress if terminal not attached
          if not sys.stdout.isatty():
              self.enabled = False
  
          self._previous_line: Optional[str] = None
  
      def __enter__(self) -> 'updatable_message':
          return self
  
      def __exit__(self, *args: Any) -> None:
          sys.stdout.write('\n')
          sys.stdout.flush()
  
      def update(self, value: str, color: Optional[str] = None) -> None:
          if not self.enabled:
              return
  
          if self._previous_line is not None:
              message = value.ljust(len(self._previous_line))
  
          else:
              message = value
  
          self._previous_line = value
  
          message = indent(
              self.key,
              value=style(
                  message,
                  fg=color or self.default_value_color),
              color=self.key_color,
              level=self.indent_level)
  
          sys.stdout.write(f"\r{message}")
          sys.stdout.flush()
  
  
  def find_fmf_root(path: str) -> List[str]:
      """
      Search trough path and return all fmf roots that exist there
  
      Returned list is ordered by path length, shortest one first.
  
      Raise `MetadataError` if no fmf root is found.
      """
      fmf_roots = []
      for root, _, files in os.walk(path):
          if not os.path.basename(root) == '.fmf':
              continue
          if 'version' in files:
              fmf_roots.append(os.path.dirname(root))
      if not fmf_roots:
          raise MetadataError(f"No fmf root present inside '{path}'.")
      fmf_roots.sort(key=lambda path: len(path))
      return fmf_roots
  
  
  #
  # JSON schema-based validation helpers
  #
  # Aims at FMF data consumed by tmt, but can be used for any structure.
  #
  
  # `Schema` represents a loaded JSON schema structure. It may be fairly complex,
  # but it's not needed to provide the exhaustive and fully detailed type since
  # tmt code is not actually "reading" it. Loaded schema is passed down to
  # jsonschema library, and while `Any` would be perfectly valid, let's use an
  # alias to make schema easier to track in our code.
  Schema = Dict[str, Any]
  SchemaStore = Dict[str, Schema]
  
  
  def _patch_plan_schema(schema: Schema, store: SchemaStore) -> None:
      """
      Resolve references to per-plugin schema known to steps. All schemas have
      been loaded into store, all that's left is to update each step in plan
      schema with the list of schemas allowed for that particular step.
  
      For each step, we create the following schema (see also plan.yaml for the
      rest of plan schema):
  
      .. code-block:: yaml
  
         <step name>:
           oneOf:
             - $ref: "/schemas/<step name>/plugin1"
             - $ref: "/schemas/<step name>/plugin2"
             ...
             - $ref: "/schemas/<step name>/pluginN"
             - type: array
               items:
                 anyOf:
                   - $ref: "/schemas/<step name>/plugin1"
                   - $ref: "/schemas/<step name>/plugin2"
                   ...
                   - $ref: "/schemas/<step name>/pluginN"
      """
  
      for step in ('discover', 'execute', 'finish', 'prepare', 'provision', 'report'):
          step_schema_prefix = f'/schemas/{step}/'
  
          step_plugin_schema_ids = [schema_id for schema_id in store.keys() if schema_id.startswith(
              step_schema_prefix) and schema_id != '/schemas/provision/hardware']
  
          refs: List[Schema] = [
              {'$ref': schema_id} for schema_id in step_plugin_schema_ids
              ]
  
          schema['properties'][step] = {
              'oneOf': refs + [
                  {
                      'type': 'array',
                      'items': {
                          'anyOf': refs
                          }
                      }
                  ]
              }
  
  
  def _load_schema(schema_filepath: str) -> Schema:
      """
      Load a JSON schema from a given filepath.
  
      A helper returning the raw loaded schema.
      """
  
      if not os.path.isabs(schema_filepath):
          schema_filepath = os.path.join(
              pkg_resources.resource_filename(
                  'tmt', 'schemas'), schema_filepath)
  
      try:
          with open(schema_filepath, 'r', encoding='utf-8') as f:
              return cast(Schema, yaml_to_dict(f.read()))
  
      except Exception as exc:
          raise FileError(f"Failed to load schema file {schema_filepath}\n{exc}")
  
  
  @functools.lru_cache(maxsize=None)
  def load_schema(schema_filepath: str) -> Schema:
      """
      Load a JSON schema from a given filepath.
  
      Recommended for general use, the method may apply some post-loading touches
      to the given schema, and unless caller is interested in the raw content of
      the file, this functions should be used instead of the real workhorse of
      schema loading, :py:func:`_load_schema`.
      """
  
      schema = _load_schema(schema_filepath)
  
      if schema.get('$id') == '/schemas/plan':
          _patch_plan_schema(schema, load_schema_store())
  
      return schema
  
  
  @functools.lru_cache(maxsize=None)
  def load_schema_store() -> SchemaStore:
      """
      Load all available JSON schemas, and put them into a "store".
  
      Schema store is a simple mapping between schema IDs and schemas.
      """
  
      store: SchemaStore = {}
  
      schema_dirpath = pkg_resources.resource_filename('tmt', 'schemas')
  
      try:
          for dirpath, dirnames, filenames in os.walk(
                  schema_dirpath, followlinks=True):
              for filename in filenames:
                  # Ignore all files but YAML files.
                  if os.path.splitext(filename)[1].lower() not in ('.yaml', '.yml'):
                      continue
  
                  schema = _load_schema(os.path.join(dirpath, filename))
  
                  store[schema['$id']] = schema
  
      except Exception as exc:
          raise FileError(f"Failed to discover schema files\n{exc}")
  
      if '/schemas/plan' not in store:
          raise FileError('Failed to discover schema for plans')
  
      _patch_plan_schema(store['/schemas/plan'], store)
  
      return store
  
  
  def _prenormalize_fmf_node(node: fmf.Tree, schema_name: str) -> fmf.Tree:
      """
      Apply the minimal possible normalization steps to nodes before validating them with schemas.
  
      tmt allows some fields to have default values, and at least ``how`` field is necessary for
      schema-based validation to work reliably. Based on ``how`` field, plan schema identifies
      the correct *plugin* schema for step validation. Without ``how``, it's hard to pick the
      correct schema.
  
      This function tries to do minimal number of changes to a given fmf node to honor the promise
      of ``how`` being optional, with known defaults for each step. It might be possible to resolve
      this purely with schemas, but since we don't know how (yet?), a Python implementation has been
      chosen to unblock schema-based validation while keeping things easier for users. This may
      change in the future, dropping the need for this pre-validation step.
  
      .. note::
  
         This function is not part of the normalization process that happens after validation. The
         purpose of this function is to make the world nice and shiny for tmt users while avoiding
         the possibility of schema becoming way too complicated, especially when we would need
         non-trivial amount fo time for experiments.
  
         The real normalization process takes place after validation, and is responsible for
         converting raw fmf data to data types and structures more suited for tmt internal
         implementation.
      """
  
      # As of now, only `how` field in plan steps seems to be required for schema-based validation
      # to work correctly, therefore ignore any other node.
      if schema_name != 'plan.yaml':
          return node
  
      # Perform the very crude and careful semi-validation. We need to set the `how` key to a default
      # value - but it's not our job to validate the general structure of node data. Walk the "happy"
      # path, touch the node only when it matches the specification of being a mapping of steps and
      # these being either mappings or lists of mappings. Whenever we notice some value does not
      # match this basic structure, ignore the step completely - its issues will be cought by schema
      # later, don't waste time on steps that do not follow specification.
  
      # Fmf data describing a plan shall be a mapping (with keys like `discover` or `adjust`).
      if not isinstance(node.data, dict):
          return node
  
      # Avoid possible circular imports
      import tmt.steps
  
      def _process_step(step_name: str, step: Dict[Any, Any]) -> None:
          """
          Process a single step configuration.
          """
  
          # If `how` is set, don't touch it, and there's nothing to do.
          if 'how' in step:
              return
  
          # Magic!
          # No, seriously: step is implemented in `tmt.steps.$step_name` package,
          # by a class `tmt.steps.$step_name.$step_name_with_capitalized_first_letter`.
          # Instead of having a set of if-elif tests, we can reach the default `how`
          # dynamically.
  
          from tmt.plugins import import_member
  
          step_module_name = f'tmt.steps.{step_name}'
          step_class_name = step_name.capitalize()
  
          step_class = import_member(step_module_name, step_class_name)
  
          if not issubclass(step_class, tmt.steps.Step):
              raise GeneralError(
                  'Possible step {step_name} implementation '
                  f'{step_module_name}.{step_class_name} is not a subclass '
                  'of tmt.steps.Step class.')
  
          step['how'] = step_class.DEFAULT_HOW
  
      def _process_step_collection(step_name: str, step_collection: Any) -> None:
          """
          Process a collection of step configurations.
          """
  
          # Ignore anything that is not a step.
          if step_name not in tmt.steps.STEPS:
              return
  
          # A single step configuration, represented as a mapping.
          if isinstance(step_collection, dict):
              _process_step(step_name, step_collection)
  
              return
  
          # Multiple step configurations, as mappings in a list
          if isinstance(step_collection, list):
              for step_config in step_collection:
                  # Unexpected, maybe instead of a mapping describing a step someone put
                  # in an integer... Ignore, schema will report it.
                  if not isinstance(step_config, dict):
                      continue
  
                  _process_step(step_name, step_config)
  
      for step_name, step_config in node.data.items():
          _process_step_collection(step_name, step_config)
  
      return node
  
  
  def validate_fmf_node(
          node: fmf.Tree, schema_name: str) -> List[Tuple[jsonschema.ValidationError, str]]:
      """ Validate a given fmf node """
  
      node = _prenormalize_fmf_node(node, schema_name)
  
      result = node.validate(load_schema(schema_name), schema_store=load_schema_store())
  
      if result.result is True:
          return []
  
      # A bit of error formatting. It is possible to use str(error), but the result
      # is a bit too JSON-ish. Let's present an error message in a way that helps
      # users to point finger on each and every issue. But don't throw the original
      # errors away!
  
      errors: List[Tuple[jsonschema.ValidationError, str]] = []
  
      for error in result.errors:
          path = f'{node.name}:{".".join(error.path)}'
  
          errors.append((error, f'{path} - {error.message}'))
  
      return errors
  
  
  # A type for callbacks given to wait()
  WaitCheckType = Callable[[], T]
  
  
  def wait(
      parent: Common,
      check: WaitCheckType[T],
      timeout: datetime.timedelta,
      tick: float = DEFAULT_WAIT_TICK,
      tick_increase: float = DEFAULT_WAIT_TICK_INCREASE
  ) -> T:
      """
      Wait for a condition to become true.
  
      To test the condition state, a ``check`` callback is called every ``tick``
      seconds until ``check`` reports a success. The callback may:
  
      * decide the condition has been fulfilled. This is a successfull outcome,
        ``check`` shall then simply return, and waiting ends. Or,
      * decide more time is needed. This is not a successfull outcome, ``check``
        shall then raise :py:clas:`WaitingIncomplete` exception, and ``wait()``
        will try again later.
  
      :param parent: "owner" of the wait process. Used for its logging capability.
      :param check: a callable responsible for testing the condition. Accepts no
          arguments. To indicate more time and attempts are needed, the callable
          shall raise :py:class:`WaitingIncomplete`, otherwise it shall return
          without exception. Its return value will be propagated by ``wait()`` up
          to ``wait()``'s. All other exceptions raised by ``check`` will propagate
          to ``wait()``'s caller as well, terminating the wait.
      :param timeout: amount of time ``wait()`` is alowed to spend waiting for
          successfull outcome of ``check`` call.
      :param tick: how many seconds to wait between two consecutive calls of
          ``check``.
      :param tick_increase: a multiplier applied to ``tick`` after every attempt.
      :returns: value returned by ``check`` reporting success.
      :raises GeneralError: when ``tick`` is not a positive integer.
      :raises WaitingTimedOutError: when time quota has been consumed.
      """
  
      if tick <= 0:
          raise GeneralError('Tick must be a positive integer')
  
      NOW = time.monotonic
  
      deadline = NOW() + timeout.total_seconds()
  
      parent.debug(
          'wait',
          f"waiting for condition '{check.__name__}' with timeout {timeout},"
          f"deadline in {timeout.total_seconds()}, checking every {tick} seconds")
  
      while True:
          now = NOW()
  
          if now > deadline:
              raise WaitingTimedOutError()
  
          try:
              ret = check()
  
              # Perform one extra check: if `check()` succeeded, but took more time than
              # allowed, it should be recognized as a failed waiting too.
              now = NOW()
  
              if now > deadline:
                  parent.debug(
                      'wait',
                      f"'{check.__name__}' finished successfully but took too much time,"
                      f"{now - deadline} over quota")
  
                  raise WaitingTimedOutError()
  
              parent.debug(
                  'wait',
                  f"'{check.__name__}' finished successfully, {deadline - now} seconds left")
  
              return ret
  
          except WaitingIncomplete:
              parent.debug(
                  'wait',
                  f"'{check.__name__}' still pending, {deadline - now} seconds left")
  
              time.sleep(tick)
  
              tick *= tick_increase
  
              continue
  
  
  class ValidateFmfMixin:
      """
      Mixin adding validation of an fmf node.
  
      Loads a schema whose name is derived from class name, and uses fmf's validate()
      method to perform the validation.
      """
  
      def _validate_fmf_node(self, node: fmf.Tree, logger: Common,
                             raise_on_validation_error: bool) -> None:
          """ Validate a given fmf node """
  
          errors = validate_fmf_node(
              node, f'{self.__class__.__name__.lower()}.yaml')
  
          if errors:
              if raise_on_validation_error:
                  raise SpecificationError(
                      f'fmf node {node.name} failed validation',
                      validation_errors=errors)
  
              for _, error_message in errors:
                  logger.warn(error_message, shift=1)
  
      def __init__(
              self,
              *,
              node: fmf.Tree,
              logger: Common,
              skip_validation: bool = False,
              raise_on_validation_error: bool = False,
              **kwargs: Any) -> None:
          # Validate *before* letting next class in line touch the data.
          if not skip_validation:
              self._validate_fmf_node(node, logger, raise_on_validation_error)
  
          kwargs.setdefault('logger', self)
          super().__init__(node=node, **kwargs)  # type: ignore[call-arg]
  
  
  # A type representing compatible sources of keys and values.
  KeySource = Union[Dict[str, Any], fmf.Tree]
  
  
  class NormalizeKeysMixin:
      """
      Mixin adding support for loading fmf keys into object attributes.
  
      When invoked, annotated class-level variables are searched for in a given source
      container - a mapping, an fmf node, etc. - and if the key of the same name as the
      variable exists, its value is "promoted" to instance variable.
  
      If a method named ``_normalize_<variable name>`` exists, it is called with the fmf
      key value as its single argument, and its return value is assigned to instance
      variable. This gives class chance to modify or transform the original value when
      needed, e.g. to convert the original value to a type more suitable for internal
      processing.
      """
  
      # If specified, keys would be iterated over in the order as listed here.
      KEYS_SHOW_ORDER: List[str] = []
  
      # NOTE: these could be static methods, self is probably useless, but that would
      # cause complications when classes assign these to their members. That makes them
      # no longer static as far as class is concerned, which means they get called with
      # `self` as the first argument. A workaround would be to assign staticmethod()-ized
      # version of them, but that's too much repetition.
      #
      # TODO: wouldn't it be nice if these could be mention in dataclass.field()?
      # It would require a clone of dataclass.field() though.
      def _normalize_string_list(self, value: Union[None, str, List[str]]) -> List[str]:
          if value is None:
              return []
  
          return [value] if isinstance(value, str) else value
  
      def _normalize_environment(self, value: Optional[Dict[str, Any]]) -> EnvironmentType:
          if value is None:
              return {}
  
          return {
              name: str(value) for name, value in value.items()
              }
  
      @classmethod
      def _iter_key_annotations(cls) -> Generator[Tuple[str, Any], None, None]:
          """
          Iterate over keys' type annotations.
  
          Keys are yielded in the order: keys declared by parent classes first, then
          keys declared by the class itself, all following the order in which keys
          were defined in their respective classes.
  
          Yields:
              pairs of key name and its annotations.
          """
  
          def _iter_class_annotations(klass: type) -> Generator[Tuple[str, Any], None, None]:
              # Skip, needs fixes to become compatible
              if klass is Common:
                  return
  
              for name, value in klass.__dict__.get('__annotations__', {}).items():
                  # Skip special fields that are not keys.
                  if name == 'KEYS_SHOW_ORDER':
                      continue
  
                  yield (name, value)
  
          # Reverse MRO to start with the most base classes first, to iterate over keys
          # in the order they are defined.
          for klass in reversed(cls.__mro__):
              yield from _iter_class_annotations(klass)
  
      @classmethod
      def keys(cls) -> Generator[str, None, None]:
          """
          Iterate over key names.
  
          Keys are yielded in the order: keys declared by parent classes first, then
          keys declared by the class itself, all following the order in which keys
          were defined in their respective classes.
  
          Yields:
              key names.
          """
  
          for keyname, _ in cls._iter_key_annotations():
              yield keyname
  
      def items(self) -> Generator[Tuple[str, Any], None, None]:
          """
          Iterate over keys and their values.
  
          Keys are yielded in the order: keys declared by parent classes first, then
          keys declared by the class itself, all following the order in which keys
          were defined in their respective classes.
  
          Yields:
              pairs of key name and its value.
          """
  
          for keyname in self.keys():
              yield (keyname, getattr(self, keyname))
  
      # TODO: exists for backward compatibility for the transition period. Once full
      # type annotations land, there should be no need for extra _keys attribute.
      @classmethod
      def _keys(cls) -> List[str]:
          """ Return a list of names of object's keys. """
  
          return list(cls.keys())
  
      def _load_keys(
              self,
              key_source: Dict[str, Any],
              key_source_name: str,
              logger: Common) -> None:
          """ Extract values for class-level attributes, and verify they match declared types. """
  
          LOG_SHIFT, LOG_LEVEL = 2, 4
  
          debug_intro = functools.partial(logger.debug, shift=LOG_SHIFT - 1, level=LOG_LEVEL)
          debug = functools.partial(logger.debug, shift=LOG_SHIFT, level=LOG_LEVEL)
  
          debug_intro('key source')
          for k, v in key_source.items():
              debug(f'{k}: {v} ({type(v)})')
  
          debug('')
  
          for keyname, keytype in self._iter_key_annotations():
              key_address = f'{key_source_name}:{keyname}'
              source_keyname = key_to_option(keyname)
  
              # Do not indent this particular entry like the rest, so it could serve
              # as a "header" for a single key processing.
              debug_intro('key', key_address)
              debug('field', source_keyname)
  
              debug('desired type', str(keytype))
  
              value: Any = None
  
              # Verbose, let's hide it a bit deeper.
              debug('dict', self.__dict__, level=LOG_LEVEL + 1)
  
              if hasattr(self, keyname):
                  # If the key exists as instance's attribute already, it is because it's been
                  # declared with a default value, and the attribute now holds said default value.
                  default_value = getattr(self, keyname)
  
                  # If the default value is a mutable container, we cannot use it directly.
                  # Should we do so, the very same default value would be assigned to multiple
                  # instances/attributes instead of each instance having its own distinct container.
                  if isinstance(default_value, (list, dict)):
                      debug('detected mutable default')
                      default_value = copy.copy(default_value)
  
                  debug('default value', str(default_value))
                  debug('default value type', str(type(default_value)))
  
                  # try+except seems to work better than get(), especially when
                  # semantic of fmf.Tree.get() is slightly different than that
                  # of dict().get().
                  try:
                      value = key_source[source_keyname]
  
                  except KeyError:
                      value = default_value
  
                  debug('raw value', str(value))
                  debug('raw value type', str(type(value)))
  
              else:
                  value = key_source.get(source_keyname)
  
                  debug('raw value', str(value))
                  debug('raw value type', str(type(value)))
  
              normalize_callback = getattr(self, f'_normalize_{keyname}', None)
  
              if normalize_callback:
                  value = normalize_callback(value)
  
                  debug('normalized value', str(value))
                  debug('normalized value type', str(type(value)))
  
              debug('final value', str(value))
              debug('final value type', str(type(value)))
  
              # Set attribute by adding it to __dict__ directly. Messing with setattr()
              # might cause re-use of mutable values by other instances.
              self.__dict__[keyname] = value
  
              # Apparently pointless, but makes the debugging output more readable.
              # There may be plenty of tests and plans and keys, a bit of spacing
              # can't hurt.
              debug('')
  
          debug_intro('normalized fields')
          for k, v in self.__dict__.items():
              debug(f'{k}: {v} ({type(v)})')
  
          debug('')
  
      def __init__(self, **kwargs: Any) -> None:
          super().__init__(**kwargs)
  
  
  class LoadFmfKeysMixin(NormalizeKeysMixin):
      def __init__(
              self,
              *,
              node: fmf.Tree,
              logger: Common,
              **kwargs: Any) -> None:
          self._load_keys(node.get(), node.name, logger)
  
          kwargs.setdefault('logger', logger)
          super().__init__(node=node, **kwargs)
  
files: tmt/utils.py