class_docstr: A restartable and failure-tolerant method for combining one or more GVCFs and Variant Datasets.
  
  Examples
  --------
  
  A Variant Dataset comprises one or more sequences. A new Variant Dataset is constructed from
  GVCF files and/or extant Variant Datasets. For example, the following produces a new Variant
  Dataset from four GVCF files containing whole genome sequences ::
  
      gvcfs = [
          'gs://bucket/sample_10123.g.vcf.bgz',
          'gs://bucket/sample_10124.g.vcf.bgz',
          'gs://bucket/sample_10125.g.vcf.bgz',
          'gs://bucket/sample_10126.g.vcf.bgz',
      ]
  
      combiner = hl.vds.new_combiner(
          output_path='gs://bucket/dataset.vds',
          temp_path='gs://1-day-temp-bucket/',
          gvcf_paths=gvcfs,
          use_genome_default_intervals=True,
      )
  
      combiner.run()
  
      vds = hl.read_vds('gs://bucket/dataset.vds')
  
  The following combines four new samples from GVCFs with multiple extant Variant Datasets::
  
      gvcfs = [
          'gs://bucket/sample_10123.g.vcf.bgz',
          'gs://bucket/sample_10124.g.vcf.bgz',
          'gs://bucket/sample_10125.g.vcf.bgz',
          'gs://bucket/sample_10126.g.vcf.bgz',
      ]
  
      vdses = [
          'gs://bucket/hgdp.vds',
          'gs://bucket/1kg.vds'
      ]
  
      combiner = hl.vds.new_combiner(
          output_path='gs://bucket/dataset.vds',
          temp_path='gs://1-day-temp-bucket/',
          save_path='gs://1-day-temp-bucket/',
          gvcf_paths=gvcfs,
          vds_paths=vdses,
          use_genome_default_intervals=True,
      )
  
      combiner.run()
  
      vds = hl.read_vds('gs://bucket/dataset.vds')
  
  
  The speed of the Variant Dataset Combiner critically depends on data partitioning. Although the
  partitioning is fully customizable, two high-quality partitioning strategies are available by
  default, one for exomes and one for genomes. These partitioning strategies can be enabled,
  respectively, with the parameters: ``use_exome_default_intervals=True`` and
  ``use_genome_default_intervals=True``.
  
  The combiner serializes itself to `save_path` so that it can be restarted after failure.
  
  Parameters
  ----------
  save_path : :class:`str`
      The location to store this VariantDatasetCombiner plan. A failed execution can be restarted
      using this plan.
  output_path : :class:`str`
      The location to store the new VariantDataset.
  temp_path : :class:`str`
      The location to store temporary intermediates. We recommend using a bucket with an automatic
      deletion or lifecycle policy.
  reference_genome : :class:`.ReferenceGenome`
      The reference genome to which all inputs (GVCFs and Variant Datasets) are aligned.
  branch_factor : :class:`int`
      The number of Variant Datasets to combine at once.
  target_records : :class:`int`
      The target number of variants per partition.
  gvcf_batch_size : :class:`int`
      The number of GVCFs to combine into a Variant Dataset at once.
  contig_recoding : :class:`dict` mapping :class:`str` to :class:`str` or :obj:`None`
      This mapping is applied to GVCF contigs before importing them into Hail. This is used to
      handle GVCFs containing invalid contig names. For example, GRCh38 GVCFs which contain the
      contig "1" instead of the correct "chr1".
  vdses : :class:`list` of :class:`.VDSMetadata`
      A list of Variant Datasets to combine. Each dataset is identified by a
      :class:`.VDSMetadata`, which is a pair of a path and the number of samples in said Variant
      Dataset.
  gvcfs : :class:`list` of :class:`.str`
      A list of paths of GVCF files to combine.
  gvcf_sample_names : :class:`list` of :class:`str` or :obj:`None`
      List of names to use for the samples from the GVCF files. Must be the same length as
      `gvcfs`. Must be specified if `gvcf_external_header` is specified.
  gvcf_external_header : :class:`str` or :obj:`None`
      A path to a file containing a VCF header which is applied to all GVCFs. Must be specified if
      `gvcf_sample_names` is specified.
  gvcf_import_intervals : :class:`list` of :class:`.Interval`
      A list of intervals defining how to partition the GVCF files. The same partitioning is used
      for all GVCF files. Finer partitioning yields more parallelism but less work per task.
  gvcf_info_to_keep : :class:`list` of :class:`str` or :obj:`None`
      GVCF ``INFO`` fields to keep in the ``gvcf_info`` entry field. By default, all fields
      except ``END`` and ``DP`` are kept.
  gvcf_reference_entry_fields_to_keep : :class:`list` of :class:`str` or :obj:`None`
      Genotype fields to keep in the reference table. If empty, the first 10,000 reference block
      rows of ``mt`` will be sampled and all fields found to be defined other than ``GT``, ``AD``,
      and ``PL`` will be entry fields in the resulting reference matrix in the dataset.
class_docstr_tok: ["A","restartable","and","failure-tolerant","method","for","combining","one","or","more","GVCFs","and","Variant","Datasets",".","Examples","--","--","--","--","A","Variant","Dataset","comprises","one","or","more","sequences",".","A","new","Variant","Dataset","is","constructed","from","GVCF","files","and\/or","extant","Variant","Datasets",".","For","example",",","the","following","produces","a","new","Variant","Dataset","from","four","GVCF","files","containing","whole","genome","sequences",":",":","gvcfs","=","[","'gs",":","\/\/bucket\/sample_10123.g.vcf.bgz","'",",","'gs",":","\/\/bucket\/sample_10124.g.vcf.bgz","'",",","'gs",":","\/\/bucket\/sample_10125.g.vcf.bgz","'",",","'gs",":","\/\/bucket\/sample_10126.g.vcf.bgz","'",",","]","combiner","=","hl.vds.new_combiner","(","output_path='gs",":","\/\/bucket\/dataset.vds","'",",","temp_path='gs",":","\/\/1-day-temp-bucket\/","'",",","gvcf_paths=gvcfs",",","use_genome_default_intervals=True",",",")","combiner.run","(",")","vds","=","hl.read_vds","(","'gs",":","\/\/bucket\/dataset.vds","'",")","The","following","combines","four","new","samples","from","GVCFs","with","multiple","extant","Variant","Datasets",":",":","gvcfs","=","[","'gs",":","\/\/bucket\/sample_10123.g.vcf.bgz","'",",","'gs",":","\/\/bucket\/sample_10124.g.vcf.bgz","'",",","'gs",":","\/\/bucket\/sample_10125.g.vcf.bgz","'",",","'gs",":","\/\/bucket\/sample_10126.g.vcf.bgz","'",",","]","vdses","=","[","'gs",":","\/\/bucket\/hgdp.vds","'",",","'gs",":","\/\/bucket\/1kg.vds'","]","combiner","=","hl.vds.new_combiner","(","output_path='gs",":","\/\/bucket\/dataset.vds","'",",","temp_path='gs",":","\/\/1-day-temp-bucket\/","'",",","save_path='gs",":","\/\/1-day-temp-bucket\/","'",",","gvcf_paths=gvcfs",",","vds_paths=vdses",",","use_genome_default_intervals=True",",",")","combiner.run","(",")","vds","=","hl.read_vds","(","'gs",":","\/\/bucket\/dataset.vds","'",")","The","speed","of","the","Variant","Dataset","Combiner","critically","depends","on","data","partitioning",".","Although","the","partitioning","is","fully","customizable",",","two","high-quality","partitioning","strategies","are","available","by","default",",","one","for","exomes","and","one","for","genomes",".","These","partitioning","strategies","can","be","enabled",",","respectively",",","with","the","parameters",":","``","use_exome_default_intervals=True","``","and","``","use_genome_default_intervals=True","``",".","The","combiner","serializes","itself","to","`","save_path","`","so","that","it","can","be","restarted","after","failure",".","Parameters","--","--","--","--","--","save_path",":",":","class",":","`","str","`","The","location","to","store","this","VariantDatasetCombiner","plan",".","A","failed","execution","can","be","restarted","using","this","plan",".","output_path",":",":","class",":","`","str","`","The","location","to","store","the","new","VariantDataset",".","temp_path",":",":","class",":","`","str","`","The","location","to","store","temporary","intermediates",".","We","recommend","using","a","bucket","with","an","automatic","deletion","or","lifecycle","policy",".","reference_genome",":",":","class",":","`",".ReferenceGenome","`","The","reference","genome","to","which","all","inputs","(","GVCFs","and","Variant","Datasets",")","are","aligned",".","branch_factor",":",":","class",":","`","int","`","The","number","of","Variant","Datasets","to","combine","at","once",".","target_records",":",":","class",":","`","int","`","The","target","number","of","variants","per","partition",".","gvcf_batch_size",":",":","class",":","`","int","`","The","number","of","GVCFs","to","combine","into","a","Variant","Dataset","at","once",".","contig_recoding",":",":","class",":","`","dict","`","mapping",":","class",":","`","str","`","to",":","class",":","`","str","`","or",":","obj",":","`","None","`","This","mapping","is","applied","to","GVCF","contigs","before","importing","them","into","Hail",".","This","is","used","to","handle","GVCFs","containing","invalid","contig","names",".","For","example",",","GRCh38","GVCFs","which","contain","the","contig","``","1","''","instead","of","the","correct","``","chr1","''",".","vdses",":",":","class",":","`","list","`","of",":","class",":","`",".VDSMetadata","`","A","list","of","Variant","Datasets","to","combine",".","Each","dataset","is","identified","by","a",":","class",":","`",".VDSMetadata","`",",","which","is","a","pair","of","a","path","and","the","number","of","samples","in","said","Variant","Dataset",".","gvcfs",":",":","class",":","`","list","`","of",":","class",":","`",".str","`","A","list","of","paths","of","GVCF","files","to","combine",".","gvcf_sample_names",":",":","class",":","`","list","`","of",":","class",":","`","str","`","or",":","obj",":","`","None","`","List","of","names","to","use","for","the","samples","from","the","GVCF","files",".","Must","be","the","same","length","as","`","gvcfs","`",".","Must","be","specified","if","`","gvcf_external_header","`","is","specified",".","gvcf_external_header",":",":","class",":","`","str","`","or",":","obj",":","`","None","`","A","path","to","a","file","containing","a","VCF","header","which","is","applied","to","all","GVCFs",".","Must","be","specified","if","`","gvcf_sample_names","`","is","specified",".","gvcf_import_intervals",":",":","class",":","`","list","`","of",":","class",":","`",".Interval","`","A","list","of","intervals","defining","how","to","partition","the","GVCF","files",".","The","same","partitioning","is","used","for","all","GVCF","files",".","Finer","partitioning","yields","more","parallelism","but","less","work","per","task",".","gvcf_info_to_keep",":",":","class",":","`","list","`","of",":","class",":","`","str","`","or",":","obj",":","`","None","`","GVCF","``","INFO","``","fields","to","keep","in","the","``","gvcf_info","``","entry","field",".","By","default",",","all","fields","except","``","END","``","and","``","DP","``","are","kept",".","gvcf_reference_entry_fields_to_keep",":",":","class",":","`","list","`","of",":","class",":","`","str","`","or",":","obj",":","`","None","`","Genotype","fields","to","keep","in","the","reference","table",".","If","empty",",","the","first","10,000","reference","block","rows","of","``","mt","``","will","be","sampled","and","all","fields","found","to","be","defined","other","than","``","GT","``",",","``","AD","``",",","and","``","PL","``","will","be","entry","fields","in","the","resulting","reference","matrix","in","the","dataset","."]
code: def _step_vdses(self):
      current_bin = original_bin = min(self._vdses)
      files_to_merge = self._vdses[current_bin][: self._branch_factor]
      if len(files_to_merge) == len(self._vdses[current_bin]):
          del self._vdses[current_bin]
      else:
          self._vdses[current_bin] = self._vdses[current_bin][
              self._branch_factor :
          ]
  
      remaining = self._branch_factor - len(files_to_merge)
      while self._num_vdses > 0 and remaining > 0:
          current_bin = min(self._vdses)
          extra = self._vdses[current_bin][-remaining:]
          if len(extra) == len(self._vdses[current_bin]):
              del self._vdses[current_bin]
          else:
              self._vdses[current_bin] = self._vdses[current_bin][
                  :-remaining
              ]
          files_to_merge = extra + files_to_merge
          remaining = self._branch_factor - len(files_to_merge)
  
      new_n_samples = sum(f.n_samples for f in files_to_merge)
      info(
          f"VDS Combine (job {self._job_id}): merging {len(files_to_merge)} datasets with {new_n_samples} samples"
      )
  
      temp_path = self._temp_out_path(f"vds-combine_job{self._job_id}")
      largest_vds = max(files_to_merge, key=lambda vds: vds.n_samples)
      vds = hl.vds.read_vds(
          largest_vds.path,
          _assert_reference_type=self._dataset_type.reference_type,
          _assert_variant_type=self._dataset_type.variant_type,
          _warn_no_ref_block_max_length=False,
      )
  
      interval_bin = floor(log(new_n_samples, self._branch_factor))
      intervals = self.__intervals_cache.get(interval_bin)
  
      if intervals is None:
          # we use the reference data since it generally has more rows than the variant data
          intervals, _ = calculate_new_intervals(
              vds.reference_data,
              self._target_records,
              os.path.join(temp_path, "interval_checkpoint.ht"),
          )
          self.__intervals_cache[interval_bin] = intervals
  
      paths = [f.path for f in files_to_merge]
      vdss = self._read_variant_datasets(paths, intervals)
      combined = combine_variant_datasets(vdss)
  
      if self.finished:
          self._write_final(combined)
          return
  
      new_path = os.path.join(temp_path, "dataset.vds")
      combined.write(
          new_path, overwrite=True, _codec_spec=FAST_CODEC_SPEC
      )
      new_bin = floor(log(new_n_samples, self._branch_factor))
      # this ensures that we don't somehow stick a vds at the end of
      # the same bin, ending up with a weird ordering issue
      if new_bin <= original_bin:
          new_bin = original_bin + 1
      self._vdses[new_bin].append(
          VDSMetadata(path=new_path, n_samples=new_n_samples)
      )
  
imports: import collections
  import hashlib
  import json
  import os
  import sys
  import uuid
  from itertools import chain
  from math import floor, log
  from typing import Collection, Dict, List, NamedTuple, Optional, Union
  import hail
  from hail.expr import HailType, tmatrix
  from hail.utils import FatalError, Interval
  from hail.utils.java import info, warning
  from .combine import combine_variant_datasets, transform_gvcf, defined_entry_fields, make_variant_stream, make_reference_stream, combine_r, calculate_even_genome_partitioning, calculate_new_intervals, combine
  from ..variant_dataset import VariantDataset
label_desc: Use image node_id 14 for calling the VariantDatasetCombiner obj's underlying member method code with example usage: obj._step_vdses() without return types