class_docstr: Wrap ``VotingClassifier`` and ``VotingRegressor`` from scikit-learn.
  
  Note that this class does not inherit from the ``Learner`` class but rather
  uses different ``Learner`` instances underlyingly.
  
  Parameters
  ----------
  learner_names : List[str]
      List of the learner names that will participate in the voting process.
  voting : Optional[str], default="hard"
      One of "hard" or "soft". If "hard", the predicted class labels
      are used for majority rule voting. If "soft", the predicted class
      label is based on the argmax of the sums of the predicted
      probabilities from each of the underlying learnrs. This parameter
      is only relevant for classification.
  custom_learner_path : Optional[:class:`skll.types.PathOrStr`], default=None
      Path to a Python file containing the definitions of any custom
      learners. Any and all custom learners in ``estimator_names`` must
      be defined in this file. If the custom learner does not inherit
      from an already existing scikit-learn estimator, it must explicitly
      define an `_estimator_type` attribute indicating whether it's a
      "classifier" or a "regressor".
  feature_scaling : str, default="none"
      How to scale the features, if at all for each estimator. Options are
      -  "with_std": scale features using the standard deviation
      -  "with_mean": center features using the mean
      -  "both": do both scaling as well as centering
      -  "none": do neither scaling nor centering
  pos_label : Optional[:class:`skll.types.LabelType`], default=None
      A string denoting the label of the class to be
      treated as the positive class in a binary classification
      setting, for each estimator. If ``None``, the class represented
      by the label that appears second when sorted is chosen as the
      positive class. For example, if the two labels in data are "A"
      and "B" and ``pos_label`` is not specified, "B" will
      be chosen as the positive class.
  min_feature_count : int, default=1
      The minimum number of examples a feature must have a nonzero
      value in to be included, for each estimator.
  model_kwargs_list : Optional[List[Dict[str, Any]]], default=None
      A list of dictionaries of keyword arguments to pass to the
      initializer for each of the estimators. There's a one-to-one
      correspondence between the order of this list and the order
      of the ``learner_names`` list.
  sampler_list : Optional[List[str]], default=None
      The samplers to use for kernel approximation, if desired, for each
      estimator. Valid values are:
      -  "AdditiveChi2Sampler"
      -  "Nystroem"
      -  "RBFSampler"
      -  "SkewedChi2Sampler"
      There's a one-to-one correspondence between the order of this list
      and the order of the ``learner_names`` list.
  sampler_kwargs_list : Optional[List[Dict[str, Any]]], default=None
      A list of dictionaries of keyword arguments to pass to the
      initializer for the specified sampler, one per estimator.
      There's a one-to-one correspondence between the order of this
      list and the order of the ``learner_names`` list.
  logger : Optional[logging.Logger], default=None
      A logging object. If ``None`` is passed, get logger from ``__name__``.
docstr: Cross-validate the meta-estimator on the given examples.
  
  We follow essentially the same methodology as in
  ``Learner.cross_validate()`` - split the examples into
  training and testing folds, and then call ``self.train()``
  on the training folds and then ``self.evaluate()`` on the
  test fold. Note that this means that underlying estimators
  with different hyperparameters may be used for each fold, as is
  the case with ``Learner.cross_validate()``.
  
  Parameters
  ----------
  examples : :class:`skll.data.featureset.FeatureSet`
      The ``FeatureSet`` instance to cross-validate learner performance on.
  stratified : bool, default=True
      Should we stratify the folds to ensure an even
      distribution of labels for each fold?
  cv_folds : Union[int, :class:`skll.types.FoldMapping`], default=10
      The number of folds to use for cross-validation, or
      a mapping from example IDs to folds.
  cv_seed: int, default=123456789
      The value for seeding the random number generator
      used to create the random folds. Note that this
      seed is *only* used if either ``grid_search`` or
      ``shuffle`` are set to ``True``.
  grid_search : bool, default=True
      Should we do grid search when training each fold?
      Note: This will make this take *much* longer.
  grid_search_folds : Union[int, :class:`skll.types.FoldMapping`], default=5
      The number of folds to use when doing the grid search, or a mapping
      from example IDs to folds.
  grid_jobs : Optional[int], default=None
      The number of jobs to run in parallel when doing the grid search.
      If ``None`` or 0, the number of grid search folds will be used.
  grid_objective : Optional[str], default=None
      The name of the objective function to use when doing the grid search.
      Must be specified if ``grid_search`` is ``True``.
  output_metrics : Optional[List[str]], default=[]
      List of additional metric names to compute in addition to the metric
      used for grid search.
  prediction_prefix : Optional[str], default=None
      If saving the predictions, this is the prefix that will be used for
      the filename. It will be followed by ``"_predictions.tsv"``
  param_grid_list : Optional[List[Dict[str, Any]]], default=None
      The list of parameters grid to search through for grid
      search, one for each underlying learner. The order of
      the dictionaries should correspond to the order If ``None``,
      the default parameter grids will be used for the underlying
      estimators.
  shuffle : bool, default=False
      Shuffle examples before splitting into folds for CV.
  save_cv_folds : bool, default=True
       Whether to save the cv fold ids or not?
  save_cv_models : bool, default=False
      Whether to save the cv models or not?
  individual_predictions : bool, default=False
      Write out the cross-validated predictions from each underlying
      learner as well.
  use_custom_folds_for_grid_search : bool, default=True
      If ``cv_folds`` is a custom dictionary, but ``grid_search_folds``
      is not, perhaps due to user oversight, should the same custom
      dictionary automatically be used for the inner grid-search
      cross-validation?
  
  Returns
  -------
  :class:`skll.types.CrossValidateTaskResults`
     A 3-tuple containing the following:
  
      List[:class:`skll.types.EvaluateTaskResults`]: the confusion matrix, overall accuracy,
      per-label PRFs, model parameters, objective function score, and
      evaluation metrics (if any) for each fold.
  
      Optional[:class:`skll.types.FoldMapping`]: dictionary containing the test-fold number
      for each id if ``save_cv_folds`` is ``True``, otherwise ``None``.
  
      Optional[List[:class:`skll.learner.voting.VotingLearner`]]: list of voting
      learners, one for each fold if ``save_cv_models`` is ``True``,
      otherwise ``None``.
  
  Raises
  ------
  ValueError
      If classification labels are not properly encoded as strings.
  ValueError
      If ``grid_search`` is ``True`` but ``grid_objective`` is ``None``.
docstr_tok: ["Cross-validate","the","meta-estimator","on","the","given","examples",".","We","follow","essentially","the","same","methodology","as","in","``","Learner.cross_validate","(",")","``","-","split","the","examples","into","training","and","testing","folds",",","and","then","call","``","self.train","(",")","``","on","the","training","folds","and","then","``","self.evaluate","(",")","``","on","the","test","fold",".","Note","that","this","means","that","underlying","estimators","with","different","hyperparameters","may","be","used","for","each","fold",",","as","is","the","case","with","``","Learner.cross_validate","(",")","``",".","Parameters","--","--","--","--","--","examples",":",":","class",":","`","skll.data.featureset.FeatureSet","`","The","``","FeatureSet","``","instance","to","cross-validate","learner","performance","on",".","stratified",":","bool",",","default=True","Should","we","stratify","the","folds","to","ensure","an","even","distribution","of","labels","for","each","fold","?","cv_folds",":","Union","[","int",",",":","class",":","`","skll.types.FoldMapping","`","]",",","default=10","The","number","of","folds","to","use","for","cross-validation",",","or","a","mapping","from","example","IDs","to","folds",".","cv_seed",":","int",",","default=123456789","The","value","for","seeding","the","random","number","generator","used","to","create","the","random","folds",".","Note","that","this","seed","is","*","only","*","used","if","either","``","grid_search","``","or","``","shuffle","``","are","set","to","``","True","``",".","grid_search",":","bool",",","default=True","Should","we","do","grid","search","when","training","each","fold","?","Note",":","This","will","make","this","take","*","much","*","longer",".","grid_search_folds",":","Union","[","int",",",":","class",":","`","skll.types.FoldMapping","`","]",",","default=5","The","number","of","folds","to","use","when","doing","the","grid","search",",","or","a","mapping","from","example","IDs","to","folds",".","grid_jobs",":","Optional","[","int","]",",","default=None","The","number","of","jobs","to","run","in","parallel","when","doing","the","grid","search",".","If","``","None","``","or","0",",","the","number","of","grid","search","folds","will","be","used",".","grid_objective",":","Optional","[","str","]",",","default=None","The","name","of","the","objective","function","to","use","when","doing","the","grid","search",".","Must","be","specified","if","``","grid_search","``","is","``","True","``",".","output_metrics",":","Optional","[","List","[","str","]","]",",","default=","[","]","List","of","additional","metric","names","to","compute","in","addition","to","the","metric","used","for","grid","search",".","prediction_prefix",":","Optional","[","str","]",",","default=None","If","saving","the","predictions",",","this","is","the","prefix","that","will","be","used","for","the","filename",".","It","will","be","followed","by","``","``","_predictions.tsv","''","``","param_grid_list",":","Optional","[","List","[","Dict","[","str",",","Any","]","]","]",",","default=None","The","list","of","parameters","grid","to","search","through","for","grid","search",",","one","for","each","underlying","learner",".","The","order","of","the","dictionaries","should","correspond","to","the","order","If","``","None","``",",","the","default","parameter","grids","will","be","used","for","the","underlying","estimators",".","shuffle",":","bool",",","default=False","Shuffle","examples","before","splitting","into","folds","for","CV",".","save_cv_folds",":","bool",",","default=True","Whether","to","save","the","cv","fold","ids","or","not","?","save_cv_models",":","bool",",","default=False","Whether","to","save","the","cv","models","or","not","?","individual_predictions",":","bool",",","default=False","Write","out","the","cross-validated","predictions","from","each","underlying","learner","as","well",".","use_custom_folds_for_grid_search",":","bool",",","default=True","If","``","cv_folds","``","is","a","custom","dictionary",",","but","``","grid_search_folds","``","is","not",",","perhaps","due","to","user","oversight",",","should","the","same","custom","dictionary","automatically","be","used","for","the","inner","grid-search","cross-validation","?","Returns","--","--","--","-",":","class",":","`","skll.types.CrossValidateTaskResults","`","A","3-tuple","containing","the","following",":","List","[",":","class",":","`","skll.types.EvaluateTaskResults","`","]",":","the","confusion","matrix",",","overall","accuracy",",","per-label","PRFs",",","model","parameters",",","objective","function","score",",","and","evaluation","metrics","(","if","any",")","for","each","fold",".","Optional","[",":","class",":","`","skll.types.FoldMapping","`","]",":","dictionary","containing","the","test-fold","number","for","each","id","if","``","save_cv_folds","``","is","``","True","``",",","otherwise","``","None","``",".","Optional","[","List","[",":","class",":","`","skll.learner.voting.VotingLearner","`","]","]",":","list","of","voting","learners",",","one","for","each","fold","if","``","save_cv_models","``","is","``","True","``",",","otherwise","``","None","``",".","Raises","--","--","--","ValueError","If","classification","labels","are","not","properly","encoded","as","strings",".","ValueError","If","``","grid_search","``","is","``","True","``","but","``","grid_objective","``","is","``","None","``","."]
code: def cross_validate(
      self,
      examples: FeatureSet,
      stratified: bool = True,
      cv_folds: Union[int, FoldMapping] = 10,
      cv_seed: int = 123456789,
      grid_search: bool = True,
      grid_search_folds: Union[int, FoldMapping] = 5,
      grid_jobs: Optional[int] = None,
      grid_objective: Optional[str] = None,
      output_metrics: List[str] = [],
      prediction_prefix: Optional[str] = None,
      param_grid_list: Optional[List[Dict[str, Any]]] = None,
      shuffle: bool = False,
      save_cv_folds: bool = True,
      save_cv_models: bool = False,
      individual_predictions: bool = False,
      use_custom_folds_for_grid_search: bool = True,
  ) -> VotingCrossValidateTaskResults:
      """
      Cross-validate the meta-estimator on the given examples.
  
      We follow essentially the same methodology as in
      ``Learner.cross_validate()`` - split the examples into
      training and testing folds, and then call ``self.train()``
      on the training folds and then ``self.evaluate()`` on the
      test fold. Note that this means that underlying estimators
      with different hyperparameters may be used for each fold, as is
      the case with ``Learner.cross_validate()``.
  
      Parameters
      ----------
      examples : :class:`skll.data.featureset.FeatureSet`
          The ``FeatureSet`` instance to cross-validate learner performance on.
      stratified : bool, default=True
          Should we stratify the folds to ensure an even
          distribution of labels for each fold?
      cv_folds : Union[int, :class:`skll.types.FoldMapping`], default=10
          The number of folds to use for cross-validation, or
          a mapping from example IDs to folds.
      cv_seed: int, default=123456789
          The value for seeding the random number generator
          used to create the random folds. Note that this
          seed is *only* used if either ``grid_search`` or
          ``shuffle`` are set to ``True``.
      grid_search : bool, default=True
          Should we do grid search when training each fold?
          Note: This will make this take *much* longer.
      grid_search_folds : Union[int, :class:`skll.types.FoldMapping`], default=5
          The number of folds to use when doing the grid search, or a mapping
          from example IDs to folds.
      grid_jobs : Optional[int], default=None
          The number of jobs to run in parallel when doing the grid search.
          If ``None`` or 0, the number of grid search folds will be used.
      grid_objective : Optional[str], default=None
          The name of the objective function to use when doing the grid search.
          Must be specified if ``grid_search`` is ``True``.
      output_metrics : Optional[List[str]], default=[]
          List of additional metric names to compute in addition to the metric
          used for grid search.
      prediction_prefix : Optional[str], default=None
          If saving the predictions, this is the prefix that will be used for
          the filename. It will be followed by ``"_predictions.tsv"``
      param_grid_list : Optional[List[Dict[str, Any]]], default=None
          The list of parameters grid to search through for grid
          search, one for each underlying learner. The order of
          the dictionaries should correspond to the order If ``None``,
          the default parameter grids will be used for the underlying
          estimators.
      shuffle : bool, default=False
          Shuffle examples before splitting into folds for CV.
      save_cv_folds : bool, default=True
           Whether to save the cv fold ids or not?
      save_cv_models : bool, default=False
          Whether to save the cv models or not?
      individual_predictions : bool, default=False
          Write out the cross-validated predictions from each underlying
          learner as well.
      use_custom_folds_for_grid_search : bool, default=True
          If ``cv_folds`` is a custom dictionary, but ``grid_search_folds``
          is not, perhaps due to user oversight, should the same custom
          dictionary automatically be used for the inner grid-search
          cross-validation?
  
      Returns
      -------
      :class:`skll.types.CrossValidateTaskResults`
         A 3-tuple containing the following:
  
          List[:class:`skll.types.EvaluateTaskResults`]: the confusion matrix, overall accuracy,
          per-label PRFs, model parameters, objective function score, and
          evaluation metrics (if any) for each fold.
  
          Optional[:class:`skll.types.FoldMapping`]: dictionary containing the test-fold number
          for each id if ``save_cv_folds`` is ``True``, otherwise ``None``.
  
          Optional[List[:class:`skll.learner.voting.VotingLearner`]]: list of voting
          learners, one for each fold if ``save_cv_models`` is ``True``,
          otherwise ``None``.
  
      Raises
      ------
      ValueError
          If classification labels are not properly encoded as strings.
      ValueError
          If ``grid_search`` is ``True`` but ``grid_objective`` is ``None``.
      """
      # Seed the random number generator so that randomized algorithms are
      # replicable.
      random_state = np.random.RandomState(cv_seed)
  
      # We need to check whether the labels in the featureset are labels
      # or continuous values. If it's the latter, we need to raise an
      # an exception since the stratified splitting in sklearn does not
      # work with continuous labels. Note that although using random folds
      # _will_ work, we want to raise an error in general since it's better
      # to encode the labels as strings anyway for classification problems.
      if self.learner_type == "classifier" and type_of_target(
          examples.labels
      ) not in [
          "binary",
          "multiclass",
      ]:
          raise ValueError(
              "Floating point labels must be encoded as strings for cross-validation."
          )
  
      # check that we have an objective since grid search is on by default
      # Note that `train()` would raise this error anyway later but it's
      # better to raise this early on so rather than after a whole bunch of
      # stuff has happened
      if grid_search and not grid_objective:
          raise ValueError(
              "Grid search is on by default. You must "
              "either specify a grid objective or turn off "
              "grid search."
          )
  
      # Shuffle so that the folds are random for the inner grid search CV.
      # If grid search is True but shuffle isn't, shuffle anyway.
      # You can't shuffle a scipy sparse matrix in place, so unfortunately
      # we make a copy of everything (and then get rid of the old version)
      if grid_search or shuffle:
          if grid_search and not shuffle:
              self.logger.warning(
                  "Training data will be shuffled to randomize "
                  "grid search folds.  Shuffling may yield "
                  "different results compared to scikit-learn."
              )
          ids, labels, features = sk_shuffle(
              examples.ids,
              examples.labels,
              examples.features,
              random_state=random_state,
          )
          examples = FeatureSet(
              examples.name,
              ids,
              labels=labels,
              features=features,
              vectorizer=examples.vectorizer,
          )
  
      # Call some setup code which will properly initialize the underlying
      # learners before they are eventually trained
      self._setup_underlying_learners(examples)
  
      # Set up the cross-validation iterator.
      kfold, cv_groups = setup_cv_fold_iterator(
          cv_folds,
          examples,
          self.learner_type,
          stratified=stratified,
          logger=self.logger,
      )
  
      # When using custom CV folds (a dictionary), if we are planning to do
      # grid search, set the grid search folds to be the same as the custom
      # cv folds unless a flag is set that explicitly tells us not to.
      # Note that this should only happen when we are using the API; otherwise
      # the configparser should take care of this even before this method is called
      if isinstance(cv_folds, dict):
          if (
              grid_search
              and use_custom_folds_for_grid_search
              and grid_search_folds != cv_folds
          ):
              self.logger.warning(
                  "The specified custom folds will be used for "
                  "the inner grid search."
              )
              grid_search_folds = cv_folds
  
      # handle each fold separately & accumulate the predictions and results
      results = []
      append_predictions = False
      saved_models: List["VotingLearner"] = []
      saved_skll_fold_ids: FoldMapping = {}
      if examples.features is not None and examples.labels is not None:
          for fold_num, (train_indices, test_indices) in enumerate(
              kfold.split(examples.features, examples.labels, cv_groups)
          ):
              # Train model
              self._model = (
                  None  # prevent feature vectorizer from being reset.
              )
              train_set = FeatureSet(
                  examples.name,
                  examples.ids[train_indices],
                  labels=examples.labels[train_indices],
                  features=examples.features[train_indices],
                  vectorizer=examples.vectorizer,
              )
  
              self.train(
                  train_set,
                  param_grid_list=param_grid_list,
                  grid_search_folds=grid_search_folds,
                  grid_search=grid_search,
                  grid_objective=grid_objective,
                  grid_jobs=grid_jobs,
                  shuffle=grid_search,
              )
  
              if save_cv_models:
                  saved_models.append(copy.deepcopy(self))
  
              # evaluate the voting meta-estimator on the test fold
              test_tuple = FeatureSet(
                  examples.name,
                  examples.ids[test_indices],
                  labels=examples.labels[test_indices],
                  features=examples.features[test_indices],
                  vectorizer=examples.vectorizer,
              )
  
              # save the results
              results.append(
                  self.evaluate(
                      test_tuple,
                      prediction_prefix=prediction_prefix,
                      append=append_predictions,
                      grid_objective=grid_objective,
                      output_metrics=output_metrics,
                      individual_predictions=individual_predictions,
                  )
              )
              append_predictions = True
  
              # save the fold number for each test ID if we were asked to
              if save_cv_folds:
                  for index in test_indices:
                      saved_skll_fold_ids[examples.ids[index]] = str(
                          fold_num
                      )
  
      # return list of results/outputs for all folds
      models = saved_models if save_cv_models else None
      skll_fold_ids = saved_skll_fold_ids if save_cv_folds else None
      return (results, skll_fold_ids, models)
  
imports: import copy
  import logging
  from importlib import import_module
  from itertools import zip_longest
  from multiprocessing import cpu_count
  from typing import Any, Dict, List, Optional, Tuple, Union
  import joblib
  import numpy
  from sklearn.ensemble import VotingClassifier, VotingRegressor
  from sklearn.utils import shuffle
  from sklearn.utils.multiclass import type_of_target
  from skll.data import FeatureSet
  from skll.data.dict_vectorizer import DictVectorizer
  from skll.learner import Learner
  from skll.types import EvaluateTaskResults, FoldMapping, LabelType, LearningCurveSizes, PathOrStr, VotingCrossValidateTaskResults
  from skll.utils.constants import MAX_CONCURRENT_PROCESSES
  from .utils import _load_learner_from_disk, _save_learner_to_disk, add_unseen_labels, compute_evaluation_metrics, get_acceptable_classification_metrics, get_acceptable_regression_metrics, get_predictions, setup_cv_fold_iterator, setup_cv_split_iterator, train_and_score, write_predictions
label_desc: Use image node_id 12 for calling the VotingLearner obj's underlying member method code with example usage: obj.cross_validate(examples, stratified, cv_folds, cv_seed, grid_search, grid_search_folds, grid_jobs, grid_objective, output_metrics, prediction_prefix, param_grid_list, shuffle, save_cv_folds, save_cv_models, individual_predictions, use_custom_folds_for_grid_search) and returns: results,  skll_fold_ids,  models