class_docstr: Wrapper class that wraps HuggingFace models as DeepChem models
  
  The class provides a wrapper for wrapping models from HuggingFace
  ecosystem in DeepChem and training it via DeepChem's api. The reason
  for this might be that you might want to do an apples-to-apples comparison
  between HuggingFace from the transformers library and DeepChem library.
  
  The `HuggingFaceModel` has a Has-A relationship by wrapping models from
  `transformers` library. Once a model is wrapped, DeepChem's API are used
  for training, prediction, evaluation and other downstream tasks.
  
  A `HuggingFaceModel` wrapper also has a `tokenizer` which tokenizes raw
  SMILES strings into tokens to be used by downstream models.  The SMILES
  strings are generally stored in the `X` attribute of deepchem.data.Dataset object'.
  This differs from the DeepChem standard workflow as tokenization is done
  on the fly here. The approach allows us to leverage `transformers` library's fast
  tokenization algorithms and other utilities like data collation, random masking of tokens
  for masked language model training etc.
  
  
  Parameters
  ----------
  model: transformers.modeling_utils.PreTrainedModel
      The HuggingFace model to wrap.
  task: str, (optional, default None)
      The task defines the type of learning task in the model. The supported tasks are
       - `mlm` - masked language modeling commonly used in pretraining
       - `mtr` - multitask regression - a task used for both pretraining base models and finetuning
       - `regression` - use it for regression tasks, like property prediction
       - `classification` - use it for classification tasks
      When the task is not specified or None, the wrapper returns raw output of the HuggingFaceModel.
      In cases where the HuggingFaceModel is a model without a task specific head, this output will be
      the last hidden states.
  tokenizer: transformers.tokenization_utils.PreTrainedTokenizer
      Tokenizer
  
  Example
  -------
  >>> import os
  >>> import tempfile
  >>> tempdir = tempfile.mkdtemp()
  
  >>> # preparing dataset
  >>> smiles = ['CN(c1ccccc1)c1ccccc1C(=O)NCC1(O)CCOCC1', 'CC[NH+](CC)C1CCC([NH2+]C2CC2)(C(=O)[O-])C1', \
  ...     'COCC(CNC(=O)c1ccc2c(c1)NC(=O)C2)OC', 'OCCn1cc(CNc2cccc3c2CCCC3)nn1', \
  ...     'CCCCCCc1ccc(C#Cc2ccc(C#CC3=CC=C(CCC)CC3)c(C3CCCCC3)c2)c(F)c1', 'nO=C(NCc1ccc(F)cc1)N1CC=C(c2c[nH]c3ccccc23)CC1']
  >>> filepath = os.path.join(tempdir, 'smiles.txt')
  >>> f = open(filepath, 'w')
  >>> f.write('\n'.join(smiles))
  253
  >>> f.close()
  
  >>> # preparing tokenizer
  >>> from tokenizers import ByteLevelBPETokenizer
  >>> from transformers.models.roberta import RobertaTokenizerFast
  >>> tokenizer = ByteLevelBPETokenizer()
  >>> tokenizer.train(files=filepath, vocab_size=1_000, min_frequency=2, special_tokens=["<s>", "<pad>", "</s>", "<unk>", "<mask>"])
  >>> tokenizer_path = os.path.join(tempdir, 'tokenizer')
  >>> os.makedirs(tokenizer_path)
  >>> result = tokenizer.save_model(tokenizer_path)
  >>> tokenizer = RobertaTokenizerFast.from_pretrained(tokenizer_path)
  
  >>> # preparing dataset
  >>> import pandas as pd
  >>> import deepchem as dc
  >>> smiles = ["CCN(CCSC)C(=O)N[C@@](C)(CC)C(F)(F)F","CC1(C)CN(C(=O)Nc2cc3ccccc3nn2)C[C@@]2(CCOC2)O1"]
  >>> labels = [3.112,2.432]
  >>> df = pd.DataFrame(list(zip(smiles, labels)), columns=["smiles", "task1"])
  >>> with dc.utils.UniversalNamedTemporaryFile(mode='w') as tmpfile:
  ...     df.to_csv(tmpfile.name)
  ...     loader = dc.data.CSVLoader(["task1"], feature_field="smiles", featurizer=dc.feat.DummyFeaturizer())
  ...     dataset = loader.create_dataset(tmpfile.name)
  
  >>> # pretraining
  >>> from deepchem.models.torch_models.hf_models import HuggingFaceModel
  >>> from transformers.models.roberta import RobertaForMaskedLM, RobertaModel, RobertaConfig
  >>> config = RobertaConfig(vocab_size=tokenizer.vocab_size)
  >>> model = RobertaForMaskedLM(config)
  >>> hf_model = HuggingFaceModel(model=model, tokenizer=tokenizer, task='mlm', model_dir='model-dir')
  >>> training_loss = hf_model.fit(dataset, nb_epoch=1)
  
  >>> # finetuning a regression model
  >>> from transformers.models.roberta import RobertaForSequenceClassification
  >>> config = RobertaConfig(vocab_size=tokenizer.vocab_size, problem_type='regression', num_labels=1)
  >>> model = RobertaForSequenceClassification(config)
  >>> hf_model = HuggingFaceModel(model=model, tokenizer=tokenizer, task='regression', model_dir='model-dir')
  >>> hf_model.load_from_pretrained()
  >>> training_loss = hf_model.fit(dataset, nb_epoch=1)
  >>> prediction = hf_model.predict(dataset)  # prediction
  >>> eval_results = hf_model.evaluate(dataset, metrics=dc.metrics.Metric(dc.metrics.mae_score))
  
  >>> # finetune a classification model
  >>> # making dataset suitable for classification
  >>> import numpy as np
  >>> y = np.random.choice([0, 1], size=dataset.y.shape)
  >>> dataset = dc.data.NumpyDataset(X=dataset.X, y=y, w=dataset.w, ids=dataset.ids)
  
  >>> from transformers import RobertaForSequenceClassification
  >>> config = RobertaConfig(vocab_size=tokenizer.vocab_size)
  >>> model = RobertaForSequenceClassification(config)
  >>> hf_model = HuggingFaceModel(model=model, task='classification', tokenizer=tokenizer)
  >>> training_loss = hf_model.fit(dataset, nb_epoch=1)
  >>> predictions = hf_model.predict(dataset)
  >>> eval_result = hf_model.evaluate(dataset, metrics=dc.metrics.Metric(dc.metrics.f1_score))
class_docstr_tok: ["Wrapper","class","that","wraps","HuggingFace","models","as","DeepChem","models","The","class","provides","a","wrapper","for","wrapping","models","from","HuggingFace","ecosystem","in","DeepChem","and","training","it","via","DeepChem","'s","api",".","The","reason","for","this","might","be","that","you","might","want","to","do","an","apples-to-apples","comparison","between","HuggingFace","from","the","transformers","library","and","DeepChem","library",".","The","`","HuggingFaceModel","`","has","a","Has-A","relationship","by","wrapping","models","from","`","transformers","`","library",".","Once","a","model","is","wrapped",",","DeepChem","'s","API","are","used","for","training",",","prediction",",","evaluation","and","other","downstream","tasks",".","A","`","HuggingFaceModel","`","wrapper","also","has","a","`","tokenizer","`","which","tokenizes","raw","SMILES","strings","into","tokens","to","be","used","by","downstream","models",".","The","SMILES","strings","are","generally","stored","in","the","`","X","`","attribute","of","deepchem.data.Dataset","object","'",".","This","differs","from","the","DeepChem","standard","workflow","as","tokenization","is","done","on","the","fly","here",".","The","approach","allows","us","to","leverage","`","transformers","`","library","'s","fast","tokenization","algorithms","and","other","utilities","like","data","collation",",","random","masking","of","tokens","for","masked","language","model","training","etc",".","Parameters","--","--","--","--","--","model",":","transformers.modeling_utils.PreTrainedModel","The","HuggingFace","model","to","wrap",".","task",":","str",",","(","optional",",","default","None",")","The","task","defines","the","type","of","learning","task","in","the","model",".","The","supported","tasks","are","-","`","mlm","`","-","masked","language","modeling","commonly","used","in","pretraining","-","`","mtr","`","-","multitask","regression","-","a","task","used","for","both","pretraining","base","models","and","finetuning","-","`","regression","`","-","use","it","for","regression","tasks",",","like","property","prediction","-","`","classification","`","-","use","it","for","classification","tasks","When","the","task","is","not","specified","or","None",",","the","wrapper","returns","raw","output","of","the","HuggingFaceModel",".","In","cases","where","the","HuggingFaceModel","is","a","model","without","a","task","specific","head",",","this","output","will","be","the","last","hidden","states",".","tokenizer",":","transformers.tokenization_utils.PreTrainedTokenizer","Tokenizer","Example","--","--","--","-",">",">",">","import","os",">",">",">","import","tempfile",">",">",">","tempdir","=","tempfile.mkdtemp","(",")",">",">",">","#","preparing","dataset",">",">",">","smiles","=","[","'CN","(","c1ccccc1",")","c1ccccc1C","(","=O",")","NCC1","(","O",")","CCOCC1","'",",","'CC","[","NH+","]","(","CC",")","C1CCC","(","[","NH2+","]","C2CC2",")","(","C","(","=O",")","[","O-","]",")","C1","'",",","\\","...","'COCC","(","CNC","(","=O",")","c1ccc2c","(","c1",")","NC","(","=O",")","C2",")","OC","'",",","'OCCn1cc","(","CNc2cccc3c2CCCC3",")","nn1","'",",","\\","...","'CCCCCCc1ccc","(","C","#","Cc2ccc","(","C","#","CC3=CC=C","(","CCC",")","CC3",")","c","(","C3CCCCC3",")","c2",")","c","(","F",")","c1","'",",","'nO=C","(","NCc1ccc","(","F",")","cc1",")","N1CC=C","(","c2c","[","nH","]","c3ccccc23",")","CC1","'","]",">",">",">","filepath","=","os.path.join","(","tempdir",",","'smiles.txt","'",")",">",">",">","f","=","open","(","filepath",",","'","w","'",")",">",">",">","f.write","(","'\\n'.join","(","smiles",")",")","253",">",">",">","f.close","(",")",">",">",">","#","preparing","tokenizer",">",">",">","from","tokenizers","import","ByteLevelBPETokenizer",">",">",">","from","transformers.models.roberta","import","RobertaTokenizerFast",">",">",">","tokenizer","=","ByteLevelBPETokenizer","(",")",">",">",">","tokenizer.train","(","files=filepath",",","vocab_size=1_000",",","min_frequency=2",",","special_tokens=","[","``","<","s",">","''",",","``","<","pad",">","''",",","``","<","\/s",">","''",",","``","<","unk",">","''",",","``","<","mask",">","''","]",")",">",">",">","tokenizer_path","=","os.path.join","(","tempdir",",","'tokenizer","'",")",">",">",">","os.makedirs","(","tokenizer_path",")",">",">",">","result","=","tokenizer.save_model","(","tokenizer_path",")",">",">",">","tokenizer","=","RobertaTokenizerFast.from_pretrained","(","tokenizer_path",")",">",">",">","#","preparing","dataset",">",">",">","import","pandas","as","pd",">",">",">","import","deepchem","as","dc",">",">",">","smiles","=","[","``","CCN","(","CCSC",")","C","(","=O",")","N","[","C","@","@","]","(","C",")","(","CC",")","C","(","F",")","(","F",")","F","''",",","''","CC1","(","C",")","CN","(","C","(","=O",")","Nc2cc3ccccc3nn2",")","C","[","C","@","@","]","2","(","CCOC2",")","O1","''","]",">",">",">","labels","=","[","3.112,2.432","]",">",">",">","df","=","pd.DataFrame","(","list","(","zip","(","smiles",",","labels",")",")",",","columns=","[","``","smiles","''",",","``","task1","''","]",")",">",">",">","with","dc.utils.UniversalNamedTemporaryFile","(","mode=","'","w","'",")","as","tmpfile",":","...","df.to_csv","(","tmpfile.name",")","...","loader","=","dc.data.CSVLoader","(","[","``","task1","''","]",",","feature_field=","''","smiles","''",",","featurizer=dc.feat.DummyFeaturizer","(",")",")","...","dataset","=","loader.create_dataset","(","tmpfile.name",")",">",">",">","#","pretraining",">",">",">","from","deepchem.models.torch_models.hf_models","import","HuggingFaceModel",">",">",">","from","transformers.models.roberta","import","RobertaForMaskedLM",",","RobertaModel",",","RobertaConfig",">",">",">","config","=","RobertaConfig","(","vocab_size=tokenizer.vocab_size",")",">",">",">","model","=","RobertaForMaskedLM","(","config",")",">",">",">","hf_model","=","HuggingFaceModel","(","model=model",",","tokenizer=tokenizer",",","task='mlm","'",",","model_dir='model-dir","'",")",">",">",">","training_loss","=","hf_model.fit","(","dataset",",","nb_epoch=1",")",">",">",">","#","finetuning","a","regression","model",">",">",">","from","transformers.models.roberta","import","RobertaForSequenceClassification",">",">",">","config","=","RobertaConfig","(","vocab_size=tokenizer.vocab_size",",","problem_type='regression","'",",","num_labels=1",")",">",">",">","model","=","RobertaForSequenceClassification","(","config",")",">",">",">","hf_model","=","HuggingFaceModel","(","model=model",",","tokenizer=tokenizer",",","task='regression","'",",","model_dir='model-dir","'",")",">",">",">","hf_model.load_from_pretrained","(",")",">",">",">","training_loss","=","hf_model.fit","(","dataset",",","nb_epoch=1",")",">",">",">","prediction","=","hf_model.predict","(","dataset",")","#","prediction",">",">",">","eval_results","=","hf_model.evaluate","(","dataset",",","metrics=dc.metrics.Metric","(","dc.metrics.mae_score",")",")",">",">",">","#","finetune","a","classification","model",">",">",">","#","making","dataset","suitable","for","classification",">",">",">","import","numpy","as","np",">",">",">","y","=","np.random.choice","(","[","0",",","1","]",",","size=dataset.y.shape",")",">",">",">","dataset","=","dc.data.NumpyDataset","(","X=dataset.X",",","y=y",",","w=dataset.w",",","ids=dataset.ids",")",">",">",">","from","transformers","import","RobertaForSequenceClassification",">",">",">","config","=","RobertaConfig","(","vocab_size=tokenizer.vocab_size",")",">",">",">","model","=","RobertaForSequenceClassification","(","config",")",">",">",">","hf_model","=","HuggingFaceModel","(","model=model",",","task='classification","'",",","tokenizer=tokenizer",")",">",">",">","training_loss","=","hf_model.fit","(","dataset",",","nb_epoch=1",")",">",">",">","predictions","=","hf_model.predict","(","dataset",")",">",">",">","eval_result","=","hf_model.evaluate","(","dataset",",","metrics=dc.metrics.Metric","(","dc.metrics.f1_score",")",")"]
docstr: Predicts output for data provided by generator.
  
  This is the private implementation of prediction. Do not
  call it directly. Instead call one of the public prediction methods.
  
  Parameters
  ----------
  generator: generator
      this should generate batches, each represented as a tuple of the form
      (inputs, labels, weights).
  transformers: list of dc.trans.Transformers
      Transformers that the input data has been transformed by.  The output
      is passed through these transformers to undo the transformations.
  uncertainty: bool
      specifies whether this is being called as part of estimating uncertainty.
      If True, it sets the training flag so that dropout will be enabled, and
      returns the values of the uncertainty outputs.
  other_output_types: list, optional
      Provides a list of other output_types (strings) to predict from model.
  
  Returns
  -------
      a NumPy array of the model produces a single output, or a list of arrays
      if it produces multiple outputs
  
  Note
  ----
  A HuggingFace model does not output uncertainity. The argument is here
  since it is also present in TorchModel. Similarly, other variables like
  other_output_types are also not used. Instead, a HuggingFace model outputs
  loss, logits, hidden state and attentions.
docstr_tok: ["Predicts","output","for","data","provided","by","generator",".","This","is","the","private","implementation","of","prediction",".","Do","not","call","it","directly",".","Instead","call","one","of","the","public","prediction","methods",".","Parameters","--","--","--","--","--","generator",":","generator","this","should","generate","batches",",","each","represented","as","a","tuple","of","the","form","(","inputs",",","labels",",","weights",")",".","transformers",":","list","of","dc.trans.Transformers","Transformers","that","the","input","data","has","been","transformed","by",".","The","output","is","passed","through","these","transformers","to","undo","the","transformations",".","uncertainty",":","bool","specifies","whether","this","is","being","called","as","part","of","estimating","uncertainty",".","If","True",",","it","sets","the","training","flag","so","that","dropout","will","be","enabled",",","and","returns","the","values","of","the","uncertainty","outputs",".","other_output_types",":","list",",","optional","Provides","a","list","of","other","output_types","(","strings",")","to","predict","from","model",".","Returns","--","--","--","-","a","NumPy","array","of","the","model","produces","a","single","output",",","or","a","list","of","arrays","if","it","produces","multiple","outputs","Note","--","--","A","HuggingFace","model","does","not","output","uncertainity",".","The","argument","is","here","since","it","is","also","present","in","TorchModel",".","Similarly",",","other","variables","like","other_output_types","are","also","not","used",".","Instead",",","a","HuggingFace","model","outputs","loss",",","logits",",","hidden","state","and","attentions","."]
code: def _predict(
      self,
      generator: Iterable[Tuple[Any, Any, Any]],
      transformers: List[Transformer],
      uncertainty: bool,
      other_output_types: Optional[OneOrMany[str]],
  ):
      """Predicts output for data provided by generator.
  
      This is the private implementation of prediction. Do not
      call it directly. Instead call one of the public prediction methods.
  
      Parameters
      ----------
      generator: generator
          this should generate batches, each represented as a tuple of the form
          (inputs, labels, weights).
      transformers: list of dc.trans.Transformers
          Transformers that the input data has been transformed by.  The output
          is passed through these transformers to undo the transformations.
      uncertainty: bool
          specifies whether this is being called as part of estimating uncertainty.
          If True, it sets the training flag so that dropout will be enabled, and
          returns the values of the uncertainty outputs.
      other_output_types: list, optional
          Provides a list of other output_types (strings) to predict from model.
  
      Returns
      -------
          a NumPy array of the model produces a single output, or a list of arrays
          if it produces multiple outputs
  
      Note
      ----
      A HuggingFace model does not output uncertainity. The argument is here
      since it is also present in TorchModel. Similarly, other variables like
      other_output_types are also not used. Instead, a HuggingFace model outputs
      loss, logits, hidden state and attentions.
      """
      results: Optional[List[List[np.ndarray]]] = None
      variances: Optional[List[List[np.ndarray]]] = None
      if uncertainty and (other_output_types is not None):
          raise ValueError(
              "This model cannot compute uncertainties and other output types simultaneously. Please invoke one at a time."
          )
      if uncertainty:
          if (
              self._variance_outputs is None
              or len(self._variance_outputs) == 0
          ):
              raise ValueError(
                  "This model cannot compute uncertainties"
              )
          if len(self._variance_outputs) != len(
              self._prediction_outputs
          ):
              raise ValueError(
                  "The number of variances must exactly match the number of outputs"
              )
      if other_output_types:
          if (
              self._other_outputs is None
              or len(self._other_outputs) == 0
          ):
              raise ValueError(
                  "This model cannot compute other outputs since no other output_types were specified."
              )
      self._ensure_built()
      self.model.eval()
      for batch in generator:
          inputs, labels, weights = batch
          inputs, _, _ = self._prepare_batch((inputs, None, None))
  
          # Invoke the model.
          output_values = self.model(**inputs)
          output_values = output_values.get("logits")
  
          if isinstance(output_values, torch.Tensor):
              output_values = [output_values]
          output_values = [
              t.detach().cpu().numpy() for t in output_values
          ]
          # Apply tranformers and record results.
          if uncertainty:
              var = [output_values[i] for i in self._variance_outputs]
              if variances is None:
                  variances = [var]
              else:
                  for i, t in enumerate(var):
                      variances[i].append(t)
          access_values = []
          if other_output_types:
              access_values += self._other_outputs
          elif self._prediction_outputs is not None:
              access_values += self._prediction_outputs
  
          if len(access_values) > 0:
              output_values = [output_values[i] for i in access_values]
  
          if len(transformers) > 0:
              if len(output_values) > 1:
                  raise ValueError(
                      "predict() does not support Transformers for models with multiple outputs."
                  )
              elif len(output_values) == 1:
                  output_values = [
                      undo_transforms(output_values[0], transformers)
                  ]
          if results is None:
              results = [[] for i in range(len(output_values))]
          for i, t in enumerate(output_values):
              results[i].append(t)
  
      # Concatenate arrays to create the final results.
      final_results = []
      final_variances = []
      if results is not None:
          for r in results:
              final_results.append(np.concatenate(r, axis=0))
  
      if uncertainty and variances is not None:
          for v in variances:
              final_variances.append(np.concatenate(v, axis=0))
          return zip(final_results, final_variances)
  
      if len(final_results) == 1:
          return final_results[0]
      else:
          return np.array(final_results)
  
file_path: deepchem/deepchem/models/torch_models/hf_models.py
imports: import logging
  import time
  from collections.abc import Sequence
  from typing import TYPE_CHECKING, Any, Callable, Iterable, List, Optional, Tuple, Union
  import numpy
  import torch
  from deepchem.models.optimizers import LearningRateSchedule
  from deepchem.models.torch_models import TorchModel
  from deepchem.trans import Transformer, undo_transforms
  from deepchem.utils.typing import LossFn, OneOrMany
  from transformers.data.data_collator import DataCollatorForLanguageModeling
  from transformers.models.auto import AutoModel, AutoModelForSequenceClassification, AutoModelForMaskedLM
label_desc: Use image node_id 5 for calling the HuggingFaceModel obj's underlying member method code with example usage: obj._predict(generator, transformers, uncertainty, other_output_types) and returns: zip, final_results, np