class_docstr: Wrapper class that wraps HuggingFace models as DeepChem models
  
  The class provides a wrapper for wrapping models from HuggingFace
  ecosystem in DeepChem and training it via DeepChem's api. The reason
  for this might be that you might want to do an apples-to-apples comparison
  between HuggingFace from the transformers library and DeepChem library.
  
  The `HuggingFaceModel` has a Has-A relationship by wrapping models from
  `transformers` library. Once a model is wrapped, DeepChem's API are used
  for training, prediction, evaluation and other downstream tasks.
  
  A `HuggingFaceModel` wrapper also has a `tokenizer` which tokenizes raw
  SMILES strings into tokens to be used by downstream models.  The SMILES
  strings are generally stored in the `X` attribute of deepchem.data.Dataset object'.
  This differs from the DeepChem standard workflow as tokenization is done
  on the fly here. The approach allows us to leverage `transformers` library's fast
  tokenization algorithms and other utilities like data collation, random masking of tokens
  for masked language model training etc.
  
  
  Parameters
  ----------
  model: transformers.modeling_utils.PreTrainedModel
      The HuggingFace model to wrap.
  task: str, (optional, default None)
      The task defines the type of learning task in the model. The supported tasks are
       - `mlm` - masked language modeling commonly used in pretraining
       - `mtr` - multitask regression - a task used for both pretraining base models and finetuning
       - `regression` - use it for regression tasks, like property prediction
       - `classification` - use it for classification tasks
      When the task is not specified or None, the wrapper returns raw output of the HuggingFaceModel.
      In cases where the HuggingFaceModel is a model without a task specific head, this output will be
      the last hidden states.
  tokenizer: transformers.tokenization_utils.PreTrainedTokenizer
      Tokenizer
  
  Example
  -------
  >>> import os
  >>> import tempfile
  >>> tempdir = tempfile.mkdtemp()
  
  >>> # preparing dataset
  >>> smiles = ['CN(c1ccccc1)c1ccccc1C(=O)NCC1(O)CCOCC1', 'CC[NH+](CC)C1CCC([NH2+]C2CC2)(C(=O)[O-])C1', \
  ...     'COCC(CNC(=O)c1ccc2c(c1)NC(=O)C2)OC', 'OCCn1cc(CNc2cccc3c2CCCC3)nn1', \
  ...     'CCCCCCc1ccc(C#Cc2ccc(C#CC3=CC=C(CCC)CC3)c(C3CCCCC3)c2)c(F)c1', 'nO=C(NCc1ccc(F)cc1)N1CC=C(c2c[nH]c3ccccc23)CC1']
  >>> filepath = os.path.join(tempdir, 'smiles.txt')
  >>> f = open(filepath, 'w')
  >>> f.write('\n'.join(smiles))
  253
  >>> f.close()
  
  >>> # preparing tokenizer
  >>> from tokenizers import ByteLevelBPETokenizer
  >>> from transformers.models.roberta import RobertaTokenizerFast
  >>> tokenizer = ByteLevelBPETokenizer()
  >>> tokenizer.train(files=filepath, vocab_size=1_000, min_frequency=2, special_tokens=["<s>", "<pad>", "</s>", "<unk>", "<mask>"])
  >>> tokenizer_path = os.path.join(tempdir, 'tokenizer')
  >>> os.makedirs(tokenizer_path)
  >>> result = tokenizer.save_model(tokenizer_path)
  >>> tokenizer = RobertaTokenizerFast.from_pretrained(tokenizer_path)
  
  >>> # preparing dataset
  >>> import pandas as pd
  >>> import deepchem as dc
  >>> smiles = ["CCN(CCSC)C(=O)N[C@@](C)(CC)C(F)(F)F","CC1(C)CN(C(=O)Nc2cc3ccccc3nn2)C[C@@]2(CCOC2)O1"]
  >>> labels = [3.112,2.432]
  >>> df = pd.DataFrame(list(zip(smiles, labels)), columns=["smiles", "task1"])
  >>> with dc.utils.UniversalNamedTemporaryFile(mode='w') as tmpfile:
  ...     df.to_csv(tmpfile.name)
  ...     loader = dc.data.CSVLoader(["task1"], feature_field="smiles", featurizer=dc.feat.DummyFeaturizer())
  ...     dataset = loader.create_dataset(tmpfile.name)
  
  >>> # pretraining
  >>> from deepchem.models.torch_models.hf_models import HuggingFaceModel
  >>> from transformers.models.roberta import RobertaForMaskedLM, RobertaModel, RobertaConfig
  >>> config = RobertaConfig(vocab_size=tokenizer.vocab_size)
  >>> model = RobertaForMaskedLM(config)
  >>> hf_model = HuggingFaceModel(model=model, tokenizer=tokenizer, task='mlm', model_dir='model-dir')
  >>> training_loss = hf_model.fit(dataset, nb_epoch=1)
  
  >>> # finetuning a regression model
  >>> from transformers.models.roberta import RobertaForSequenceClassification
  >>> config = RobertaConfig(vocab_size=tokenizer.vocab_size, problem_type='regression', num_labels=1)
  >>> model = RobertaForSequenceClassification(config)
  >>> hf_model = HuggingFaceModel(model=model, tokenizer=tokenizer, task='regression', model_dir='model-dir')
  >>> hf_model.load_from_pretrained()
  >>> training_loss = hf_model.fit(dataset, nb_epoch=1)
  >>> prediction = hf_model.predict(dataset)  # prediction
  >>> eval_results = hf_model.evaluate(dataset, metrics=dc.metrics.Metric(dc.metrics.mae_score))
  
  >>> # finetune a classification model
  >>> # making dataset suitable for classification
  >>> import numpy as np
  >>> y = np.random.choice([0, 1], size=dataset.y.shape)
  >>> dataset = dc.data.NumpyDataset(X=dataset.X, y=y, w=dataset.w, ids=dataset.ids)
  
  >>> from transformers import RobertaForSequenceClassification
  >>> config = RobertaConfig(vocab_size=tokenizer.vocab_size)
  >>> model = RobertaForSequenceClassification(config)
  >>> hf_model = HuggingFaceModel(model=model, task='classification', tokenizer=tokenizer)
  >>> training_loss = hf_model.fit(dataset, nb_epoch=1)
  >>> predictions = hf_model.predict(dataset)
  >>> eval_result = hf_model.evaluate(dataset, metrics=dc.metrics.Metric(dc.metrics.f1_score))
class_docstr_tok: ["Wrapper","class","that","wraps","HuggingFace","models","as","DeepChem","models","The","class","provides","a","wrapper","for","wrapping","models","from","HuggingFace","ecosystem","in","DeepChem","and","training","it","via","DeepChem","'s","api",".","The","reason","for","this","might","be","that","you","might","want","to","do","an","apples-to-apples","comparison","between","HuggingFace","from","the","transformers","library","and","DeepChem","library",".","The","`","HuggingFaceModel","`","has","a","Has-A","relationship","by","wrapping","models","from","`","transformers","`","library",".","Once","a","model","is","wrapped",",","DeepChem","'s","API","are","used","for","training",",","prediction",",","evaluation","and","other","downstream","tasks",".","A","`","HuggingFaceModel","`","wrapper","also","has","a","`","tokenizer","`","which","tokenizes","raw","SMILES","strings","into","tokens","to","be","used","by","downstream","models",".","The","SMILES","strings","are","generally","stored","in","the","`","X","`","attribute","of","deepchem.data.Dataset","object","'",".","This","differs","from","the","DeepChem","standard","workflow","as","tokenization","is","done","on","the","fly","here",".","The","approach","allows","us","to","leverage","`","transformers","`","library","'s","fast","tokenization","algorithms","and","other","utilities","like","data","collation",",","random","masking","of","tokens","for","masked","language","model","training","etc",".","Parameters","--","--","--","--","--","model",":","transformers.modeling_utils.PreTrainedModel","The","HuggingFace","model","to","wrap",".","task",":","str",",","(","optional",",","default","None",")","The","task","defines","the","type","of","learning","task","in","the","model",".","The","supported","tasks","are","-","`","mlm","`","-","masked","language","modeling","commonly","used","in","pretraining","-","`","mtr","`","-","multitask","regression","-","a","task","used","for","both","pretraining","base","models","and","finetuning","-","`","regression","`","-","use","it","for","regression","tasks",",","like","property","prediction","-","`","classification","`","-","use","it","for","classification","tasks","When","the","task","is","not","specified","or","None",",","the","wrapper","returns","raw","output","of","the","HuggingFaceModel",".","In","cases","where","the","HuggingFaceModel","is","a","model","without","a","task","specific","head",",","this","output","will","be","the","last","hidden","states",".","tokenizer",":","transformers.tokenization_utils.PreTrainedTokenizer","Tokenizer","Example","--","--","--","-",">",">",">","import","os",">",">",">","import","tempfile",">",">",">","tempdir","=","tempfile.mkdtemp","(",")",">",">",">","#","preparing","dataset",">",">",">","smiles","=","[","'CN","(","c1ccccc1",")","c1ccccc1C","(","=O",")","NCC1","(","O",")","CCOCC1","'",",","'CC","[","NH+","]","(","CC",")","C1CCC","(","[","NH2+","]","C2CC2",")","(","C","(","=O",")","[","O-","]",")","C1","'",",","\\","...","'COCC","(","CNC","(","=O",")","c1ccc2c","(","c1",")","NC","(","=O",")","C2",")","OC","'",",","'OCCn1cc","(","CNc2cccc3c2CCCC3",")","nn1","'",",","\\","...","'CCCCCCc1ccc","(","C","#","Cc2ccc","(","C","#","CC3=CC=C","(","CCC",")","CC3",")","c","(","C3CCCCC3",")","c2",")","c","(","F",")","c1","'",",","'nO=C","(","NCc1ccc","(","F",")","cc1",")","N1CC=C","(","c2c","[","nH","]","c3ccccc23",")","CC1","'","]",">",">",">","filepath","=","os.path.join","(","tempdir",",","'smiles.txt","'",")",">",">",">","f","=","open","(","filepath",",","'","w","'",")",">",">",">","f.write","(","'\\n'.join","(","smiles",")",")","253",">",">",">","f.close","(",")",">",">",">","#","preparing","tokenizer",">",">",">","from","tokenizers","import","ByteLevelBPETokenizer",">",">",">","from","transformers.models.roberta","import","RobertaTokenizerFast",">",">",">","tokenizer","=","ByteLevelBPETokenizer","(",")",">",">",">","tokenizer.train","(","files=filepath",",","vocab_size=1_000",",","min_frequency=2",",","special_tokens=","[","``","<","s",">","''",",","``","<","pad",">","''",",","``","<","\/s",">","''",",","``","<","unk",">","''",",","``","<","mask",">","''","]",")",">",">",">","tokenizer_path","=","os.path.join","(","tempdir",",","'tokenizer","'",")",">",">",">","os.makedirs","(","tokenizer_path",")",">",">",">","result","=","tokenizer.save_model","(","tokenizer_path",")",">",">",">","tokenizer","=","RobertaTokenizerFast.from_pretrained","(","tokenizer_path",")",">",">",">","#","preparing","dataset",">",">",">","import","pandas","as","pd",">",">",">","import","deepchem","as","dc",">",">",">","smiles","=","[","``","CCN","(","CCSC",")","C","(","=O",")","N","[","C","@","@","]","(","C",")","(","CC",")","C","(","F",")","(","F",")","F","''",",","''","CC1","(","C",")","CN","(","C","(","=O",")","Nc2cc3ccccc3nn2",")","C","[","C","@","@","]","2","(","CCOC2",")","O1","''","]",">",">",">","labels","=","[","3.112,2.432","]",">",">",">","df","=","pd.DataFrame","(","list","(","zip","(","smiles",",","labels",")",")",",","columns=","[","``","smiles","''",",","``","task1","''","]",")",">",">",">","with","dc.utils.UniversalNamedTemporaryFile","(","mode=","'","w","'",")","as","tmpfile",":","...","df.to_csv","(","tmpfile.name",")","...","loader","=","dc.data.CSVLoader","(","[","``","task1","''","]",",","feature_field=","''","smiles","''",",","featurizer=dc.feat.DummyFeaturizer","(",")",")","...","dataset","=","loader.create_dataset","(","tmpfile.name",")",">",">",">","#","pretraining",">",">",">","from","deepchem.models.torch_models.hf_models","import","HuggingFaceModel",">",">",">","from","transformers.models.roberta","import","RobertaForMaskedLM",",","RobertaModel",",","RobertaConfig",">",">",">","config","=","RobertaConfig","(","vocab_size=tokenizer.vocab_size",")",">",">",">","model","=","RobertaForMaskedLM","(","config",")",">",">",">","hf_model","=","HuggingFaceModel","(","model=model",",","tokenizer=tokenizer",",","task='mlm","'",",","model_dir='model-dir","'",")",">",">",">","training_loss","=","hf_model.fit","(","dataset",",","nb_epoch=1",")",">",">",">","#","finetuning","a","regression","model",">",">",">","from","transformers.models.roberta","import","RobertaForSequenceClassification",">",">",">","config","=","RobertaConfig","(","vocab_size=tokenizer.vocab_size",",","problem_type='regression","'",",","num_labels=1",")",">",">",">","model","=","RobertaForSequenceClassification","(","config",")",">",">",">","hf_model","=","HuggingFaceModel","(","model=model",",","tokenizer=tokenizer",",","task='regression","'",",","model_dir='model-dir","'",")",">",">",">","hf_model.load_from_pretrained","(",")",">",">",">","training_loss","=","hf_model.fit","(","dataset",",","nb_epoch=1",")",">",">",">","prediction","=","hf_model.predict","(","dataset",")","#","prediction",">",">",">","eval_results","=","hf_model.evaluate","(","dataset",",","metrics=dc.metrics.Metric","(","dc.metrics.mae_score",")",")",">",">",">","#","finetune","a","classification","model",">",">",">","#","making","dataset","suitable","for","classification",">",">",">","import","numpy","as","np",">",">",">","y","=","np.random.choice","(","[","0",",","1","]",",","size=dataset.y.shape",")",">",">",">","dataset","=","dc.data.NumpyDataset","(","X=dataset.X",",","y=y",",","w=dataset.w",",","ids=dataset.ids",")",">",">",">","from","transformers","import","RobertaForSequenceClassification",">",">",">","config","=","RobertaConfig","(","vocab_size=tokenizer.vocab_size",")",">",">",">","model","=","RobertaForSequenceClassification","(","config",")",">",">",">","hf_model","=","HuggingFaceModel","(","model=model",",","task='classification","'",",","tokenizer=tokenizer",")",">",">",">","training_loss","=","hf_model.fit","(","dataset",",","nb_epoch=1",")",">",">",">","predictions","=","hf_model.predict","(","dataset",")",">",">",">","eval_result","=","hf_model.evaluate","(","dataset",",","metrics=dc.metrics.Metric","(","dc.metrics.f1_score",")",")"]
docstr: Train this model on data from a generator.
  
  Parameters
  ----------
  generator: generator
      this should generate batches, each represented as a tuple of the form
      (inputs, labels, weights).
  max_checkpoints_to_keep: int
      the maximum number of checkpoints to keep.  Older checkpoints are discarded.
  checkpoint_interval: int
      the frequency at which to write checkpoints, measured in training steps.
      Set this to 0 to disable automatic checkpointing.
  restore: bool
      if True, restore the model from the most recent checkpoint and continue training
      from there.  If False, retrain the model from scratch.
  variables: list of torch.nn.Parameter
      the variables to train.  If None (the default), all trainable variables in
      the model are used.
  loss: function
      a function of the form f(outputs, labels, weights) that computes the loss
      for each batch.  If None (the default), the model's standard loss function
      is used.
  callbacks: function or list of functions
      one or more functions of the form f(model, step) that will be invoked after
      every step.  This can be used to perform validation, logging, etc.
  all_losses: Optional[List[float]], optional (default None)
      If specified, all logged losses are appended into this list. Note that
      you can call `fit()` repeatedly with the same list and losses will
      continue to be appended.
  
  Returns
  -------
  The average loss over the most recent checkpoint interval
  
  Note
  ----
  A HuggingFace model can return embeddings (last hidden state), attentions.
  Support must be added to return the embeddings to the user, so that it can
  be used for other downstream applications.
docstr_tok: ["Train","this","model","on","data","from","a","generator",".","Parameters","--","--","--","--","--","generator",":","generator","this","should","generate","batches",",","each","represented","as","a","tuple","of","the","form","(","inputs",",","labels",",","weights",")",".","max_checkpoints_to_keep",":","int","the","maximum","number","of","checkpoints","to","keep",".","Older","checkpoints","are","discarded",".","checkpoint_interval",":","int","the","frequency","at","which","to","write","checkpoints",",","measured","in","training","steps",".","Set","this","to","0","to","disable","automatic","checkpointing",".","restore",":","bool","if","True",",","restore","the","model","from","the","most","recent","checkpoint","and","continue","training","from","there",".","If","False",",","retrain","the","model","from","scratch",".","variables",":","list","of","torch.nn.Parameter","the","variables","to","train",".","If","None","(","the","default",")",",","all","trainable","variables","in","the","model","are","used",".","loss",":","function","a","function","of","the","form","f","(","outputs",",","labels",",","weights",")","that","computes","the","loss","for","each","batch",".","If","None","(","the","default",")",",","the","model","'s","standard","loss","function","is","used",".","callbacks",":","function","or","list","of","functions","one","or","more","functions","of","the","form","f","(","model",",","step",")","that","will","be","invoked","after","every","step",".","This","can","be","used","to","perform","validation",",","logging",",","etc",".","all_losses",":","Optional","[","List","[","float","]","]",",","optional","(","default","None",")","If","specified",",","all","logged","losses","are","appended","into","this","list",".","Note","that","you","can","call","`","fit","(",")","`","repeatedly","with","the","same","list","and","losses","will","continue","to","be","appended",".","Returns","--","--","--","-","The","average","loss","over","the","most","recent","checkpoint","interval","Note","--","--","A","HuggingFace","model","can","return","embeddings","(","last","hidden","state",")",",","attentions",".","Support","must","be","added","to","return","the","embeddings","to","the","user",",","so","that","it","can","be","used","for","other","downstream","applications","."]
code: def fit_generator(
      self,
      generator: Iterable[Tuple[Any, Any, Any]],
      max_checkpoints_to_keep: int = 5,
      checkpoint_interval: int = 1000,
      restore: bool = False,
      variables: Optional[List[torch.nn.Parameter]] = None,
      loss: Optional[LossFn] = None,
      callbacks: Union[Callable, List[Callable]] = [],
      all_losses: Optional[List[float]] = None,
  ) -> float:
      """Train this model on data from a generator.
  
      Parameters
      ----------
      generator: generator
          this should generate batches, each represented as a tuple of the form
          (inputs, labels, weights).
      max_checkpoints_to_keep: int
          the maximum number of checkpoints to keep.  Older checkpoints are discarded.
      checkpoint_interval: int
          the frequency at which to write checkpoints, measured in training steps.
          Set this to 0 to disable automatic checkpointing.
      restore: bool
          if True, restore the model from the most recent checkpoint and continue training
          from there.  If False, retrain the model from scratch.
      variables: list of torch.nn.Parameter
          the variables to train.  If None (the default), all trainable variables in
          the model are used.
      loss: function
          a function of the form f(outputs, labels, weights) that computes the loss
          for each batch.  If None (the default), the model's standard loss function
          is used.
      callbacks: function or list of functions
          one or more functions of the form f(model, step) that will be invoked after
          every step.  This can be used to perform validation, logging, etc.
      all_losses: Optional[List[float]], optional (default None)
          If specified, all logged losses are appended into this list. Note that
          you can call `fit()` repeatedly with the same list and losses will
          continue to be appended.
  
      Returns
      -------
      The average loss over the most recent checkpoint interval
  
      Note
      ----
      A HuggingFace model can return embeddings (last hidden state), attentions.
      Support must be added to return the embeddings to the user, so that it can
      be used for other downstream applications.
      """
      if not isinstance(callbacks, SequenceCollection):
          callbacks = [callbacks]
      self._ensure_built()
      self.model.train()
      avg_loss = 0.0
      last_avg_loss = 0.0
      averaged_batches = 0
      if variables is None:
          optimizer = self._pytorch_optimizer
          lr_schedule = self._lr_schedule
      else:
          var_key = tuple(variables)
          if var_key in self._optimizer_for_vars:
              optimizer, lr_schedule = self._optimizer_for_vars[var_key]
          else:
              optimizer = self.optimizer._create_pytorch_optimizer(
                  variables
              )
              if isinstance(
                  self.optimizer.learning_rate, LearningRateSchedule
              ):
                  lr_schedule = self.optimizer.learning_rate._create_pytorch_schedule(
                      optimizer
                  )
              else:
                  lr_schedule = None
              self._optimizer_for_vars[var_key] = (
                  optimizer,
                  lr_schedule,
              )
      time1 = time.time()
  
      # Main training loop.
  
      for batch in generator:
          if restore:
              self.restore()
              restore = False
          inputs: OneOrMany[torch.Tensor]
          inputs, labels, weights = self._prepare_batch(batch)
  
          optimizer.zero_grad()
          outputs = self.model(**inputs)
  
          if self._loss_outputs is not None:
              outputs = [outputs[i] for i in self._loss_outputs]
          batch_loss = outputs.get("loss")
          batch_loss.backward()
          optimizer.step()
          if lr_schedule is not None:
              lr_schedule.step()
          self._global_step += 1
          current_step = self._global_step
  
          avg_loss += batch_loss
  
          # Report progress and write checkpoints.
          averaged_batches += 1
          should_log = current_step % self.log_frequency == 0
          if should_log:
              avg_loss = float(avg_loss) / averaged_batches
              logger.info(
                  "Ending global_step %d: Average loss %g"
                  % (current_step, avg_loss)
              )
              if all_losses is not None:
                  all_losses.append(avg_loss)
              # Capture the last avg_loss in case of return since we're resetting to 0 now
              last_avg_loss = avg_loss
              avg_loss = 0.0
              averaged_batches = 0
  
          if (
              checkpoint_interval > 0
              and current_step % checkpoint_interval
              == checkpoint_interval - 1
          ):
              self.save_checkpoint(max_checkpoints_to_keep)
          for c in callbacks:
              c(self, current_step)
          if self.tensorboard and should_log:
              self._log_scalar_to_tensorboard(
                  "loss", batch_loss, current_step
              )
          if (self.wandb_logger is not None) and should_log:
              all_data = dict({"train/loss": batch_loss})
              self.wandb_logger.log_data(all_data, step=current_step)
  
      # Report final results.
      if averaged_batches > 0:
          avg_loss = float(avg_loss) / averaged_batches
          logger.info(
              "Ending global_step %d: Average loss %g"
              % (current_step, avg_loss)
          )
          if all_losses is not None:
              all_losses.append(avg_loss)
          last_avg_loss = avg_loss
  
      if checkpoint_interval > 0:
          self.save_checkpoint(max_checkpoints_to_keep)
  
      time2 = time.time()
      logger.info(
          "TIMING: model fitting took %0.3f s" % (time2 - time1)
      )
      return last_avg_loss
  
file_path: deepchem/deepchem/models/torch_models/hf_models.py
imports: import logging
  import time
  from collections.abc import Sequence
  from typing import TYPE_CHECKING, Any, Callable, Iterable, List, Optional, Tuple, Union
  import numpy
  import torch
  from deepchem.models.optimizers import LearningRateSchedule
  from deepchem.models.torch_models import TorchModel
  from deepchem.trans import Transformer, undo_transforms
  from deepchem.utils.typing import LossFn, OneOrMany
  from transformers.data.data_collator import DataCollatorForLanguageModeling
  from transformers.models.auto import AutoModel, AutoModelForSequenceClassification, AutoModelForMaskedLM
label_desc: Use image node_id 4 for calling the HuggingFaceModel obj's underlying member method code with example usage: obj.fit_generator(generator, max_checkpoints_to_keep, checkpoint_interval, restore, variables, loss, callbacks, all_losses) and returns: last_avg_loss