class_docstr: Wrapper class that wraps HuggingFace models as DeepChem models
  
  The class provides a wrapper for wrapping models from HuggingFace
  ecosystem in DeepChem and training it via DeepChem's api. The reason
  for this might be that you might want to do an apples-to-apples comparison
  between HuggingFace from the transformers library and DeepChem library.
  
  The `HuggingFaceModel` has a Has-A relationship by wrapping models from
  `transformers` library. Once a model is wrapped, DeepChem's API are used
  for training, prediction, evaluation and other downstream tasks.
  
  A `HuggingFaceModel` wrapper also has a `tokenizer` which tokenizes raw
  SMILES strings into tokens to be used by downstream models.  The SMILES
  strings are generally stored in the `X` attribute of deepchem.data.Dataset object'.
  This differs from the DeepChem standard workflow as tokenization is done
  on the fly here. The approach allows us to leverage `transformers` library's fast
  tokenization algorithms and other utilities like data collation, random masking of tokens
  for masked language model training etc.
  
  
  Parameters
  ----------
  model: transformers.modeling_utils.PreTrainedModel
      The HuggingFace model to wrap.
  task: str, (optional, default None)
      The task defines the type of learning task in the model. The supported tasks are
       - `mlm` - masked language modeling commonly used in pretraining
       - `mtr` - multitask regression - a task used for both pretraining base models and finetuning
       - `regression` - use it for regression tasks, like property prediction
       - `classification` - use it for classification tasks
      When the task is not specified or None, the wrapper returns raw output of the HuggingFaceModel.
      In cases where the HuggingFaceModel is a model without a task specific head, this output will be
      the last hidden states.
  tokenizer: transformers.tokenization_utils.PreTrainedTokenizer
      Tokenizer
  
  Example
  -------
  >>> import os
  >>> import tempfile
  >>> tempdir = tempfile.mkdtemp()
  
  >>> # preparing dataset
  >>> smiles = ['CN(c1ccccc1)c1ccccc1C(=O)NCC1(O)CCOCC1', 'CC[NH+](CC)C1CCC([NH2+]C2CC2)(C(=O)[O-])C1', \
  ...     'COCC(CNC(=O)c1ccc2c(c1)NC(=O)C2)OC', 'OCCn1cc(CNc2cccc3c2CCCC3)nn1', \
  ...     'CCCCCCc1ccc(C#Cc2ccc(C#CC3=CC=C(CCC)CC3)c(C3CCCCC3)c2)c(F)c1', 'nO=C(NCc1ccc(F)cc1)N1CC=C(c2c[nH]c3ccccc23)CC1']
  >>> filepath = os.path.join(tempdir, 'smiles.txt')
  >>> f = open(filepath, 'w')
  >>> f.write('\n'.join(smiles))
  253
  >>> f.close()
  
  >>> # preparing tokenizer
  >>> from tokenizers import ByteLevelBPETokenizer
  >>> from transformers.models.roberta import RobertaTokenizerFast
  >>> tokenizer = ByteLevelBPETokenizer()
  >>> tokenizer.train(files=filepath, vocab_size=1_000, min_frequency=2, special_tokens=["<s>", "<pad>", "</s>", "<unk>", "<mask>"])
  >>> tokenizer_path = os.path.join(tempdir, 'tokenizer')
  >>> os.makedirs(tokenizer_path)
  >>> result = tokenizer.save_model(tokenizer_path)
  >>> tokenizer = RobertaTokenizerFast.from_pretrained(tokenizer_path)
  
  >>> # preparing dataset
  >>> import pandas as pd
  >>> import deepchem as dc
  >>> smiles = ["CCN(CCSC)C(=O)N[C@@](C)(CC)C(F)(F)F","CC1(C)CN(C(=O)Nc2cc3ccccc3nn2)C[C@@]2(CCOC2)O1"]
  >>> labels = [3.112,2.432]
  >>> df = pd.DataFrame(list(zip(smiles, labels)), columns=["smiles", "task1"])
  >>> with dc.utils.UniversalNamedTemporaryFile(mode='w') as tmpfile:
  ...     df.to_csv(tmpfile.name)
  ...     loader = dc.data.CSVLoader(["task1"], feature_field="smiles", featurizer=dc.feat.DummyFeaturizer())
  ...     dataset = loader.create_dataset(tmpfile.name)
  
  >>> # pretraining
  >>> from deepchem.models.torch_models.hf_models import HuggingFaceModel
  >>> from transformers.models.roberta import RobertaForMaskedLM, RobertaModel, RobertaConfig
  >>> config = RobertaConfig(vocab_size=tokenizer.vocab_size)
  >>> model = RobertaForMaskedLM(config)
  >>> hf_model = HuggingFaceModel(model=model, tokenizer=tokenizer, task='mlm', model_dir='model-dir')
  >>> training_loss = hf_model.fit(dataset, nb_epoch=1)
  
  >>> # finetuning a regression model
  >>> from transformers.models.roberta import RobertaForSequenceClassification
  >>> config = RobertaConfig(vocab_size=tokenizer.vocab_size, problem_type='regression', num_labels=1)
  >>> model = RobertaForSequenceClassification(config)
  >>> hf_model = HuggingFaceModel(model=model, tokenizer=tokenizer, task='regression', model_dir='model-dir')
  >>> hf_model.load_from_pretrained()
  >>> training_loss = hf_model.fit(dataset, nb_epoch=1)
  >>> prediction = hf_model.predict(dataset)  # prediction
  >>> eval_results = hf_model.evaluate(dataset, metrics=dc.metrics.Metric(dc.metrics.mae_score))
  
  >>> # finetune a classification model
  >>> # making dataset suitable for classification
  >>> import numpy as np
  >>> y = np.random.choice([0, 1], size=dataset.y.shape)
  >>> dataset = dc.data.NumpyDataset(X=dataset.X, y=y, w=dataset.w, ids=dataset.ids)
  
  >>> from transformers import RobertaForSequenceClassification
  >>> config = RobertaConfig(vocab_size=tokenizer.vocab_size)
  >>> model = RobertaForSequenceClassification(config)
  >>> hf_model = HuggingFaceModel(model=model, task='classification', tokenizer=tokenizer)
  >>> training_loss = hf_model.fit(dataset, nb_epoch=1)
  >>> predictions = hf_model.predict(dataset)
  >>> eval_result = hf_model.evaluate(dataset, metrics=dc.metrics.Metric(dc.metrics.f1_score))
class_docstr_tok: ["Wrapper","class","that","wraps","HuggingFace","models","as","DeepChem","models","The","class","provides","a","wrapper","for","wrapping","models","from","HuggingFace","ecosystem","in","DeepChem","and","training","it","via","DeepChem","'s","api",".","The","reason","for","this","might","be","that","you","might","want","to","do","an","apples-to-apples","comparison","between","HuggingFace","from","the","transformers","library","and","DeepChem","library",".","The","`","HuggingFaceModel","`","has","a","Has-A","relationship","by","wrapping","models","from","`","transformers","`","library",".","Once","a","model","is","wrapped",",","DeepChem","'s","API","are","used","for","training",",","prediction",",","evaluation","and","other","downstream","tasks",".","A","`","HuggingFaceModel","`","wrapper","also","has","a","`","tokenizer","`","which","tokenizes","raw","SMILES","strings","into","tokens","to","be","used","by","downstream","models",".","The","SMILES","strings","are","generally","stored","in","the","`","X","`","attribute","of","deepchem.data.Dataset","object","'",".","This","differs","from","the","DeepChem","standard","workflow","as","tokenization","is","done","on","the","fly","here",".","The","approach","allows","us","to","leverage","`","transformers","`","library","'s","fast","tokenization","algorithms","and","other","utilities","like","data","collation",",","random","masking","of","tokens","for","masked","language","model","training","etc",".","Parameters","--","--","--","--","--","model",":","transformers.modeling_utils.PreTrainedModel","The","HuggingFace","model","to","wrap",".","task",":","str",",","(","optional",",","default","None",")","The","task","defines","the","type","of","learning","task","in","the","model",".","The","supported","tasks","are","-","`","mlm","`","-","masked","language","modeling","commonly","used","in","pretraining","-","`","mtr","`","-","multitask","regression","-","a","task","used","for","both","pretraining","base","models","and","finetuning","-","`","regression","`","-","use","it","for","regression","tasks",",","like","property","prediction","-","`","classification","`","-","use","it","for","classification","tasks","When","the","task","is","not","specified","or","None",",","the","wrapper","returns","raw","output","of","the","HuggingFaceModel",".","In","cases","where","the","HuggingFaceModel","is","a","model","without","a","task","specific","head",",","this","output","will","be","the","last","hidden","states",".","tokenizer",":","transformers.tokenization_utils.PreTrainedTokenizer","Tokenizer","Example","--","--","--","-",">",">",">","import","os",">",">",">","import","tempfile",">",">",">","tempdir","=","tempfile.mkdtemp","(",")",">",">",">","#","preparing","dataset",">",">",">","smiles","=","[","'CN","(","c1ccccc1",")","c1ccccc1C","(","=O",")","NCC1","(","O",")","CCOCC1","'",",","'CC","[","NH+","]","(","CC",")","C1CCC","(","[","NH2+","]","C2CC2",")","(","C","(","=O",")","[","O-","]",")","C1","'",",","\\","...","'COCC","(","CNC","(","=O",")","c1ccc2c","(","c1",")","NC","(","=O",")","C2",")","OC","'",",","'OCCn1cc","(","CNc2cccc3c2CCCC3",")","nn1","'",",","\\","...","'CCCCCCc1ccc","(","C","#","Cc2ccc","(","C","#","CC3=CC=C","(","CCC",")","CC3",")","c","(","C3CCCCC3",")","c2",")","c","(","F",")","c1","'",",","'nO=C","(","NCc1ccc","(","F",")","cc1",")","N1CC=C","(","c2c","[","nH","]","c3ccccc23",")","CC1","'","]",">",">",">","filepath","=","os.path.join","(","tempdir",",","'smiles.txt","'",")",">",">",">","f","=","open","(","filepath",",","'","w","'",")",">",">",">","f.write","(","'\\n'.join","(","smiles",")",")","253",">",">",">","f.close","(",")",">",">",">","#","preparing","tokenizer",">",">",">","from","tokenizers","import","ByteLevelBPETokenizer",">",">",">","from","transformers.models.roberta","import","RobertaTokenizerFast",">",">",">","tokenizer","=","ByteLevelBPETokenizer","(",")",">",">",">","tokenizer.train","(","files=filepath",",","vocab_size=1_000",",","min_frequency=2",",","special_tokens=","[","``","<","s",">","''",",","``","<","pad",">","''",",","``","<","\/s",">","''",",","``","<","unk",">","''",",","``","<","mask",">","''","]",")",">",">",">","tokenizer_path","=","os.path.join","(","tempdir",",","'tokenizer","'",")",">",">",">","os.makedirs","(","tokenizer_path",")",">",">",">","result","=","tokenizer.save_model","(","tokenizer_path",")",">",">",">","tokenizer","=","RobertaTokenizerFast.from_pretrained","(","tokenizer_path",")",">",">",">","#","preparing","dataset",">",">",">","import","pandas","as","pd",">",">",">","import","deepchem","as","dc",">",">",">","smiles","=","[","``","CCN","(","CCSC",")","C","(","=O",")","N","[","C","@","@","]","(","C",")","(","CC",")","C","(","F",")","(","F",")","F","''",",","''","CC1","(","C",")","CN","(","C","(","=O",")","Nc2cc3ccccc3nn2",")","C","[","C","@","@","]","2","(","CCOC2",")","O1","''","]",">",">",">","labels","=","[","3.112,2.432","]",">",">",">","df","=","pd.DataFrame","(","list","(","zip","(","smiles",",","labels",")",")",",","columns=","[","``","smiles","''",",","``","task1","''","]",")",">",">",">","with","dc.utils.UniversalNamedTemporaryFile","(","mode=","'","w","'",")","as","tmpfile",":","...","df.to_csv","(","tmpfile.name",")","...","loader","=","dc.data.CSVLoader","(","[","``","task1","''","]",",","feature_field=","''","smiles","''",",","featurizer=dc.feat.DummyFeaturizer","(",")",")","...","dataset","=","loader.create_dataset","(","tmpfile.name",")",">",">",">","#","pretraining",">",">",">","from","deepchem.models.torch_models.hf_models","import","HuggingFaceModel",">",">",">","from","transformers.models.roberta","import","RobertaForMaskedLM",",","RobertaModel",",","RobertaConfig",">",">",">","config","=","RobertaConfig","(","vocab_size=tokenizer.vocab_size",")",">",">",">","model","=","RobertaForMaskedLM","(","config",")",">",">",">","hf_model","=","HuggingFaceModel","(","model=model",",","tokenizer=tokenizer",",","task='mlm","'",",","model_dir='model-dir","'",")",">",">",">","training_loss","=","hf_model.fit","(","dataset",",","nb_epoch=1",")",">",">",">","#","finetuning","a","regression","model",">",">",">","from","transformers.models.roberta","import","RobertaForSequenceClassification",">",">",">","config","=","RobertaConfig","(","vocab_size=tokenizer.vocab_size",",","problem_type='regression","'",",","num_labels=1",")",">",">",">","model","=","RobertaForSequenceClassification","(","config",")",">",">",">","hf_model","=","HuggingFaceModel","(","model=model",",","tokenizer=tokenizer",",","task='regression","'",",","model_dir='model-dir","'",")",">",">",">","hf_model.load_from_pretrained","(",")",">",">",">","training_loss","=","hf_model.fit","(","dataset",",","nb_epoch=1",")",">",">",">","prediction","=","hf_model.predict","(","dataset",")","#","prediction",">",">",">","eval_results","=","hf_model.evaluate","(","dataset",",","metrics=dc.metrics.Metric","(","dc.metrics.mae_score",")",")",">",">",">","#","finetune","a","classification","model",">",">",">","#","making","dataset","suitable","for","classification",">",">",">","import","numpy","as","np",">",">",">","y","=","np.random.choice","(","[","0",",","1","]",",","size=dataset.y.shape",")",">",">",">","dataset","=","dc.data.NumpyDataset","(","X=dataset.X",",","y=y",",","w=dataset.w",",","ids=dataset.ids",")",">",">",">","from","transformers","import","RobertaForSequenceClassification",">",">",">","config","=","RobertaConfig","(","vocab_size=tokenizer.vocab_size",")",">",">",">","model","=","RobertaForSequenceClassification","(","config",")",">",">",">","hf_model","=","HuggingFaceModel","(","model=model",",","task='classification","'",",","tokenizer=tokenizer",")",">",">",">","training_loss","=","hf_model.fit","(","dataset",",","nb_epoch=1",")",">",">",">","predictions","=","hf_model.predict","(","dataset",")",">",">",">","eval_result","=","hf_model.evaluate","(","dataset",",","metrics=dc.metrics.Metric","(","dc.metrics.f1_score",")",")"]
docstr: Load HuggingFace model from a pretrained checkpoint.
  
  The utility can be used for loading a model from a checkpoint.
  Given `model_dir`, it checks for existing checkpoint in the directory.
  If a checkpoint exists, the models state is loaded from the checkpoint.
  
  If the option `from_hf_checkpoint` is set as True, then it loads a pretrained
  model using HuggingFace models `from_pretrained` method. This option
  interprets model_dir as a model id of a pretrained model hosted inside a model repo
  on huggingface.co or path to directory containing model weights saved using `save_pretrained`
  method of a HuggingFace model.
  
  Parameter
  ----------
  model_dir: str
      Directory containing model checkpoint
  from_hf_checkpoint: bool, default False
      Loads a pretrained model from HuggingFace checkpoint.
  
  Example
  -------
  >>> from transformers import RobertaTokenizerFast
  >>> tokenizer = RobertaTokenizerFast.from_pretrained("seyonec/PubChem10M_SMILES_BPE_60k")
  
  >>> from deepchem.models.torch_models.hf_models import HuggingFaceModel
  >>> from transformers.models.roberta import RobertaForMaskedLM, RobertaModel, RobertaConfig
  >>> config = RobertaConfig(vocab_size=tokenizer.vocab_size)
  >>> model = RobertaForMaskedLM(config)
  >>> pretrain_model = HuggingFaceModel(model=model, tokenizer=tokenizer, task='mlm', model_dir='model-dir')
  >>> pretrain_model.save_checkpoint()
  
  >>> from transformers import RobertaForSequenceClassification
  >>> config = RobertaConfig(vocab_size=tokenizer.vocab_size)
  >>> model = RobertaForSequenceClassification(config)
  >>> finetune_model = HuggingFaceModel(model=model, task='classification', tokenizer=tokenizer, model_dir='model-dir')
  
  >>> finetune_model.load_from_pretrained()
docstr_tok: ["Load","HuggingFace","model","from","a","pretrained","checkpoint",".","The","utility","can","be","used","for","loading","a","model","from","a","checkpoint",".","Given","`","model_dir","`",",","it","checks","for","existing","checkpoint","in","the","directory",".","If","a","checkpoint","exists",",","the","models","state","is","loaded","from","the","checkpoint",".","If","the","option","`","from_hf_checkpoint","`","is","set","as","True",",","then","it","loads","a","pretrained","model","using","HuggingFace","models","`","from_pretrained","`","method",".","This","option","interprets","model_dir","as","a","model","id","of","a","pretrained","model","hosted","inside","a","model","repo","on","huggingface.co","or","path","to","directory","containing","model","weights","saved","using","`","save_pretrained","`","method","of","a","HuggingFace","model",".","Parameter","--","--","--","--","--","model_dir",":","str","Directory","containing","model","checkpoint","from_hf_checkpoint",":","bool",",","default","False","Loads","a","pretrained","model","from","HuggingFace","checkpoint",".","Example","--","--","--","-",">",">",">","from","transformers","import","RobertaTokenizerFast",">",">",">","tokenizer","=","RobertaTokenizerFast.from_pretrained","(","``","seyonec\/PubChem10M_SMILES_BPE_60k","''",")",">",">",">","from","deepchem.models.torch_models.hf_models","import","HuggingFaceModel",">",">",">","from","transformers.models.roberta","import","RobertaForMaskedLM",",","RobertaModel",",","RobertaConfig",">",">",">","config","=","RobertaConfig","(","vocab_size=tokenizer.vocab_size",")",">",">",">","model","=","RobertaForMaskedLM","(","config",")",">",">",">","pretrain_model","=","HuggingFaceModel","(","model=model",",","tokenizer=tokenizer",",","task='mlm","'",",","model_dir='model-dir","'",")",">",">",">","pretrain_model.save_checkpoint","(",")",">",">",">","from","transformers","import","RobertaForSequenceClassification",">",">",">","config","=","RobertaConfig","(","vocab_size=tokenizer.vocab_size",")",">",">",">","model","=","RobertaForSequenceClassification","(","config",")",">",">",">","finetune_model","=","HuggingFaceModel","(","model=model",",","task='classification","'",",","tokenizer=tokenizer",",","model_dir='model-dir","'",")",">",">",">","finetune_model.load_from_pretrained","(",")"]
code: def load_from_pretrained(  # type: ignore
      self,
      model_dir: Optional[str] = None,
      from_hf_checkpoint: bool = False,
  ):
      """Load HuggingFace model from a pretrained checkpoint.
  
      The utility can be used for loading a model from a checkpoint.
      Given `model_dir`, it checks for existing checkpoint in the directory.
      If a checkpoint exists, the models state is loaded from the checkpoint.
  
      If the option `from_hf_checkpoint` is set as True, then it loads a pretrained
      model using HuggingFace models `from_pretrained` method. This option
      interprets model_dir as a model id of a pretrained model hosted inside a model repo
      on huggingface.co or path to directory containing model weights saved using `save_pretrained`
      method of a HuggingFace model.
  
      Parameter
      ----------
      model_dir: str
          Directory containing model checkpoint
      from_hf_checkpoint: bool, default False
          Loads a pretrained model from HuggingFace checkpoint.
  
      Example
      -------
      >>> from transformers import RobertaTokenizerFast
      >>> tokenizer = RobertaTokenizerFast.from_pretrained("seyonec/PubChem10M_SMILES_BPE_60k")
  
      >>> from deepchem.models.torch_models.hf_models import HuggingFaceModel
      >>> from transformers.models.roberta import RobertaForMaskedLM, RobertaModel, RobertaConfig
      >>> config = RobertaConfig(vocab_size=tokenizer.vocab_size)
      >>> model = RobertaForMaskedLM(config)
      >>> pretrain_model = HuggingFaceModel(model=model, tokenizer=tokenizer, task='mlm', model_dir='model-dir')
      >>> pretrain_model.save_checkpoint()
  
      >>> from transformers import RobertaForSequenceClassification
      >>> config = RobertaConfig(vocab_size=tokenizer.vocab_size)
      >>> model = RobertaForSequenceClassification(config)
      >>> finetune_model = HuggingFaceModel(model=model, task='classification', tokenizer=tokenizer, model_dir='model-dir')
  
      >>> finetune_model.load_from_pretrained()
      """
      if model_dir is None:
          model_dir = self.model_dir
  
      if from_hf_checkpoint:
          # FIXME Transformers library has an api like AutoModel.from_pretrained. It allows to
          # initialise and create a model instance directly without requiring a class instance initialisation step.
          # To use `load_from_pretrained` in DeepChem, we need to follow a two step process
          # of initialising class instance and then loading weights via `load_from_pretrained`.
          if self.task == "mlm":
              self.model = AutoModelForMaskedLM.from_pretrained(
                  model_dir
              )
          elif self.task in ["mtr", "regression", "classification"]:
              self.model = (
                  AutoModelForSequenceClassification.from_pretrained(
                      model_dir
                  )
              )
          else:
              self.model = AutoModel.from_pretrained(model_dir)
      elif not from_hf_checkpoint:
          checkpoints = sorted(self.get_checkpoints(model_dir))
          if len(checkpoints) == 0:
              raise ValueError("No checkpoint found")
          else:
              checkpoint = checkpoints[0]
              data = torch.load(checkpoint, map_location=self.device)
              self.model.load_state_dict(
                  data["model_state_dict"], strict=False
              )
  
file_path: deepchem/deepchem/models/torch_models/hf_models.py
imports: import logging
  import time
  from collections.abc import Sequence
  from typing import TYPE_CHECKING, Any, Callable, Iterable, List, Optional, Tuple, Union
  import numpy
  import torch
  from deepchem.models.optimizers import LearningRateSchedule
  from deepchem.models.torch_models import TorchModel
  from deepchem.trans import Transformer, undo_transforms
  from deepchem.utils.typing import LossFn, OneOrMany
  from transformers.data.data_collator import DataCollatorForLanguageModeling
  from transformers.models.auto import AutoModel, AutoModelForSequenceClassification, AutoModelForMaskedLM
label_desc: Use image node_id 2 for calling the HuggingFaceModel obj's underlying member method code with example usage: obj.load_from_pretrained(model_dir, from_hf_checkpoint) without return types