class_docstr: Wrapper class that wraps HuggingFace models as DeepChem models
  
  The class provides a wrapper for wrapping models from HuggingFace
  ecosystem in DeepChem and training it via DeepChem's api. The reason
  for this might be that you might want to do an apples-to-apples comparison
  between HuggingFace from the transformers library and DeepChem library.
  
  The `HuggingFaceModel` has a Has-A relationship by wrapping models from
  `transformers` library. Once a model is wrapped, DeepChem's API are used
  for training, prediction, evaluation and other downstream tasks.
  
  A `HuggingFaceModel` wrapper also has a `tokenizer` which tokenizes raw
  SMILES strings into tokens to be used by downstream models.  The SMILES
  strings are generally stored in the `X` attribute of deepchem.data.Dataset object'.
  This differs from the DeepChem standard workflow as tokenization is done
  on the fly here. The approach allows us to leverage `transformers` library's fast
  tokenization algorithms and other utilities like data collation, random masking of tokens
  for masked language model training etc.
  
  
  Parameters
  ----------
  model: transformers.modeling_utils.PreTrainedModel
      The HuggingFace model to wrap.
  task: str, (optional, default None)
      The task defines the type of learning task in the model. The supported tasks are
       - `mlm` - masked language modeling commonly used in pretraining
       - `mtr` - multitask regression - a task used for both pretraining base models and finetuning
       - `regression` - use it for regression tasks, like property prediction
       - `classification` - use it for classification tasks
      When the task is not specified or None, the wrapper returns raw output of the HuggingFaceModel.
      In cases where the HuggingFaceModel is a model without a task specific head, this output will be
      the last hidden states.
  tokenizer: transformers.tokenization_utils.PreTrainedTokenizer
      Tokenizer
  
  Example
  -------
  >>> import os
  >>> import tempfile
  >>> tempdir = tempfile.mkdtemp()
  
  >>> # preparing dataset
  >>> smiles = ['CN(c1ccccc1)c1ccccc1C(=O)NCC1(O)CCOCC1', 'CC[NH+](CC)C1CCC([NH2+]C2CC2)(C(=O)[O-])C1', \
  ...     'COCC(CNC(=O)c1ccc2c(c1)NC(=O)C2)OC', 'OCCn1cc(CNc2cccc3c2CCCC3)nn1', \
  ...     'CCCCCCc1ccc(C#Cc2ccc(C#CC3=CC=C(CCC)CC3)c(C3CCCCC3)c2)c(F)c1', 'nO=C(NCc1ccc(F)cc1)N1CC=C(c2c[nH]c3ccccc23)CC1']
  >>> filepath = os.path.join(tempdir, 'smiles.txt')
  >>> f = open(filepath, 'w')
  >>> f.write('\n'.join(smiles))
  253
  >>> f.close()
  
  >>> # preparing tokenizer
  >>> from tokenizers import ByteLevelBPETokenizer
  >>> from transformers.models.roberta import RobertaTokenizerFast
  >>> tokenizer = ByteLevelBPETokenizer()
  >>> tokenizer.train(files=filepath, vocab_size=1_000, min_frequency=2, special_tokens=["<s>", "<pad>", "</s>", "<unk>", "<mask>"])
  >>> tokenizer_path = os.path.join(tempdir, 'tokenizer')
  >>> os.makedirs(tokenizer_path)
  >>> result = tokenizer.save_model(tokenizer_path)
  >>> tokenizer = RobertaTokenizerFast.from_pretrained(tokenizer_path)
  
  >>> # preparing dataset
  >>> import pandas as pd
  >>> import deepchem as dc
  >>> smiles = ["CCN(CCSC)C(=O)N[C@@](C)(CC)C(F)(F)F","CC1(C)CN(C(=O)Nc2cc3ccccc3nn2)C[C@@]2(CCOC2)O1"]
  >>> labels = [3.112,2.432]
  >>> df = pd.DataFrame(list(zip(smiles, labels)), columns=["smiles", "task1"])
  >>> with dc.utils.UniversalNamedTemporaryFile(mode='w') as tmpfile:
  ...     df.to_csv(tmpfile.name)
  ...     loader = dc.data.CSVLoader(["task1"], feature_field="smiles", featurizer=dc.feat.DummyFeaturizer())
  ...     dataset = loader.create_dataset(tmpfile.name)
  
  >>> # pretraining
  >>> from deepchem.models.torch_models.hf_models import HuggingFaceModel
  >>> from transformers.models.roberta import RobertaForMaskedLM, RobertaModel, RobertaConfig
  >>> config = RobertaConfig(vocab_size=tokenizer.vocab_size)
  >>> model = RobertaForMaskedLM(config)
  >>> hf_model = HuggingFaceModel(model=model, tokenizer=tokenizer, task='mlm', model_dir='model-dir')
  >>> training_loss = hf_model.fit(dataset, nb_epoch=1)
  
  >>> # finetuning a regression model
  >>> from transformers.models.roberta import RobertaForSequenceClassification
  >>> config = RobertaConfig(vocab_size=tokenizer.vocab_size, problem_type='regression', num_labels=1)
  >>> model = RobertaForSequenceClassification(config)
  >>> hf_model = HuggingFaceModel(model=model, tokenizer=tokenizer, task='regression', model_dir='model-dir')
  >>> hf_model.load_from_pretrained()
  >>> training_loss = hf_model.fit(dataset, nb_epoch=1)
  >>> prediction = hf_model.predict(dataset)  # prediction
  >>> eval_results = hf_model.evaluate(dataset, metrics=dc.metrics.Metric(dc.metrics.mae_score))
  
  >>> # finetune a classification model
  >>> # making dataset suitable for classification
  >>> import numpy as np
  >>> y = np.random.choice([0, 1], size=dataset.y.shape)
  >>> dataset = dc.data.NumpyDataset(X=dataset.X, y=y, w=dataset.w, ids=dataset.ids)
  
  >>> from transformers import RobertaForSequenceClassification
  >>> config = RobertaConfig(vocab_size=tokenizer.vocab_size)
  >>> model = RobertaForSequenceClassification(config)
  >>> hf_model = HuggingFaceModel(model=model, task='classification', tokenizer=tokenizer)
  >>> training_loss = hf_model.fit(dataset, nb_epoch=1)
  >>> predictions = hf_model.predict(dataset)
  >>> eval_result = hf_model.evaluate(dataset, metrics=dc.metrics.Metric(dc.metrics.f1_score))
class_docstr_tok: ["Wrapper","class","that","wraps","HuggingFace","models","as","DeepChem","models","The","class","provides","a","wrapper","for","wrapping","models","from","HuggingFace","ecosystem","in","DeepChem","and","training","it","via","DeepChem","'s","api",".","The","reason","for","this","might","be","that","you","might","want","to","do","an","apples-to-apples","comparison","between","HuggingFace","from","the","transformers","library","and","DeepChem","library",".","The","`","HuggingFaceModel","`","has","a","Has-A","relationship","by","wrapping","models","from","`","transformers","`","library",".","Once","a","model","is","wrapped",",","DeepChem","'s","API","are","used","for","training",",","prediction",",","evaluation","and","other","downstream","tasks",".","A","`","HuggingFaceModel","`","wrapper","also","has","a","`","tokenizer","`","which","tokenizes","raw","SMILES","strings","into","tokens","to","be","used","by","downstream","models",".","The","SMILES","strings","are","generally","stored","in","the","`","X","`","attribute","of","deepchem.data.Dataset","object","'",".","This","differs","from","the","DeepChem","standard","workflow","as","tokenization","is","done","on","the","fly","here",".","The","approach","allows","us","to","leverage","`","transformers","`","library","'s","fast","tokenization","algorithms","and","other","utilities","like","data","collation",",","random","masking","of","tokens","for","masked","language","model","training","etc",".","Parameters","--","--","--","--","--","model",":","transformers.modeling_utils.PreTrainedModel","The","HuggingFace","model","to","wrap",".","task",":","str",",","(","optional",",","default","None",")","The","task","defines","the","type","of","learning","task","in","the","model",".","The","supported","tasks","are","-","`","mlm","`","-","masked","language","modeling","commonly","used","in","pretraining","-","`","mtr","`","-","multitask","regression","-","a","task","used","for","both","pretraining","base","models","and","finetuning","-","`","regression","`","-","use","it","for","regression","tasks",",","like","property","prediction","-","`","classification","`","-","use","it","for","classification","tasks","When","the","task","is","not","specified","or","None",",","the","wrapper","returns","raw","output","of","the","HuggingFaceModel",".","In","cases","where","the","HuggingFaceModel","is","a","model","without","a","task","specific","head",",","this","output","will","be","the","last","hidden","states",".","tokenizer",":","transformers.tokenization_utils.PreTrainedTokenizer","Tokenizer","Example","--","--","--","-",">",">",">","import","os",">",">",">","import","tempfile",">",">",">","tempdir","=","tempfile.mkdtemp","(",")",">",">",">","#","preparing","dataset",">",">",">","smiles","=","[","'CN","(","c1ccccc1",")","c1ccccc1C","(","=O",")","NCC1","(","O",")","CCOCC1","'",",","'CC","[","NH+","]","(","CC",")","C1CCC","(","[","NH2+","]","C2CC2",")","(","C","(","=O",")","[","O-","]",")","C1","'",",","\\","...","'COCC","(","CNC","(","=O",")","c1ccc2c","(","c1",")","NC","(","=O",")","C2",")","OC","'",",","'OCCn1cc","(","CNc2cccc3c2CCCC3",")","nn1","'",",","\\","...","'CCCCCCc1ccc","(","C","#","Cc2ccc","(","C","#","CC3=CC=C","(","CCC",")","CC3",")","c","(","C3CCCCC3",")","c2",")","c","(","F",")","c1","'",",","'nO=C","(","NCc1ccc","(","F",")","cc1",")","N1CC=C","(","c2c","[","nH","]","c3ccccc23",")","CC1","'","]",">",">",">","filepath","=","os.path.join","(","tempdir",",","'smiles.txt","'",")",">",">",">","f","=","open","(","filepath",",","'","w","'",")",">",">",">","f.write","(","'\\n'.join","(","smiles",")",")","253",">",">",">","f.close","(",")",">",">",">","#","preparing","tokenizer",">",">",">","from","tokenizers","import","ByteLevelBPETokenizer",">",">",">","from","transformers.models.roberta","import","RobertaTokenizerFast",">",">",">","tokenizer","=","ByteLevelBPETokenizer","(",")",">",">",">","tokenizer.train","(","files=filepath",",","vocab_size=1_000",",","min_frequency=2",",","special_tokens=","[","``","<","s",">","''",",","``","<","pad",">","''",",","``","<","\/s",">","''",",","``","<","unk",">","''",",","``","<","mask",">","''","]",")",">",">",">","tokenizer_path","=","os.path.join","(","tempdir",",","'tokenizer","'",")",">",">",">","os.makedirs","(","tokenizer_path",")",">",">",">","result","=","tokenizer.save_model","(","tokenizer_path",")",">",">",">","tokenizer","=","RobertaTokenizerFast.from_pretrained","(","tokenizer_path",")",">",">",">","#","preparing","dataset",">",">",">","import","pandas","as","pd",">",">",">","import","deepchem","as","dc",">",">",">","smiles","=","[","``","CCN","(","CCSC",")","C","(","=O",")","N","[","C","@","@","]","(","C",")","(","CC",")","C","(","F",")","(","F",")","F","''",",","''","CC1","(","C",")","CN","(","C","(","=O",")","Nc2cc3ccccc3nn2",")","C","[","C","@","@","]","2","(","CCOC2",")","O1","''","]",">",">",">","labels","=","[","3.112,2.432","]",">",">",">","df","=","pd.DataFrame","(","list","(","zip","(","smiles",",","labels",")",")",",","columns=","[","``","smiles","''",",","``","task1","''","]",")",">",">",">","with","dc.utils.UniversalNamedTemporaryFile","(","mode=","'","w","'",")","as","tmpfile",":","...","df.to_csv","(","tmpfile.name",")","...","loader","=","dc.data.CSVLoader","(","[","``","task1","''","]",",","feature_field=","''","smiles","''",",","featurizer=dc.feat.DummyFeaturizer","(",")",")","...","dataset","=","loader.create_dataset","(","tmpfile.name",")",">",">",">","#","pretraining",">",">",">","from","deepchem.models.torch_models.hf_models","import","HuggingFaceModel",">",">",">","from","transformers.models.roberta","import","RobertaForMaskedLM",",","RobertaModel",",","RobertaConfig",">",">",">","config","=","RobertaConfig","(","vocab_size=tokenizer.vocab_size",")",">",">",">","model","=","RobertaForMaskedLM","(","config",")",">",">",">","hf_model","=","HuggingFaceModel","(","model=model",",","tokenizer=tokenizer",",","task='mlm","'",",","model_dir='model-dir","'",")",">",">",">","training_loss","=","hf_model.fit","(","dataset",",","nb_epoch=1",")",">",">",">","#","finetuning","a","regression","model",">",">",">","from","transformers.models.roberta","import","RobertaForSequenceClassification",">",">",">","config","=","RobertaConfig","(","vocab_size=tokenizer.vocab_size",",","problem_type='regression","'",",","num_labels=1",")",">",">",">","model","=","RobertaForSequenceClassification","(","config",")",">",">",">","hf_model","=","HuggingFaceModel","(","model=model",",","tokenizer=tokenizer",",","task='regression","'",",","model_dir='model-dir","'",")",">",">",">","hf_model.load_from_pretrained","(",")",">",">",">","training_loss","=","hf_model.fit","(","dataset",",","nb_epoch=1",")",">",">",">","prediction","=","hf_model.predict","(","dataset",")","#","prediction",">",">",">","eval_results","=","hf_model.evaluate","(","dataset",",","metrics=dc.metrics.Metric","(","dc.metrics.mae_score",")",")",">",">",">","#","finetune","a","classification","model",">",">",">","#","making","dataset","suitable","for","classification",">",">",">","import","numpy","as","np",">",">",">","y","=","np.random.choice","(","[","0",",","1","]",",","size=dataset.y.shape",")",">",">",">","dataset","=","dc.data.NumpyDataset","(","X=dataset.X",",","y=y",",","w=dataset.w",",","ids=dataset.ids",")",">",">",">","from","transformers","import","RobertaForSequenceClassification",">",">",">","config","=","RobertaConfig","(","vocab_size=tokenizer.vocab_size",")",">",">",">","model","=","RobertaForSequenceClassification","(","config",")",">",">",">","hf_model","=","HuggingFaceModel","(","model=model",",","task='classification","'",",","tokenizer=tokenizer",")",">",">",">","training_loss","=","hf_model.fit","(","dataset",",","nb_epoch=1",")",">",">",">","predictions","=","hf_model.predict","(","dataset",")",">",">",">","eval_result","=","hf_model.evaluate","(","dataset",",","metrics=dc.metrics.Metric","(","dc.metrics.f1_score",")",")"]
code: def _prepare_batch(self, batch: Tuple[Any, Any, Any]):
      smiles_batch, y, w = batch
      tokens = self.tokenizer(
          smiles_batch[0].tolist(), padding=True, return_tensors="pt"
      )
  
      if self.task == "mlm":
          inputs, labels = self.data_collator.torch_mask_tokens(
              tokens["input_ids"]
          )
          inputs = {
              "input_ids": inputs.to(self.device),
              "labels": labels.to(self.device),
              "attention_mask": tokens["attention_mask"].to(
                  self.device
              ),
          }
          return inputs, None, w
      elif self.task in ["regression", "classification", "mtr"]:
          if y is not None:
              # y is None during predict
              y = torch.from_numpy(y[0])
              if self.task == "regression" or self.task == "mtr":
                  y = y.float().to(self.device)
              elif self.task == "classification":
                  y = y.long().to(self.device)
          for key, value in tokens.items():
              tokens[key] = value.to(self.device)
  
          inputs = {**tokens, "labels": y}
          return inputs, y, w
  
file_path: deepchem/deepchem/models/torch_models/hf_models.py
imports: import logging
  import time
  from collections.abc import Sequence
  from typing import TYPE_CHECKING, Any, Callable, Iterable, List, Optional, Tuple, Union
  import numpy
  import torch
  from deepchem.models.optimizers import LearningRateSchedule
  from deepchem.models.torch_models import TorchModel
  from deepchem.trans import Transformer, undo_transforms
  from deepchem.utils.typing import LossFn, OneOrMany
  from transformers.data.data_collator import DataCollatorForLanguageModeling
  from transformers.models.auto import AutoModel, AutoModelForSequenceClassification, AutoModelForMaskedLM
label_desc: Use image node_id 3 for calling the HuggingFaceModel obj's underlying member method code with example usage: obj._prepare_batch(batch) and returns: inputs,  None,  w, inputs,  y,  w