class_docstr: An internal TorchModel class.
  
  In this class, we define the various layers and establish a sequential model for the Molecular Attention Transformer.
  We also define the forward call of this model in the forward function.
  
  References
  ----------
  .. [1] Lukasz Maziarka et al. "Molecule Attention Transformer" Graph Representation Learning workshop and Machine Learning and the Physical Sciences workshop at NeurIPS 2019. 2020. https://arxiv.org/abs/2002.08264
  
  Examples
  --------
  >>> import deepchem as dc
  >>> import pandas as pd
  >>> import numpy as np
  >>> smiles = ['CC', 'CCC',  'CCCC', 'CCCCC', 'CCCCCCC']
  >>> vals = [1.35, 6.72, 5.67, 1.23, 1.76]
  >>> df = pd.DataFrame(list(zip(smiles, vals)), columns = ['smiles', 'y'])
  >>> loader = dc.data.CSVLoader(tasks=['y'], feature_field='smiles', featurizer=dc.feat.MATFeaturizer())
  >>> df.to_csv('test.csv')
  >>> dataset = loader.create_dataset('test.csv')
  >>> model = dc.models.torch_models.MAT()
  >>> # To simulate input data, we will generate matrices for a single molecule.
  >>> vals = dataset.X[0]
  >>> node = vals.node_features
  >>> adj = vals.adjacency_matrix
  >>> dist = vals.distance_matrix
  >>> # We will now utilize a helper function defined in MATModel to get our matrices ready, and convert them into a batch consisting of a single molecule.
  >>> helper = dc.models.torch_models.MATModel()
  >>> node_features = helper.pad_sequence(torch.tensor(node).unsqueeze(0).float())
  >>> adjacency = helper.pad_sequence(torch.tensor(adj).unsqueeze(0).float())
  >>> distance = helper.pad_sequence(torch.tensor(dist).unsqueeze(0).float())
  >>> inputs = [node_features, adjacency, distance]
  >>> inputs = [x.astype(np.float32) if x.dtype == np.float64 else x for x in inputs]
  >>> # Get the forward call of the model for this batch.
  >>> output = model(inputs)
class_docstr_tok: ["An","internal","TorchModel","class",".","In","this","class",",","we","define","the","various","layers","and","establish","a","sequential","model","for","the","Molecular","Attention","Transformer",".","We","also","define","the","forward","call","of","this","model","in","the","forward","function",".","References","--","--","--","--","--","..","[","1","]","Lukasz","Maziarka","et","al",".","``","Molecule","Attention","Transformer","''","Graph","Representation","Learning","workshop","and","Machine","Learning","and","the","Physical","Sciences","workshop","at","NeurIPS","2019",".","2020.","https",":","\/\/arxiv.org\/abs\/2002.08264","Examples","--","--","--","--",">",">",">","import","deepchem","as","dc",">",">",">","import","pandas","as","pd",">",">",">","import","numpy","as","np",">",">",">","smiles","=","[","'CC","'",",","'CCC","'",",","'CCCC","'",",","'CCCCC","'",",","'CCCCCCC","'","]",">",">",">","vals","=","[","1.35",",","6.72",",","5.67",",","1.23",",","1.76","]",">",">",">","df","=","pd.DataFrame","(","list","(","zip","(","smiles",",","vals",")",")",",","columns","=","[","'smiles","'",",","'","y","'","]",")",">",">",">","loader","=","dc.data.CSVLoader","(","tasks=","[","'","y","'","]",",","feature_field='smiles","'",",","featurizer=dc.feat.MATFeaturizer","(",")",")",">",">",">","df.to_csv","(","'test.csv","'",")",">",">",">","dataset","=","loader.create_dataset","(","'test.csv","'",")",">",">",">","model","=","dc.models.torch_models.MAT","(",")",">",">",">","#","To","simulate","input","data",",","we","will","generate","matrices","for","a","single","molecule",".",">",">",">","vals","=","dataset.X","[","0","]",">",">",">","node","=","vals.node_features",">",">",">","adj","=","vals.adjacency_matrix",">",">",">","dist","=","vals.distance_matrix",">",">",">","#","We","will","now","utilize","a","helper","function","defined","in","MATModel","to","get","our","matrices","ready",",","and","convert","them","into","a","batch","consisting","of","a","single","molecule",".",">",">",">","helper","=","dc.models.torch_models.MATModel","(",")",">",">",">","node_features","=","helper.pad_sequence","(","torch.tensor","(","node",")",".unsqueeze","(","0",")",".float","(",")",")",">",">",">","adjacency","=","helper.pad_sequence","(","torch.tensor","(","adj",")",".unsqueeze","(","0",")",".float","(",")",")",">",">",">","distance","=","helper.pad_sequence","(","torch.tensor","(","dist",")",".unsqueeze","(","0",")",".float","(",")",")",">",">",">","inputs","=","[","node_features",",","adjacency",",","distance","]",">",">",">","inputs","=","[","x.astype","(","np.float32",")","if","x.dtype","==","np.float64","else","x","for","x","in","inputs","]",">",">",">","#","Get","the","forward","call","of","the","model","for","this","batch",".",">",">",">","output","=","model","(","inputs",")"]
docstr: Initialization for the internal MAT class.
  
  Parameters
  ----------
  dist_kernel: str
      Kernel activation to be used. Can be either 'softmax' for softmax or 'exp' for exponential, for the self-attention layer.
  n_encoders: int
      Number of encoder layers in the encoder block.
  lambda_attention: float
      Constant to be multiplied with the attention matrix in the self-attention layer.
  lambda_distance: float
      Constant to be multiplied with the distance matrix in the self-attention layer.
  h: int
      Number of attention heads for the self-attention layer.
  sa_hsize: int
      Size of dense layer in the self-attention layer.
  sa_dropout_p: float
      Dropout probability for the self-attention layer.
  output_bias: bool
      If True, dense layers will use bias vectors in the self-attention layer.
  d_input: int
      Size of input layer in the feed-forward layer.
  d_hidden: int
      Size of hidden layer in the feed-forward layer. Will also be used as d_output for the MATEmbedding layer.
  d_output: int
      Size of output layer in the feed-forward layer.
  activation: str
      Activation function to be used in the feed-forward layer.
      Can choose between 'relu' for ReLU, 'leakyrelu' for LeakyReLU, 'prelu' for PReLU,
      'tanh' for TanH, 'selu' for SELU, 'elu' for ELU and 'linear' for linear activation.
  n_layers: int
      Number of layers in the feed-forward layer.
  ff_dropout_p: float
      Dropout probability in the feeed-forward layer.
  encoder_hsize: int
      Size of Dense layer for the encoder itself.
  encoder_dropout_p: float
      Dropout probability for connections in the encoder layer.
  embed_input_hsize: int
      Size of input layer for the MATEmbedding layer.
  embed_dropout_p: float
      Dropout probability for the MATEmbedding layer.
  gen_aggregation_type: str
      Type of aggregation to be used. Can be 'grover', 'mean' or 'contextual'.
  gen_dropout_p: float
      Dropout probability for the MATGenerator layer.
  gen_n_layers: int
      Number of layers in MATGenerator.
  gen_attn_hidden: int
      Size of hidden attention layer in the MATGenerator layer.
  gen_attn_out: int
      Size of output attention layer in the MATGenerator layer.
  gen_d_output: int
      Size of output layer in the MATGenerator layer.
docstr_tok: ["Initialization","for","the","internal","MAT","class",".","Parameters","--","--","--","--","--","dist_kernel",":","str","Kernel","activation","to","be","used",".","Can","be","either","'softmax","'","for","softmax","or","'exp","'","for","exponential",",","for","the","self-attention","layer",".","n_encoders",":","int","Number","of","encoder","layers","in","the","encoder","block",".","lambda_attention",":","float","Constant","to","be","multiplied","with","the","attention","matrix","in","the","self-attention","layer",".","lambda_distance",":","float","Constant","to","be","multiplied","with","the","distance","matrix","in","the","self-attention","layer",".","h",":","int","Number","of","attention","heads","for","the","self-attention","layer",".","sa_hsize",":","int","Size","of","dense","layer","in","the","self-attention","layer",".","sa_dropout_p",":","float","Dropout","probability","for","the","self-attention","layer",".","output_bias",":","bool","If","True",",","dense","layers","will","use","bias","vectors","in","the","self-attention","layer",".","d_input",":","int","Size","of","input","layer","in","the","feed-forward","layer",".","d_hidden",":","int","Size","of","hidden","layer","in","the","feed-forward","layer",".","Will","also","be","used","as","d_output","for","the","MATEmbedding","layer",".","d_output",":","int","Size","of","output","layer","in","the","feed-forward","layer",".","activation",":","str","Activation","function","to","be","used","in","the","feed-forward","layer",".","Can","choose","between","'relu","'","for","ReLU",",","'leakyrelu","'","for","LeakyReLU",",","'prelu","'","for","PReLU",",","'tanh","'","for","TanH",",","'selu","'","for","SELU",",","'elu","'","for","ELU","and","'linear","'","for","linear","activation",".","n_layers",":","int","Number","of","layers","in","the","feed-forward","layer",".","ff_dropout_p",":","float","Dropout","probability","in","the","feeed-forward","layer",".","encoder_hsize",":","int","Size","of","Dense","layer","for","the","encoder","itself",".","encoder_dropout_p",":","float","Dropout","probability","for","connections","in","the","encoder","layer",".","embed_input_hsize",":","int","Size","of","input","layer","for","the","MATEmbedding","layer",".","embed_dropout_p",":","float","Dropout","probability","for","the","MATEmbedding","layer",".","gen_aggregation_type",":","str","Type","of","aggregation","to","be","used",".","Can","be","'grover","'",",","'mean","'","or","'contextual","'",".","gen_dropout_p",":","float","Dropout","probability","for","the","MATGenerator","layer",".","gen_n_layers",":","int","Number","of","layers","in","MATGenerator",".","gen_attn_hidden",":","int","Size","of","hidden","attention","layer","in","the","MATGenerator","layer",".","gen_attn_out",":","int","Size","of","output","attention","layer","in","the","MATGenerator","layer",".","gen_d_output",":","int","Size","of","output","layer","in","the","MATGenerator","layer","."]
code: def __init__(
      self,
      dist_kernel: str = "softmax",
      n_encoders=8,
      lambda_attention: float = 0.33,
      lambda_distance: float = 0.33,
      h: int = 16,
      sa_hsize: int = 1024,
      sa_dropout_p: float = 0.0,
      output_bias: bool = True,
      d_input: int = 1024,
      d_hidden: int = 1024,
      d_output: int = 1024,
      activation: str = "leakyrelu",
      n_layers: int = 1,
      ff_dropout_p: float = 0.0,
      encoder_hsize: int = 1024,
      encoder_dropout_p: float = 0.0,
      embed_input_hsize: int = 36,
      embed_dropout_p: float = 0.0,
      gen_aggregation_type: str = "mean",
      gen_dropout_p: float = 0.0,
      gen_n_layers: int = 1,
      gen_attn_hidden: int = 128,
      gen_attn_out: int = 4,
      gen_d_output: int = 1,
      **kwargs,
  ):
      """
      Initialization for the internal MAT class.
  
      Parameters
      ----------
      dist_kernel: str
          Kernel activation to be used. Can be either 'softmax' for softmax or 'exp' for exponential, for the self-attention layer.
      n_encoders: int
          Number of encoder layers in the encoder block.
      lambda_attention: float
          Constant to be multiplied with the attention matrix in the self-attention layer.
      lambda_distance: float
          Constant to be multiplied with the distance matrix in the self-attention layer.
      h: int
          Number of attention heads for the self-attention layer.
      sa_hsize: int
          Size of dense layer in the self-attention layer.
      sa_dropout_p: float
          Dropout probability for the self-attention layer.
      output_bias: bool
          If True, dense layers will use bias vectors in the self-attention layer.
      d_input: int
          Size of input layer in the feed-forward layer.
      d_hidden: int
          Size of hidden layer in the feed-forward layer. Will also be used as d_output for the MATEmbedding layer.
      d_output: int
          Size of output layer in the feed-forward layer.
      activation: str
          Activation function to be used in the feed-forward layer.
          Can choose between 'relu' for ReLU, 'leakyrelu' for LeakyReLU, 'prelu' for PReLU,
          'tanh' for TanH, 'selu' for SELU, 'elu' for ELU and 'linear' for linear activation.
      n_layers: int
          Number of layers in the feed-forward layer.
      ff_dropout_p: float
          Dropout probability in the feeed-forward layer.
      encoder_hsize: int
          Size of Dense layer for the encoder itself.
      encoder_dropout_p: float
          Dropout probability for connections in the encoder layer.
      embed_input_hsize: int
          Size of input layer for the MATEmbedding layer.
      embed_dropout_p: float
          Dropout probability for the MATEmbedding layer.
      gen_aggregation_type: str
          Type of aggregation to be used. Can be 'grover', 'mean' or 'contextual'.
      gen_dropout_p: float
          Dropout probability for the MATGenerator layer.
      gen_n_layers: int
          Number of layers in MATGenerator.
      gen_attn_hidden: int
          Size of hidden attention layer in the MATGenerator layer.
      gen_attn_out: int
          Size of output attention layer in the MATGenerator layer.
      gen_d_output: int
          Size of output layer in the MATGenerator layer.
      """
  
      super(MAT, self).__init__()
  
      self.embedding = layers.MATEmbedding(
          d_input=embed_input_hsize,
          d_output=d_hidden,
          dropout_p=embed_dropout_p,
      )
  
      self.encoder = nn.ModuleList(
          [
              layers.MATEncoderLayer(
                  dist_kernel=dist_kernel,
                  lambda_attention=lambda_attention,
                  lambda_distance=lambda_distance,
                  h=h,
                  sa_hsize=sa_hsize,
                  sa_dropout_p=sa_dropout_p,
                  output_bias=output_bias,
                  d_input=d_input,
                  d_hidden=d_hidden,
                  d_output=d_output,
                  activation=activation,
                  n_layers=n_layers,
                  ff_dropout_p=ff_dropout_p,
                  encoder_hsize=encoder_hsize,
                  encoder_dropout_p=encoder_dropout_p,
              )
              for _ in range(n_encoders)
          ]
      )
  
      self.generator = layers.MATGenerator(
          hsize=d_input,
          aggregation_type=gen_aggregation_type,
          d_output=gen_d_output,
          n_layers=gen_n_layers,
          dropout_p=gen_dropout_p,
          attn_hidden=gen_attn_hidden,
          attn_out=gen_attn_out,
      )
  
file_path: deepchem/deepchem/models/torch_models/mat.py
imports: import torch
  import torch.nn
  import numpy
  from deepchem.models.torch_models import layers
  from deepchem.models.torch_models.torch_model import TorchModel
  from deepchem.models.losses import L2Loss
  from typing import Any
label_desc: Use image node_id 1 to create a new MAT object from inherited base classes: nn with example: obj = MAT(dist_kernel, n_encoders, lambda_attention, lambda_distance, h, sa_hsize, sa_dropout_p, output_bias, d_input, d_hidden, d_output, activation, n_layers, ff_dropout_p, encoder_hsize, encoder_dropout_p, embed_input_hsize, embed_dropout_p, gen_aggregation_type, gen_dropout_p, gen_n_layers, gen_attn_hidden, gen_attn_out, gen_d_output)