paddlenlp/ops/faster_transformer/transformer/encoder.py
def encoder_layer_forward(self,
                          src,
                          src_mask,
                          cache=None,
                          sequence_id_offset=None,
                          trt_seq_len=None):
    """
    Redefines `forward` function of `paddle.nn.TransformerEncoderLayer` for
    integrating FasterTransformer for inference.

    The original `forward` function would not be replaced unless
    `enable_faster_encoder` is called by objects of its base class. After
    replacing, objects of `paddle.nn.TransformerEncoderLayer` also have the
    same member variables as before.

    After inference, `disable_faster_encoder` could be called to restore the
    `forward` function of `paddle.nn.TransformerEncoder` and
    `paddle.nn.TransformerEncoderLayer`.

    Args:
        src (Tensor):
            The input of Transformer encoder layer. It is a tensor with shape
            `[batch_size, sequence_length, d_model]`. The data type should be
            float32 or float64.
        src_mask (Tensor, optional):
            A tensor used in multi-head attention to prevents attention to some
            unwanted positions, usually the paddings or the subsequent
            positions. It is a tensor with shape `[batch_size, 1, 1, sequence_length]`.
            When the data type is bool, the unwanted positions have `False`
            values and the others have `True` values. When the data type is int,
            the unwanted positions have 0 values and the others have 1 values.
            When the data type is float, the unwanted positions have `-INF`
            values and the others have 0 values. It can be None when nothing
            wanted or needed to be prevented attention to. Defaults to None.

    Returns:
        src(Tensor|tuple):
            It is a tensor that has the same shape and data type as `enc_input`,
            representing the output of Transformer encoder layer. Or a tuple if
            `cache` is not None, except for encoder layer output, the tuple
            includes the new cache which is same as input `cache` argument but
            `incremental_cache` has an incremental length. See
            `paddle.nn.MultiHeadAttention.gen_cache` and
            `paddle.nn.MultiHeadAttention.forward` for more details.
    """
    if cache is not None:
        raise NotImplementedError("cache in encoder is not supported now")

    src = infer_transformer_encoder(
        input=src,
        q_weight=self.self_attn.q_proj.weight,
        q_bias=self.self_attn.q_proj.bias,
        k_weight=self.self_attn.k_proj.weight,
        k_bias=self.self_attn.k_proj.bias,
        v_weight=self.self_attn.v_proj.weight,
        v_bias=self.self_attn.v_proj.bias,
        attn_out_weight=self.self_attn.out_proj.weight,
        attn_out_bias=self.self_attn.out_proj.bias,
        attn_mask=src_mask,
        norm1_weight=self.norm1.weight,
        norm1_bias=self.norm1.bias,
        norm2_weight=self.norm2.weight,
        norm2_bias=self.norm2.bias,
        ffn_inter_weight=self.linear1.weight,
        ffn_inter_bias=self.linear1.bias,
        ffn_out_weight=self.linear2.weight,
        ffn_out_bias=self.linear2.bias,
        # sequence_id_offset=paddle.to_tensor([]),
        # trt_seqlen_offset=paddle.to_tensor([]),
        # amax_list=paddle.to_tensor([]),  # int8 mode is not supported.
        n_head=self._config['nhead'],
        size_per_head=self._config['d_model'] // self._config['nhead'],
        use_gelu=self._config['activation'] == 'gelu',
        normalize_before=self._config['normalize_before'] == True)

    return src

    Redefines `forward` function of `paddle.nn.TransformerEncoderLayer` for
    integrating FasterTransformer for inference.

    The original `forward` function would not be replaced unless
    `enable_faster_encoder` is called by objects of its base class. After
    replacing, objects of `paddle.nn.TransformerEncoderLayer` also have the
    same member variables as before.

    After inference, `disable_faster_encoder` could be called to restore the
    `forward` function of `paddle.nn.TransformerEncoder` and
    `paddle.nn.TransformerEncoderLayer`.

    Args:
        src (Tensor):
            The input of Transformer encoder layer. It is a tensor with shape
            `[batch_size, sequence_length, d_model]`. The data type should be
            float32 or float64.
        src_mask (Tensor, optional):
            A tensor used in multi-head attention to prevents attention to some
            unwanted positions, usually the paddings or the subsequent
            positions. It is a tensor with shape `[batch_size, 1, 1, sequence_length]`.
            When the data type is bool, the unwanted positions have `False`
            values and the others have `True` values. When the data type is int,
            the unwanted positions have 0 values and the others have 1 values.
            When the data type is float, the unwanted positions have `-INF`
            values and the others have 0 values. It can be None when nothing
            wanted or needed to be prevented attention to. Defaults to None.

    Returns:
        src(Tensor|tuple):
            It is a tensor that has the same shape and data type as `enc_input`,
            representing the output of Transformer encoder layer. Or a tuple if
            `cache` is not None, except for encoder layer output, the tuple
            includes the new cache which is same as input `cache` argument but
            `incremental_cache` has an incremental length. See
            `paddle.nn.MultiHeadAttention.gen_cache` and
            `paddle.nn.MultiHeadAttention.forward` for more details.
    
Redefines `forward` function of `paddle.nn.TransformerEncoderLayer` for
integrating FasterTransformer for inference.

The original `forward` function would not be replaced unless
`enable_faster_encoder` is called by objects of its base class. After
replacing, objects of `paddle.nn.TransformerEncoderLayer` also have the
same member variables as before.


Redefines ` forward ` function of ` paddle . nn . TransformerEncoderLayer ` for integrating FasterTransformer for inference . The original ` forward ` function would not be replaced unless ` enable_faster_encoder ` is called by objects of its base class . After replacing objects of ` paddle . nn . TransformerEncoderLayer ` also have the same member variables as before .
def encoder_layer_forward(self,
                          src,
                          src_mask,
                          cache=None,
                          sequence_id_offset=None,
                          trt_seq_len=None):
    if cache is not None:
        raise NotImplementedError("cache in encoder is not supported now")
    src = infer_transformer_encoder(
        input=src,
        q_weight=self.self_attn.q_proj.weight,
        q_bias=self.self_attn.q_proj.bias,
        k_weight=self.self_attn.k_proj.weight,
        k_bias=self.self_attn.k_proj.bias,
        v_weight=self.self_attn.v_proj.weight,
        v_bias=self.self_attn.v_proj.bias,
        attn_out_weight=self.self_attn.out_proj.weight,
        attn_out_bias=self.self_attn.out_proj.bias,
        attn_mask=src_mask,
        norm1_weight=self.norm1.weight,
        norm1_bias=self.norm1.bias,
        norm2_weight=self.norm2.weight,
        norm2_bias=self.norm2.bias,
        ffn_inter_weight=self.linear1.weight,
        ffn_inter_bias=self.linear1.bias,
        ffn_out_weight=self.linear2.weight,
        ffn_out_bias=self.linear2.bias,
        n_head=self._config['nhead'],
        size_per_head=self._config['d_model'] // self._config['nhead'],
        use_gelu=self._config['activation'] == 'gelu',
        normalize_before=self._config['normalize_before'] == True)
    return src
def encoder_layer_forward ( self , src , src_mask , cache = None , sequence_id_offset = None , trt_seq_len = None ) : if cache is not None : raise NotImplementedError ( "cache in encoder is not supported now" ) src = infer_transformer_encoder ( input = src , q_weight = self . self_attn . q_proj . weight , q_bias = self . self_attn . q_proj . bias , k_weight = self . self_attn . k_proj . weight , k_bias = self . self_attn . k_proj . bias , v_weight = self . self_attn . v_proj . weight , v_bias = self . self_attn . v_proj . bias , attn_out_weight = self . self_attn . out_proj . weight , attn_out_bias = self . self_attn . out_proj . bias , attn_mask = src_mask , norm1_weight = self . norm1 . weight , norm1_bias = self . norm1 . bias , norm2_weight = self . norm2 . weight , norm2_bias = self . norm2 . bias , ffn_inter_weight = self . linear1 . weight , ffn_inter_bias = self . linear1 . bias , ffn_out_weight = self . linear2 . weight , ffn_out_bias = self . linear2 . bias , n_head = self . _config [ 'nhead' ] , size_per_head = self . _config [ 'd_model' ] // self . _config [ 'nhead' ] , use_gelu = self . _config [ 'activation' ] == 'gelu' , normalize_before = self . _config [ 'normalize_before' ] == True ) return src
Redefines `forward` function of `paddle.nn.TransformerEncoderLayer` for
integrating FasterTransformer for inference.
Redefines ` forward ` function of ` paddle . nn . TransformerEncoderLayer ` for integrating FasterTransformer for inference .
"""
    Redefines `forward` function of `paddle.nn.TransformerEncoderLayer` for
    integrating FasterTransformer for inference.

    The original `forward` function would not be replaced unless
    `enable_faster_encoder` is called by objects of its base class. After
    replacing, objects of `paddle.nn.TransformerEncoderLayer` also have the
    same member variables as before.

    After inference, `disable_faster_encoder` could be called to restore the
    `forward` function of `paddle.nn.TransformerEncoder` and
    `paddle.nn.TransformerEncoderLayer`.

    Args:
        src (Tensor):
            The input of Transformer encoder layer. It is a tensor with shape
            `[batch_size, sequence_length, d_model]`. The data type should be
            float32 or float64.
        src_mask (Tensor, optional):
            A tensor used in multi-head attention to prevents attention to some
            unwanted positions, usually the paddings or the subsequent
            positions. It is a tensor with shape `[batch_size, 1, 1, sequence_length]`.
            When the data type is bool, the unwanted positions have `False`
            values and the others have `True` values. When the data type is int,
            the unwanted positions have 0 values and the others have 1 values.
            When the data type is float, the unwanted positions have `-INF`
            values and the others have 0 values. It can be None when nothing
            wanted or needed to be prevented attention to. Defaults to None.

    Returns:
        src(Tensor|tuple):
            It is a tensor that has the same shape and data type as `enc_input`,
            representing the output of Transformer encoder layer. Or a tuple if
            `cache` is not None, except for encoder layer output, the tuple
            includes the new cache which is same as input `cache` argument but
            `incremental_cache` has an incremental length. See
            `paddle.nn.MultiHeadAttention.gen_cache` and
            `paddle.nn.MultiHeadAttention.forward` for more details.
    """ # sequence_id_offset=paddle.to_tensor([]), # trt_seqlen_offset=paddle.to_tensor([]), # amax_list=paddle.to_tensor([]),  # int8 mode is not supported.
It is a tensor that has the same shape and data type as `enc_input`,
representing the output of Transformer encoder layer. Or a tuple if
`cache` is not None, except for encoder layer output, the tuple
includes the new cache which is same as input `cache` argument but
`incremental_cache` has an incremental length. See
`paddle.nn.MultiHeadAttention.gen_cache` and
`paddle.nn.MultiHeadAttention.forward` for more details.
It is a tensor that has the same shape and data type as ` enc_input ` representing the output of Transformer encoder layer . Or a tuple if ` cache ` is not None except for encoder layer output the tuple includes the new cache which is same as input ` cache ` argument but ` incremental_cache ` has an incremental length . See ` paddle . nn . MultiHeadAttention . gen_cache ` and ` paddle . nn . MultiHeadAttention . forward ` for more details .
src(Tensor|tuple)
self
src
The input of Transformer encoder layer. It is a tensor with shape
`[batch_size, sequence_length, d_model]`. The data type should be
float32 or float64.
The input of Transformer encoder layer . It is a tensor with shape ` [ batch_size sequence_length d_model ] ` . The data type should be float32 or float64 .
src_mask
A tensor used in multi-head attention to prevents attention to some
unwanted positions, usually the paddings or the subsequent
positions. It is a tensor with shape `[batch_size, 1, 1, sequence_length]`.
When the data type is bool, the unwanted positions have `False`
values and the others have `True` values. When the data type is int,
the unwanted positions have 0 values and the others have 1 values.
When the data type is float, the unwanted positions have `-INF`
values and the others have 0 values. It can be None when nothing
wanted or needed to be prevented attention to. Defaults to None.
A tensor used in multi - head attention to prevents attention to some unwanted positions usually the paddings or the subsequent positions . It is a tensor with shape ` [ batch_size 1 1 sequence_length ] ` . When the data type is bool the unwanted positions have ` False ` values and the others have ` True ` values . When the data type is int the unwanted positions have 0 values and the others have 1 values . When the data type is float the unwanted positions have ` - INF ` values and the others have 0 values . It can be None when nothing wanted or needed to be prevented attention to . Defaults to None .
cache
sequence_id_offset
trt_seq_len