layer Normalization in pytorch?
shouldn't the layer normalization of x = torch.tensor([[1.5,0,0,0,0]]) be [[1.5,-0.5,-0.5,-0.5]] ? according to this paper paper and the equation from the pytorch doc. But the torch.nn.LayerNorm gives [[ 1.7320, -0.5773, -0.5773, -0.5773]]

Here is the example code:

x = torch.tensor([[1.5,.0,.0,.0]])
layerNorm = torch.nn.LayerNorm(4, elementwise_affine = False)

y1 = layerNorm(x)

mean = x.mean(-1, keepdim = True)
var = x.var(-1, keepdim = True)
y2 = (x-mean)/torch.sqrt(var+layerNorm.eps)


where:

y1 == tensor([[ 1.7320, -0.5773, -0.5773, -0.5773]])
y2 == tensor([[ 1.5000, -0.5000, -0.5000, -0.5000]])


Yet another simplified implementation of a Layer Norm layer with bare PyTorch.
from typing import Tuple
import torch


def layer_norm(
    x: torch.Tensor, dim: Tuple[int], eps: float = 0.00001
) -&gt; torch.Tensor:
    mean = torch.mean(x, dim=dim, keepdim=True)
    var = torch.square(x - mean).mean(dim=dim, keepdim=True)
    return (x - mean) / torch.sqrt(var + eps)


def test_that_results_match() -&gt; None:
    dims = (1, 2)
    X = torch.normal(0, 1, size=(3, 3, 3))

    indices = torch.tensor(dims)
    normalized_shape = torch.tensor(X.size()).index_select(0, indices)
    orig_layer_norm = torch.nn.LayerNorm(normalized_shape)

    y = orig_layer_norm(X)
    y_hat = layer_norm(X, dim=dims)

    assert torch.allclose(y, y_hat)

Note, that original implementation also has trainable parameters  and β (see the docs).
