bias dimension definition in coding neural network
In the following Figure showing the code for defining the dimension of the bias b1 term, I wonder why the first dimension of bias b1 is not the batch size? Does it mean then it just assumes this bias is applied to all batches then? 

If I specify the bias b1 dimension to be (batch_size, 256) then does it mean i am applying a different b1 to different batch? But theoretically it should still work right? Also what is the difference between tensor (256), (256,) and (256,1)...? 

Figure: dimension definition of nn

The weights and biases of your neural network layer are not specified in terms of batch size.

eg: w1 = torch.randn(784,256) :
This is a 2D matrix you're going to use for a matrix multiply.
784 is the dimension of your input image without considering batch size. (I'm guessing this is for mnist? it looks like you're flattening the 2d images to a 1d vector so 28*28=784).

256 is your output dimension of your output (how many logits you're using)

Similarly, b1 = torch.randn(256):
This is an 1D vector you're just adding to the logits.

256 is the dimension of the logits

Pytorch automaticallly broadcasts (repeats) these over the batch dimension for all your operations, so it doesn't matter what the batch size was.

Eg. eg in adding, b1 is automatically repeated over the first dimension, so it's actual shape for the add is (batch_size, 256).

By convention, pytorch "aligns" dimensions from right to left.


if any are missing, it then repeats the tensor over the missing dimension
If any dimension is 1, it repeats the tensor over that dimension to match the other operand.
Eg (copied from the docs on broadcasting)


&gt;&gt;&gt; x=torch.empty(5,7,3)
&gt;&gt;&gt; y=torch.empty(5,7,3)
# same shapes are always broadcastable (i.e. the above rules always hold)

&gt;&gt;&gt; x=torch.empty((0,))
&gt;&gt;&gt; y=torch.empty(2,2)
# x and y are not broadcastable, because x does not have at least 1 dimension

# can line up trailing dimensions
&gt;&gt;&gt; x=torch.empty(5,3,4,1)
&gt;&gt;&gt; y=torch.empty(  3,1,1)
# x and y are broadcastable.
# 1st trailing dimension: both have size 1
# 2nd trailing dimension: y has size 1
# 3rd trailing dimension: x size == y size
# 4th trailing dimension: y dimension doesn't exist

# but:
&gt;&gt;&gt; x=torch.empty(5,2,4,1)
&gt;&gt;&gt; y=torch.empty(  3,1,1)
# x and y are not broadcastable, because in the 3rd trailing dimension 2 != 3


This is really convenient because it means you don't have to redefine your neural net every time you want to use a different batch_size

Here's a link if you want to learn more about broadcasting in pytorch




  Also what is the difference between tensor (256), (256,) and (256,1)


the first two are exactly the same; python generally allows for trailing commas in tuple expressions. You are creating a 1D vector of 256 elements.

The last one is different; you are creating a 2D tensor where the first dimension is 256 and the second dimension is 1. The underlying data is the same, and it doesn't matter as long as you're consistent about which you're using, but if you mix them, it often leads to undesired behavior:

Eg: 

a = torch.randn(256)
b = torch.randn(256)
c = a + b
c.shape
&gt;&gt;&gt; torch.Size([256])


Simple: they just add element-wise.

But notice what happens when one of them is shaped (-1,1):

b = b.view(-1,1) # -1 here means torch will infer the shape of this dimension based on the known size of the tensor and all other specified dimensions
b.shape
&gt;&gt;&gt; torch.Size([256, 1])

c = a + b


Now because of broadcasting rules


a is repeated over the first dimension so it has the same number of dimensions as b, so it automatically interpretes a as tensor(256,256)
b is repeated so it's last dimension (1) now matches the dimension of a (256) 


so:

c.shape
&gt;&gt;&gt; torch.Size([256, 256])




Hint: The broadcasting rules can be hard to remember, and are often the source of bugs. When in doubt about tensor shapes, it's worth running your code in an interpreter line by line with dummy data and just checking what the shape of each tensor is eg print(torch.mm(input,w1).shape)
