Why when the batch size increased, the epoch time will also increasing?
Epoch time means the time required to train for an epoch.
From my point of view, when the batch size increased, the epoch time will reduce. But actually I observed that when the batch size increased, the epoch time would reduce at first as we expect, but then increase. The relationship between batch size and epoch is like a v-shaped function. For example, when I change batch size from 1 to 256, the epoch time reduce monotonically. But when I change batch size from 256 to 1024, the epoch time will increase monotonically.
I've observed this so many times that I don't think it's a coincidence. I thought about the reason for this phenomenon from several aspects, but none of them were right.
Considering an simple case, one use batchsize 1024, other use batchsize 512. The calculation time of the former must be less than twice the calculation time of the latter, and same is true for the time to transfer data to the GPU. So whatever, the epoch time of batchsize1024 must be smaller than batchsize 512 instead of larger.
Is it because the algorithm adopted by pytorch is not effective in large matrices multiplication?
My configuration is as follows, I use pytorch, the training process is very standard, and I have observed this phenomenon too many times in different tasks, so I don't think it is accidental. My code is as follows.
               for epoch in range(1, total_epoch + 1):
                    for x, y in tqdm(self.loader):
                    with autocast():
                        loss = self.criterion(self.model(x), y)
                    self.optimizer.zero_grad()
                    scaler.scale(loss).backward()
                    scaler.unscale_(self.optimizer)
                    scaler.step(self.optimizer)
                    scaler.update()
                    self.optimizer.step()


As you already noticed, there are many factors that may affect epoch-time/batch-time.
Some of these factors may be specific to your machine's configuration. In order to get an accurate answer, you'll need to have a more detailed breakdown of the running time. One way of doing so would be using a profiler.
Try and be more specific about what is taking longer as batch size changes:

Loading data from disk
Moving data from CPU mem to device mem
Forward pass
Backward pass
Updating the model

Once you pinpoint the cause of latency, you can speculate as to the reason for it.
For instance, if loading the data from disk is the bottleneck, you might need more workers in the DataLoader as you increase batch size. In that case, you might be hitting a &quot;ceiling&quot; of the number of cores you have on your CPU.
Bottom line - there's not enough information in your question for a good answer.
