Have you encountered the similar problem like loss jitter during training?
Background: It's about loss jittering which generates at the beginning stage of every training epoch. When the dataloader loads the first batch data to feed into the network, the loss value always rises suddenly, then returns to normal from the second batch and continues to decline. The curve is so strange. I need your help!



    for epoch in range(begin_epoch, end_epoch):
        print('PROGRESS: %.2f%%' % (100.0 * epoch / end_epoch))

        # set epoch as random seed of sampler while distributed training
        if train_sampler is not None and hasattr(train_sampler, 'set_epoch'):
            train_sampler.set_epoch(epoch)

        # reset metrics
        metrics.reset()

        # set net to train mode
        net.train()

        # clear the paramter gradients
        # optimizer.zero_grad()

        # init end time
        end_time = time.time()

        if isinstance(lr_scheduler, torch.optim.lr_scheduler.ReduceLROnPlateau):
            name, value = validation_monitor.metrics.get()
            val = value[name.index(validation_monitor.host_metric_name)]
            lr_scheduler.step(val, epoch)

        # training
        train_loader_iter = iter(train_loader)
        for nbatch in range(total_size):
            try:
                batch = next(train_loader_iter)
            except StopIteration:
                print('reset loader .. ')
                train_loader_iter = iter(train_loader)
                batch = next(train_loader_iter)
            global_steps = total_size * epoch + nbatch

            os.environ['global_steps'] = str(global_steps)

            # record time
            data_in_time = time.time() - end_time

            # transfer data to GPU
            data_transfer_time = time.time()
            batch = to_cuda(batch)
            data_transfer_time = time.time() - data_transfer_time

            # forward
            forward_time = time.time()
            outputs, loss = net(*batch)
            loss = loss.mean()
            if gradient_accumulate_steps &gt; 1:
                loss = loss / gradient_accumulate_steps
            forward_time = time.time() - forward_time

            # backward
            backward_time = time.time()
            if fp16:
                with amp.scale_loss(loss, optimizer) as scaled_loss:
                    scaled_loss.backward()
            else:
                loss.backward()
            backward_time = time.time() - backward_time

            optimizer_time = time.time()
            if (global_steps + 1) % gradient_accumulate_steps == 0:

                # clip gradient
                if clip_grad_norm &gt; 0:
                    if fp16:
                        total_norm = torch.nn.utils.clip_grad_norm_(amp.master_params(optimizer),
                                                                    clip_grad_norm)
                    else:
                        total_norm = torch.nn.utils.clip_grad_norm_(net.parameters(),
                                                                    clip_grad_norm)
                    if writer is not None:
                        writer.add_scalar(tag='grad-para/Total-Norm',
                                        scalar_value=float(total_norm),
                                        global_step=global_steps)

                optimizer.step()
                
                # step LR scheduler
                if lr_scheduler is not None and not isinstance(lr_scheduler,
                                                            torch.optim.lr_scheduler.ReduceLROnPlateau):
                    lr_scheduler.step()

                # clear the parameter gradients
                optimizer.zero_grad()
            optimizer_time = time.time() - optimizer_time

            # update metric
            metric_time = time.time()
            metrics.update(outputs)
            if writer is not None and nbatch % 50 == 0:
                with torch.no_grad():
                    for group_i, param_group in enumerate(optimizer.param_groups):
                        writer.add_scalar(tag='Initial-LR/Group_{}'.format(group_i),
                                        scalar_value=param_group['initial_lr'],
                                        global_step=global_steps)
                        writer.add_scalar(tag='LR/Group_{}'.format(group_i),
                                        scalar_value=param_group['lr'],
                                        global_step=global_steps)
                    writer.add_scalar(tag='Train-Loss',
                                    scalar_value=float(loss.item()),
                                    global_step=global_steps)
                    name, value = metrics.get()
                    for n, v in zip(name, value):
                        if 'Logits' in n:
                            writer.add_scalar(tag='Train-Logits/' + n,
                                            scalar_value=v,
                                            global_step=global_steps)
                        else:
                            writer.add_scalar(tag='Train-' + n,
                                            scalar_value=v,
                                            global_step=global_steps)
                    for k, v in outputs.items():
                        if 'score' in k:
                            writer.add_histogram(tag=k,
                                                 values=v,
                                                 global_step=global_steps)

            metric_time = time.time() - metric_time


You have a batch in your dataset that have high loss, that's it. 

It is not that common that people store metrics for every batch, usually it is the average over epoch (or average over multiple batch steps) that is stored. You won't see such spikes if you will store averages.

You also could reduce these spikes by shuffling your data so that the problematic batch is spread out across the epoch. In general it is a good practice to do so at the beginning of each epoch.
