How to find a good LR?
I'm very new to PyTorch and working on a project of mine to solve a Sudoku board. What I'm doing is giving the network a Tensor that has the board (9x9) and another 2 values, the first being the row and the second being the column.
My network:

class Network(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(9 * 9 + 2, 32)  # board + row and col
        self.fc2 = nn.Linear(32, 32)
        self.fc3 = nn.Linear(32, 9)  # the number in that spot

    def forward(self, x):
        x = f.relu(self.fc1(x))
        x = f.relu(self.fc2(x))
        x = self.fc3(x)
        return f.log_softmax(x, dim=1)



I have a for loop that iterates over each board, and then iterate on each block in the 9x9 grid and input the board with the row and column to the network.

optimizer = optim.Adam(net.parameters(), lr=0.1)
scheduler = lr_scheduler.CosineAnnealingLR(optimizer, len(quizzes), eta_min=0)
for i, board in enumerate(quizzes):
    new_board = [[val for val in row] for row in board]  # Don't affect original board
    for row in range(9):
        for col in range(9):
            if new_board[row][col] != 0:
                continue
            row_col[0] = row  # the row value
            row_col[1] = col  # the col value

            final_tensor = board_tensor.view(-1, 9 * 9 + 2)
            output = net(final_tensor)  # type: Tensor
            solution_num = solutions[i][row][col]
            solution_tensor = torch.tensor(solution_num - 1, dtype=torch.long).reshape(-1)  # do -1 because it needs to match the node.
            optimizer.zero_grad()
            loss = f.nll_loss(output, solution_tensor)
            loss.backward()
            optimizer.step()
            scheduler.step(epoch=epoch)
            new_board[row][col] = solution_num  # Add the new value into the board.
            avg_loss += loss.item()
            count += 1
    if i % 100 == 0:
        print(f"Loss: {round(avg_loss / count, 3)}. {i} / {len(quizzes)}. {epoch} / {EPOCHS}")
        avg_loss = 0
        count = 0


Now, my issue is that the network is just guessing the same number over and over. Every time I reset the network it guesses a different number, but after a bit it just stays constant and doesn't guess any other number. This of course makes the accuracy 11.111% (1/9) and I don't know how to get past this. I tried using MSELoss instead of NLL_loss but it didn't change any result, and switched between optim.Adam to optim.SGE. I am quite new to this whole topic so I don't know which functions I should be using (log_softmax, Adam, SGE, and all those types of functions for loss / optimization).

Does anyone know where I messed up? I tried changing the learning rate and adding a weight decay too, but that didn't help

I don't suspect your issue is with parameter tuning. I suspect that the model is unable to learn how to solve this problem in this way.

To start: given a single cell on an unsolved (but let's assume solvable) Sudoku board, it's likely that other cells need to be solved in order to be able to know the correct value for the current cell. Asking your network to implicitly solve the entire puzzle to write down the answer for a single cell doesn't really make sense. Further, I'm not sure this task is something ML can solve in the approach you're using.

If you really have your heart set on solving this using ML, add a tenth "I don't know the answer" class. Then alter your dataset to be aware of when a given cell is not yet knowable. This will still end up with a model that is limited to the skill level of the person creating the dataset though. This will also be a non-trivial amount of extra work and I think there are better ways for you to spend your time learning about ML.

Side note: Sudoku is a graph colouring problem. Or depending on how you want to deal with it, a constrained integer program. Kinda weird to use ML to solve a solved problem IMHO.
