Cannot improve model accuracy
I am building a general-purpose NN that would classify images (Dog/No Dog) and movie reviews(Good/Bad). I have to stick to a very specific architecture and loss function so changing these two seems out of the equation. My architecture is a two-layer network with relu followed by a sigmoid and a cross-entropy loss function. With 1000 epochs and a learning rate of around .001 I am getting 100 percent training accuracy and .72 testing accuracy.I was looking for suggestions to improve my testing accuracy.This is the layout of what I have:

def train_net(epochs,batch_size,train_x,train_y,model_size,lr):
  n_x,n_h,n_y=model_size
  model = Net(n_x, n_h, n_y)
  optim = torch.optim.Adam(model.parameters(),lr=0.005)
  loss_function = nn.BCELoss() 
  train_losses = []
  accuracy = []
  for epoch in range(epochs):
    count=0
    model.train() 
    train_loss = []
    batch_accuracy = []
    for idx in range(0, train_x.shape[0], batch_size):

      batch_x = torch.from_numpy(train_x[idx : idx + batch_size]).float() 
      batch_y = torch.from_numpy(train_y[:,idx : idx + batch_size]).float()    
      model_output = model(batch_x) 
      batch_accuracy=[]  
      loss = loss_function(model_output, batch_y) 
      train_loss.append(loss.item())

      preds = model_output &gt; 0.5
      nb_correct = (preds == batch_y).sum()
      count+=nb_correct.item()                            
      optim.zero_grad()
      loss.backward()
# Scheduler made it worse 
# scheduler.step(loss.item())  
  optim.step()

    if epoch % 100 == 1:
      train_losses.append(train_loss)
      print("Iteration : {}, Training loss: {} ,Accuracy %: {}".format(epoch,np.mean(train_loss),(count/train_x.shape[0])*100))              
  plt.plot(np.squeeze(train_losses))
  plt.ylabel('loss')
  plt.xlabel('iterations (per tens)')
  plt.title("Learning rate =" + str(lr))
  plt.show()
  return model


My model parameters:

batch_size = 32
lr = 0.0001
epochs = 1500

n_x = 12288     # num_px * num_px * 3
n_h = 7
n_y = 1
model_size=n_x,n_h,n_y
model=train_net(epochs,batch_size,train_x,train_y,model_size,or)


and this is the testing phase.

model.eval()  #Setting the model to eval mode, hence making it deterministic.
test_loss = []
count=0;
loss_function = nn.BCELoss()

for idx in range(0, test_x.shape[0], batch_size):
  with torch.no_grad():   
    batch_x = torch.from_numpy(test_x[idx : idx + batch_size]).float() 
    batch_y = torch.from_numpy(test_y[:,idx : idx + batch_size]).float()
    model_output = model(batch_x)
    preds = model_output &gt; 0.5
    loss = loss_function(model_output, batch_y)
    test_loss.append(loss.item())
    nb_correct = (preds == batch_y).sum()
    count+=nb_correct.item()  

print("test loss: {},test accuracy: {}".format(np.mean(test_loss),count/test_x.shape[0]))


Things I have tried:
Messing around with the learning rate, having momentum, using schedulers and changing batch sizes.Of course these were mainly guesses and not based on any valid assumptions.

The issue you're facing is overfitting. With 100% accuracy on the training set, your model is effectively memorizing the training set, then failing to generalize to unseen samples. The good news is this is a very common major challenge!

You need regularization. One method is dropout, whereby on different training epochs a random set of the NN connections are dropped, forcing the network to "learn" alternate pathways and weights, and softening sharp peaks in parameter space. Since you need to keep your architecture and loss function the same, you won't be able to add such an option in (though for completeness, read this article for a description and implementation of dropout in PyTorch).

Given your constraints, you'll want to use something like L2 or L1 weight regularization. This typically shows up in the way of adding an additional term to the cost/loss function, which penalizes large weights. In PyTorch, L2 regularization is implemented via the torch.optim construct, with the option weight_decay. (See documentation: torch.optim, search for 'L2')

For your code, try something like:

def train_net(epochs,batch_size,train_x,train_y,model_size,lr):
  ...
  optim = torch.optim.Adam(model.parameters(),...,weight_decay=0.01)
  ...

