How to initialise the Weights for specific task and Backpropagation modification
My model is used to predict values based on an minimising a loss function L. But, the loss function doesn’t have a single global minima value, but rather a large number of places where it achieves global minima.
So, the model is based like this:
Model Input is [nXn] tensor (let’s say: inp=[ [i_11, i_12, i_13, ..., i_1n],[i_21, i_22, ..., i_2n],...,[i_n1,i_n2, ..., i_nn] ]) and model output is [nX1] tensor (let’s say: out1=[o_1, o_2,..., o_n ])
Output tensor is out1 is passed in a function f to get out2 (let’s say: f(o_1, o_2, o_3,..., o_n)=[O_1, O_2, O_3, ..., O_n] )
These 2 values (i.e., out1 and out2) are minimised using MSELoss i.e., Loss = ||out1 - out2||
Now, there are a lot of values for [o_1, o_2, ..., o_n] for which the Loss goes to minimum.
But, I want the values of [o_1, o_2, ..., o_n] for which |o_1| + |o_2| + |o_3| + ... + |o_n| is maximum
Right now, the weights are initialised randomly:
self.weight = torch.nn.parameter.Parameter(torch.FloatTensor(in_features, out_features)) for some value of in_features and out_features

But by doing this, I am getting the values of [o_1, o_2, ..., o_n] for which |o_1| + |o_2| + |o_3| + ... + |o_n| is minimum.
I know this problem can be solved by without using deep-learning, but I am trying to get the results like this for some task computation.
Is there a way to change this to get the largest values predicted at the output of the neural net?
Or is there any other technique (backpropagation change) to change it to get the desired largest valued output?
Thanks in advance.
EDIT 1:
Based on the answer, out1=[o_1, o_2,..., o_n ] is tending to zero-valued tensor. In the initial epochs, out2=[O_1, O_2, O_3, ..., O_n] takes very large values, but subsequently comes down to lower values.
A snippet of code below will give the idea:
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import numpy as np


class Model(nn.Module):
    def __init__(self, inp_l, hid_l, out_l=1):
        super(Model, self).__init__()
     
        self.lay1 = nn.Linear(inp_l ,hid_l)
        self.lay2 = nn.Linear(hid_l ,out_l)
        self.dp = nn.Dropout(p=0.5)
        
    def forward(self, inp):
                
        self.out1= torch.tensor([]).float()
        
        for row in range(x.shape[0]):
            y = self.lay1(inp[row])
            y = F.relu(y)
            y = self.dp(y.float())
            y = self.lay2(y)
            y = F.relu(y)
            self.out1= torch.cat((self.out1, y))
            
        return self.out1.view(inp.shape[0],-1)

def function_f(inp, out1):
    '''
    Some functional computation is done to return out2.
    '''
    return out2

def train_model(epoch):
    model.train()
    t = time.time()
    optimizer.zero_grad()
    out1 = model(inp)
    
    out2 = function_f(inp, out1)
    
    loss1 = ((out1-out2)**2).mean()
    loss2 = -out1.abs().mean()
    loss_train = loss1 + loss2
    
    loss_train.backward(retain_graph=True)

    optimizer.step()
    
    if epoch%40==0:
        print('Epoch: {:04d}'.format(epoch+1),
                  'loss_train: {:.4f}'.format(loss_train.item()),
                  'time: {:.4f}s'.format(time.time() - t))

model= Model(inp_l=10, hid_l=5, out_l=1)
optimizer = optim.Adam(model.parameters(), lr=0.001)

inp = torch.randint(100, (10, 10))

for ep in range(100):
       train_model(ep)


But, out1 value goes to trivial solution i.e., zero-valued tensor which is the minimum valued solution. As mentioned before EDIT, I want to get the max-valued solution.
Thank you.

I am not sure I understand what you want.
Your weight initialization is overly complicated as well, you may just do:
self.weight = torch.nn.Linear(in_features, out_featues)

If you want to have the largest value of a batch of inputs you may simply do:
    y = self.weight(x)
    return y.max(dim=0)[0]

But I am not entirely sure that is what you meant with your question.
EDIT:
It seems you have two objectives. The first thing I would try is to convert both of them in losses to be minimized by the optimizer.
    loss1 = MSE(out1, out2)
    loss2 = - out1.abs().mean()
    loss = loss1 + loss2

minimizing loss will simutaneously minimize the MSE between out1 and out2 and maximize the absolute values of out1. (minimizing - out1.abs().mean() is the same as maximizing out1.abs().mean()).
Notice that it is possible your neural net will just create large biases and zero the weights as a lazy solution for the objective. You may turn of biases to avoid the problem, but I would still expect some other training problems.
