Instruction: Pytorch - extracting predicted values to an array

Input:
I'm a new pytorch user and moderate experience with Tensorflow/Keras. The pytorch examples are fantastic. I've worked through the demand forecasting lab using the Temporal Fusion Transform (https://pytorch-forecasting.readthedocs.io/en/latest/tutorials/stallion.html).
All makes sense but haven't figured out how to save the predicted values in notebook section #20 to a numpy array.
Section #20,
*new_raw_predictions, new_x = best_tft.predict(new_prediction_data, mode="raw", return_x=True)*
I see the values in the tensors, print(new_raw_predictions) ,
like this --
*{'prediction': tensor([[[3.4951e+00, 1.7341e+01,     2.7446e+01,  ..., 6.3175e+01,
9.0240e+01, 1.2589e+02],
[1.1698e+01, 2.3643e+01, 3.3291e+01,  ..., 6.6374e+01,
9.1148e+01, 1.3173e+02],
I've seen some similar questions asked here but none seem to work. All attempts result in a similar error so I'm missing something fundamental about pytorch and the output tensor; I always get 'AttributeError: 'dict' object has no attribute new_raw_predictions'
A few examples of what's been tried:
*new_raw_predictions.cpu().numpy() new_raw_predictions.detach().cpu().numpy() new_raw_predictions.numpy()*
Goal is to save the predicted output so I can compare changes to the model. Thanks in advance!

Output:
It all depends on how you've created your model, because pytorch can return values however you specify. In your case, it looks like it returns a dictionary, of which 'prediction' is a key. You can convert to numpy using the command you supplied above, but with one change:
preds = new_raw_predictions['prediction'].detach().cpu().numpy()

of course if it's not on the GPU you don't need to use .detach().cpu(), just .numpy()


Instruction: Avoid Loop for Selecting Dimensions in a Batch Tensor in PyTorch

Input:
I have a batch tensor and another tensor having indices of the dimensions to select from  batch tensor. At present, I am looping around batch tensor as shown below in the code snippet:
import torch

# create tensors to represent our data in torch format
batch_size = 8
batch_data = torch.rand(batch_size, 3, 240, 320)

# notice that channels_id has 8 elements, i.e., = batch_size
channels_id = torch.tensor([2, 0, 2, 1, 0, 2, 1, 0])

This is how I am selecting dimensions inside a for loop and then stacking to convert a single tensor:
batch_out = torch.stack([batch_i[channel_i] for batch_i, channel_i in zip(batch_data, channels_id)])
batch_out.size()  # prints torch.Size([8, 240, 320])

It works fine. However, is there a better PyTorch way to achieve the same?

Output:
As per the hint from @Shai, I could make it work using the torch.gather function. Below is the complete code:
import torch

# create tensors to represent our data in torch format
batch_size = 8
batch_data = torch.rand(batch_size, 3, 240, 320)

# notice that channels_id has 8 elements, i.e., batch_size
channels_id = torch.tensor([2, 0, 2, 1, 0, 2, 1, 0])

# resizing channels_id to (8 , 1, 240, 320)
channels_id = channels_id.view(-1, 1, 1, 1).repeat((1, 1) + batch_data.size()[-2:])

batch_out = torch.gather(batch_data, 1, channels_id).squeeze()
batch_out.size()  # prints torch.Size([8, 240, 320])


Instruction: How to test my own image on a MNIST trained network?

Input:
This is my first time trying to train a network and use PyTorch, so please forgive me if this is considered simple.
I have a pretrained AlexNet network that was modified to classify 3 classes, which I've already trained on MNIST that I mapped to 3 different labels.
class Net( nn.Module ) :
    def __init__( self ) :
        super( Net, self ).__init__()

        self.model = models.alexnet( pretrained = True )
        # changed in_channels from 3 to 1 bc images are black and white
        self.model.features[ 0 ] = nn.Conv2d( 1, 64, kernel_size = 11, stride = 4, padding = 2 )

        # binary classifier -> 3 out_features
        self.model.classifier[ 4 ] = nn.Linear( 4096, 1024 )
        self.model.classifier[ 6 ] = nn.Linear( 1024, 3 )

    def forward( self, x ):
        return self.model( x )

model = Net().to( device )

I want to test this on a single .png image that I drew, which is already 255x255, and in black and white. I would like the predicted label. This is the code I have so far for preprocessing the image:
from PIL import Image
import matplotlib.pyplot as plt
import cv2
image_8 = Image.open( "eight.png" ).convert('L')

image_8 = list( image_8.getdata())
normalized_8 = [(255 - x) * 1.0 / 255.0 for x in image_8 ]
tensor_8 = torch.FloatTensor( normalized_8 )

pred = model( tensor_8 )

from which I got the following error: Expected 4-dimensional input for 4-dimensional weight [64, 1, 11, 11], but got 1-dimensional input of size [50176] instead. So this is clearly the wrong way to do things, but I'm not sure how to proceed.

Output:
Change your inference code to the following. Images are not intended to be flattened into 1d.
import matplotlib.pyplot as plt
import cv2
image_8 = cv2.imread("eight.png")

# following line may or may not be necessary
image_8 = cv2.cvtColor(image_8, cv2.COLOR_BGR2GRAY)

# you can divide numpy arrays by a constant natively
image_8 /= 255.

# This makes a 4d tensor (batched image) with shape [1, channels, width, height]
image_8 = torch.Tensor(tensor_8).unsqueeze(axis=0)

pred = model(image_8)

If the image is still 3d (shape of [1, width, height]), add a second .unsqueeze(axis=0).


Instruction: using DQN to solve shortest path

Input:
I'm trying to find out if DQN can solve the shortest path algorithm
so I have this Dataframe which contains a source which has nodes id ,end which represents the destination and also has nodes id, and the weights which represent the distance of the edge and then I converted the data frame into a graph theory as following
DataFrame

    source  end weight
0   688615041   208456626   15.653688122127072
1   688615041   1799221665  10.092266065922756
2   1799221657  1799221660  8.673942902872051
3   1799221660  1799221665  15.282152665774992
4   1799221660  2003461246  25.85307821157314
5   1799221660  299832604   75.99884525624508
6   299832606   2003461227  4.510148061854331
7   299832606   2003461246  10.954119220974723
8   299832606   2364408910  4.903114362426424
9   1731824802  2003461235  6.812335798968233
10  1799221677  208456626   8.308567154008992
11  208456626   2003461246  14.56512909988425
12  208456626   1250468692  16.416527267975034
13  1011881546  1250468696  12.209773608913697
14  1011881546  2003461246  7.477102764665149
15  2364408910  1130166767  9.780352545373274
16  2364408910  2003461246  6.660771089602594
17  2364408910  2003461237  3.125301826317477
18  2364408911  2003461240  3.836966849565568
19  2364408911  2003461246  6.137847950353395
20  2364408911  2003461247  7.399469477211698
21  2364408911  2003461237  3.90876793066916
22  1250468692  1250468696  8.474825189804282
23  1250468701  2003461247  4.539111170687284
24  2003461235  2003461246  12.400601105777394
25  2003461246  2003461247  12.437602668573737

and the graph looks like this
pos = nx.spring_layout(g)
edge_labels = nx.get_edge_attributes(g, 'weight')
nx.draw(g, pos, node_size=100)
nx.draw_networkx_edge_labels(g, pos, edge_labels, font_size=8)
nx.draw_networkx_labels(g, pos, font_size=10)
plt.title("Syntethic representation of the City")
plt.show()
print('Total number of Nodes: '+str(len(g.nodes)))

graph
Now I used DQN in a fixed state from node number 1130166767 as a start to node number 1731824802 as a goal.
this is the whole code of mine
class Network(nn.Module):
  def __init__(self,input_dim,n_action):
    super(Network,self).__init__()
    self.f1=nn.Linear(input_dim,128)
    self.f2=nn.Linear(128,64)
    self.f3=nn.Linear(64,32)
    self.f4=nn.Linear(32,n_action)
    #self.optimizer=optim.Adam(self.parameters(),lr=lr)
    #self.loss=nn.MSELoss()
    self.device=T.device('cuda' if T.cuda.is_available() else 'cpu')
    self.to(self.device)

      def forward(self,x):
        x=F.relu(self.f1(x))
        x=F.relu(self.f2(x))
        x=F.relu(self.f3(x))
        x=self.f4(x)
        return x

      def act(self,obs):
        #state=T.tensor(obs).to(device)
        state=obs.to(self.device)
        actions=self.forward(state)
        action=T.argmax(actions).item()

        return action

device=T.device('cuda' if T.cuda.is_available() else 'cpu')
print(device)

num_states = len(g.nodes)*1
### if we need to train a specific set of nodes for ex 10 we *10
num_actions = len(g.nodes)
print("Expected number of States are: "+str(num_states))
print("Expected number of action are: "+str(num_actions))

#num_action*2=when we would like to convert the state into onehotvector we need to concatinate the two vector 22+22
online=Network(num_actions*2,num_actions)
target=Network(num_actions*2,num_actions)
target.load_state_dict(online.state_dict())
optimizer=T.optim.Adam(online.parameters(),lr=5e-4)

#create a dictionary that have encoded index for each node
#to solve this isssu
#reset()=476562122273
#number of state < 476562122273
enc_node={}
dec_node={}
for index,nd in enumerate(g.nodes):
  enc_node[nd]=index
  dec_node[index]=nd

def wayenc(current_node,new_node,type=1):
  #encoded
  if type==1: #distance
    if new_node in g[current_node]:
      rw=g[current_node][new_node]['weight']*-1
      return rw,True
    rw=-5000
    return rw,False

def rw_function(current,action):
  #current_node
  #new_node
  beta=1 #between 1 and 0
  current=dec_node[current]
  new_node=dec_node[action]
  rw0,link=wayenc(current,new_node)
  rw1=0
  frw=rw0*beta+(1-beta)*rw1

  return frw,link

def state_enc(dst, end,n=len(g.nodes)):
    return dst+n*end

def state_dec(state,n=len(g.nodes)):
    dst = state%n
    end = (state-dst)/n
    return dst, int(end)

def step(state,action):
    done=False
    current_node , end = state_dec(state)

    new_state = state_enc(action,end)

    rw,link=rw_function(current_node,action)

    if not link:
        new_state = state
        return new_state,rw,False

    elif action == end:
        rw = 10000 #500*12
        done=True

    return new_state,rw,done

def reset():
  state=state_enc(enc_node[1130166767],enc_node[1731824802])
  return state

def state_to_vector(current_node,end_node):
  n=len(g.nodes)
  source_state_zeros=[0.]*n
  source_state_zeros[current_node]=1

  end_state_zeros=[0.]*n
  end_state_zeros[end_node]=1.
  vector=source_state_zeros+end_state_zeros
  return vector

#return a list of list converted from state to vectors
def list_of_vecotrs(new_obses_t):
  list_new_obss_t=new_obses_t.tolist()
  #convert to integer
  list_new_obss_t=[int(v) for v in list_new_obss_t]
  vector_list=[]
  for state in list_new_obss_t:
    s,f=state_dec(state)
    vector=state_to_vector(s,f)
    vector_list.append(vector)
  return vector_list

#fill the replay buffer
#replay_buffer=[]
rew_buffer=[0]
penalties=[]
episode_reward=0.0
batch_size=num_actions*2
buffer_size=100000
min_replay_size=int(buffer_size*0.20)
target_update_freq=1000
flag=0
action_list=np.arange(0,len(g.nodes)).tolist()
replay_buffer=deque(maxlen=buffer_size)

#populate the experience network
obs=reset()
#obs,end=state_dec(start,len(g.nodes))
for _ in tqdm(range(min_replay_size)):
  action=np.random.choice(action_list)
  new_obs,rew,done=step(obs,action)
  transition=(obs,action,rew,done,new_obs)
  replay_buffer.append(transition)
  obs=new_obs
  if done:
    obs=reset()

#main training loop
obs=reset()
episodes=100000
start=1
end=0.1
decay=episodes
gamma=0.99
epsilon=0.5

gamma_list=[]
mean_reward=[]
done_location=[]
loss_list=[]
number_of_episodes=[]
stat_dict={'episodes':[],'epsilon':[],'explore_exploit':[],'time':[]}

for i in tqdm(range(episodes)):
  itr=0
  #epsilon=np.interp(i,[0,decay],[start,end])
  #gamma=np.interp(i,[0,decay],[start,end])
  epsilon=np.exp(-i/(episodes/3))
  rnd_sample=random.random()

  stat_dict['episodes'].append(i)
  stat_dict['epsilon'].append(epsilon)

  #choose an action
  if rnd_sample <=epsilon:
    action=np.random.choice(action_list)
    stat_dict['explore_exploit'].append('explore')

  else:
    source,end=state_dec(obs)
    v_obs=state_to_vector(source,end)
    t_obs=T.tensor(v_obs)
    action=online.act(t_obs)
    stat_dict['explore_exploit'].append('exploit')

  #fill transition and append to replay buffer

  new_obs,rew,done=step(obs,action)

  transition=(obs,action,rew,done,new_obs)
  replay_buffer.append(transition)
  obs=new_obs
  episode_reward+=rew

  if done:
    obs=reset()
    rew_buffer.append(episode_reward)
    episode_reward=0.0
    done_location.append(i)

  #start gradient step

  transitions=random.sample(replay_buffer,batch_size)

  obses=np.asarray([t[0] for t in transitions])
  actions=np.asarray([t[1] for t in transitions])
  rews=np.asarray([t[2] for t in transitions])
  dones=np.asarray([t[3] for t in transitions])
  new_obses=np.asarray([t[4] for t in transitions])

  obses_t=T.as_tensor(obses,dtype=T.float32).to(device)
  actions_t=T.as_tensor(actions,dtype=T.int64).to(device).unsqueeze(-1)
  rews_t=T.as_tensor(rews,dtype=T.float32).to(device)
  dones_t=T.as_tensor(dones,dtype=T.float32).to(device)
  new_obses_t=T.as_tensor(new_obses,dtype=T.float32).to(device)

  list_new_obses_t=T.tensor(list_of_vecotrs(new_obses_t)).to(device)
  target_q_values=target(list_new_obses_t)##

  max_target_q_values=target_q_values.max(dim=1,keepdim=False)[0]
  targets=rews_t+gamma*(1-dones_t)*max_target_q_values

  list_obses_t=T.tensor(list_of_vecotrs(obses_t)).to(device)
  q_values=online(list_obses_t)
  action_q_values=T.gather(input=q_values,dim=1,index=actions_t)

  #warning UserWarning: Using a target size (torch.Size([24, 24])) that is different to the input size (torch.Size([24, 1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
  targets=targets.unsqueeze(-1)
  loss=nn.functional.mse_loss(action_q_values,targets)
  #loss=rmsle(action_q_values,targets)
  loss_list.append(loss.item())

  optimizer.zero_grad()
  loss.backward()
  optimizer.step()

  #plot
  mean_reward.append(np.mean(rew_buffer))
  number_of_episodes.append(i)
  gamma_list.append(gamma)
  dec = {'number_of_episodes':number_of_episodes,'mean_reward':mean_reward,'gamma':gamma_list}
  #clear_output(wait=True)
  #sns.lineplot(data=dec, x="number_of_episodes", y="mean_reward")
  #plt.show()

  if i % target_update_freq==0:
    target.load_state_dict(online.state_dict())
  if i % 1000 ==0:
    print('step',i,'avg rew',round(np.mean(rew_buffer),2))
    pass

now if you can see from thephotos
nither the rewards are increasing or the loss is decreasing, I tried the following

increasing and decreasing the learning rate

changing target_update_freq from 100 1000  1000

I tried changing the state representation from Onehotvector to [state, end] and sent it as pair.

i tried to change the loss function from mse_loss,smooth_l1,... etc

i tried to increase the number of episodes

adding another layer to NN network
7.changing how the decay of epsilon works linear ,exponential

most of these solutions are from questions on Stacked, but nothing works for me
How can I improve the performance? or in another ward? How can I increase the rewards?

Output:
It seems your problem needs just parameters tunning only

I change your learning rate into 0.02
I changed the dimension of the states that been send to NN

class Network(nn.Module):
  def __init__(self,input_dim,n_action):
    super(Network,self).__init__()
    self.f1=nn.Linear(input_dim,128)
    self.f2=nn.Linear(128,64)
    self.f3=nn.Linear(64,32)
    self.f4=nn.Linear(32,n_action)
    #self.optimizer=optim.Adam(self.parameters(),lr=lr)
    #self.loss=nn.MSELoss()
    self.device=T.device('cuda' if T.cuda.is_available() else 'cpu')
    self.to(self.device)

  def forward(self,x):
    x=F.relu(self.f1(x))
    x=F.relu(self.f2(x))
    x=F.relu(self.f3(x))
    x=self.f4(x)
    return x

  def act(self,obs):
    #state=T.tensor(obs).to(device)
    state=obs.to(self.device)
    actions=self.forward(state)
    action=T.argmax(actions).item()

    return action

device=T.device('cuda' if T.cuda.is_available() else 'cpu')
print(device)
num_states = len(g.nodes)**2
num_actions = len(g.nodes)
online=Network(num_actions*2,num_actions)
target=Network(num_actions*2,num_actions)
target.load_state_dict(online.state_dict())
optimizer=T.optim.Adam(online.parameters(),lr=1e-2)
enc_node={}
dec_node={}
for index,nd in enumerate(g.nodes):
  enc_node[nd]=index
  dec_node[index]=nd

def wayenc(current_node,new_node,type=1):
  #encoded
  if type==1: #distance
    if new_node in g[current_node]:
      rw=g[current_node][new_node]['weight']*-1
      return rw,True
    rw=-1000
    return rw,False
def rw_function(current,action):
  beta=1
  current=dec_node[current]
  new_node=dec_node[action]
  rw0,link=wayenc(current,new_node)
  rw1=0
  frw=rw0*beta+(1-beta)*rw1

  return frw,link

def state_enc(dst, end,n=len(g.nodes)):
    return dst+n*end

def state_dec(state,n=len(g.nodes)):
    dst = state%n
    end = (state-dst)/n
    return dst, int(end)

def step(state,action):
    done=False
    current_node , end = state_dec(state)

    new_state = state_enc(action,end)

    rw,link=rw_function(current_node,action)

    if not link:
        new_state = state
        return new_state,rw,False

    elif action == end:
        rw = 10000
        done=True

    return new_state,rw,done

def reset():
  state=state_enc(enc_node[1130166767],enc_node[1731824802])
  return state

def state_to_vector(current_node,end_node):
  n=len(g.nodes)
  source_state_zeros=[0.]*n
  source_state_zeros[current_node]=1

  end_state_zeros=[0.]*n
  end_state_zeros[end_node]=1.
  vector=source_state_zeros+end_state_zeros
  return vector

#return a list of list converted from state to vectors
def list_of_vecotrs(new_obses_t):
  list_new_obss_t=new_obses_t.tolist()
  #convert to integer
  list_new_obss_t=[int(v) for v in list_new_obss_t]
  vector_list=[]
  for state in list_new_obss_t:
    s,f=state_dec(state)
    vector=state_to_vector(s,f)
    vector_list.append(vector)
  return vector_list

  #replay_buffer=[]
rew_buffer=[0]
penalties=[]
episode_reward=0.0
#batch_size=num_actions*2
batch_size=32
buffer_size=50000
min_replay_size=int(buffer_size*0.25)

target_update_freq=1000
flag=0
action_list=np.arange(0,len(g.nodes)).tolist()
replay_buffer=deque(maxlen=min_replay_size)

#populate the experience network
obs=reset()
#obs,end=state_dec(start,len(g.nodes))
for _ in tqdm(range(min_replay_size)):
  action=np.random.choice(action_list)
  new_obs,rew,done=step(obs,action)
  transition=(obs,action,rew,done,new_obs)
  replay_buffer.append(transition)
  obs=new_obs
  if done:
    obs=reset()

#main training loop
obs=reset()
episodes=70000
start=1
end=0.1
decay=episodes
gamma=0.99
epsilon=0.5

gamma_list=[]
mean_reward=[]
done_location=[]
loss_list=[]
number_of_episodes=[]
stat_dict={'episodes':[],'epsilon':[],'explore_exploit':[],'time':[]}

for i in tqdm(range(episodes)):
  itr=0

  epsilon=np.exp(-i/(episodes/2))
  rnd_sample=random.random()

  stat_dict['episodes'].append(i)
  stat_dict['epsilon'].append(epsilon)

  if rnd_sample <=epsilon:
    action=np.random.choice(action_list)
    stat_dict['explore_exploit'].append('explore')

  else:
    source,end=state_dec(obs)
    v_obs=state_to_vector(source,end)
    t_obs=T.tensor([v_obs])
    action=online.act(t_obs)
    stat_dict['explore_exploit'].append('exploit')

  new_obs,rew,done=step(obs,action)

  transition=(obs,action,rew,done,new_obs)
  replay_buffer.append(transition)
  obs=new_obs
  episode_reward+=rew

  if done:
    obs=reset()
    rew_buffer.append(episode_reward)
    episode_reward=0.0
    done_location.append(i)

  batch_size=32
  transitions=random.sample(replay_buffer,batch_size)

  obses=np.asarray([t[0] for t in transitions])
  actions=np.asarray([t[1] for t in transitions])
  rews=np.asarray([t[2] for t in transitions])
  dones=np.asarray([t[3] for t in transitions])
  new_obses=np.asarray([t[4] for t in transitions])

  obses_t=T.as_tensor(obses,dtype=T.float32).to(device)
  actions_t=T.as_tensor(actions,dtype=T.int64).to(device).unsqueeze(-1)
  rews_t=T.as_tensor(rews,dtype=T.float32).to(device)
  dones_t=T.as_tensor(dones,dtype=T.float32).to(device)
  new_obses_t=T.as_tensor(new_obses,dtype=T.float32).to(device)

  list_new_obses_t=T.tensor(list_of_vecotrs(new_obses_t)).to(device)
  target_q_values=target(list_new_obses_t)##
  #target_q_values=target(obses_t)

  max_target_q_values=target_q_values.max(dim=1,keepdim=False)[0]
  targets=rews_t+gamma*(1-dones_t)*max_target_q_values
  targets=targets.unsqueeze(-1)

  list_obses_t=T.tensor(list_of_vecotrs(obses_t)).to(device)
  q_values=online(list_obses_t)
  #q_values=online(obses_t)
  action_q_values=T.gather(input=q_values,dim=1,index=actions_t)

  loss=nn.functional.mse_loss(action_q_values,targets)
  loss_list.append(loss.item())

  optimizer.zero_grad()
  loss.backward()
  optimizer.step()

  mean_reward.append(np.mean(rew_buffer))
  number_of_episodes.append(i)
  gamma_list.append(gamma)
  dec = {'number_of_episodes':number_of_episodes,'mean_reward':mean_reward,'gamma':gamma_list}

  if i % target_update_freq==0:
    target.load_state_dict(online.state_dict())
  if i % 1000 ==0:
    print('step',i,'avg rew',round(np.mean(rew_buffer),2))
    pass
  if i==5000:
    pass

I did run this script and it gave me a good performance changing the learning rate helped a lot


Instruction: How to fix PyTorch RuntimeError: CUDA error: out of memory?

Input:
I'm trying to train my Pytorch model on a remote server using a GPU. However, the training phase doesn't start, and I have the following error instead: RuntimeError: CUDA error: out of memory
I reinstalled Pytorch with Cuda 11 in case my version of Cuda is not compatible with the GPU I use (NVidia GeForce RTX 3080). It still doesn't work.
I also ran this command torch.cuda.empty_cache(). And it still doesn't work.
When I run the code below in my interpreter it still displays RuntimeError: CUDA error: out of memory
import torch
print(torch.rand(1, device="cuda"))

However, it works on cpu.
import torch
print(torch.rand(1, device="cpu"))

When I run the command nvidia-smi I have the following output:

How can I fix it?

Output:
The problem here is that the GPU that you are trying to use is already occupied by another process. The steps for checking this are:

Use nvidia-smi in the terminal. This will check if your GPU
drivers are installed and the load of the GPUS. If it fails, or doesn't show your gpu, check your driver installation.
If the GPU shows >0% GPU Memory Usage, that means that it is already being used by another process. You can close it (Don't do that in a shared environment!) or launch it in the other GPU, if you have another one free.


Instruction: TF depth_to_space not same as Torch's PixelShuffle when output channels > 1?

Input:
I noticed something interesting when trying to port a torch-trained model to tensorflow (TF). When the output channels of a PixelShuffle operation are greater than one, the corresponding depth_to_space function in TF is not equivalent (Note: I convert the input to TF to NHWC and the output back to NCHW). I was wondering whether this expected behavior OR there is a misunderstanding?
Specifically,
# Torch
torch_out = nn.PixelShuffle(2)(input)

and
# TF/Keras
input = np.transpose(input, (0, 2, 3, 1)) // Convert to NHWC
keras_input = keras.layers.Input(shape=input.shape[1:])
keras_d2s = keras.layers.Lambda(lambda x: tf.nn.depth_to_space(x, 2))(input)
...
keras_out = np.transpose(keras_d2s, (0, 3, 1, 2)) // Convert back to NCHW

and
keras_out != torch_out

Here is a testbench:
import numpy as np

import torch
import tensorflow as tf

from torch import nn
from tensorflow import keras

class Shuffle(nn.Module):
    def __init__(self, s, k, ic):
        super(Shuffle, self).__init__()
        self.shuffle = nn.PixelShuffle(s)

    def forward(self, inputs):
        return self.shuffle(inputs)

def main():
    sz = 4

    h = 3
    w = 3
    k = 3
    ic = 8
    s = 2

    input = np.arange(0, ic * h * w, dtype=np.float32).reshape(ic, h, w)
    input = input[np.newaxis]
    torch_input = torch.from_numpy(input)

    shuffle_model = Shuffle(s, k, ic)
    shuffle_out = shuffle_model(torch_input).detach().numpy()
    print('Shuffle out:', shuffle_out.shape)
    print(shuffle_out)

    input = np.transpose(input, (0, 2, 3, 1))

    keras_input = keras.layers.Input(shape=input.shape[1:])
    keras_d2s = keras.layers.Lambda(lambda x: tf.nn.depth_to_space(x, s))(keras_input)
    keras_model = keras.Model(keras_input, keras_d2s)
    keras_out = keras_model.predict(input)

    keras_out = np.transpose(keras_out, (0, 3, 1, 2))

    print('Keras out:', keras_out.shape)
    print(keras_out)
    equal = np.allclose(shuffle_out, keras_out)
    print('Equal?', equal)

if __name__ == '__main__':
    main()

Output: They are indeed different. If you want them to match you need to shuffle the channels of one of the inputs. Or if the pixelshuffle/depth_to_space layer follows a convolution layer you can shuffle the channels of the weights of the convolution. Specifically, if oc is the number of output channels and s is the block_size then you need to permute the channels of the convolution's weights in TF using [i + oc * j for i in range(oc) for j in range(s ** 2)] (yields something like [0, 2, 4, 1, 3, 5]).


Instruction: How to find contents of an NGC Docker image?

Input:
The NVIDIA NGC container catalog has a broad range of GPU-optimised containers for common activities such as deep learning. How does one find what is inside the Docker images?
For example, I require an image with Pytorch 1.4 and Python 3.6 or 3.7, but the Pytorch tags go from pytorch:17.10 to pytorch:21.06-py3 (where xx.xx is the container version). Is there somewhere a list of what is installed in each container, or even better the container Dockerfile that was used to build the images?

Output:
The details of pytorch NGC containers are listed at PyTorch Release Notes at the bottom of the pytorch NGC page.
All other deep learning frameworks related documentation is also at NVIDIA Deep Learning Frameworks.


Instruction: RandomAdjustSharpness gives IndexError: tuple index out of range

Input:
While using RandomAdjustSharpness, my code throws the following error - IndexError: tuple index out of range. I followed the instructions given over here - https://pytorch.org/vision/stable/transforms.html and therefore am confused with this error.
Here is my code -
import math, random
from sklearn.datasets import load_sample_images
import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim
import torch.autograd as autograd
import torch.nn.functional as F
import torchvision.transforms as transforms
import matplotlib.pyplot as plt

def random_crop(imgs):

    imgs = torch.tensor(imgs)
    change = torch.nn.Sequential(
        transforms.RandomCrop(427),
        transforms.RandomAdjustSharpness(1, p=1)
    )
    imgs = change(imgs).numpy()
    return imgs

###Obtaining a random image and preprocessing it!##
dataset = load_sample_images()
first_img_data = dataset.images[0]
first_img_data  = first_img_data.reshape(-1, 427, 640)
first_img_data = first_img_data[1, :, :]
#first_img_data = first_img_data[0:84, 0:84].reshape(-1, 84,84)
# first_img_data = torch.tensor(first_img_data)
plt.figure()
plt.imshow(np.squeeze(first_img_data))

foo = random_crop(first_img_data)
plt.figure()

plt.imshow(np.squeeze(foo))
plt.show()

Output:
you need to a dimension to your tensor like this
torch.tensor([imgs])


Instruction: AttributeError: module 'torchvision.models' has no attribute 'mobilenet_v3_small'

Input:
I am unable to import mobilenet_v3_small from torchvision. I have the below code:
import torch
import torch.nn as nn
import torch.utils.data
from torch.autograd import Variable
import torch.nn.functional as F
import math
import numpy as np
import torchvision.models as models

class feature_extraction(nn.Module):
    def __init__(self):
        super().__init__()

        self.mobilenet = models.mobilenet_v3_small(pretrained=True)

I am getting the error:
self.mobilenet = models.mobilenet_v3_small(pretrained=True)
AttributeError: module 'torchvision.models' has no attribute 'mobilenet_v3_small'

I have the below versions:
cudatoolkit               11.0.221             h6bb024c_0
torch                     1.7.0                    pypi_0    pypi
torchaudio                0.7.2                      py37    pytorch
torchvision               0.8.1                    pypi_0    pypi

Python version is 3.7

Output: mobilenet_v3_small is not available in torchvision 0.8.1. If you want to use it you need to upgrade to 0.10.0 (stable version) or at least 0.9.0.


Instruction: How to cluster the nodes of the Cora dataset based on their in-degree values?

Input:
I want to cluster the nodes of Cora dataset in a way that each cluster only contains the nodes with the same in-degree value. I can code something as follows:
import torch
from torch_geometric.datasets import Planetoid
from torch_geometric.utils import degree

dataset = Planetoid('./data','CORA')
data = dataset[0]
n = data.num_nodes
indegree = degree(data.edge_index[1], n, dtype=torch.long)
counts = torch.bincount(indegree)

But since I don't access the index value of the nodes, I don't know how to place each node in which cluster?

Output:
You can use return_inverse in torch.unique to recover the indices.
The nodes with the same value i in indices belong to the same cluster because they all have a degree equal to indegree_class[i].
indegree_class, indices = torch.unique(indegree, return_inverse=True)


Instruction: Extracting Meaningful Error Message from 'RuntimeError: CUDA error: device-side assert triggered' on Google Colab in Pytorch

Input:
I am experiencing the following error while training a generative network via Pytorch 1.9.0+cu102:
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

While using a Google Colaboratory GPU session. This segment was triggered on either one of these two lines:
running_loss += loss.item()

or
target = target.to(device)

It produces the error on the first line when I am first running the notebook, and the second line each subsequent time I try to run the block. The first error occurs after training for 3 batches. The second error happens on the first batch. I can confirm that the device is cuda0, that device is available, and target is a pytorch tensor. Naturally, I tried to take the advice of the error and run:
!CUDA_LAUNCH_BLOCKING=1

and
os.system('CUDA_LAUNCH_BLOCKING=1')

However, neither of these lines changes the error message. According to a different post, this is because colab is running these lines in a subshell. The error does not occur when running on CPU, and I do not have access to a GPU device besides the GPU on Colab. While this question has been asked in many different forms, no answers are particularly helpful to me because they either recommend passing the aforementioned line, are about a situation fundamentally different from my own (such as training a classifier with an inappropriate number of classes), or recommend a solution which I have already tried, such as resetting the runtime or switching to CPU.
I am hoping to gain insight into the following questions:

Is there a way for me to get a more specific error message? Efforts to set the launch blocking variable have been unsuccessful.
How could it be that I am getting this error on two seemingly very different lines? How could it be that my network trains for 3 batches (it is always 3), but fails on the fourth?
Does this situation remind anyone of an error that they have encountered previously, and have a possible route for ameliorating it given the limited information I can extract?

Output:
I was successfully able to get more information about the error by executing:
os.environ['CUDA_LAUNCH_BLOCKING'] = "1"

BEFORE importing torch. This allowed me to get a more detailed traceback and ultimately diagnose the problem as an inappropriate loss function.


Instruction: Is it possible to expand nodes of a frozen neural network model by width?

Input:
I want to expand the number of nodes of a frozen neural network model by width in pytorch. I want to do something like what shown in the below image where grey are frozen weights and green are newly added trainable weights.
 .
I have an initial model which takes 3 inputs and gives one output back, this model also has two hidden layers with nodes h1=5 and h2=3 respectively. I created the model in pytorch and frozen the weights.
import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
    def __init__(self):
        super(Net,self).__init__()
        self.fc1 = nn.Linear(3, 5)
        self.fc2 = nn.Linear(5, 3)
        self.fc3 = nn.Linear(3, 1)
    def forward(self,x):
        x = self.fc1(x)
        x = self.fc2(x)
        x = self.fc3(x)
        return x

print(Net())

model = Net()
X = torch.rand(5,3)
y = model(X)
print(y)

# Freeze layers
for param in model.parameters():
    param.requires_grad = False

Now I want to expand this model by adding trainable nodes to h1=5+2, h2=3+1 and output= 1+1. Only the newly added nodes should be trainable and all other weights should be frozen, and those frozen weights should have the same weight as the parent model. Can this be done in pytorch or in tensorflow?

Output:
There are 2 things need to be done
1. Expand the layers
You really should use ModuleList or ModuleDict to create layers because that would mean you can use loop. I know eval or setattr also work but they tend to break something else so I don't want to use them.
There's 2 ways I can think of. One is directly replace the weight with something bigger and the other one is create a bigger layer and replace the whole layer.
# Replace the weight with randomly generated tensor
fc1_newweight = torch.rand(7, 3)
fc1_newbias = torch.rand(7)
fc1_shape = model.fc1.weight.shape
fc1_newweight[:fc1_shape[0], :fc1_shape[1]] = model.fc1.weight.clone()
fc1_newbias[:fc1_shape[0]] = model.fc1.bias.clone()
model.fc1.weight = torch.nn.Parameter(fc1_newweight)
model.fc1.bias = torch.nn.Parameter(fc1_newbias)

# Replace the weight with the random generated weights from the new layer
fc2_shape = model.fc2.weight.shape
fc2 = nn.Linear(7, 4)
fc2_weight = fc2.state_dict()
fc2_weight['weight'][:fc2_shape[0], :fc2_shape[1]] = model.fc2.weight.clone()
fc2_weight['bias'][:fc2_shape[0]] = model.fc2.bias.clone()
fc2.load_state_dict(fc2_weight)
model.fc2.weight = torch.nn.Parameter(fc2_weight['weight'])
model.fc2.bias = torch.nn.Parameter(fc2_weight['bias'])

# Replace the whole layer
fc3_shape = model.fc3.weight.shape
fc3 = nn.Linear(4, 2)
fc3_weight = fc3.state_dict()
fc3_weight['weight'][:fc3_shape[0], :fc3_shape[1]] = model.fc3.weight.clone()
fc3_weight['bias'][:fc3_shape[0]] = model.fc3.bias.clone()
fc3.load_state_dict(fc3_weight)
model.fc3 = fc3

I'd prefer the 2. or 3. over 1. because the weights will be generated using nn.init.kaiming_uniform instead of just uniform.
2. Select what to be trainable
This is tricky because you can't just set require_grad on only some elements of the weights because you'll get RuntimeError: you can only change requires_grad flags of leaf variables.
But something like this should be a good enough substitute. Again, using ModuleList will make the code here look a lot better too.
y = model(x)
loss = criterion(y, target)
loss.backward()

model.fc1.weight.grad[:fc1_shape[0], :fc1_shape[1]] = 0
model.fc1.bias.grad[:fc1_shape[0]] = 0
model.fc2.weight.grad[:fc2_shape[0], :fc2_shape[1]] = 0
model.fc2.bias.grad[:fc2_shape[0]] = 0
model.fc3.weight.grad[:fc3_shape[0], :fc3_shape[1]] = 0
model.fc3.bias.grad[:fc3_shape[0]] = 0

optimizer.step()


Instruction: In my PyTorch train iterator, how do I resolve the ValueError: only one element tensors can be converted to Python scalars?

Input:
How do I solve the ValueError: only one element tensors can be converted to Python scalars?
I am closely following a tutorial on building a Question Answering Bot in PyTorch. However, at training, my code is unable to save the checkpoints, giving me aforementioned ValueError. The error happens at torch.save(torch.tensor(train_loss_set), os.path.join(output_dir, 'training_loss.pt'))
Below is my code corresponding to the train iterator:
num_train_epochs = 1

print("***** Running training *****")
print("  Num examples = %d" % len(dataset))
print("  Num Epochs = %d" % num_train_epochs)
print("  Batch size = %d" % batch_size)
print("  Total optimization steps = %d" % (len(train_dataloader) // num_train_epochs))

model.zero_grad()
train_iterator = trange(num_train_epochs, desc="Epoch")
set_seed()

for _ in train_iterator:
    epoch_iterator = tqdm(train_dataloader, desc="Iteration")
    for step, batch in enumerate(epoch_iterator):
        if step < global_step + 1:
            continue

        model.train()
        batch = tuple(t.to(device) for t in batch)

        inputs = {'input_ids':       batch[0],
                'attention_mask':  batch[1],
                'token_type_ids':  batch[2],
                'start_positions': batch[3],
                'end_positions':   batch[4]}

        outputs = model(**inputs)

        loss = outputs[0]
        train_loss_set.append(loss)
        loss.sum().backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)

        tr_loss += loss.sum().item()
        optimizer.step()
        model.zero_grad()
        global_step += 1

        if global_step % 1000 == 0:
            print("Train loss: {}".format(tr_loss/global_step))
            output_dir = 'checkpoints/checkpoint-{}'.format(global_step)
            if not os.path.exists(output_dir):
                os.makedirs(output_dir)
            model_to_save = model.module if hasattr(model, 'module') else model  # Take care of distributed/parallel training
            model_to_save.save_pretrained(output_dir)
            torch.save(torch.tensor(train_loss_set), os.path.join(output_dir, 'training_loss.pt'))
            print("Saving model checkpoint to %s" % output_dir)

Edit
print(train_loss_set[:10]) returns the following:
[tensor([5.7099, 5.7395], device='cuda:0', grad_fn=<GatherBackward>), tensor([5.2470, 5.4016], device='cuda:0', grad_fn=<GatherBackward>), tensor([5.1311, 5.0390], device='cuda:0', grad_fn=<GatherBackward>), tensor([4.4326, 4.8475], device='cuda:0', grad_fn=<GatherBackward>), tensor([3.4740, 3.9955], device='cuda:0', grad_fn=<GatherBackward>), tensor([4.8710, 4.5907], device='cuda:0', grad_fn=<GatherBackward>), tensor([4.4294, 4.3013], device='cuda:0', grad_fn=<GatherBackward>), tensor([2.7536, 2.9540], device='cuda:0', grad_fn=<GatherBackward>), tensor([3.8989, 3.3436], device='cuda:0', grad_fn=<GatherBackward>), tensor([3.3534, 3.2532], device='cuda:0', grad_fn=<GatherBackward>)]

Could this have to do with the fact that I'm using DataParallel?

Output:
It's a weird behavior of pytorch.
Basically you can't create a Tensor using a list(s) of Tensors.
But there's 3 things you can do.

Is, you don't need torch.tensor when saving a list of tensors so this should work.

torch.save(train_loss_set, os.path.join(output_dir, 'training_loss.pt'))

Use torch.stack instead.

torch.save(torch.stack(train_loss_set), os.path.join(output_dir, 'training_loss.pt'))

This kinda counter intuitive but you can convert the tensors inside to ndarray. And you can use torch.tensor

train_loss_set.append(loss.cpu().detach().numpy())


Instruction: TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first (Segmentation using yolact edge)

Input:
I am running segmentation on yolact edge. I am trying to find coordinates of the minimu and maximum x and y pixel coordinated of the mask using my own algorithm.
I am trying to convert the values of a tuple to numpy. However I am getting the follwoing errror
TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.

Code

    xmin = []
    xmax = []
    y = []
    print(np.shape(t[3]))
    print(type(t[3][:][:][:]))
    #row = (t[3][1][360][:]==1).nonzero(as_tuple=True)
    for i in range (0, 2):
        t_cpu = t[3].clone().detach().cpu()
        horizontal_translation = torch.where(t[3][i][:][:]==1)
        print(horizontal_translation)
        horizontal_translation_numpy = np.asarray(horizontal_translation[1])
        x_min = np.amin(horizontal_translation_numpy)
        x_max = np.amax(horizontal_translation_numpy)
        np.append(xmin,x_min)
        np.append(xmax, x_max)
    print(xmin)
    print(xmax)

Note:
t is a pytorch tensor that is output by the default program that contains mask data in t[3]. How do I fix this?
Output:
torch.Size([2, 720, 1280])
<class 'torch.Tensor'>
(tensor([105, 105, 105,  ..., 503, 503, 503]), tensor([427, 428, 429,  ..., 468, 469, 470]))
Traceback (most recent call last):
  File "eval.py", line 1303, in <module>
    evaluate(net, dataset)
  File "eval.py", line 928, in evaluate
    evalimage(net, inp, out, detections=detections, image_id="0")
  File "eval.py", line 621, in evalimage
    img_numpy = prep_display(preds, frame, None, None, undo_transform=False)
  File "eval.py", line 198, in prep_display
    horizontal_translation_numpy = np.asarray(horizontal_translation[1])
  File "/home/nvidia/.local/lib/python3.6/site-packages/numpy/core/_asarray.py", line 83, in asarray
    return array(a, dtype, copy=False, order=order)
  File "/home/nvidia/.local/lib/python3.6/site-packages/torch/tensor.py", line 480, in __array__
    return self.numpy()
TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.

Output:
This should work
xmin = []
xmax = []
y = []
print(np.shape(t[3]))
print(type(t[3][:][:][:]))
#row = (t[3][1][360][:]==1).nonzero(as_tuple=True)
for i in range (0, 2):
    t_cpu = t[3].clone().detach().cpu()
    horizontal_translation = torch.where(t[3][i][:][:]==1)
    print(horizontal_translation)
    horizontal_translation_numpy = horizontal_translation.cpu().numpy()
    x_min = np.amin(horizontal_translation_numpy)
    x_max = np.amax(horizontal_translation_numpy)
    np.append(xmin,x_min)
    np.append(xmax, x_max)
print(xmin)
print(xmax)


Instruction: How can I apply group normalization after a full-connection layer?

Input:
How can I apply Group Normalization after a full-connection layer? Say the output of the full-connection layer is 1024. And the group normalization layer is using 16 groups.
self.gn1 = nn.GroupNorm(16, hidden_size)
h1 = F.relu(self.gn1(self.fc1(x))))

Am I right? How should we understand the group normalization if it is applied to the output of a full-connection layer?

Output:
Your code is correct, but let's see what happens in a small example.
The output of a fully-connected layer is usually a 2D-tensor with shape (batch_size, hidden_size) so I will focus on this kind of input, but remember that GroupNorm supports tensors with an arbitrary number of dimensions. In fact, GroupNorm works always on the last dimension of the tensor.
GroupNorm treats all the samples in the batch as independent and it creates n_groups from the last dimension of the tensor, as you can see from the image.

When the input tensor is 2D, the cube in the image becomes a square because there is no third vertical dimension, so in practice the normalization is performed on fixed-size consecutive pieces of the rows of the input matrix.
Let's see an example with some code.
import torch
import torch.nn as nn

batch_size = 2
hidden_size = 32
n_groups = 8
group_size = hidden_size // n_groups # = 4

# Input tensor that can be the result of a fully-connected layer
x = torch.rand(batch_size, hidden_size)

# GroupNorm with affine disabled to simplify the inspection of results
gn1 = nn.GroupNorm(n_groups, hidden_size, affine=False)
r = gn1(x)

# The rows are split into n_groups (8) groups of size group_size (4)
# and the normalization is applied to these pieces of rows.
# We can check it for the first group x[0, :group_size] with the following code
first_group = x[0, :group_size]
normalized_first_group = (first_group - first_group.mean())/torch.sqrt(first_group.var(unbiased=False) + gn1.eps)
print(r[0, :4])
print(normalized_first_group)
if(torch.allclose(r[0, :4], normalized_first_group)):
    print('The result on the first group is the expected one')


Instruction: How to prevent appending to a list to fill gpu memory

Input:
I run a model with pytorch with batch inference. I want to save the output from that model to a list, because I need it later for other calculations. The problem is, that if I append the output from my model to my list "output", it fills the gpu memory.
The output from the model is a list.
Is there a way to move the list to RAM?
if __name__ == "__main__":
        output = []
        with torch.no_grad():
            for i in input_split:
                try:
                    preds = model(i)
                    output.append(preds)
                    del preds
                    gc.collect()
                    torch.cuda.empty_cache()

                except RuntimeError as e:
                    print("Failed")

Output:
It is because the tensors you get from preds = model(i) are still in GPU.
You can just take them out of the GPU before appending them to the list
output = []
with torch.no_grad():
    for i in input_split:
        preds = model(i)
        output.append(preds.cpu())

And when you want to use them again in GPU then just put them into GPU one by one
for data in output:
    data = data.cuda()

Edit 1. The Detectron2
This is similar to the above answer, except a little bit more complicated.
The output of Detectron2 is list[dict]. Which means if you have more than 1 images in the batch you'll have to create an empty list as buffer, like
output = []
with torch.no_grad():
    for i in input_split:
        preds = model(i)

        buffer = []
        for pred in preds:
            pred['instances']['pred_boxes'] = pred['instances']['pred_boxes'].cpu()
            pred['instances']['scores'] = pred['instances']['scores'].cpu()
            pred['instances']['pred_classes'] = pred['instances']['pred_classes'].cpu()
            pred['instances']['pred_masks'] = pred['instances']['pred_masks'].cpu()
            pred['instances']['pred_keypoints'] = pred['instances']['pred_keypoints'].cpu()
            buffer.append(pred)
        output.append(buffer)

You maybe able to use pointer and get rid of buffer like this too
output = []
with torch.no_grad():
    for i in input_split:
        preds = model(i)

        for pred in preds:
            pred['instances']['pred_boxes'] = pred['instances']['pred_boxes'].cpu()
            pred['instances']['scores'] = pred['instances']['scores'].cpu()
            pred['instances']['pred_classes'] = pred['instances']['pred_classes'].cpu()
            pred['instances']['pred_masks'] = pred['instances']['pred_masks'].cpu()
            pred['instances']['pred_keypoints'] = pred['instances']['pred_keypoints'].cpu()
        output.append(preds)


Instruction: Torch.nn has a specific activation function?

Input:
My network has a output layer with Relu activation function, but I want the output is something like "Relu+1", that is I want the output is all bigger than 1 and has the same shape of Relu function.
How should I change my torch.nn network?
My code is like:
self.actor = nn.Sequential(
                     nn.Linear(state_dim, 256),
                     nn.ReLU(),
                     nn.Linear(256, 256),
                     nn.ReLU(),
                     nn.Linear(256, action_dim),
                     nn.ReLU()
)

Output:
There are 2 ways I can think of.

Make self.actor an nn.Module object

class Actor(nn.Module):
    def __int__(self, state_dim, action_dim):
        super().__init__()
        self.linear1 = nn.Linear(state_dim, 256)
        self.relu = nn.ReLU()
        self.linear2 = nn.Linear(256, 256)
        self.linear3 = nn.Linear(256, action_dim)

    def forward(self, x):
        x = self.linear1(x)
        x = self.relu(x)
        x = self.linear2(x)
        x = self.relu(x)
        x = self.linear2(x)
        x = self.relu(x) + 1
        return x

class ......
    self.actor = Actor(state_dim, action_dim)

Make a Module class to do that and add it to the self.actor

class Add1(nn.Module):
    def forward(self, x):
        return x + 1

class ......
    self.actor = nn.Sequential(
                     nn.Linear(state_dim, 256),
                     nn.ReLU(),
                     nn.Linear(256, 256),
                     nn.ReLU(),
                     nn.Linear(256, action_dim),
                     nn.ReLU(),
                     Add1()
)


Instruction: What causes the Pytorch nn Module to return a 1 or 0 for the argmax of this array?

Input:
I have an array containing four random numbers that an argmax function should be returning a 0, 1, 2, or 3, but when the argmax is called from inside the nn.Module model it is always 0 or 1.
I would just like to know how and why it is always getting a 1 or 0 from the four numbers in the array.
Below I have the nn.Module and a comparison of a random array of len 3 computed inside and outside of the model using the function act (Net.act).
from torch import nn
import torch

import random

class Network(nn.Module):
    def __init__(self):
        super().__init__()

        self.net = nn.Sequential(
            nn.Linear(4, 64),
            nn.Tanh(),
            nn.Linear(64, 2))

    def forward(self, x):
        return self.net(x)

    def act(self, obs):
        obs_t = torch.as_tensor(obs, dtype=torch.float32, device=device)
        q_values = self(obs_t.unsqueeze(0))
        max_q_index = torch.argmax(q_values, dim=1)[0]
        action = max_q_index.detach().item()

        return action

Net = Network()

This is a side by side comparison of a random array of len 3 inside and outside of the act function in the nn.Module (Net).
for _ in range(20):
    z = np.array([random.uniform(-1, 1) for _ in range(4)])

    obs_t = torch.as_tensor(z, dtype=torch.float32, device=device)
    q_values = (obs_t.unsqueeze(0))
    max_q_index = torch.argmax(q_values, dim=1)[0]
    action = max_q_index.detach().item()

    print(action, Net.act(z))

Output
3 0
0 0
2 0
0 0
2 0
0 0
1 0
2 0
3 0
0 1
0 0
1 1
3 1
3 0
1 0
0 1
3 0
3 0
0 0
1 0

Output: self(obs_t.unsqueeze(0)) returns a 2 column matrix because the last layer of your model (nn.Linear(64, 2)) is defined to output two columns. max_q_index contains the column index of the largest value in each row of the model output (column index because dim=1). Since there are only 2 columns, max_q_index can only have values of 0 or 1.


Instruction: How to compare training and test performance in a Faster RCNN object detection model

Input:
I'm following a tutorial here for implementing a Faster RCNN against a custom dataset using PyTorch.
This is my training loop:
for images, targets in metric_logger.log_every(data_loader, print_freq, header):
    # FOR GPU
    images = list(image.to(device) for image in images)
    targets = [{k: v.to(device) for k, v in t.items()} for t in targets]

    # Train the model
    loss_dict = model(images, targets)

    # reduce losses over all GPUs for logging purposes
    losses = sum(loss for loss in loss_dict.values())
    loss_dict_reduced = reduce_dict(loss_dict)
    losses_reduced = sum(loss for loss in loss_dict_reduced.values())
    loss_value = losses_reduced.item()

The metric logger (defined here) outputs the following to the console during training:
Epoch: [0]  [  0/226]  eta: 0:07:57  lr: 0.000027  loss: 6.5019 (6.5019)  loss_classifier: 0.8038 (0.8038)  loss_box_reg: 0.1398 (0.1398)  loss_objectness: 5.2717 (5.2717)  loss_rpn_box_reg: 0.2866 (0.2866)  time: 2.1142  data: 0.1003  max mem: 3827
Epoch: [0]  [ 30/226]  eta: 0:02:28  lr: 0.000693  loss: 1.3016 (2.4401)  loss_classifier: 0.2914 (0.4067)  loss_box_reg: 0.2294 (0.2191)  loss_objectness: 0.3558 (1.2913)  loss_rpn_box_reg: 0.3749 (0.5230)  time: 0.7128  data: 0.0923  max mem: 4341

After an epoch has finished, I call an evaluate method which outputs the following:
Test:  [  0/100]  eta: 0:00:25  model_time: 0.0880 (0.0880)  evaluator_time: 0.1400 (0.1400)  time: 0.2510  data: 0.0200  max mem: 4703
Test:  [ 99/100]  eta: 0:00:00  model_time: 0.0790 (0.0786)  evaluator_time: 0.0110 (0.0382)  time: 0.1528  data: 0.0221  max mem: 4703
Test: Total time: 0:00:14 (0.1401 s / it)
Averaged stats: model_time: 0.0790 (0.0786)  evaluator_time: 0.0110 (0.0382)
Accumulating evaluation results...
DONE (t=0.11s).
IoU metric: bbox
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.263
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.346
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.304
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.208
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.308
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.013
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.027
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.175
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.311
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.264
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.351
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.086

I'm a bit confused by the differing metrics used during training and testing - I had wanted to plot training + validation loss (or the equivalent IoU values) so I can visualise training and testing performance, as well as checking if any overfitting is occurring.
My question is, how can I compare the model's training and testing performance?

Output:
The evaluate() function here doesn't calculate any loss. And look at how the loss is calculate in train_one_epoch() here, you actually need model to be in train mode. And make it like the train_one_epoch() except without updating the weight, like
@torch.no_grad()
def evaluate_loss(model, data_loader, device):
    model.train()
    metric_logger = utils.MetricLogger(delimiter="  ")
    header = 'Test:'
    for images, targets in metric_logger.log_every(data_loader, 100, header):
        images = list(image.to(device) for image in images)
        targets = [{k: v.to(device) for k, v in t.items()} for t in targets]

        loss_dict = model(images, targets)

        losses = sum(loss for loss in loss_dict.values())

        # reduce losses over all GPUs for logging purposes
        loss_dict_reduced = utils.reduce_dict(loss_dict)
        losses_reduced = sum(loss for loss in loss_dict_reduced.values())

        metric_logger.update(loss=losses_reduced, **loss_dict_reduced)

But since you need the model to be in eval mode to get bounding boxes. If you need mAP you'll need a loops of the original code too.


Instruction: What is smart way to get batched gather?

Input:
I have two matrices, A and B, with shapes (n, m, k) and (n, m) respectively. n is the batch size, m is the amount of data in a batch, and k is the feature size.
Each element of B is an index less than m (specifically B = torch.randint(high=m, shape=(n,m))).
I want to implement [A[i][B[i]] for i in range(n)] in a smarter way.
Is there a better way in pytorch to implement this without doing for loop?

Output:
You can use
a[torch.arange(n)[:, None], b]

An example:
>>> n, m, k = 3, 2, 5
>>> a = torch.arange(30).view(n, m, k)
>>> b = torch.randint(high=m, size=(n,m))

# first indexer (of shape (n, 1))
>>> torch.arange(n)[:, None]

tensor([[0],
        [1],
        [2]])

# second indexer
>>> b

tensor([[1, 0],
        [0, 1],
        [1, 1]])

The indexers have the shape (3, 1) and (3, 2) respectively so they'll be broadcasted to (3, 2) to effectively have
tensor([[0, 0],
        [1, 1],
        [2, 2]])

and
tensor([[1, 0],
        [0, 1],
        [1, 1]])

which says: for the first row, take 1st (k,) array and put the result and take 0th (k,) array and put the result. This fills in a (m, k) array in the output which is repeated n times for each row,
to get
>>> a[torch.arange(n)[:, None], b]

tensor([[[ 5,  6,  7,  8,  9],
         [ 0,  1,  2,  3,  4]],

        [[10, 11, 12, 13, 14],
         [15, 16, 17, 18, 19]],

        [[25, 26, 27, 28, 29],
         [25, 26, 27, 28, 29]]])

comparing with list comprehension:
>>> [a[i][b[i]] for i in range(n)]

[tensor([[5, 6, 7, 8, 9],
         [0, 1, 2, 3, 4]]),
 tensor([[10, 11, 12, 13, 14],
         [15, 16, 17, 18, 19]]),
 tensor([[25, 26, 27, 28, 29],
         [25, 26, 27, 28, 29]])]


Instruction: Getting this while using pytorch transforms--->TypeError: integer argument expected, got float

Input:
I cloned transfer-learning-library repo and working on maximum classifier discrepancy. I am trying to change the augmentation but getting the following error
Traceback (most recent call last):
  File "mcd.py", line 378, in <module>
    main(args)
  File "mcd.py", line 145, in main
    results = validate(val_loader, G, F1, F2, args)
  File "mcd.py", line 290, in validate
    for i, (images, target) in enumerate(val_loader):
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 521, in __next__
    data = self._next_data()
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 1203, in _next_data
    return self._process_data(data)
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 1229, in _process_data
    data.reraise()
  File "/usr/local/lib/python3.7/dist-packages/torch/_utils.py", line 425, in reraise
    raise self.exc_type(msg)
TypeError: Caught TypeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "../../../common/vision/datasets/imagelist.py", line 48, in __getitem__
    img = self.transform(img)
  File "/usr/local/lib/python3.7/dist-packages/torchvision/transforms/transforms.py", line 60, in __call__
    img = t(img)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/torchvision/transforms/transforms.py", line 750, in forward
    return F.perspective(img, startpoints, endpoints, self.interpolation, fill)
  File "/usr/local/lib/python3.7/dist-packages/torchvision/transforms/functional.py", line 647, in perspective
    return F_pil.perspective(img, coeffs, interpolation=pil_interpolation, fill=fill)
  File "/usr/local/lib/python3.7/dist-packages/torchvision/transforms/functional_pil.py", line 289, in perspective
    return img.transform(img.size, Image.PERSPECTIVE, perspective_coeffs, interpolation, **opts)
  File "/usr/local/lib/python3.7/dist-packages/PIL/Image.py", line 2371, in transform
    im = new(self.mode, size, fillcolor)
  File "/usr/local/lib/python3.7/dist-packages/PIL/Image.py", line 2578, in new
    return im._new(core.fill(mode, size, color))
TypeError: integer argument expected, got float

The previous code was
 # Data loading code
    normalize = T.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
    if args.center_crop:
        train_transform = T.Compose([
            ResizeImage(256),
            T.CenterCrop(224),
            T.RandomHorizontalFlip(),
            T.ToTensor(),
            normalize
        ])
    else:
        train_transform = T.Compose([
            ResizeImage(256),
            T.RandomResizedCrop(224),
            T.RandomHorizontalFlip(),
            T.ToTensor(),
            normalize
        ])
    val_transform = T.Compose([
        ResizeImage(256),
        T.CenterCrop(224),
        T.ToTensor(),
        normalize
    ])

I just added  T.RandomPerspective(distortion_scale = 0.8, p=0.5, fill=0.6) for val_transform.
Before this I also added few other transforms for train_transform but still got the same error.
What could be the problem?

Output:
The fill argument needs to be an integer.
This transform does not support the fill parameter for Tensor types; therefore, if you wish to use the fill parameter, then you must use this transform before the ToTensor transform. At this point, the data is integral.


Instruction: How is get predict accuracy score in Bert Classification

Input:
I am using Bert Classifier for my Chatbot project. I perform the necessary tokenizer operations for the incoming text message. Then I insert it into the model and make a prediction. How can I get the accuracy of this estimate?
for text in test_texts:
    encoded_dict = tokenizer.encode_plus(
        text,
        add_special_tokens=True,
        max_length=max_len,
        pad_to_max_length=True,
        return_attention_mask=True,
        return_tensors='pt',
    )

    input_ids.append(encoded_dict['input_ids'])
    attention_masks.append(encoded_dict['attention_mask'])

input_ids = torch.cat(input_ids, dim=0)
attention_masks = torch.cat(attention_masks, dim=0)
print("input_ids ",input_ids)
print("attention_masks ",attention_masks)

batch_size = 32

prediction_data = TensorDataset(input_ids, attention_masks)
prediction_sampler = SequentialSampler(prediction_data)
prediction_dataloader = DataLoader(prediction_data, sampler=prediction_sampler, batch_size=batch_size)

print("prediction_data ",prediction_data)
print("prediction_sampler ",prediction_sampler)
print("prediction_dataloader ",prediction_dataloader)
model.eval()
predictions, true_labels = [], []

for batch in prediction_dataloader:
    batch = tuple(t.to(device) for t in batch)
    b_input_ids, b_input_mask = batch
    print("b input ids",b_input_ids)
    with torch.no_grad():
        outputs = model(b_input_ids, token_type_ids=None,
                        attention_mask=b_input_mask.to(device))

    logits = outputs[0]
    logits = logits.detach().cpu().numpy()
    label_ids = b_input_mask.to('cpu').numpy()

    predictions.append(logits)
    true_labels.append(label_ids)
    print("logits ",logits)

    print("label_ids ",label_ids)
    print("true_labels ",true_labels)

print('Prediction completed')

prediction_set = []

for i in range(len(true_labels)):
    pred_labels_i = np.argmax(predictions[i], axis=1).flatten()
    prediction_set.append(pred_labels_i)

prediction= [item for sublist in prediction_set for item in sublist]

print("prediction:", prediction[0])

I am looking for a percentage value. will respond or pass depending on the result of this percentage value.

Output:
Accuracy can be directly computed using some libraries.
For example, you can use sklearn:
from sklearn.metrics import accuracy_score
print("Accuracy:", accuracy_score(true_labels, predictions)) # Value between 0 and 1

print("Accuracy Percentage {} %:".format(100*accuracy_score(true_labels, predictions))) # Value between 0 and 100
``


Instruction: How can I avoid getting overlapping keypoints during inference?

Input:
I have been using Detectron2 for recognizing 4 keypoints on each image,
My dummy dataset consists of 1000 images, and I applied augmentations.
def build_train_loader(cls, cfg):
    augs = [
           T.RandomFlip(prob=0.5,horizontal=True),
           T.RandomFlip(prob=0.5,horizontal=False,vertical=True),
           T.RandomRotation(angle=[0, 180]),
           T.RandomSaturation(0.9, 1.9)
           ]
    return build_detection_train_loader(cfg,
                                        mapper=DatasetMapper(cfg,
                                                is_train=True,
                                                augmentations=augs)
                                    )

I have checked the images after those transforms which I have applied (each type of transform was tested separately), and it seems it has done well, the keypoints are positioned correctly.
Now after the training phase (keypoint_rcnn_R_50_FPN_3x.yaml),
I get some identical keypoints, which means in many images the keypoints overlap,
Here are few samples from my results:
[[[180.4211, 332.8872,   0.7105],
[276.3517, 369.3892,   0.7390],
[276.3517, 366.9956,   0.4788],
[220.5920, 296.9836,   0.9515]]]

And from another image:
[[[611.8049, 268.8926,   0.7576],
[611.8049, 268.8926,   1.2022],
[699.7122, 261.2566,   1.7348],
[724.5556, 198.2591,   1.4403]]]

I have compared the inference's results with augmentations and without,
and it seems with augmentation the keypoints are barely getting recognized . gosh, How can it be?
Can someone please suggest any idea how to overcome those kind of mistakes?
what am I doing wrong?
Thank you!
I have added a link to my google colab notebook:
https://colab.research.google.com/drive/1uIzvB8vCWdGrT7qnz2d2npEYCqOxET5S?usp=sharing

Output:
The problem is that there's nothing unique about the different corners of the rectangle. However, in your annotation and in your loss function there is an implicit assumption that the order of the corners is significant:
The corners are labeled in a specific order and the network is trained to output the corners in that specific order.
However, when you augment the dataset, by flipping and rotating the images, you change the implicit order of the corners and now the net does not know which of the four corners to predict at each time.
As far as I can see you have two ways of addressing this issue:

Explicitly force order on the corners:
Make sure that no matter what augmentation the image underwent, for each rectangle the ground truth points are ordered "top left", "top right", "bottom left", "bottom right". This means you'll have to transform the coordinates of the corners (as you are doing now), but also reorder them.
Adding this consistency should help your model overcome the ambiguity in identifying the different corners.

Make the loss invariant to the order of the predicted corners:
Suppose your ground truth rectangle span the domain [0, 1]x[0, 1]: the four corners you should predict are [[0, 0], [1, 1], [1, 0], [0, 1]]. Note that if you predict [[1, 1], [0, 0], [0, 1], [1, 0]] your loss is very high, although you predicted the right corners just in a different order than annotated ones.
Therefore, you should make youy loss invariant to the order of the predicted points:

where pi(i) is a permutation of the corners.


Instruction: Wrong tensor type when trying to do the HuggingFace tutorial (pytorch)

Input:
I've recently been trying to get hands on experience with the transformer library from Hugging Face. Since I'm an absolute noob when it comes to using Pytorch (and Deep Learning in general), I started with the introduction that can be found here.
Here is the code to install dependencies :
#!pip install transformers
!pip install transformers[sentencepiece] # includes transformers dependencies
!pip install datasets # datasets from huggingface hub
!pip install tqdm

Here's the code they propose to use to fine-tune BERT the MNPR dataset (used in the GLUE benchmark). This dataset includes two sentences per "sample", so in the tokenizer we have to use sentence1 and sentence2.
from datasets import load_dataset
from transformers import AutoTokenizer, DataCollatorWithPadding
from torch.utils.data import DataLoader
from transformers import AutoModelForSequenceClassification
from transformers import AdamW
from transformers import get_scheduler
import torch
from tqdm.auto import tqdm

raw_datasets = load_dataset("glue", "mrpc")
checkpoint = "bert-base-uncased"

# functions defining how the tokenizer works
tokenizer = AutoTokenizer.from_pretrained(checkpoint)

def tokenize_function(example):
  return tokenizer(example["sentence1"], example["sentence2"], truncation=True)

# tokenizer will use dynamic padding (https://huggingface.co/course/chapter3/2?fw=pt)
tokenized_datasets = raw_datasets.map(tokenize_function, batched=True)
data_collator = DataCollatorWithPadding(tokenizer=tokenizer)

# remove unecessary columns from data and format in torch tensors
tokenized_datasets = tokenized_datasets.remove_columns(
  ["sentence1", "sentence2", "idx"]
)
tokenized_datasets = tokenized_datasets.rename_column("label", "labels")
tokenized_datasets.set_format("torch")

train_dataloader = DataLoader(
  tokenized_datasets["train"], shuffle=True, batch_size=8, collate_fn=data_collator
)
eval_dataloader = DataLoader(
  tokenized_datasets["validation"], batch_size=8, collate_fn=data_collator
)

# loading model and training requirements
model = AutoModelForSequenceClassification.from_pretrained(checkpoint, num_labels=2)

optimizer = AdamW(model.parameters(), lr=5e-5)

num_epochs = 3
num_training_steps = num_epochs * len(train_dataloader)
lr_scheduler = get_scheduler(
  "linear",
  optimizer=optimizer,
  num_warmup_steps=0,
  num_training_steps=num_training_steps
)
print(num_training_steps)

device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
model.to(device)

progress_bar = tqdm(range(num_training_steps))

# training loop:
model.train()
for epoch in range(num_epochs):
  for batch in train_dataloader:
    batch = {k: v.to(device) for k, v in batch.items()}
    outputs = model(**batch)
    loss = outputs.loss
    loss.backward()

    optimizer.step()
    lr_scheduler.step()
    optimizer.zero_grad()
    progress_bar.update(1)
    # assert 1==0

This works perfectly fine for me in Google Colab. I wanted to do the same thing with another dataset sst2. The code I use is very similar to the one above. The only few lines of code that change are the lines to import the data and the tokenizer (we have one sentence per feature instead of two). I have double-checked and the tokenizer works fine. Here is my code :
# imports
import torch
from datasets import load_dataset # datasets from huggingface
# tokenization
from transformers import AutoTokenizer, DataCollatorWithPadding
from torch.utils.data import DataLoader
# training
from transformers import AdamW, AutoModelForSequenceClassification, get_scheduler
from tqdm.auto import tqdm

# Hyperparameters
batch_size = 8
learning_rate = 5e-5
num_epochs = 3
num_warmup_steps = 0

# load dataset and choosing checkpoint
raw_datasets = load_dataset("glue", "sst2")
checkpoint = "bert-base-uncased"
# load tokenizer
tokenizer = AutoTokenizer.from_pretrained(checkpoint)

# tokenization of dataset
def tokenize_function(example):
  return tokenizer(example["sentence"], truncation=True)

data_collator = DataCollatorWithPadding(tokenizer=tokenizer)
tokenized_datasets = raw_datasets.map(tokenize_function, batched=True)
tokenized_datasets = tokenized_datasets.remove_columns(["sentence", "idx"])
tokenized_datasets = tokenized_datasets.rename_column("label", "labels")
tokenized_datasets.set_format("torch")

# setting DataLoader
train_dataloader = DataLoader(
  tokenized_datasets["train"], shuffle=True, batch_size=batch_size, collate_fn=data_collator
)
eval_dataloader = DataLoader(
  tokenized_datasets["validation"], batch_size=batch_size, collate_fn=data_collator
)

# import model
model = AutoModelForSequenceClassification.from_pretrained(checkpoint, num_labels=1)

# setup training loop
optimizer = AdamW(model.parameters(), lr=learning_rate)

num_training_steps = num_epochs * len(train_dataloader)
print(num_training_steps)
lr_scheduler = get_scheduler(
    "linear",
    optimizer=optimizer,
    num_warmup_steps=num_warmup_steps,
    num_training_steps=num_training_steps
)
# chose device (GPU or CPU)
device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
model.to(device)

progress_bar = tqdm(range(num_training_steps))

model.train()
for epoch in range(num_epochs):
  for batch in train_dataloader:
    batch = {k: v.to(device) for k, v in batch.items()}
    for k,v in batch.items():
      print(f"key={k},v.dtype={v.dtype}, type(v)={type(v)}")
    outputs = model(**batch)
    loss = outputs.loss
    loss.backward()

    optimizer.step()
    lr_scheduler.step()
    optimizer.zero_grad()
    progress_bar.update(1)

And here's the error I get :
RuntimeError                              Traceback (most recent call last)
<ipython-input-11-7893d7715ac2> in <module>()
     69     outputs = model(**batch)
     70     loss = outputs.loss
---> 71     loss.backward()
     72
     73     optimizer.step()

1 frames
/usr/local/lib/python3.7/dist-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
    147     Variable._execution_engine.run_backward(
    148         tensors, grad_tensors_, retain_graph, create_graph, inputs,
--> 149         allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
    150
    151

RuntimeError: Found dtype Long but expected Float

This seems like a very silly mistake, but like I said I'm an absolute pytorch noob and it's difficult for me to know where to start solving this issue. I have checked the type of the values in batch.items() and in both cases, they are all torch.int64 (or torch.long). I tried to change the attention_mask and input_ids values to torch.float32, but I got the same error message.
Thanks in advance.
Python version and packages :

python 3.7.20
Pytorch 1.9.0+cu102
transformers 4.8.2
GPU : Tesla T4 (also tried with tesla P4)

Output:
I found the source of the problem. The problem comes from the line
model = AutoModelForSequenceClassification.from_pretrained(checkpoint, num_labels=1)

Since the dataset has 2 classes, the correct way of calling the model should be
model = AutoModelForSequenceClassification.from_pretrained(checkpoint, num_labels=2)

With this modification, my code now works.


Instruction: AttributeError: 'numpy.ndarray' object has no attribute 'unsqueeze'

Input:
I'm running a training code using pyhtorch and numpy.
This is the plot_example function:
def plot_example(low_res_folder, gen):
    files=os.listdir(low_res_folder)

    gen.eval()
    for file in files:
        image=Image.open("test_images/" + file)
        with torch.no_grad():
            upscaled_img=gen(
                config1.both_transform(image=np.asarray(image))["image"]
                .unsqueeze(0)
                .to(config1.DEVICE)
            )
        save_image(upscaled_img * 0.5 + 0.5, f"saved/{file}")
    gen.train()

The problem I have is that the unsqueeze attribute raises the error:
File "E:\Downloads\esrgan-tf2-masteren\modules\train1.py", line 58, in train_fn
    plot_example("test_images/", gen)

File "E:\Downloads\esrgan-tf2-masteren\modules\utils1.py", line 46, in plot_example
    config1.both_transform(image=np.asarray(image))["image"]

AttributeError: 'numpy.ndarray' object has no attribute 'unsqueeze'

The network is GAN network and gen() represents the Generator.

Output:
Make sure image is a tensor in the shape of [batch size, channels, height, width] before entering any Pytorch layers.
Here you have
image=np.asarray(image)
I would remove this numpy conversion and keep it a torch.tensor.
Or if you really want it to be a numpy array, then right before it enters your generator make sure to use torch.from_numpy() as shown in this documentation on your numpy image before it gets unsqueezed: https://pytorch.org/docs/stable/generated/torch.from_numpy.html
This function is ofcourse an alternative if you don't want to get rid of that original conversion.
Sarthak Jain


Instruction: How to compute the parameter importance in pytorch?

Input:
I want to develop a lifelong learning system,so i need to prevent important parameter from changing.I read related paper 'Memory Aware Synapses: Learning what (not) to forget',a method was mentioned,I need to calculate the gradient of each parameter conresponding to each input image,so how should i write my code in pytorch?
'Memory Aware Synapses: Learning what (not) to forget'

Output:
You can do it using standard optimization procedure and .backward() method on your loss function.
First, scaling as defined in your link:
class Scaler:
    def __init__(self, parameters, delta):
        self.parameters = parameters
        self.delta = delta

    def step(self):
        """Multiplies gradients in place."""
        for param in self.parameters:
            if param.grad is None:
                raise ValueError("backward() has to be called before running scaler")

            param.grad *= self.delta

One can use it just like optimizer.step(), see below (see comments):
model = torch.nn.Sequential(
    torch.nn.Linear(10, 100), torch.nn.ReLU(), torch.nn.Linear(100, 1)
)

scaler = Scaler(model.parameters(), delta=0.001)
optimizer = torch.optim.Adam(model.parameters())
criterion = torch.nn.MSELoss()

X, y = torch.randn(64, 10), torch.randn(64)

# Optimization loop
EPOCHS = 10
for _ in range(EPOCHS):
    output = model(X)
    loss = criterion(output, y)

    loss.backward()  # Now model has the gradients

    optimizer.step()  # Optimize model's parameters

    print(next(model.parameters()).grad)

    scaler.step()  # Scaler gradients

    optimizer.zero_grad()  # Zero gradient before next step

After scaler.step() you will have gradient scaled available inside param.grad for each parameter (just like those are accessed within Scaler's step method) so you can do whatever you want with them.


Instruction: Getting the autograd counter of a tensor in PyTorch

Input:
I am using PyTorch for training a network. I was going through the autograd documentation and here it is mentioned that for each tensor there is a counter that the autograd implements to track the "version" of any tensor. How can I get this counter for any tensor in the graph?
Reason why I need it.
I have encountered the autograd error

[torch.cuda.FloatTensor [x, y, z]], which is output 0 of torch::autograd::CopySlices, is at version 7; expected version 6 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

This is not new to me and I have been successful in handling it before. This time around I am not able to see why the tensor would be at version 7 instead of being at 6. To answer this, I would want to know the version at any given point in the run.
Thanks.

Output:
It can be obtained through the command tensor_name._version.
As an example of how to use it, following MSE is provided.
import torch

a = torch.zeros(10, 5)
print(a._version)  # prints 0
a[:, 1] = 1
print(a._version)  # prints 1


Instruction: What is the correct way of encoding a large batch of documents with sentence transformers/pytorch?

Input:
I am having issues encoding a large number of documents (more than a million) with the sentence_transformers library.
Given a very similar corpus list of strings. When I do:
from sentence_transformers import SentenceTransformer
embedder = SentenceTransformer('msmarco-distilbert-base-v2')
corpus_embeddings = embedder.encode(corpus, convert_to_tensor=False)

After some hours, the process seems to be stuck, as it never finishes and when checking the process viewer nothing is running.
As I am suspicious that this is a ram issue (the GPU board doesn't have enough memory to fit everything in a single step) I tried to split the corpus into batches, transform them into NumPy arrays, and concat them into a single matrix as follows:
from itertools import zip_longest
from sentence_transformers import SentenceTransformer, util
import torch
from loguru import logger
import glob
from natsort import natsorted

def grouper(iterable, n, fillvalue=np.nan):
    args = [iter(iterable)] * n
    return zip_longest(*args, fillvalue=fillvalue)

embedder = SentenceTransformer('msmarco-distilbert-base-v2')

for j, e in enumerate(list(grouper(corpus, 3))):
    try:
#         print('------------------')
        for i  in filter(lambda v: v==v, e):
            corpus_embeddings=embedder.encode(i, convert_to_tensor=False)
            torch.save(corpus_embeddings, f'/Users/user/Downloads/embeddings_part_{j}.npy')
    except TypeError:
        print(j, e)
        logger.debug("TypeError in batch {batch_num}", batch_num=j)

l = []
for e in natsorted(glob.glob("/Users/user/Downloads/*.npy")):
    l.append(torch.load(e))
    corpus_embeddings = np.vstack(l)
corpus_embeddings

Nevertheless, the above procedure doesn't seem to work. The reason is that when I try with a small sample of the corpus with and without the batch approach the matrices I get are different for example:
Without batch approach:
array([[-0.6828216 , -0.26541945,  0.31026787, ...,  0.19941986,
         0.02366139,  0.4489861 ],
       [-0.45781   , -0.02955275,  1.0897563 , ..., -0.20077021,
        -0.37821707,  0.2248317 ],
       [ 0.8532193 , -0.13642257, -0.8872398 , ..., -0.57482916,
         0.12760726, -0.66986346],
       ...,
       [-0.04036704,  0.06745373, -0.6010259 , ..., -0.08174597,
        -0.18513843, -0.64744204],
       [-0.30782765, -0.04935509, -0.11624689, ...,  0.10423593,
        -0.14073376, -0.09206307],
       [-0.77139395, -0.08119706,  0.43753916, ...,  0.1653319 ,
         0.06861683, -0.16276269]], dtype=float32)

With batch approach:
array([[ 0.8532191 , -0.13642241, -0.8872397 , ..., -0.5748289 ,
         0.12760736, -0.6698637 ],
       [ 0.3679317 , -0.21968201,  0.9932826 , ..., -0.86282325,
        -0.04683857,  0.18995859],
       [ 0.23026675,  0.69587034, -0.8116473 , ...,  0.23903558,
         0.413471  , -0.23438476],
       ...,
       [ 0.923319  ,  0.4152724 , -0.3153545 , ..., -0.6863369 ,
         0.01149149, -0.51300013],
       [-0.30782777, -0.04935484, -0.11624689, ...,  0.10423636,
        -0.1407339 , -0.09206269],
       [-0.77139413, -0.08119693,  0.43753892, ...,  0.16533189,
         0.06861652, -0.16276267]], dtype=float32)

What is the correct way of doing the above batch procedure?
UPDATE
After inspecting the above batch procedure, I found that I was able to get the same matrix output with and without the batching when I set to 1 the batch size of the above code (enumerate(list(grouper(corpus, 1)))). Therefore, my question is, what is the correct way of applying the encoder to a large set of documents?

Output:
This line here sorts the input by text length before doing the encode. I have no idea why.
So, either comment those lines out or copy them to your code like
length_sorted_idx = np.argsort([-embedder._text_length(sen) for sen in corpus])
corpus_sorted = [corpus[idx] for idx in length_sorted_idx]

Then use the corpus_sorted to encode and map the output back using length_sorted_idx.
Or just encode it one by one and you won't need to care about which output is from which text.


Instruction: IndexError: list index out of range in prediction of images

Input:
I am doing predictions on images where I write all classes' names and in the test folder, I have 20 images. Please give me some hint as, why I am getting error? How we can check the indices of the model?
Code
import numpy as np
import sys, random
import torch
from torchvision import models, transforms
from PIL import Image
from pathlib import Path
import matplotlib.pyplot as plt
import glob

# Paths for image directory and model
IMDIR = './test'
MODEL = 'checkpoint/resnet18/Monday_31_May_2021_21h_25m_05s/resnet18-1000-regular.pth'

# Load the model for testing
model = models.resnet18()

model.named_children()

torch.save(model.state_dict, MODEL)
model.eval()

# Class labels for prediction
class_names = ['BC', 'BK', 'CC', 'CL', 'CM', 'DF', 'DG', 'DS', 'HL', 'IF', 'JD', 'JS', 'LD', 'LP', 'LS', 'PO', 'RI',
               'SD', 'SG', 'TO']

# Retreive 9 random images from directory
files = Path(IMDIR).resolve().glob('*.*')
print(files)

images = random.sample(list(files), 1)
print(images)
# Configure plots
fig = plt.figure(figsize=(9, 9))
rows, cols = 3, 3

# Preprocessing transformations
preprocess = transforms.Compose([
    transforms.Resize((256, 256)),
    # transforms.CenterCrop(size=224),
    transforms.ToTensor(),
    transforms.Normalize(0.5306, 0.1348)
])

# Enable gpu mode, if cuda available
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

# Perform prediction and plot results
with torch.no_grad():
    for num, img in enumerate(images):
        img = Image.open(img).convert('RGB')
        inputs = preprocess(img).unsqueeze(0).cpu()
        outputs = model(inputs)
        _, preds = torch.max(outputs, 1)
        print(preds)
        label = class_names[preds]
        plt.subplot(rows, cols, num + 1)
        plt.title("Pred: " + label)
        plt.axis('off')
        plt.imshow(img)
'''
Sample run: python test.py test
'''

Traceback
Traceback (most recent call last):
  File "/media/khawar/HDD_Khawar/CVPR/pytorch-cifar100/test_box.py", line 57, in <module>
    label = class_names[preds]
IndexError: list index out of range

Output:
Your error stems from the fact that you don't do any modification to the linear layers of your resnet model.
I suggest adding this code:
# What you have
model = models.resnet18()

# What you need
model.fc = nn.Sequential(
    nn.Linear(model.fc.in_features, len(class_names)))

This changes the last linear layers to outputting the correct amount of nodes
Sarthak