Instruction: Object Detection inference using multi-gpu & multi threading, Pytorch

Input:
I am trying to detect objects in a video using multiple GPUs. I want to distribute frames to GPUs for inference to increase total process time. I succeeded running inference in single gpu, but failed to run on multiple GPUs.

I thought dividing frames per number of gpus and processing inference would decrease the time. If there is another way I can decrease running time, I would be glad to receive suggestions.

I am using pre-trained model provided by Pytorch. What I tried is as follows:
1. I read the video and divide frames by number of gpus I have(currently two NVIDIA GeForce GTX 1080 Ti)
2. Then, I distributed frames to gpus and process object detection inference.
(Later I planned to use multi-threads to dynamically distribute frames per number of gpus, but currently I made it static)

The same method I tried worked well in Tensorflow using with tf.device() and I am trying to make it possible in Pytorch as well.

pytorch_multithread.py

...
    def detection_gpu(frame_list, device_name, device, detect, model):
        model.to(device)
        model.eval()

        for frame in frame_list:
            start = time.time()
            detect.bounding_box_rcnn(frame, model=model)
            end = time.time()

            cv2.putText(frame, '{:.2f}ms'.format((end - start) * 1000), (40, 40), cv2.FONT_HERSHEY_SIMPLEX, 0.75,
                        (255, 0, 0),
                        2)

            cv2.imshow(str(device_name), frame)

            if cv2.waitKey(1) & 0xFF == ord('q'):
                break

    def main():
        args = arg_parse()

        VIDEO_PATH = args.video

        print("Loading network.....")
        model = models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
        print("Network successfully loaded")

        num_gpus = torch.cuda.device_count()
        if torch.cuda.is_available() and num_gpus > 1:
            device = ["cuda:{}".format(i) for i in range(num_gpus)]
        elif num_gpus == 1:
            device = "cuda"
        else:
            device = "cpu"
        # class names ex) person, car, truck, and etc.
        PATH_TO_LABELS = "labels/mscoco_labels.names"

        # load detection class, default confidence threshold is 0.5
        if num_gpus>1:
            detect = [DetectBoxes(PATH_TO_LABELS, device[i], conf_threshold=args.confidence) for i in range(num_gpus)]
        else:
            detect = [DetectBoxes(PATH_TO_LABELS, device, conf_threshold=args.confidence) for i in range(1)]

        cap = cv2.VideoCapture(VIDEO_PATH)

        # find number of gpus that is available
        frame_length = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))

        # TODO: CPU환경 고려하기
        # divide frames of video by number of gpus
        div = frame_length // num_gpus
        divide_point = [i for i in range(frame_length) if i != 0 and i % div == 0]
        divide_point.pop()

        frame_list = []
        fragments = []
        count = 0
        while cap.isOpened():
            hasFrame, frame = cap.read()
            if not hasFrame:
                frame_list.append(fragments)
                break
            if count in divide_point:
                frame_list.append(fragments)
                fragments = []
            fragments.append(frame)
            count += 1
        cap.release()

        detection_gpu(frame_list[0], 0, device[0], detect[0], model)
        detection_gpu(frame_list[1], 1, device[1], detect[1], model)
        # Process object detection using threading
        # thread_detection = [ThreadWithReturnValue(target=detection_gpu,
        #                                           args=(frame_list[i], i, detect, model))
        #                     for i in range(num_gpus)]
        #
        #
        # final_list = []
        # # Begin operating threads
        # for th in thread_detection:
        #     th.start()
        #
        # # Once tasks are completed get return value (frames) and put to new list
        # for th in thread_detection:
        #     final_list.extend(th.join())
        cv2.destroyAllWindows()

detection_boxes_pytorch.py

    def bounding_box_rcnn(self, frame, model):
            print(self.device)
            # Image is converted to image Tensor
            transform = transforms.Compose([transforms.ToTensor()])
            img = transform(frame).to(self.device)
            with torch.no_grad():
                # The image is passed through model to get predictions
                pred = model([img])

            # classes, bounding boxes, confidence scores are gained
            # only classes and bounding boxes > confThershold are passed to draw_boxes
            pred_class = [self.classes[i] for i in list(pred[0]['labels'].cpu().clone().numpy())]
            pred_boxes = [[(i[0], i[1]), (i[2], i[3])] for i in list(pred[0]['boxes'].detach().cpu().clone().numpy())]
            pred_score = list(pred[0]['scores'].detach().cpu().clone().numpy())
            pred_t = [pred_score.index(x) for x in pred_score if x > self.confThreshold][-1]
            pred_colors = [i for i in list(pred[0]['labels'].cpu().clone().numpy())]
            pred_boxes = pred_boxes[:pred_t + 1]
            pred_class = pred_class[:pred_t + 1]

            for i in range(len(pred_boxes)):
                left = int(pred_boxes[i][0][0])
                top = int(pred_boxes[i][0][1])
                right = int(pred_boxes[i][1][0])
                bottom = int(pred_boxes[i][1][1])

                color = STANDARD_COLORS[pred_colors[i] % len(STANDARD_COLORS)]

                self.draw_boxes(frame, pred_class[i], pred_score[i], left, top, right, bottom, color)

The error I get is as follows:

    Traceback (most recent call last):
      File "C:/Users/username/Desktop/Object_Detection_Video_AllInOne/pytorch_multithread.py", line 133, in <module>
        main()
      File "C:/Users/username/Desktop/Object_Detection_Video_AllInOne/pytorch_multithread.py", line 113, in main
        detection_gpu(frame_list[1], 1, device[1], detect[1], model)
      File "C:/Users/username/Desktop/Object_Detection_Video_AllInOne/pytorch_multithread.py", line 39, in detection_gpu
        detect.bounding_box_rcnn(frame, model=model)
      File "C:\Users\username\Desktop\Object_Detection_Video_AllInOne\p_utils\detection_boxes_pytorch.py", line 64, in bounding_box_rcnn
        pred = model([img])
      File "C:\Users\username\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\nn\modules\module.py", line 493, in __call__
        result = self.forward(*input, **kwargs)
      File "C:\Users\username\AppData\Local\Programs\Python\Python37\lib\site-packages\torchvision\models\detection\generalized_rcnn.py", line 51, in forward
        proposals, proposal_losses = self.rpn(images, features, targets)
      File "C:\Users\username\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\nn\modules\module.py", line 493, in __call__
        result = self.forward(*input, **kwargs)
      File "C:\Users\username\AppData\Local\Programs\Python\Python37\lib\site-packages\torchvision\models\detection\rpn.py", line 409, in forward
        proposals = self.box_coder.decode(pred_bbox_deltas.detach(), anchors)
      File "C:\Users\username\AppData\Local\Programs\Python\Python37\lib\site-packages\torchvision\models\detection\_utils.py", line 168, in decode
        rel_codes.reshape(sum(boxes_per_image), -1), concat_boxes
      File "C:\Users\username\AppData\Local\Programs\Python\Python37\lib\site-packages\torchvision\models\detection\_utils.py", line 199, in decode_single
        pred_ctr_x = dx * widths[:, None] + ctr_x[:, None]
    RuntimeError: binary_op(): expected both inputs to be on same device, but input a is on cuda:1 and input b is on cuda:0

Output: Pytorch provides DataParallel module to run a model on mutiple GPUs. Detailed documentation of DataParallel and toy example can be found here and here.


Instruction: Fine Tuning Pretrained Model MobileNet_V2 in Pytorch

Input:
I am new to pyTorch and I am trying to Create a Classifier where I have around 10 kinds of Images Folder Dataset, for this task I am using Pretrained model( MobileNet_v2 ) but the problem is I am not able to change the FC layer of it. There is not model.fc attribute.
Can anyone help me to do this.
Thanks

Output:
Do something like below:

import torch
model = torch.hub.load('pytorch/vision', 'mobilenet_v2', pretrained=True)
print(model.classifier)

model.classifier[1] = torch.nn.Linear(in_features=model.classifier[1].in_features, out_features=10)
print(model.classifier)

output:

Sequential(
  (0): Dropout(p=0.2)
  (1): Linear(in_features=1280, out_features=1000, bias=True)
)
Sequential(
  (0): Dropout(p=0.2)
  (1): Linear(in_features=1280, out_features=10, bias=True)
)

Note: you would need torch >= 1.1.0 to use torch.hub.


Instruction: How the parenthesis are used right after the object's name?

Input:
I want to know about the parenthesis after the Object's name.

I am learning AI and building the AI model, now in the Tutorial's code the author has written a line which is containing the Parenthesis right after the object's name which is : self.model(...)

Where self.model is the Object of the Network class.
How the objects are having parenthesis being an object, not a function?
Now I want to know about the parenthesis after the Object's name.

class Network(nn.Module):

    def __init__(self, input_size, nb_action):
        super(Network, self).__init__()
        self.input_size = input_size
        self.nb_action = nb_action
        self.fc1 = nn.Linear(input_size, 30)
        self.fc2 = nn.Linear(30, nb_action)

    def forward(self, state):
        x = F.relu(self.fc1(state))
        q_values = self.fc2(x)
        return q_values

class Dqn():

    def __init__(self, input_size, nb_action, gamma):
        self.gamma = gamma
        self.reward_window = []
        self.model = Network(input_size, nb_action)
        self.memory = ReplayMemory(100000)
        self.optimizer = optim.Adam(self.model.parameters(), lr = 0.001)
        self.last_state = torch.Tensor(input_size).unsqueeze(0)
        self.last_action = 0
        self.last_reward = 0

    def select_action(self, state):
        probs = F.softmax(self.model(Variable(state, volatile = True))*100) # <-- The problem is here where the self.model object is CALLED with Parenthesis.
        action = probs.multinomial(10)
        return action.data[0,0]

Output:
In python, everything is an object. The functions you create and call are also objects. Anything in python that can be called is a callable object.

However, if you want a class object in python to be a callable object, the __call__ method must be defined inside the class.

When the object is called, the __call__(self, ...) method gets called.

Here is an example.

class Foo:
    def __init__(self, x=0):
        self.x = x

    def __str__(self):
        return f'{self.__class__.__name__}({self.x})'

    def __call__(self, *args):
        print(f'{self} has been called with {args} as arguments')

f1 = Foo(5)
f2 = Foo()

f1()  # f1.__call__()
f2()  # f2.__call__()
f1(1, 2, 3)  # f1.__call__(1, 2, 3)
f2(10, 20)  # f2.__call__(10, 20)

Output:

Foo(5) has been called with () as arguments
Foo(0) has been called with () as arguments
Foo(5) has been called with (1, 2, 3) as arguments
Foo(0) has been called with (10, 20) as arguments

This is how you can make a python object a callable object.


Instruction: Pytorch Grayscale input to Vgg

Input:
I am new to pytorch and I want to use Vgg for transfer learning.
I want to delete the fully connected layers and add some new fully connected layers. Also rather than RGB input I want to use grayscale input. For this I will add the weights of the input layer and get a single weight. So the three channel's weights will be added.

I achieved deleting the fully connected layers but I am having trouble with grayscale part. I add the three weights together and form a new weight. Then I try to change the state dict of the vgg model but this gives me error. The networks code is below:

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
    vgg=models.vgg16(pretrained = True).features[:30]

    w1=vgg.state_dict()['0.weight'][:,0,:,:] #first channel of first input layer's weight
    w2=vgg.state_dict()['0.weight'][:,1,:,:]
    w3=vgg.state_dict()['0.weight'][:,2,:,:]
    w4=w1+w2+w3 # add the three weigths of the channels
    w4=w4.unsqueeze(1) # make it 4 dimensional

    a=vgg.state_dict()#create a new statedict
    a['0.weight']=w4 #replace the new state dict's weigt

    vgg.load_state_dict(a) # this line gives the error,load the new state dict

    self.vgg =nn.Sequential(vgg)
    self.fc1 = nn.Linear(14*14*512, 1000)
    self.fc2 = nn.Linear(1000, 2)

def forward(self, x):
    x = self.vgg(x)
    x = x.view(-1, 14 * 14 * 512)
    x = F.relu(self.fc1(x))
    x = self.fc2(x)
    return x

This gives an error of:

  RuntimeError: Error(s) in loading state_dict for Sequential:  size
  mismatch for 0.weight: copying a param with shape torch.Size([64, 1,
  3, 3]) from checkpoint, the shape in current model is torch.Size([64,
  3, 3, 3]).

So it doesn't allow me to replace the weight with a different sized weight. Is there a solution for this problem or is there anything other that I can try. All I want to do is use the vgg's layers up to fully connected layers and change the first layers weights.

Output:
In short: The Error is caused by Mismatch between pretrained model parameters and the vgg model
Reason: You modified the parameters in pretrained model from [64,3,3,3] -> [64,1,3,3] by adding, but you didn't change the structure of VGG, which still needs a [64,3,3,3] shape of input.
Resolution: Remove the first convolution layer of VGG structure and add a new one which makes it to fit you input shape.


Instruction: Trouble Converting LSTM Pytorch Model to ONNX

Input:
I am trying to export my LSTM Anomally-Detection Pytorch model to ONNX, but I'm experiencing errors. Please take a look at my code below.

Note: My data is shaped as [2685, 5, 6].
Here is where I define my model:

class Model(torch.nn.Module):
    def __init__(self, input_dim, hidden_dim, layer_dim):
        super(Model, self).__init__()
        self.hidden_dim = hidden_dim
        self.layer_dim = layer_dim
        self.lstm = nn.LSTM(input_dim, hidden_dim, layer_dim, batch_first=True)
        self.fc1 = torch.nn.Linear(hidden_dim, hidden_dim)
        self.fc2 = torch.nn.Linear(hidden_dim, input_dim)
    def forward(self, x):
        h0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim).requires_grad_()
        c0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim).requires_grad_()
        out, (hn, cn) = self.lstm(x, (h0.detach(), c0.detach()))
        out = self.fc1(out)
        out = self.fc2(out)
        return out

input_dim = 6
hidden_dim = 3
layer_dim = 2
model = Model(input_dim, hidden_dim, layer_dim)

I can train it and test with it fine. But the problem comes when exporting:

model.eval()
import torch.onnx
torch_out = torch.onnx.export(model,
                         torch.randn(2685, 5, 6),
                         "onnx_model.onnx",
                         export_params = True
                        )

But I have the following error:

LSTM(6, 3, num_layers=2, batch_first=True)
Linear(in_features=3, out_features=3, bias=True)
Linear(in_features=3, out_features=6, bias=True)
['input_1', 'l_lstm_LSTM', 'l_fc1_Linear', 'l_fc2_Linear', 'l_lstm_LSTM', 'l_fc1_Linear', 'l_fc2_Linear', 'l_lstm_LSTM', 'l_fc1_Linear', 'l_fc2_Linear', 'l_lstm_LSTM', 'l_fc1_Linear', 'l_fc2_Linear', 'l_lstm_LSTM', 'l_fc1_Linear', 'l_fc2_Linear', 'l_lstm_LSTM', 'l_fc1_Linear', 'l_fc2_Linear', 'l_lstm_LSTM', 'l_fc1_Linear', 'l_fc2_Linear', 'l_lstm_LSTM', 'l_fc1_Linear', 'l_fc2_Linear', 'l_lstm_LSTM', 'l_fc1_Linear', 'l_fc2_Linear', 'l_lstm_LSTM', 'l_fc1_Linear', 'l_fc2_Linear']

/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/onnx/symbolic.py:173: UserWarning: ONNX export failed on RNN/GRU/LSTM because batch_first not supported
  warnings.warn("ONNX export failed on " + op + " because " + msg + " not supported")

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-264-28c6c55537ab> in <module>()
     10                          torch.randn(2685, 5, 6),
     11                          "onnx_model.onnx",
---> 12                          export_params = True
     13                         )

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/onnx/__init__.py in export(*args, **kwargs)
     23 def export(*args, **kwargs):
     24     from torch.onnx import utils
---> 25     return utils.export(*args, **kwargs)
     26
     27

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/onnx/utils.py in export(model, args, f, export_params, verbose, training, input_names, output_names, aten, export_raw_ir, operator_export_type, opset_version, _retain_param_name, do_constant_folding, strip_doc_string)
    129             operator_export_type=operator_export_type, opset_version=opset_version,
    130             _retain_param_name=_retain_param_name, do_constant_folding=do_constant_folding,
--> 131             strip_doc_string=strip_doc_string)
    132
    133

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/onnx/utils.py in _export(model, args, f, export_params, verbose, training, input_names, output_names, operator_export_type, export_type, example_outputs, propagate, opset_version, _retain_param_name, do_constant_folding, strip_doc_string)
    367         if export_params:
    368             proto, export_map = graph._export_onnx(params_dict, opset_version, defer_weight_export, operator_export_type,
--> 369                                                    strip_doc_string)
    370         else:
    371             proto, export_map = graph._export_onnx({}, opset_version, False, operator_export_type, strip_doc_string)

RuntimeError: ONNX export failed: Couldn't export operator aten::lstm

Defined at:
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/rnn.py(522): forward_impl
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/rnn.py(539): forward_tensor
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/rnn.py(559): forward
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py(481): _slow_forward
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py(491): __call__
<ipython-input-255-468cef410a2c>(14): forward
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py(481): _slow_forward
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py(491): __call__
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/jit/__init__.py(294): forward
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py(493): __call__
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/jit/__init__.py(231): get_trace_graph
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/onnx/utils.py(225): _trace_and_get_graph_from_model
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/onnx/utils.py(266): _model_to_graph
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/onnx/utils.py(363): _export
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/onnx/utils.py(131): export
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/onnx/__init__.py(25): export
<ipython-input-264-28c6c55537ab>(12): <module>
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/IPython/core/interactiveshell.py(2963): run_code
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/IPython/core/interactiveshell.py(2903): run_ast_nodes
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/IPython/core/interactiveshell.py(2785): _run_cell
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/IPython/core/interactiveshell.py(2662): run_cell
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/ipykernel/zmqshell.py(537): run_cell
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/ipykernel/ipkernel.py(208): do_execute
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/ipykernel/kernelbase.py(399): execute_request
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/ipykernel/kernelbase.py(233): dispatch_shell
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/ipykernel/kernelbase.py(283): dispatcher
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/tornado/stack_context.py(276): null_wrapper
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py(432): _run_callback
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py(480): _handle_recv
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py(450): _handle_events
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/tornado/stack_context.py(276): null_wrapper
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/tornado/platform/asyncio.py(117): _handle_events
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/asyncio/events.py(145): _run
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/asyncio/base_events.py(1432): _run_once
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/asyncio/base_events.py(422): run_forever
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/tornado/platform/asyncio.py(127): start
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/ipykernel/kernelapp.py(486): start
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/traitlets/config/application.py(658): launch_instance
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/ipykernel/__main__.py(3): <module>
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/runpy.py(85): _run_code
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/runpy.py(193): _run_module_as_main

Graph we tried to export:
graph(%input.1 : Float(2685, 5, 6),
      %lstm.weight_ih_l0 : Float(12, 6),
      %lstm.weight_hh_l0 : Float(12, 3),
      %lstm.bias_ih_l0 : Float(12),
      %lstm.bias_hh_l0 : Float(12),
      %lstm.weight_ih_l1 : Float(12, 3),
      %lstm.weight_hh_l1 : Float(12, 3),
      %lstm.bias_ih_l1 : Float(12),
      %lstm.bias_hh_l1 : Float(12),
      %fc1.weight : Float(3, 3),
      %fc1.bias : Float(3),
      %fc2.weight : Float(6, 3),
      %fc2.bias : Float(6)):
  %13 : Long() = onnx::Constant[value={0}](), scope: Model
  %14 : Tensor = onnx::Shape(%input.1), scope: Model
  %15 : Long() = onnx::Gather[axis=0](%14, %13), scope: Model
  %16 : Long() = onnx::Constant[value={2}](), scope: Model
  %17 : Long() = onnx::Constant[value={3}](), scope: Model
  %18 : Tensor = onnx::Unsqueeze[axes=[0]](%16)
  %19 : Tensor = onnx::Unsqueeze[axes=[0]](%15)
  %20 : Tensor = onnx::Unsqueeze[axes=[0]](%17)
  %21 : Tensor = onnx::Concat[axis=0](%18, %19, %20)
  %22 : Float(2, 2685, 3) = onnx::ConstantOfShape[value={0}](%21), scope: Model
  %23 : Long() = onnx::Constant[value={0}](), scope: Model
  %24 : Tensor = onnx::Shape(%input.1), scope: Model
  %25 : Long() = onnx::Gather[axis=0](%24, %23), scope: Model
  %26 : Long() = onnx::Constant[value={2}](), scope: Model
  %27 : Long() = onnx::Constant[value={3}](), scope: Model
  %28 : Tensor = onnx::Unsqueeze[axes=[0]](%26)
  %29 : Tensor = onnx::Unsqueeze[axes=[0]](%25)
  %30 : Tensor = onnx::Unsqueeze[axes=[0]](%27)
  %31 : Tensor = onnx::Concat[axis=0](%28, %29, %30)
  %32 : Float(2, 2685, 3) = onnx::ConstantOfShape[value={0}](%31), scope: Model
  %33 : Long() = onnx::Constant[value={1}](), scope: Model/LSTM[lstm]
  %34 : Long() = onnx::Constant[value={2}](), scope: Model/LSTM[lstm]
  %35 : Double() = onnx::Constant[value={0}](), scope: Model/LSTM[lstm]
  %36 : Long() = onnx::Constant[value={0}](), scope: Model/LSTM[lstm]
  %37 : Long() = onnx::Constant[value={0}](), scope: Model/LSTM[lstm]
  %38 : Long() = onnx::Constant[value={1}](), scope: Model/LSTM[lstm]
  %input.2 : Float(2685!, 5!, 3), %40 : Float(2, 2685, 3), %41 : Float(2, 2685, 3) = aten::lstm(%input.1, %22, %32, %lstm.weight_ih_l0, %lstm.weight_hh_l0, %lstm.bias_ih_l0, %lstm.bias_hh_l0, %lstm.weight_ih_l1, %lstm.weight_hh_l1, %lstm.bias_ih_l1, %lstm.bias_hh_l1, %33, %34, %35, %36, %37, %38), scope: Model/LSTM[lstm]
  %42 : Float(3!, 3!) = onnx::Transpose[perm=[1, 0]](%fc1.weight), scope: Model/Linear[fc1]
  %43 : Float(2685, 5, 3) = onnx::MatMul(%input.2, %42), scope: Model/Linear[fc1]
  %44 : Float(2685, 5, 3) = onnx::Add(%43, %fc1.bias), scope: Model/Linear[fc1]
  %45 : Float(3!, 6!) = onnx::Transpose[perm=[1, 0]](%fc2.weight), scope: Model/Linear[fc2]
  %46 : Float(2685, 5, 6) = onnx::MatMul(%44, %45), scope: Model/Linear[fc2]
  %47 : Float(2685, 5, 6) = onnx::Add(%46, %fc2.bias), scope: Model/Linear[fc2]
  return (%47)

What does this mean? What should I do to export properly?

Output:
If you're coming here from Google the previous answers are no longer up to date. ONNX now supports an LSTM operator. Take care as exporting from PyTorch will fix the input sequence length by default unless you use the dynamic_axes parameter.
Below is a minimal LSTM export example I adapted from the torch.onnx FAQ
import torch
import onnx
from torch import nn
import numpy as np
import onnxruntime.backend as backend
import numpy as np

torch.manual_seed(0)

layer_count = 4

model = nn.LSTM(10, 20, num_layers=layer_count, bidirectional=True)
model.eval()

with torch.no_grad():
    input = torch.randn(1, 3, 10)
    h0 = torch.randn(layer_count * 2, 3, 20)
    c0 = torch.randn(layer_count * 2, 3, 20)
    output, (hn, cn) = model(input, (h0, c0))

    # default export
    torch.onnx.export(model, (input, (h0, c0)), 'lstm.onnx')
    onnx_model = onnx.load('lstm.onnx')
    # input shape [5, 3, 10]
    print(onnx_model.graph.input[0])

    # export with `dynamic_axes`
    torch.onnx.export(model, (input, (h0, c0)), 'lstm.onnx',
                    input_names=['input', 'h0', 'c0'],
                    output_names=['output', 'hn', 'cn'],
                    dynamic_axes={'input': {0: 'sequence'}, 'output': {0: 'sequence'}})
    onnx_model = onnx.load('lstm.onnx')
    # input shape ['sequence', 3, 10]

# Check export
y, (hn, cn) = model(input, (h0, c0))
y_onnx, hn_onnx, cn_onnx = backend.run(
    onnx_model,
    [input.numpy(), h0.numpy(), c0.numpy()],
    device='CPU'
)

np.testing.assert_almost_equal(y_onnx, y.detach(), decimal=5)
np.testing.assert_almost_equal(hn_onnx, hn.detach(), decimal=5)
np.testing.assert_almost_equal(cn_onnx, cn.detach(), decimal=5)

I've tested this example with:
torch==1.4.0,
onnx=1.7.0


Instruction: Should the embedding layer be changed during training a neural network?

Input:
I'm a new one for the field of deep learning and Pytorch.

Recently when I learn one of the pytorch tutorial example for NER task, I found the embedding of nn.Embedding changed during the training.

So my question is should the embedding be changed during training the network?

And if I want to load a pre-trained embedding (for example, trained word2vec embedding) into a PyTorch embedding layer, should the pre-trained embedding also be changed during the train process?

Or how could I prevent updating the embeddings?

Thank you.

Output:
One can either learn embeddings during the task, finetune them for task at hand or leave as they are (provided they have been learned in some fashion before).

In the last case, with standard embeddings like word2vec one eventually finetunes (using small learning rate), but uses vocabulary and embeddings provided. When it comes to current SOTA like BERT fine-tuning on your data should always be done, but in unsupervised way (as trained originally).

Easiest way to use them is static method of torch.nn.Embedding.from_pretrained (docs) and provide Tensor with your pretrained data.

If you want the layer to be trainable, pass freeze=False, by default it's not as you want.


Instruction: RuntimeError: size mismatch, m1: [192 x 68], m2: [1024 x 68] at /opt/conda/conda-bld/pytorch_/work/aten/src/THC/generic/THCTensorMathBlas.cu:268

Input:
I'm getting a size mismatch error that I can't understand.

(Pdb) self.W_di
Linear(in_features=68, out_features=1024, bias=True)
(Pdb) indices.size()
torch.Size([32, 6, 68])
(Pdb) self.W_di(indices)
*** RuntimeError: size mismatch, m1: [192 x 68], m2: [1024 x 68] at /opt/conda/conda-bld/pytorch_1556653099582/work/aten/src/THC/generic/THCTensorMathBlas.cu:268

Why is there a mismatch?
Maybe because of the way I defined the weight in forward (instead of _init_)?

This is how I defined self.W_di:

def forward(self):

    if self.W_di is None:
        self.W_di_weight = nn.Parameter(torch.randn(mL_n * 2,1024).to(device))
        self.W_di_bias = nn.Parameter(torch.ones(1024).to(device))

    self.W_di = nn.Linear(mL_n * 2, 1024)
    self.W_di.weight = self.W_di_weight
    self.W_di.bias = self.W_di_bias

    result = self.W_di(indices)

Any pointer would be highly appreciated!

Output:
Check my answer in here in general you may set

self.W_di = nn.Linear(mL_n * 2, 68)

Or increase the in features.


Instruction: Assign Keras/TF/PyTorch layer to hardware type

Input:
Suppose we have the following architecture:

Multiple CNN layers
RNN layer
(Time-distributed) Dense classification layer

We want to train this architecture now. Our fancy GPU is very fast at solving the CNN layers. Although using a lower clockrate, it can perform many convolutions in parallel, thus the speed. Our fancy CPU however is faster for the (very long) resulting time series, because the time steps cannot be parallelized, and the processing profits from the higher CPU clockrate. So the (supposedly) smart idea for execution would look like this:

Multiple CNN layers (run on GPU)
RNN layer (run on CPU)
(Time-distributed) Dense classification layer (run on GPU/CPU)

This lead me to two important questions:

Is it possible, with any of the frameworks mentioned in the title, to distribute certain layers to certain hardware, and how?

If it is possible, would the overhead for the additional memory operations, e.g. tranferring between GPU-/CPU-RAM, render the whole idea useless?

Output:
Basically, in Pytorch you can control the device on which variables/parameters reside. AFAIK, it is your responsibility to make sure that for each operation all the arguments reside on the same device: i.e., you cannot conv(x, y) where x is on GPU and y is on CPU.

This is done via pytorch's .to() method that moves a module/variable .to('cpu') or .to('cuda:0')


Instruction: Translating Pytorch program into Keras: different results

Input:
I have translated a pytorch program into keras.

A working Pytorch program:

import numpy as np
import cv2
import torch
import torch.nn as nn
from skimage import segmentation

np.random.seed(1)
torch.manual_seed(1)

fi = "in.jpg"

class MyNet(nn.Module):
    def __init__(self, n_inChannel, n_outChannel):
        super(MyNet, self).__init__()

        self.seq = nn.Sequential(
            nn.Conv2d(n_inChannel, n_outChannel, kernel_size=3, stride=1, padding=1),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(n_outChannel),

            nn.Conv2d(n_outChannel, n_outChannel, kernel_size=3, stride=1, padding=1),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(n_outChannel),

            nn.Conv2d(n_outChannel, n_outChannel, kernel_size=1, stride=1, padding=0),
            nn.BatchNorm2d(n_outChannel)
        )

    def forward(self, x):
        return self.seq(x)

im = cv2.imread(fi)
data = torch.from_numpy(np.array([im.transpose((2, 0, 1)).astype('float32')/255.]))
data = data.cuda()

labels = segmentation.slic(im, compactness=100, n_segments=10000)
labels = labels.flatten()
u_labels = np.unique(labels)
label_indexes = np.array([np.where(labels == u_label)[0] for u_label in u_labels])

n_inChannel = 3
n_outChannel = 100

model = MyNet(n_inChannel, n_outChannel)
model.cuda()
model.train()
loss_fn = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)

label_colours = np.random.randint(255,size=(100,3))

for batch_idx in range(100):
    optimizer.zero_grad()
    output = model( data )[ 0 ]
    output = output.permute( 1, 2, 0 ).view(-1, n_outChannel)
    ignore, target = torch.max( output, 1 )

    im_target = target.data.cpu().numpy()
    nLabels = len(np.unique(im_target))

    im_target_rgb = np.array([label_colours[ c % 100 ] for c in im_target]) # correct position of "im_target"
    im_target_rgb = im_target_rgb.reshape( im.shape ).astype( np.uint8 )

    for inds in label_indexes:
        u_labels_, hist = np.unique(im_target[inds], return_counts=True)
        im_target[inds] = u_labels_[np.argmax(hist, 0)]

    target = torch.from_numpy(im_target)
    target = target.cuda()

    loss = loss_fn(output, target)

    loss.backward()
    optimizer.step()

    print (batch_idx, '/', 100, ':', nLabels, loss.item())

    if nLabels <= 3:
        break
fo = "out.jpg"
cv2.imwrite(fo, im_target_rgb)

(source: https://github.com/kanezaki/pytorch-unsupervised-segmentation/blob/master/demo.py)

My translation into Keras:

import cv2
import numpy as np
from skimage import segmentation
from keras.layers import Conv2D, BatchNormalization, Input, Reshape
from keras.models import Model
import keras.backend as k
from keras.optimizers import SGD, Adam
from skimage.util import img_as_float
from skimage import io
from keras.models import Sequential

np.random.seed(0)

fi = "in.jpg"
im = cv2.imread(fi).astype(float)/255.

labels = segmentation.slic(im, compactness=100, n_segments=10000)
labels = labels.flatten()
print (labels.shape)
u_labels = np.unique(labels)

label_indexes = [np.where(labels == u_label)[0] for u_label in np.unique(labels)]

n_channels = 100

model = Sequential()

model.add (  Conv2D(n_channels, kernel_size=3, activation='relu', input_shape=im.shape, padding='same'))
model.add( BatchNormalization())

model.add( Conv2D(n_channels, kernel_size=3, activation='relu', padding='same'))
model.add( BatchNormalization())

model.add( Conv2D(n_channels, kernel_size=1, padding='same'))
model.add( BatchNormalization())

model.add( Reshape((im.shape[0] * im.shape[1], n_channels)))

img = np.expand_dims(im,0)
print (img.shape)
output = model.predict(img)

print (output.shape)

im_target = np.argmax(output[0], 1)
print (im_target.shape)

for inds in label_indexes:
    u_labels_, hist = np.unique(im_target[inds], return_counts=True)
    im_target[inds] = u_labels_[np.argmax(hist, 0)]

def custom_loss(loss_target, loss_output):
  return k.categorical_crossentropy(target=k.stack(loss_target), output=k.stack(loss_output), from_logits=True)

model.compile(optimizer=SGD(lr=0.1, momentum=0.9), loss=custom_loss)

model.fit(img, output, epochs=100, batch_size=1, verbose=1)

pred_result = model.predict(x=[img])[0]
print (pred_result.shape)

target = np.argmax(pred_result, 1)
print (target.shape)

nLabels = len(np.unique(target))

label_colours = np.random.randint(255, size=(100, 3))

im_target_rgb = np.array([label_colours[c % 100] for c in im_target])

im_target_rgb = im_target_rgb.reshape(im.shape).astype(np.uint8)

cv2.imwrite("out.jpg", im_target_rgb)

However, Keras output is really different than of pytorch

Input image:

Pytorch result:

Keras result:

Could someone help me for this translation?

Edit 1:

I corrected two errors as advised by @sebrockm
1. removed `relu` from last conv layer
2. added `from_logits = True` in the loss function

Also, changed the no. of conv layers from 4 to 3 to match with the original code.

However, output image did not improve than before and the `loss` are resulted in negative:

Epoch 99/100
1/1 [==============================] - 0s 92ms/step - loss: -22.8380

Epoch 100/100
1/1 [==============================] - 0s 99ms/step - loss: -23.039

I think that the Keras code lacks connection between model and output. However, could not figure out to make this connection.

Output:
Two major mistakes that I see (likely related):

The last convolutional layer in the original model does not have an activation function, while your translation uses relu.
The original model uses CrossEntropyLoss as loss function, while your model uses categorical_crossentropy with logits=False (a default argument). Without mathematical background the difference is tricky to explain, but in short: CrossEntropyLoss has a softmax built in, that's why the model doesn't have one on the last layer. To do the same in keras, use k.categorical_crossentropy(..., logits=True). "logits" means the input values are expected not to be "softmaxed", i.e. all values can be arbitrary. Currently, your loss function expects the output values to be "softmaxed", i.e. all values must be between 0 and 1 (and sum up to 1).

Update:

One other mistake, likely a huge one: In Keras, you calculate the output once in the beginning and never change it from there on. Then you train your model to fit on this initially generated output.

In the original pytorch code, target (which is the variable being trained on) gets updated in every training loop.
So, you cannot use Keras' fit method which is designed for doing the entire training for you (given fixed training data). You will have to replicate the training loop manually, just as it is done in the pytorch code. I'm not sure if this is easily doable with the API Keras provides. train_on_batch is one method you surely will need in your manual loop. You will have to do some more work, I'm afraid...


Instruction: What's the difference between tf.nn.ctc_loss with pytorch.nn.CTCLoss

Input:
For the same input and label:

the output of pytorch.nn.CTCLoss is 5.74,
the output of tf.nn.ctc_loss is 129.69,
but the output of math.log(tf ctc loss) is 4.86

So what's the difference between pytorch.nn.CTCLoss with tf.nn.ctc_loss?

tf: 1.13.1
pytorch: 1.1.0

I had try to these:

log_softmax the input, and then send it to pytorch.nn.CTCLoss,
tf.nn.log_softmax the input, and then send it to tf.nn.ctc_loss
directly send the input to tf.nn.ctc_loss
directly send the input to tf.nn.ctc_loss, and then math.log(output of tf.nn.ctc_loss)

In the case 2, case 3, and case 4, the result of calculation is difference from pytorch.nn.CTCLoss

from torch import nn
import torch
import tensorflow as tf
import math

time_step = 50  # Input sequence length
vocab_size = 20  # Number of classes
batch_size = 16  # Batch size
target_sequence_length = 30  # Target sequence length

def dense_to_sparse(dense_tensor, sequence_length):
    indices = tf.where(tf.sequence_mask(sequence_length))
    values = tf.gather_nd(dense_tensor, indices)
    shape = tf.shape(dense_tensor, out_type=tf.int64)
    return tf.SparseTensor(indices, values, shape)

def compute_loss(x, y, x_len):
    ctclosses = tf.nn.ctc_loss(
        y,
        tf.cast(x, dtype=tf.float32),
        x_len,
        preprocess_collapse_repeated=False,
        ctc_merge_repeated=False,
        ignore_longer_outputs_than_inputs=False
    )
    ctclosses = tf.reduce_mean(ctclosses)

    with tf.Session() as sess:
        ctclosses = sess.run(ctclosses)
        print(f"tf ctc loss: {ctclosses}")
        print(f"tf log(ctc loss): {math.log(ctclosses)}")

minimum_target_length = 10

ctc_loss = nn.CTCLoss(blank=vocab_size - 1)
x = torch.randn(time_step, batch_size, vocab_size)  # [size] = T,N,C
y = torch.randint(0, vocab_size - 2, (batch_size, target_sequence_length), dtype=torch.long)  # low, high, [size]

x_lengths = torch.full((batch_size,), time_step, dtype=torch.long)  # Length of inputs
y_lengths = torch.randint(minimum_target_length, target_sequence_length, (batch_size,),
                          dtype=torch.long)  # Length of targets can be variable (even if target sequences are constant length)

loss = ctc_loss(x.log_softmax(2).detach(), y, x_lengths, y_lengths)
print(f"torch ctc loss: {loss}")

x = x.numpy()
y = y.numpy()
x_lengths = x_lengths.numpy()
y_lengths = y_lengths.numpy()
x = tf.cast(x, dtype=tf.float32)
y = tf.cast(dense_to_sparse(y, y_lengths), dtype=tf.int32)
compute_loss(x, y, x_lengths)

I expect the output of tf.nn.ctc_loss is same with the output of pytorch.nn.CTCLoss, but actually they are not, but how can i make them same?

Output:
The automatic mean reduction of the CTCLoss of pytorch is not the same as computing all the individual losses, and then doing the mean (as you are doing in the Tensorflow implementation). Indeed from the doc of CTCLoss (pytorch):

``'mean'``: the output losses will be divided by the target lengths and
            then the mean over the batch is taken.

To obtain the same value:

1- Change the reduction method to sum:

ctc_loss = nn.CTCLoss(reduction='sum')

2- Divide the loss computed by the batch_size:

loss = ctc_loss(x.log_softmax(2).detach(), y, x_lengths, y_lengths)
loss = (loss.item())/batch_size

3- Change the parameter ctc_merge_repeated of Tensorflow to True (I am assuming it is the case in the pytorch CTC as well)

    ctclosses = tf.nn.ctc_loss(
    y,
    tf.cast(x, dtype=tf.float32),
    x_len,
    preprocess_collapse_repeated=False,
    ctc_merge_repeated=True,
    ignore_longer_outputs_than_inputs=False
)

You will now get very close results between the pytorch loss and the tensorflow loss (without taking the log of the value). The small difference remaining probably comes from slight differences in between the implementations.
In my last three runs, I got the following values:

pytorch loss : 113.33 vs tf loss = 113.52
pytorch loss : 116.30 vs tf loss = 115.57
pytorch loss : 115.67 vs tf loss = 114.54


Instruction: How to get the full Jacobian of a derivative in PyTorch?

Input:
Lets consider a simple tensor x and lets define another one which depends on x and have multiple dimension : y = (x, 2x, x^2).

How can I have the full gradient dy/dx = (1,2,x) ?

For example lets take the code :

import torch
from torch.autograd import grad

x = 2 * torch.ones(1)
x.requires_grad = True
y = torch.cat((x, 2*x, x*x))
# dy_dx = ???

This is what I have I unsuccessfuly tried so far :

>>> dy_dx = grad(y, x, grad_outputs=torch.ones_like(y), create_graph=True)
(tensor([7.], grad_fn=<AddBackward0>),)
>>> dy_dx = grad(y, x, grad_outputs=torch.Tensor([1,0,0]), create_graph=True)
(tensor([1.], grad_fn=<AddBackward0>),)
>>> dy_dx = grad(y, [x,x,x], grad_outputs=torch.eye(3), create_graph=True)
(tensor([7.], grad_fn=<AddBackward0>),)

Each times I got only part of the gradient or an accumulated version...

I know I could use a for loop using the second expression like

dy_dx = torch.zeros_like(y)
coord = torch.zeros_like(y)
for i in range (y.size(0)):
    coord[i] = 1
    dy_dx[i], = grad(y, x, grad_outputs=coord, create_graph=True)
    coord[i] = 0

However, as I am handeling with high dimensions tensors, this for loop could take too much time to compute. Moreover, there must be a way to perform the full jacobian without acuumulating the gradient...

Does anyone has the solution ? Or an alternative ?

Output: torch.autograd.grad in PyTorch is aggregated. To have a vector auto-differentiated with respect to the input, use torch.autograd.functional.jacobian.


Instruction: Why are Pytorch and Keras implementations giving vastly different results?

Input:
I am trying to train a 1-D ConvNet for time series classification as shown in this paper (refer to FCN om Fig. 1b) https://arxiv.org/pdf/1611.06455.pdf

The Keras implementation is giving me vastly superior performance. Could someone explain why is that the case?

The code for Pytorch is as follow:

class Net(torch.nn.Module):
def __init__(self):
    super(Net, self).__init__()
    self.conv1 = nn.Conv1d(x_train.shape[1], 128, 8)
    self.bnorm1 = nn.BatchNorm1d(128)
    self.conv2 = nn.Conv1d(128, 256, 5)
    self.bnorm2 = nn.BatchNorm1d(256)
    self.conv3 = nn.Conv1d(256, 128, 3)
    self.bnorm3 = nn.BatchNorm1d(128)
    self.dense = nn.Linear(128, nb_classes)

def forward(self, x):
   c1=self.conv1(x)
   b1 = F.relu(self.bnorm1(c1))
   c2=self.conv2(b1)
   b2 = F.relu(self.bnorm2(c2))
   c3=self.conv3(b2)
   b3 = F.relu(self.bnorm3(c3))
   output = torch.mean(b3, 2)
   dense1=self.dense(output)
   return F.softmax(dense1)

 model = Net()
 criterion = nn.CrossEntropyLoss()
 optimizer = torch.optim.SGD(model.parameters(), lr=0.5, momentum=0.99)
 losses=[]
 for t in range(1000):
     y_pred_1= model(x_train.float())
     loss_1 = criterion(y_pred_1, y_train.long())
     print(t, loss_1.item())
     optimizer.zero_grad()
     loss_1.backward()
     optimizer.step()

For comparison, I use the following code for Keras:

x = keras.layers.Input(x_train.shape[1:])
conv1 = keras.layers.Conv1D(128, 8, padding='valid')(x)
conv1 = keras.layers.BatchNormalization()(conv1)
conv1 = keras.layers.Activation('relu')(conv1)
conv2 = keras.layers.Conv1D(256, 5, padding='valid')(conv1)
conv2 = keras.layers.BatchNormalization()(conv2)
conv2 = keras.layers.Activation('relu')(conv2)
conv3 = keras.layers.Conv1D(128, 3, padding='valid')(conv2)
conv3 = keras.layers.BatchNormalization()(conv3)
conv3 = keras.layers.Activation('relu')(conv3)
full = keras.layers.GlobalAveragePooling1D()(conv3)
out = keras.layers.Dense(nb_classes, activation='softmax')(full)

model = keras.models.Model(inputs=x, outputs=out)
optimizer = keras.optimizers.SGD(lr=0.5, decay=0.0, momentum=0.99)
model.compile(loss='categorical_crossentropy', optimizer=optimizer)
hist = model.fit(x_train, Y_train, batch_size=x_train.shape[0], nb_epoch=2000)

The only difference I see between the two is the initialization but however, the results are just vastly different. For reference, I use the same preprocessing as follows for both the datasets, with a subtle difference in input shapes, for Pytorch (Batch_Size, Channels, Length) and for Keras: (Batch_Size, Length, Channels).

Output:
The reason of different results is due to different default parameters of layers and optimizer. For example in pytorch decay-rate of batch-norm is considered as 0.9, whereas in keras it is 0.99. Like that, there may be other variation in default parameters.

If you use same parameters and fixed random seed for initialization, there won't be much difference in the result for both library.


Instruction: How can I compute the mean of values selected from a vector A from an indexing vector B?

Input:
I have a vector of values, for example:

import torch
v = torch.rand(6)

tensor([0.0811, 0.9658, 0.1901, 0.0872, 0.8895, 0.9647])

and an index to select values from v:

index = torch.tensor([0, 1, 0, 2, 0, 2])

I want to produce a vector mean which would compute the mean values of v grouped by indexes from index.

In this example:

mean = torch.tensor([(0.0811 + 0.1901 + 0.8895) / 3, 0.9658, (0.0872 + 0.9647) / 2, 0, 0, 0])

tensor([0.3869, 0.9658, 0.5260, 0.0000, 0.0000, 0.0000])

Output:
One possible solution using a combination of torch.bincount and Tensor.index_add():

v = torch.tensor([0.0811, 0.9658, 0.1901, 0.0872, 0.8895, 0.9647])
index = torch.tensor([0, 1, 0, 2, 0, 2])

bincount() gets the total for each index use in index:

bincount = torch.bincount(index, minlength=6)
# --> tensor([3, 1, 2, 0, 0, 0])

index_add() adds from v in the order given by index:

numerator = torch.zeros(6)
numerator = numerator.index_add(0, index, v)
# --> tensor([1.1607, 0.9658, 1.0520, 0.0000, 0.0000, 0.0000])

Replace zeros with 1.0 in bincount to prevent division by 0
and convert from int to float:

div = bincount.float()
div[bincount == 0] = 1.0
# --> tensor([3., 1., 2., 1., 1., 1.])

mean = num/div
# --> tensor([0.3869, 0.9658, 0.5260, 0.0000, 0.0000, 0.0000])


Instruction: How to get entire dataset from dataloader in PyTorch

Input:
How to load entire dataset from the DataLoader? I am getting only one batch of dataset.

This is my code

dataloader = torch.utils.data.DataLoader(dataset=dataset, batch_size=64)
images, labels = next(iter(dataloader))

Output:
You can set batch_size=dataset.__len__() in case dataset is torch Dataset, else something like batch_szie=len(dataset) should work.

Beware, this might require a lot of memory depending upon your dataset.


Instruction: How to fix the error "TypeError: forward()... - CUDA

Input:
For my bachelor thesis I need to train a network with some music similarity data using GPU and CUDA.

Tried to fix the problem several times with different approaches, but none of them worked.

use_cuda = torch.cuda.is_available()

BSG_model = bayesian_skipgram(V, EMBEDDING_DIM)

if use_cuda:
    BSG_model.cuda()
optimizer = torch.optim.Adam(BSG_model.parameters(), lr=0.005)
BSG_model.train()
loss_progress = []
iter_time = time.time()

dataloader = DataLoader(data, batch_size=16, shuffle=True)

print("N_batches", len(dataloader))

for epoch in range(1):
    for i, batch in enumerate(dataloader):
        batch_start = time.time()
        main_word = batch[:,0]
        context_word = batch[:,1:]

    #print("Main word:,", main_word.shape, context_word.shape)

    optimizer.zero_grad()

    if use_cuda:
        loss = BSG_model.forward(main_word.cuda(), context_word.cuda(), use_cuda=True)
    else:
        loss = BSG_model.forward(main_word, context_word)

    loss.backward()
    optimizer.step()
    batch_end = time.time() - batch_start
    if i % 10 == 0:
        print("epoch, batch ", epoch, i)
        loss_progress.append(loss.item())
        print(time.time()-iter_time)
        print(loss)
        iter_time = time.time()

"The expected result should be that the model starts to train the embeddings..."
"The output is the following:"

TypeError                                 Traceback (most recent call last)
<ipython-input-36-c69aba816e22> in <module>()
     34
     35         if use_cuda:
---> 36             loss = BSG_model.forward(main_word.cuda(), context_word.cuda(), use_cuda=True)
     37         else:
     38             loss = BSG_model.forward(main_word, context_word)

 TypeError: forward() got an unexpected keyword argument 'use_cuda'

Output:
The error message is quite self explanatory:

  TypeError: forward() got an unexpected keyword argument 'use_cuda'

You call forward function like this

oss = BSG_model.forward(main_word.cuda(), context_word.cuda(), use_cuda=True)

with two positional arguments: (main_word.cuda(), context_word.cuda() and one keyword arguement: use_cuda=True.

Keyword arguments means that when the function is declared/defined it has an argument with the same name. For instance:

def forward(self, word, context, use_cuda):
  ...

Is a declaration of forward function with use_cuda argument.

However, it seems like you are calling forward with use_cuda keyword argument, but the function forward you are using does not have a use_cuda argument at all!

Please look carefully at the way your BSG_model.forward function is defined.


Instruction: How to rotate set of 3d points using angle axis to rotation matrix?

Input: I am trying to rotate a set of 3d points and I am looking at this function from the kornia library. If I try to rotate a point around the z-axis by pi/2, my input(axis angle representation) should be [0, 0, pi/2]. When I use this as input into the function, it returns a 4x4 rotation matrix. However, I don't know how to apply this 4x4 matrix to my data because it is Nx3. What do I do with the output matrix? Thanks!

Output: if you look at their source, they are only updating  3 rows and 3 columns of torch.eye(4) tensor. So I think  rotation_matrix[..., :3, :3] should provide you with the correct rotation matrix.


Instruction: How to define several layers via a loop in __init__ for Pytorch?

Input:
I am trying to define a multi-task model in Pytorch where I need a different set of layers for different tasks. I face problems in defining layers, especially if I use a for loop to store different layers in a list then I get an error from optimizer stating that model.parameters() is an empty list, which in fact it is.

Following is the code:

x_trains=[]
y_trains=[]

num_tasks=2
for i in range(num_tasks):
    x_trains.append(torch.from_numpy(np.random.rand(100,1,50)).float())
    y_trains.append(torch.from_numpy(np.array([np.random.randint(10) for i in range(100)])).long())
nb_classes=10

class Net(torch.nn.Module):
def __init__(self):
    super(Net, self).__init__()

    self.all_task_layers=[]
    for i in range(num_tasks):
        self.all_task_layers.append(nn.Conv1d(1, 128, 8))
        self.all_task_layers.append(nn.BatchNorm1d(128))
        self.all_task_layers.append(nn.Conv1d(128, 256, 5))
        self.all_task_layers.append(nn.BatchNorm1d(256))
        self.all_task_layers.append(nn.Conv1d(256, 128, 3))
        self.all_task_layers.append(nn.BatchNorm1d(128))
        self.all_task_layers.append(nn.Linear(128, nb_classes))
    #self.dict_layers_for_tasks[i][1]
    self.all_b1s=[]
    self.all_b2s=[]
    self.all_b3s=[]
    self.all_dense1s=[]

def forward(self, x_num_tasks):
    for i in range(0,len(self.all_task_layers),num_tasks):
        self.all_b1s.append(F.relu(self.all_task_layers[i+1](self.all_task_layers[i+0](x_num_tasks[i]))))

    for i in range(0,len(self.all_task_layers),num_tasks):
        self.all_b2s.append(F.relu(self.all_task_layers[i+3](self.all_task_layers[i+2](self.all_b1s[i]))))

    for i in range(0,len(self.all_task_layers),num_tasks):
        self.all_b3s.append(F.relu(self.all_task_layers[i+5](self.all_task_layers[i+4](self.all_b2s[i]))))

    for i in range(0,len(self.all_task_layers),num_tasks):
        self.all_dense1s.append(self.all_task_layers[i+6](self.all_b3s[i]))

    return self.all_dense1s

model = Net()
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

losses=[]
for t in range(50):
    y_preds= model(x_trains)
    optimizer.zero_grad()
    for i in range(num_tasks):
        loss=criterion(y_preds[i], y_trains[i])
        losses.append(loss)
        print(t,i,loss.item())
        loss.backward(retain_graph=True)
        optimizer.step()

The same model if I try to initialize the layers for the two tasks with two sets of Conv->BatchNorm->Conv->BatchNorm->Conv->BatchNorm->GlobalAveragePooling->Linear then it would work. But if I have let’s say 5 tasks, then it can get pretty messy, and that is why I created a list of layers, and indexed them. For example, the first 7 layers are for task 1 and then the last 7 for task 2. But then the model.parameters() is giving me an empty list. How else could I do it? Or is there a simple fix, I am overlooking?

Output:
you should use nn.ModuleList() to wrap the list. for example
x_trains = nn.ModuleList(x_trains)

see PyTorch : How to properly create a list of nn.Linear()


Instruction: Colab not recognizing local gpu

Input:
Im trying to train a Neural Network that I wrote, but it seems that colab is not recognizing the gtx 1050 on my laptop. I can't use their cloud GPU's for this task, because I run into memory constraints

print(cuda.is_available())

is returning False

Output:
Indeed you gotta select the local runtime accelerator to use GPUs or TPUs, go to Runtime then Change runtime type like in the picture:

And then change it to GPU (takes some secs):


Instruction: Best practices for generating a random seeds to seed Pytorch?

Input:
What I really want is to seed the dataset and dataloader. I am adapting code from:

https://gist.github.com/kevinzakka/d33bf8d6c7f06a9d8c76d97a7879f5cb

Anyone know how to seed this properly? What are the best practices for seeding things in Pytorch.

Honestly, I have no idea if there is an algorithm specific way for GPU vs CPU. I care mostly about general pytorch and make sure my code is "truly random". Specially when it uses GPU I guess...

related:

https://discuss.pytorch.org/t/best-practices-for-seeding-random-numbers-on-gpu/18751
https://discuss.pytorch.org/t/the-random-seed/19516/4
https://discuss.pytorch.org/t/best-practices-for-generating-a-random-seed-to-seed-pytorch/52894/2

My answer was deleted and here is its content:

I don't know if this is the best for pytorch but this is what seems the best for any programming language:

Usually the best random sample you could get in any programming language is generated through the operating system. In Python, you can use the os module:

random_data = os.urandom(4)

In this way you get a cryptographic safe random byte sequence which you may convert in a numeric data type for using as a seed.

seed = int.from_bytes(random_data, byteorder="big")

EDIT: the snippets of code works only on Python 3

'''
Greater than 4 I get this error:

ValueError: Seed must be between 0 and 2**32 - 1
'''

RAND_SIZE = 4

Output:
Have a look at https://pytorch.org/docs/stable/notes/randomness.html

This is what I use

def seed_everything(seed=42):
  random.seed(seed)
  os.environ['PYTHONHASHSEED'] = str(seed)
  np.random.seed(seed)
  torch.manual_seed(seed)
  torch.backends.cudnn.deterministic = True
  torch.backends.cudnn.benchmark = False

the two last parameters (cudnn) are for GPU


Instruction: How to use non-square padding for deconvnet in PyTorch

Input:
Thank you for you attention. I hopes to use nn.ConvTranspose2d to expand dimension of a tensor in PyTorch.(from (N,C,4,4) to (N,C,8,8)) However, I find that if I want to keep the kernel size to 3 and stride to 2. I need to set the padding to [[0,0],[0,0],[0,1],[0,1]](only one side to H and W), which is not a square.

I learnt that in tensorflow we can set the padding like [[0,0],[0,0],[0,1],[0,1]], but in torch, the padding can only be a value for each dimension, which means the padding is on both sides.

So, I am wondering is there any way to do this in PyTorch?

Here's some code of details

import torch
import torch.nn as nn

a = torch.rand([100, 80, 4, 4])
a.shape
nn.ConvTranspose2d(80, 40, kernel_size=3, stride=2, padding=0)(a).shape
nn.ConvTranspose2d(80, 40, kernel_size=3, stride=2, padding=1)(a).shape
nn.ConvTranspose2d(80, 40, kernel_size=3, stride=2, padding=(0,1,0,1))(a).shape

>>> torch.Size([100, 80, 4, 4])
>>> torch.Size([100, 40, 9, 9])
>>> torch.Size([100, 40, 7, 7])
>>> RuntimeError: expected padding to be a single integer value or a list of 2 values to match the convolution dimensions, but got padding=[0, 1, 0, 1]

Output:
You can use F.pad from functional API.

b= nn.ConvTranspose2d(80, 40, kernel_size=3, stride=2, padding=1)(a)
c = F.pad(b, (0,1,0,1),"constant", 0)


Instruction: outputs are different between ONNX and pytorch

Input:
I try to convert my pytorch Resnet50 model to ONNX and do inference. The conversion procedural makes no errors, but the final result of onnx model from onnxruntime has large gaps with the result of origin model from pytorch.
What is possible solution ?

Version of ONNX: 1.5.0
Version of pytorch: 1.1.0
CUDA: 9.0
System: Ubuntu 18.06
Python: 3.5

Here is the code of conversion

import torch
import models
from collections import OrderedDict
state_dict = "/home/yx-wan/newhome/workspace/filter-pruning-geometric-median/scripts/snapshots/resnet50-rate-0.7/best.resnet50.GM_0.7_76.82.pth.tar"
arch = 'resnet50'

def import_sparse(model,state_dict):
    new_state_dict = OrderedDict()
    for k, v in state_dict.items():
        name = k[7:]  # remove `module.`
        new_state_dict[name] = v
    model.load_state_dict(new_state_dict)
    print("sparse_model_loaded")
    return model

# initialize model
model = models.__dict__[arch](pretrained=False).cuda()

checkpoint = torch.load(state_dict)
model = import_sparse(model, checkpoint['state_dict'])
print("Top 1 precise of model: {}".format(checkpoint['best_prec1']))
dummy_input =torch.randn(1, 3, 224, 224).cuda()
torch.onnx.export(model, dummy_input, "{}.onnx".format(arch), verbose=True)

Here is the result checking code

import sys

from onnxruntime.datasets import get_example
import onnxruntime
import cv2
import numpy as np
import torch
import models
import onnxruntime
from collections import  OrderedDict
from my_tools import resize_img

def import_sparse(model,checkpoint):
    new_state_dict = OrderedDict()
    for k, v in checkpoint['state_dict'].items():
        name = k[7:]  # remove `module.`
        new_state_dict[name] = v
    model.load_state_dict(new_state_dict)
    return model

image_path = "./img652.jpg"
onnx_model_path = "/workplace/workspace/filter-pruning-geometric-median/resnet50.onnx"
ckpt="./scripts/snapshots/resnet50-rate-0.7/best.resnet50.GM_0.7_76.82.pth.tar"

img_ori = cv2.imread(image_path)  # BGR
img = cv2.cvtColor(img_ori, cv2.COLOR_BGR2RGB)
img, ratio_h, ratio_w = resize_img(img,224,224)
img = img - np.array([123.68, 116.78, 103.94],dtype=np.float32)
img_batch = np.expand_dims(img, 0)

# NHWC -> NCHW
img_batch = np.transpose(img_batch,[0,3,1,2])

example_model = get_example(onnx_model_path)
sess = onnxruntime.InferenceSession(example_model)
input_name = sess.get_inputs()[0].name
print("Input name  :", input_name)
input_shape = sess.get_inputs()[0].shape
print("Input shape :", input_shape)
input_type = sess.get_inputs()[0].type
print("Input type  :", input_type)
output_name = sess.get_outputs()[0].name
print("Output name  :", output_name)
output_shape = sess.get_outputs()[0].shape
print("Output shape :", output_shape)
output_type = sess.get_outputs()[0].type
print("Output type  :", output_type)

print("Input data shape{}".format(img_batch.shape))
assert(list(input_shape) == list(img_batch.shape))
result_onnx = sess.run([output_name], {input_name: img_batch})

# initialize model
model = models.__dict__["resnet50"]()
checkpoint = torch.load(ckpt,map_location='cpu')
best_prec1 = checkpoint['best_prec1']
model = import_sparse(model,checkpoint)
img_batch = torch.FloatTensor(img_batch)
with torch.no_grad():
    result_torch = model(img_batch)
    result_torch = result_torch.numpy()

print("max onnx-torch:{}".format(np.max(result_onnx-result_torch)))

And the output (with some warning but I think is doesn't matter) of checking code is

2019-08-09 02:59:21.378599853 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer4.2.bn3.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378654931 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer4.2.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378665235 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer4.2.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378675069 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer4.1.bn3.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378686874 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer4.0.downsample.1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378698995 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer4.1.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378718700 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer3.5.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378729567 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer3.4.bn3.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378739657 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer3.4.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378752091 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer3.3.bn3.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378762533 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer3.3.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378771168 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer3.2.bn3.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378781705 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer3.2.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378792325 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer3.4.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378802071 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer3.1.bn3.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378812061 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer4.0.bn3.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378822884 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer3.1.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378834198 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer3.0.downsample.1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378845176 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer1.2.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378859324 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer2.0.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378869709 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer2.0.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378883281 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer3.5.bn3.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378893302 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer2.3.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378904876 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer1.1.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378915507 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer1.0.bn3.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378926638 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer4.0.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378938115 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer3.0.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378948686 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378958670 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer3.2.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378969125 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer4.1.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378979556 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer2.1.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378990553 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer2.2.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379001126 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer1.2.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379011508 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer1.0.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379021900 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer1.0.downsample.1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379033504 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer2.2.bn3.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379044076 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer1.2.bn3.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379064049 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer1.1.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379076654 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer2.0.bn3.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379089769 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer2.1.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379102140 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer3.0.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379114598 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer3.3.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379133520 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer2.2.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379144015 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer2.3.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379155771 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer1.1.bn3.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379167084 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer2.3.bn3.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379178303 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer3.0.bn3.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379189605 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer4.0.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379199974 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer3.1.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379211042 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer2.0.downsample.1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379221800 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer3.5.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379232566 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer1.0.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379243442 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer2.1.bn3.num_batches_tracked'. It is not used by any node and should be removed from the model.
Input name  : 0
Input shape : [1, 3, 224, 224]
Input type  : tensor(float)
Output name  : 503
Output shape : [1, 1000]
Output type  : tensor(float)
Input data shape(1, 3, 224, 224)
max onnx-torch:104.89282989501953

Output:
Problem solve by adding model.eval() before running inference of pytorch model in test code. Solution is from the link

model = models.__dict__["resnet50"]()
checkpoint = torch.load(ckpt,map_location='cpu')
best_prec1 = checkpoint['best_prec1']
model = import_sparse(model,checkpoint)
model.eval()
img_batch = torch.FloatTensor(img_batch)
with torch.no_grad():
    result_torch = model(img_batch)
    result_torch = result_torch.numpy()


Instruction: pytorch: inception v3 parameter empty error

Input:
I am using inception_v3 from torchvision.models as my base model and adding an FC layer at the end to get features. However, I am getting an empty parameter error.

import torch
import torch.nn as nn
import torchvision.models as models

class Baseline(nn.Module):
    def __init__(self, out_size):
        super().__init__()
        model = models.inception_v3(pretrained=True)
        model.fc = nn.Linear(2048, out_size)
        model.aux_logits = False

        # Freeze model weights
        for param in model.parameters():
            param.requires_grad = False

        self.parameters = nn.ParameterList()

    def forward(self, image):
        x = model(image)
        x = x.view(x.size(0), -1)
        x = model.fc = (x)
        return x

Output:
My understanding is that you are updating self.parameters with an empty nn.ParameterList  which is not required here.

self.parameters will already have all the parameters your Baseline class has, including those of inception_v3 and nn.Linear. When you are updating them at the end with an empty list, you are essentially deleting all of the previously stored parameters.


Instruction: How to get unique value along 1 dim of a 2d array in pytorch?

Input:
I have a 2d tensor now, which may have repeated elements along a dim, like

tmp = torch.tensor([[1,2,3,2,4],[0,5,6,7,2],[3,4,5,3,5],[7,5,6,7,7]])

I hope to get unique elements along dim=1, the result should be like this

result = [[1,2,3,4],[0,5,6,7,2],[3,4,5],[5,6,7]]

Is there a way could get the result without using for- loop?

I tried using torch.unique,  like this,

result=[]
for i in range(tmp.shape[0]):
    t = tmp[i,:]
    result.append(torch.unique(t))

It works, but time-consuming.

Output:
You cannot call unique on n-rank tensor when n>=2.
This is because in PyTorch there are no jagged array tensors.

tmp = torch.tensor([[1,2,3,2,4],[0,5,6,7,2],[3,4,5,3,5],[7,5,6,7,7]])
%timeit tmpt =torch.unbind(tmp); [torch.unique(t) for t in tmpt]

This returned 39.3 µs, while your original loop took

tmp = torch.tensor([[1,2,3,2,4],[0,5,6,7,2],[3,4,5,3,5],[7,5,6,7,7]])
result=[]
%timeit for i in range(tmp.shape[0]):    t = tmp[i,:] ;    result.append(torch.unique(t))

54 µs on average


Instruction: Pytorch-Implement the same model in pytorch and keras but got different results

Input:
I am learning pytorch and want to practice it with an keras example (https://keras.io/examples/lstm_seq2seq/), this is a seq2seq 101 example which translate eng to fra on char-level features (no embedding).

Keras code is below:

from keras.models import Model
from keras.layers import Input, LSTM, Dense
import numpy as np

batch_size = 64  # Batch size for training.
epochs = 100  # Number of epochs to train for.
latent_dim = 256  # Latent dimensionality of the encoding space.
num_samples = 10000  # Number of samples to train on.
# Path to the data txt file on disk.
data_path = 'fra-eng/fra.txt'

# Vectorize the data.
input_texts = []
target_texts = []
input_characters = set()
target_characters = set()
with open(data_path, 'r', encoding='utf-8') as f:
    lines = f.read().split('\n')
for line in lines[: min(num_samples, len(lines) - 1)]:
    input_text, target_text = line.split('\t')
    # We use "tab" as the "start sequence" character
    # for the targets, and "\n" as "end sequence" character.
    target_text = '\t' + target_text + '\n'
    input_texts.append(input_text)
    target_texts.append(target_text)
    for char in input_text:
        if char not in input_characters:
            input_characters.add(char)
    for char in target_text:
        if char not in target_characters:
            target_characters.add(char)

input_characters = sorted(list(input_characters))
target_characters = sorted(list(target_characters))
num_encoder_tokens = len(input_characters)
num_decoder_tokens = len(target_characters)
max_encoder_seq_length = max([len(txt) for txt in input_texts])
max_decoder_seq_length = max([len(txt) for txt in target_texts])

print('Number of samples:', len(input_texts))
print('Number of unique input tokens:', num_encoder_tokens)
print('Number of unique output tokens:', num_decoder_tokens)
print('Max sequence length for inputs:', max_encoder_seq_length)
print('Max sequence length for outputs:', max_decoder_seq_length)

input_token_index = dict(
    [(char, i) for i, char in enumerate(input_characters)])
target_token_index = dict(
    [(char, i) for i, char in enumerate(target_characters)])

encoder_input_data = np.zeros(
    (len(input_texts), max_encoder_seq_length, num_encoder_tokens),
    dtype='float32')
decoder_input_data = np.zeros(
    (len(input_texts), max_decoder_seq_length, num_decoder_tokens),
    dtype='float32')
decoder_target_data = np.zeros(
    (len(input_texts), max_decoder_seq_length, num_decoder_tokens),
    dtype='float32')

for i, (input_text, target_text) in enumerate(zip(input_texts, target_texts)):
    for t, char in enumerate(input_text):
        encoder_input_data[i, t, input_token_index[char]] = 1.
    for t, char in enumerate(target_text):
        # decoder_target_data is ahead of decoder_input_data by one timestep
        decoder_input_data[i, t, target_token_index[char]] = 1.
        if t > 0:
            # decoder_target_data will be ahead by one timestep
            # and will not include the start character.
            decoder_target_data[i, t - 1, target_token_index[char]] = 1.

# Define an input sequence and process it.
encoder_inputs = Input(shape=(None, num_encoder_tokens))
encoder = LSTM(latent_dim, return_state=True)
encoder_outputs, state_h, state_c = encoder(encoder_inputs)
# We discard `encoder_outputs` and only keep the states.
encoder_states = [state_h, state_c]

# Set up the decoder, using `encoder_states` as initial state.
decoder_inputs = Input(shape=(None, num_decoder_tokens))
# We set up our decoder to return full output sequences,
# and to return internal states as well. We don't use the
# return states in the training model, but we will use them in inference.
decoder_lstm = LSTM(latent_dim, return_sequences=True, return_state=True)
decoder_outputs, _, _ = decoder_lstm(decoder_inputs,
                                     initial_state=encoder_states)
decoder_dense = Dense(num_decoder_tokens, activation='softmax')
decoder_outputs = decoder_dense(decoder_outputs)

# Define the model that will turn
# `encoder_input_data` & `decoder_input_data` into `decoder_target_data`
model = Model([encoder_inputs, decoder_inputs], decoder_outputs)

# Run training
model.compile(optimizer='rmsprop', loss='categorical_crossentropy')
model.fit([encoder_input_data, decoder_input_data], decoder_target_data,
          batch_size=batch_size,
          epochs=epochs,
          validation_split=0.2)
# Save model
model.save('s2s.h5')

# Next: inference mode (sampling).
# Here's the drill:
# 1) encode input and retrieve initial decoder state
# 2) run one step of decoder with this initial state
# and a "start of sequence" token as target.
# Output will be the next target token
# 3) Repeat with the current target token and current states

# Define sampling models
encoder_model = Model(encoder_inputs, encoder_states)

decoder_state_input_h = Input(shape=(latent_dim,))
decoder_state_input_c = Input(shape=(latent_dim,))
decoder_states_inputs = [decoder_state_input_h, decoder_state_input_c]
decoder_outputs, state_h, state_c = decoder_lstm(
    decoder_inputs, initial_state=decoder_states_inputs)
decoder_states = [state_h, state_c]
decoder_outputs = decoder_dense(decoder_outputs)
decoder_model = Model(
    [decoder_inputs] + decoder_states_inputs,
    [decoder_outputs] + decoder_states)

# Reverse-lookup token index to decode sequences back to
# something readable.
reverse_input_char_index = dict(
    (i, char) for char, i in input_token_index.items())
reverse_target_char_index = dict(
    (i, char) for char, i in target_token_index.items())

def decode_sequence(input_seq):
    # Encode the input as state vectors.
    states_value = encoder_model.predict(input_seq)

    # Generate empty target sequence of length 1.
    target_seq = np.zeros((1, 1, num_decoder_tokens))
    # Populate the first character of target sequence with the start character.
    target_seq[0, 0, target_token_index['\t']] = 1.

    # Sampling loop for a batch of sequences
    # (to simplify, here we assume a batch of size 1).
    stop_condition = False
    decoded_sentence = ''
    while not stop_condition:
        output_tokens, h, c = decoder_model.predict(
            [target_seq] + states_value)

        # Sample a token
        sampled_token_index = np.argmax(output_tokens[0, -1, :])
        sampled_char = reverse_target_char_index[sampled_token_index]
        decoded_sentence += sampled_char

        # Exit condition: either hit max length
        # or find stop character.
        if (sampled_char == '\n' or
           len(decoded_sentence) > max_decoder_seq_length):
            stop_condition = True

        # Update the target sequence (of length 1).
        target_seq = np.zeros((1, 1, num_decoder_tokens))
        target_seq[0, 0, sampled_token_index] = 1.

        # Update states
        states_value = [h, c]

    return decoded_sentence

for seq_index in range(100):
    # Take one sequence (part of the training set)
    # for trying out decoding.
    input_seq = encoder_input_data[seq_index: seq_index + 1]
    decoded_sentence = decode_sequence(input_seq)
    print('-')
    print('Input sentence:', input_texts[seq_index])
    print('Decoded sentence:', decoded_sentence)

I want to implement this exact same model using pytorch, below is my code:

from __future__ import unicode_literals, print_function, division
from io import open
import unicodedata
import string
import re
import random
import numpy as np
import torch
import torch.nn as nn
from torch import optim
import torch.nn.functional as F

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
data_path = './eng_fra.txt'

# Vectorize the data.
input_texts = []
target_texts = []
input_characters = set()
target_characters = set()
with open(data_path, 'r', encoding='utf-8') as f:
    lines = f.read().split('\n')
for line in lines[: min(num_samples, len(lines) - 1)]:
    #print('line:',line)
    input_text, target_text = line.split('\t')
    # We use "tab" as the "start sequence" character
    # for the targets, and "\n" as "end sequence" character.
    target_text = '\t' + target_text + '\n' # why?
   # print('input_text and target_text:',input_text, target_text)
    input_texts.append(input_text)
    target_texts.append(target_text)
    for char in input_text:
        if char not in input_characters:
            input_characters.add(char)
    for char in target_text:
        if char not in target_characters:
            target_characters.add(char)

input_characters = sorted(list(input_characters))
target_characters = sorted(list(target_characters))
num_encoder_tokens = len(input_characters)
print('input_characters',input_characters)
num_decoder_tokens = len(target_characters)
print('target_characters',target_characters)
max_encoder_seq_length = max([len(txt) for txt in input_texts])
max_decoder_seq_length = max([len(txt) for txt in target_texts])
print('max_encoder_seq_length and max_decoder_seq_length',max_encoder_seq_length,max_decoder_seq_length)

input_token_index = dict(
    [(char, i) for i, char in enumerate(input_characters)])
target_token_index = dict(
    [(char, i) for i, char in enumerate(target_characters)])

# define the shapes
encoder_input_data = np.zeros(
    (len(input_texts), max_encoder_seq_length, num_encoder_tokens),
    dtype='float32')
decoder_input_data = np.zeros(
    (len(input_texts), max_decoder_seq_length, num_decoder_tokens),
    dtype='float32')
decoder_target_data = np.zeros(
    (len(input_texts), max_decoder_seq_length, num_decoder_tokens),
    dtype='float32')

# one hot encoding for each word in each sentence
for i, (input_text, target_text) in enumerate(zip(input_texts, target_texts)):
    for t, char in enumerate(input_text):
        encoder_input_data[i, t, input_token_index[char]] = 1.
    for t, char in enumerate(target_text):
        # decoder_target_data is ahead of decoder_input_data by one timestep
        decoder_input_data[i, t, target_token_index[char]] = 1.
        if t > 0:
            # decoder_target_data will be ahead by one timestep
            # and will not include the start character.
            decoder_target_data[i, t - 1, target_token_index[char]] = 1.
encoder_input_data=torch.Tensor(encoder_input_data).to(device)
decoder_input_data=torch.Tensor(decoder_input_data).to(device)
decoder_target_data=torch.Tensor(decoder_target_data).to(device)

class encoder(nn.Module):
  def __init__(self):
    super(encoder,self).__init__()
    self.LSTM=nn.LSTM(input_size=num_encoder_tokens,hidden_size=256,batch_first=True)
  def forward(self,x):
    out,(h,c)=self.LSTM(x)
    return h,c

class decoder(nn.Module):
  def __init__(self):
    super(decoder,self).__init__()
    self.LSTM=nn.LSTM(input_size=num_decoder_tokens,hidden_size=256,batch_first=True)
    self.FC=nn.Linear(256,num_decoder_tokens)

  def forward(self,x, hidden):
    out,(h,c)=self.LSTM(x,hidden)
    out=self.FC(out)
    return out,(h,c)

class seq2seq(nn.Module):
  def __init__(self,encoder,decoder):
    super(seq2seq,self).__init__()
    self.encoder=encoder
    self.decoder=decoder

  def forward(self,encode_input_data,decode_input_data):
    hidden, cell = self.encoder(encode_input_data)
    output, (hidden, cell) = self.decoder(decode_input_data, (hidden, cell))
    return output

encoder=encoder().to(device)
# encoder_loss = nn.CrossEntropyLoss() # CrossEntropyLoss compute softmax internally in pytorch
# encoder_optimizer = torch.optim.Adam(encoder.parameters(), lr=0.001)

decoder=decoder().to(device)
# decoder_loss = nn.CrossEntropyLoss() # CrossEntropyLoss compute softmax internally in pytorch
# decoder_optimizer = torch.optim.Adam(decoder.parameters(), lr=0.001)

model=seq2seq(encoder,decoder).to(device)
optimizer = optim.RMSprop(model.parameters(),lr=0.01)
loss_fun=nn.CrossEntropyLoss()

# model.train()

num_epochs=50
batches=np.array_split(range(decoder_target_data.shape[0]),100)
total_step=len(batches)
for epoch in range(num_epochs):

  for i,batch_ids in enumerate(batches):
    encoder_input=encoder_input_data[batch_ids]
    decoder_input=decoder_input_data[batch_ids]
    decoder_target=decoder_target_data[batch_ids]

    output = model(encoder_input, decoder_input)
    loss=loss_fun(output.view(-1,93).to(device),decoder_target.view(-1,93).max(dim=1)[1].to(device))

    # Backward and optimize
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    if (i+1) % 20 == 0:
        print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'
               .format(epoch+1, num_epochs, i+1, total_step, loss.item()))

# Reverse-lookup token index to decode sequences back to
# something readable.
reverse_input_char_index = dict(
    (i, char) for char, i in input_token_index.items())
reverse_target_char_index = dict(
    (i, char) for char, i in target_token_index.items())

def decode_sequence(input_seq):
    # Encode the input as state vectors.
    h,c=model.encoder(input_seq)

    # Generate empty target sequence of length 1.
    # Populate the first character of target sequence with the start character.
    target_seq = torch.zeros((1, 1, num_decoder_tokens)).to(device)
    target_seq[0, 0, target_token_index['\t']] = 1.

    # Sampling loop for a batch of sequences
    # (to simplify, here we assume a batch of size 1).
    stop_condition = False
    decoded_sentence = ''
    while not stop_condition:
        output_tokens, (h_t, c_t) = model.decoder(target_seq,(h,c))

        # Sample a token
        sampled_token_index = output_tokens.view(-1,93).squeeze(0).max(dim=0)[1].item()
        sampled_char = reverse_target_char_index[sampled_token_index]
        decoded_sentence += sampled_char

        # Exit condition: either hit max length
        # or find stop character.
        if (sampled_char == '\n' or
           len(decoded_sentence) > max_decoder_seq_length):
            stop_condition = True

        # Update the target sequence (of length 1).
        target_seq = torch.zeros((1, 1, num_decoder_tokens)).to(device)
        target_seq[0, 0, sampled_token_index] = 1.

        # Update states
        h,c=h_t,c_t

    return decoded_sentence

for seq_index in range(100):
    # Take one sequence (part of the training set)
    # for trying out decoding.
    input_seq = encoder_input_data[seq_index: seq_index + 1]
    decoded_sentence = decode_sequence(input_seq)
    print('-')
    print('Input sentence:', input_texts[seq_index])
    print('Decoded sentence:', decoded_sentence)

As you can see, I used exactly the same data processing and model structure. My pytorch version can run without error, but the performance seems worse than the original keras version by comparing the translation results.

One thing might cause error is the loss function (cross_entropy). In pytorch, cross_entropy loss function seems does not support one-hot labels directly, which I need to change the label to integer. However I don't think this should make big difference.

If you want to run the models, the data can be downloaded from:
https://github.com/jinfagang/pytorch_chatbot/blob/master/datasets/eng-fra.txt

Did I do something wrong in my code? Many thanks

Output:
One way to look at the issue would be:

Fixing seeds to the same value in both Pytorch and Keras, albeit it cannot really guarantee the same output.
Weight initialization in Pytorch is different from Keras. Make sure they have the same weight initialization functions
I've been using for a problem of mine and I can say that even with 1 and 2 being setup identically, there is a high probability of getting the same results (that could be due to the way how Pytorch is implemented).

Hope that helps! Please update us if you managed to resolve your issue.


Instruction: One hot encoding a segmented image using pytorch

Input:
I have a segmented image as a tensor of size [1,1,256,256]. The image is a binary segmented image. I want to one hot encode it to get an image of size [1,2,256,256].
I tried torch.nn.functional.one_hot(img, 2). But it gave me an image of size [1,256,256,2]. How do I get the desired tensor?

Output:
Try to use transpose():

img_one_hot = torch.nn.functional.one_hot(img, 2).transpose(1, 4).squeeze(-1)

transpose(1, 4) - swaps 1st and 4th dimension, returning the tensor of the shape of [1, 2, 256, 256, 1], squeeze(-1) removes the last dim resulting in [1 , 2, 256, 256] shaped tensor.


Instruction: How to run Tensorboard in parallel

Input:
https://github.com/NVIDIA/DeepRecommender

According to the above page, I tried to run the NVIDIA's DeepRecommender program.After I activated the pytorch, I run the program as below but it failed.

[I run this Command]

$ python run.py --gpu_ids 0 \
 --path_to_train_data Netflix/NF_TRAIN \
 --path_to_eval_data Netflix/NF_VALID \
 --hidden_layers 512,512,1024 \
 --non_linearity_type selu \
 --batch_size 128 \
 --logdir model_save \
 --drop_prob 0.8 \
 --optimizer momentum \
 --lr 0.005 \
 --weight_decay 0 \
 --aug_step 1 \
 --noise_prob 0 \
 --num_epochs 12 \
 --summary_frequency 1000

[The comments of the Guide.]

  Note that you can run Tensorboard in parallel

  $ tensorboard --logdir=model_save

[My Question]

The guide says as above.I don't know how to run in parallel.Please tell me the way. Shoud I open 2 terminal  windows?

[Enviroment]

The detail of the enviroment is as follow.
---> Ubuntu 18.04 LTS, python 3.6, Pytorch 1.2.0, CUDA V10.1.168

[The 1st trial]

After I activated the pytorch,

 $source activate pytorch

 $python run.py --gpu_ids 0 \  (The long parameters are abbreviated here.)

[The Error messages of the 1st trial]

Traceback (most recent call last):
  File "run.py", line 13, in
    from logger import Logger
  File "/home/user/NVIDIA_DeepRecommender/DeepRecommender-mp_branch/logge                                                                                r.py", line 4, in
    import tensorflow as tf
ModuleNotFoundError: No module named 'tensorflow'

[The 2nd trial]

After I activated the tensorflow-gpu,

 $ source activate tensorflow-gpu

 $python run.py --gpu_ids 0 \  (The long parameters are abbreviated here.)

[The Error messages of the 2nd trial.]

Traceback (most recent call last):
  File "run.py", line 2, in
    import torch
ModuleNotFoundError: No module named 'torch'

[Expected result]

  $ python run.py --gpu_ids 0 \
   The program can run with no error and finish training the model.

Output: Try either installing tensorflow-gpu in your pytorch environment or pytorch in your tensorflow-gpu environemnt and use that environment to run your program.


Instruction: Matplotlib Pylot - Images are being displayed in low resolution (pixel to pixel)

Input:
When I display some samples photos from the dataset I use, the previews of images are displayed in low resolution (they look like very low-resolution photos). How I can I display the images without losing their resolutions?

Here are my transformations which are used to move the data to the tensor and apply some transformations using PyTorch functions:

data_transforms = transforms.Compose([
    transforms.Resize((50, 50)),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                         std=[0.229, 0.224, 0.225])
])

Then I load the data through DataLoader:

train_loader = DataLoader(face_train_dataset,
                          batch_size=train_batch_size, shuffle=False,
                          num_workers=4)

Finally, I display some previews for the sample photos which are retrieved using the DataLoader object:

example_data = example_data.cpu()
example_targets = example_targets.cpu()
for i in range(6):
    plt.subplot(2, 3, i + 1)
    plt.tight_layout()
    plt.imshow(example_data[i][0], cmap='gray', interpolation='none')
    plt.title('{}'.format(folders[example_targets[i]]))

plt.show()

p.s. Images are in tiff format.

Output:
What resolution are you expecting?

One of the transformations you are applying is

transforms.Resize((50, 50))

That is, you are reducing the input images resolution to 50 by 50 pixels. This is the resolution you are getting when you plot the images.

In order to have a more graceful display of the low-res images you might want to consider changing the interpolation method of imshow to

plt.imshow(example_data[i][0], cmap='gray', interpolation='bicubic')


Instruction: Keyword arguments in torch.nn.Sequential (pytroch)

Input:
a question regarding keywords in torch.nn.Sequential, it is possible in some way to forward keywords to specific models in a sequence?

    model = torch.nn.Sequential(model_0, MaxPoolingChannel(1))
    res = model(input_ids_2, keyword_test=mask)

here, keyword_test should be forwarded only to the first model.

Thank a lot and best regards!

my duplicate from - https://discuss.pytorch.org/t/keyword-arguments-in-torch-nn-sequential/53282

Output:
No; you cannot. This is only possible if all models passed to the nn.Sequential expects the argument you are trying to pass in their forward method (at least at the time of writing this).

Two workarounds could be (I'm not aware of the whole case, but anticipated from the question):

If your value is static, why not initializing your first model with that value, and access it during computation with self.keyword_test.
In case the value is dynamic, you could have it as an inherent property in the input; hence, you can access it, also, during computation with input_ids_2.keyword_test


Instruction: Getting distributed package doesn't have mpi built in error

Input:
I have been trying to write a distributed application using pytorch. I have been following tutorial here. Over there, I am using the "MPI Backend" option. According to that, I need to follow the basic steps to install pytorch and then install openmpi as conda install -c conda-forge openmpi

Unfortunately, whenever I try to run a script using mpirun mpiexec -n 2 python ptdist.py, I get the following error RuntimeError: Distributed package doesn't have MPI built in. I believe this is happening because of error in import ProcessGroupMPI code here in python.

I have tried to install openmpi from their source code as well as sudo apt-get install python-mpi4py, but am still facing the same error.

I also tried pip install mpi4py but that also does not help

Does anyone know what is the problem?

Output:
From https://medium.com/@esaliya/pytorch-distributed-with-mpi-acb84b3ae5fd

  The MPI backend, though supported, is not available unless you compile PyTorch from its source

This suggests you should first install your favorite MPI library, and possibly mpi4py built on top of it, and then build pytorch from sources at last.