RuntimeError: cudnn RNN backward can only be called in training mode
I have seen this problem the first time, I never encountered such an error in previous Python projects. Here is my training code:
def train(net, opt, criterion,ucf_train, batchsize,i):
    opt.zero_grad()
    total_loss = 0
    net=net.eval()
    net=net.train()
    for vid in range(i*batchsize,i*batchsize+batchsize,1):
    
        output=infer(net,ucf_train[vid])
        m=get_label_no(ucf_train[vid])
        m=m.cuda( )
        loss = criterion(output,m)
        loss.backward(retain_graph=True)
        total_loss += loss 
        opt.step()       #updates wghts and biases

    return total_loss/n_points

code for infer(net,input)
def infer(net, name):
    net.eval()
    hidden_0 = net.init_hidden()
    hidden_1 = net.init_hidden()
    hidden_2 = net.init_hidden()
    video_path = fetch_ucf_video(name)
    cap = cv2.VideoCapture(video_path)
    resize=(224,224)
    T=FrameCapture(video_path)
    print(T)
    lim=T-(T%20)-2
    i=0
    while(1):
      ret, frame2 = cap.read()
      frame2= cv2.resize(frame2, resize)
    #  print(type(frame2))
      if (i%20==0 and i&lt;lim):
          input=normalize(frame2)     
          input=input.cuda()       
          output,hidden_0,hidden_1, hidden_2  = net(input, hidden_0, hidden_1, hidden_2)
      elif (i&gt;=lim):
          break
      i=i+1 
    op=output  
    torch.cuda.empty_cache() 
    op=op.cuda() 
    return op 

I am getting this error, I tried with model.train() following this where net is my model:
 RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-62-42238f3f6877&gt; in &lt;module&gt;()
----&gt; 1 train(net1,opt,criterion,ucf_train,1,0)

2 frames
/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
    125     Variable._execution_engine.run_backward(
    126         tensors, grad_tensors, retain_graph, create_graph,
--&gt; 127         allow_unreachable=True)  # allow_unreachable flag
    128 
    129 

RuntimeError: cudnn RNN backward can only be called in training mode


You should remove the net.eval() call that comes right after the def infer(net, name):
It needs to be removed because you call this infer function inside your training code. Your model needs to be in train mode throughout the the whole training.
And you never set your model back to train after calling eval as well, so that is the root of the exception you are getting. If you want to use this infer code in your test cases, you can cover that case with an if.
Also the net.eval() that comes right after the total_loss=0 assignment is not useful since you call net.train() right after that. You can also remove that one since it gets neutralized right in next line.
The updated code
def train(net, opt, criterion,ucf_train, batchsize,i):
    opt.zero_grad()
    total_loss = 0
    net=net.train()
    for vid in range(i*batchsize,i*batchsize+batchsize,1):
        output=infer(net,ucf_train[vid])
        m=get_label_no(ucf_train[vid])
        m=m.cuda( )
        loss = criterion(output,m)
        loss.backward(retain_graph=True)
        total_loss += loss 
        opt.step()       #updates wghts and biases

    return total_loss/n_points

code for infer(net,input)
def infer(net, name, is_train=True):
    if not is_train:
        net.eval()
    hidden_0 = net.init_hidden()
    hidden_1 = net.init_hidden()
    hidden_2 = net.init_hidden()
    video_path = fetch_ucf_video(name)
    cap = cv2.VideoCapture(video_path)
    resize=(224,224)
    T=FrameCapture(video_path)
    print(T)
    lim=T-(T%20)-2
    i=0
    while(1):
      ret, frame2 = cap.read()
      frame2= cv2.resize(frame2, resize)
      #  print(type(frame2))
      if (i%20==0 and i&lt;lim):
          input=normalize(frame2)     
          input=input.cuda()       
          output,hidden_0,hidden_1, hidden_2  = net(input, hidden_0, hidden_1, hidden_2)
      elif (i&gt;=lim):
          break
      i=i+1 
    op=output  
    torch.cuda.empty_cache() 
    op=op.cuda() 
    return op 

