title: Removing last 2 layers from a BERT classifier results in " 'tuple' object has no attribute 'dim' " error. Why?
body: <p>I fine tuned a huggingface transformer using Keras (with ktrain) and then reloaded the model in Pytorch.</p>
  
  <p>I want to access the third to last layer (<code>pre_classifier</code>), so I removed the two last layers:</p>
  
  <pre><code>BERT2 = torch.nn.Sequential(*(list(BERT.children())[:-2])) 
  </code></pre>
  
  <p>Running an encoded sentence through this yields the following error message:</p>
  
  <pre><code>AttributeError                            Traceback (most recent call last)
  &lt;ipython-input-38-640702475573&gt; in &lt;module&gt;
  ----&gt; 1 ans2=BERT2(torch.tensor([e1]))
        2 print (ans2)
  
  C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
      539             result = self._slow_forward(*input, **kwargs)
      540         else:
  --&gt; 541             result = self.forward(*input, **kwargs)
      542         for hook in self._forward_hooks.values():
      543             hook_result = hook(self, input, result)
  
  C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\container.py in forward(self, input)
       90     def forward(self, input):
       91         for module in self._modules.values():
  ---&gt; 92             input = module(input)
       93         return input
       94 
  
  C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
      539             result = self._slow_forward(*input, **kwargs)
      540         else:
  --&gt; 541             result = self.forward(*input, **kwargs)
      542         for hook in self._forward_hooks.values():
      543             hook_result = hook(self, input, result)
  
  C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\linear.py in forward(self, input)
       85 
       86     def forward(self, input):
  ---&gt; 87         return F.linear(input, self.weight, self.bias)
       88 
       89     def extra_repr(self):
  
  C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\functional.py in linear(input, weight, bias)
     1366         - Output: :math:`(N, *, out\_features)`
     1367     """
  -&gt; 1368     if input.dim() == 2 and bias is not None:
     1369         # fused op is marginally faster
     1370         ret = torch.addmm(bias, input, weight.t())
  
  AttributeError: 'tuple' object has no attribute 'dim'
  </code></pre>
  
  <p>Meanwhile deleting the classifier entirely (all three layers)</p>
  
  <pre><code>BERT3 = torch.nn.Sequential(*(list(BERT.children())[:-3])) 
  </code></pre>
  
  <p>Yields the expected tensor (within a size 1 tuple) with the expected shape (<code>[sentence_num,token_num,768]</code>).</p>
  
  <p>Why does the removal of two (but not three) layers breaks the model?
  And how can I access the <code>pre_classifier</code> results?</p>
  
  <p>It is not accessible by setting <code>config</code> with <code>output_hidden_states=True</code> as this flag returns the hidden values of the BERT transformer stack, not those of the classifier layers downstream to it.</p>
  
  <p>--</p>
  
  <h2>PS</h2>
  
  <p>The code used to initialize the BERT model:</p>
  
  <pre><code>def collect_data_for_FT():
  
      from sklearn.datasets import fetch_20newsgroups
      train_data = fetch_20newsgroups(subset='train', shuffle=True, random_state=42)
      test_data =  fetch_20newsgroups(subset='test', shuffle=True, random_state=42)
  
      print('size of training set: %s' % (len(train_b['data'])))
      print('size of validation set: %s' % (len(test_b['data'])))
      print('classes: %s' % (train_b.target_names))
  
      x_train = train_data.data
      y_train = train_data.target
      x_test = test_data.data
      y_test = test_data.target
  
      return(x_train,y_train,x_test,y_test)
  
  
   bert_name = 'distilbert-base-uncased'
      from transformers import DistilBertForSequenceClassification,AutoConfig,AutoTokenizer
  import os
  dir_path = os.getcwd()
  dir_path=os.path.join(dir_path,'models')
  
  config = AutoConfig.from_pretrained(bert_name,num_labels=20) # change model configuration to access hidden values.
  
  try:
      BERT = DistilBertForSequenceClassification.from_pretrained(dir_path,config=config)
      print ("Finetuned predictor loaded")
  except:
      import tensorflow.keras as keras
      print ("No finetuned predictor found.\nTraining.")
      (x_train,y_train,x_test,y_test)=collect_data_for_FT()
      ####
      # prework:
      import ktrain
      from ktrain import text
      t = text.Transformer(bert_name, maxlen=500, classes=train_b.target_names)
      trn = t.preprocess_train(x_train, y_train)
      val = t.preprocess_test(x_test, y_test)
      pre_trained_model = t.get_classifier()
      learner = ktrain.get_learner(pre_trained_model, train_data=trn, val_data=val, batch_size=6)    
      ####
  
      ####
      # Find best learning rate
      learner.lr_find()
      learner.lr_plot()
      ####
  
      learner.fit_onecycle(2e-4, 4) # choosen based on the learning rate/loss plot.
  
      ####
      # prepare and save:
      predictor = ktrain.get_predictor(learner.model, preproc=t)
      predictor.save('my_distilbertbase_predictor')
      predictor.model.save_pretrained(dir_path)
      ####
      BERT = DistilBertForSequenceClassification.from_pretrained(os.path.join(dir_path), from_tf=True,config=config) # re-load tensorflow to pytorch
      BERT.save_pretrained(dir_path) # save as a "full blooded" pytorch model
      BERT = DistilBertForSequenceClassification.from_pretrained(dir_path,config=config)  # re-load
      from tensorflow.keras import backend as K
      K.clear_session() # loading from tensorflow takes up space and the GPU. This releases it/
  </code></pre>