title: How to test a model before fine-tuning in Pytorch Lightning?
body: <p>Doing things on Google Colab.</p>
  <ul>
  <li>transformers: 4.10.2</li>
  <li>pytorch-lightning: 1.2.7</li>
  </ul>
  <pre class="lang-py prettyprint-override"><code>import torch
  from torch.utils.data import DataLoader
  from transformers import BertJapaneseTokenizer, BertForSequenceClassification
  import pytorch_lightning as pl
  
  dataset_for_loader = [
      {'data':torch.tensor([0,1]), 'labels':torch.tensor(0)},
      {'data':torch.tensor([2,3]), 'labels':torch.tensor(1)},
      {'data':torch.tensor([4,5]), 'labels':torch.tensor(2)},
      {'data':torch.tensor([6,7]), 'labels':torch.tensor(3)},
  ]
  loader = DataLoader(dataset_for_loader, batch_size=2)
  
  for idx, batch in enumerate(loader):
      print(f'# batch {idx}')
      print(batch)
  
  category_list = [
      'dokujo-tsushin',
      'it-life-hack',
      'kaden-channel',
      'livedoor-homme',
      'movie-enter',
      'peachy',
      'smax',
      'sports-watch',
      'topic-news'
  ]
  
  tokenizer = BertJapaneseTokenizer.from_pretrained(MODEL_NAME)
  
  max_length = 128
  dataset_for_loader = []
  for label, category in enumerate(tqdm(category_list)):
      # file ./text has lots of articles, categorized by category
      # and they are just plain texts, whose content begins from forth line
      for file in glob.glob(f'./text/{category}/{category}*'):
          lines = open(file).read().splitlines()
          text = '\n'.join(lines[3:])
          encoding = tokenizer(
              text,
              max_length=max_length, 
              padding='max_length',
              truncation=True
          )
          encoding['labels'] = label
          encoding = { k: torch.tensor(v) for k, v in encoding.items() }
          dataset_for_loader.append(encoding)
  
  SEED=lambda:0.0
  
  # random.shuffle(dataset_for_loader) # ランダムにシャッフル
  random.shuffle(dataset_for_loader,SEED)
  n = len(dataset_for_loader)
  n_train = int(0.6*n)
  n_val = int(0.2*n)
  dataset_train = dataset_for_loader[:n_train]
  dataset_val = dataset_for_loader[n_train:n_train+n_val]
  dataset_test = dataset_for_loader[n_train+n_val:]
  
  dataloader_train = DataLoader(
      dataset_train, batch_size=32, shuffle=True    
  ) 
  dataloader_val = DataLoader(dataset_val, batch_size=256)
  dataloader_test = DataLoader(dataset_test, batch_size=256)
  
  class BertForSequenceClassification_pl(pl.LightningModule):
      def __init__(self, model_name, num_labels, lr):
          super().__init__()
          self.save_hyperparameters()
          self.bert_sc = BertForSequenceClassification.from_pretrained(
              model_name,
              num_labels=num_labels
          )
  
      def training_step(self, batch, batch_idx):
          output = self.bert_sc(**batch)
          loss = output.loss
          self.log('train_loss', loss)
          return loss
  
      def validation_step(self, batch, batch_idx):
          output = self.bert_sc(**batch)
          val_loss = output.loss
          self.log('val_loss', val_loss)
  
      def test_step(self, batch, batch_idx):
          labels = batch.pop('labels')
          output = self.bert_sc(**batch)
          labels_predicted = output.logits.argmax(-1)
          num_correct = ( labels_predicted == labels ).sum().item()
          accuracy = num_correct/labels.size(0)
          self.log('accuracy', accuracy)
  
      def configure_optimizers(self):
          return torch.optim.Adam(self.parameters(), lr=self.hparams.lr)
  
  checkpoint = pl.callbacks.ModelCheckpoint(
      monitor='val_loss',
      mode='min',
      save_top_k=1,
      save_weights_only=True,
      dirpath='model/',
  )
  trainer = pl.Trainer(
      gpus=1,
      max_epochs=10,
      callbacks = [checkpoint]
  )
  
  model = BertForSequenceClassification_pl(
      MODEL_NAME, num_labels=9, lr=1e-5
  )
  
  ### (a) ###
  
  # I think this is where I am doing fine-tuning
  trainer.fit(model, dataloader_train, dataloader_val)
  
  # this is to score after fine-tuning
  test = trainer.test(test_dataloaders=dataloader_test)
  print(f'Accuracy: {test[0][&quot;accuracy&quot;]:.2f}')
  </code></pre>
  <p>But I am not really sure how to do a test before fine-tuning, in order to compare two models before and after fine-tuning, in order to show how effective fine-tuning is.</p>
  <p>Inserting the following two lines to <code>### (a) ###</code>:</p>
  <pre class="lang-py prettyprint-override"><code>test = trainer.test(test_dataloaders=dataloader_test)
  print(f'Accuracy: {test[0][&quot;accuracy&quot;]:.2f}')
  </code></pre>
  <p>I got this result:</p>
  <pre class="lang-py prettyprint-override"><code>---------------------------------------------------------------------------
  AttributeError                            Traceback (most recent call last)
  &lt;ipython-input-13-c8b2c67f2d5c&gt; in &lt;module&gt;()
        9 
       10 # 6-19
  ---&gt; 11 test = trainer.test(test_dataloaders=dataloader_test)
       12 print(f'Accuracy: {test[0][&quot;accuracy&quot;]:.2f}')
       13 
  
  /usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py in test(self, model, test_dataloaders, ckpt_path, verbose, datamodule)
      896         self.verbose_test = verbose
      897 
  --&gt; 898         self._set_running_stage(RunningStage.TESTING, model or self.lightning_module)
      899 
      900         # If you supply a datamodule you can't supply train_dataloader or val_dataloaders
  
  /usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py in _set_running_stage(self, stage, model_ref)
      563         the trainer and the model
      564         &quot;&quot;&quot;
  --&gt; 565         model_ref.running_stage = stage
      566         self._running_stage = stage
      567 
  
  AttributeError: 'NoneType' object has no attribute 'running_stage'
  </code></pre>
  <p>I noticed that <a href="https://pytorch-lightning.readthedocs.io/en/latest/common/trainer.html#fit" rel="nofollow noreferrer"><code>Trainer.fit()</code> can take <code>None</code> as arguments other than <code>model</code></a>, so I tried this:</p>
  <pre class="lang-py prettyprint-override"><code>trainer.fit(model)
  test=trainer.test(test_dataloaders=dataloader_test)
  print(f'Accuracy: {test[0][&quot;accuracy&quot;]:.2f}')
  </code></pre>
  <p>The result:</p>
  <pre class="lang-py prettyprint-override"><code>MisconfigurationException: No `train_dataloader()` method defined. Lightning `Trainer` expects as minimum a `training_step()`, `train_dataloader()` and `configure_optimizers()` to be defined.
  </code></pre>
  <p>Thanks.</p>