title: How to create a data preprocessing pipeline in pytorch outside the Dataloader class?
body: <p>I am trying to make a model for data with 40 features which have to classified into 10 classes. I am new to PyTorch and this is my first project in it.</p>
  <p>I am given a custom Dataset class (<strong>which I am not allowed to change</strong>) which is as follows:</p>
  <pre><code>class MyData(Dataset):
      def _init_(self, mode):
          with open(mode+'.pkl', 'rb') as handle:
              data = pickle.load(handle)
              self.X = data['x'].astype('float')
              self.y = data['y'].astype('long')
  
      def _len_(self):
          return len(self.X)
  
      def _getitem_(self, idx):
          if torch.is_tensor(idx):
              idx = idx.tolist()
  
          sample = (self.X[idx], self.y[idx])
  
          return sample
  </code></pre>
  <p>I have done some preprocessing on the data like normalization and then trained and saved the model. As I wasn't allowed to change the dataset class, I made the changes outside of it and then used the <code>DataLoader</code> method. The preprocessing is as follows :</p>
  <pre><code>train_data=MyData(&quot;train&quot;)
  features, labels = train_data[:]
  
  df = pd.DataFrame(features)
  
  x = df.values
  min_max_scaler = preprocessing.MinMaxScaler()
  x_scaled = min_max_scaler.fit_transform(x)
  
  input_array = x_scaled
  output_array = labels
  
  inputs = torch.Tensor(input_array)
  targets = torch.Tensor(output_array).type(torch.LongTensor)
  dataset = TensorDataset(inputs, targets)
  train_ds, val_ds = random_split(dataset, [3300, 300])
  
  
  batch_size = 300
  n_epochs = 200
  log_interval = 10
  train_losses = []
  train_counter = []
  test_losses = []
  
  train_loader = DataLoader(train_ds, batch_size, shuffle=True)
  val_loader = DataLoader(val_ds, batch_size)
  test_counter = [i*len(train_loader.dataset) for i in range(n_epochs + 1)]
  </code></pre>
  <p>After this I define the training and testing functions ( and remove the print statements as the autograder will not be able to grade my assignment if I do so) as follows:</p>
  <pre><code>def train(epoch):
    model.train()
    for batch_idx, (data, target) in enumerate(train_loader):
      optimizer.zero_grad()
      output = model(data.double())
      loss = criterion(output, target)
      loss.backward()
      optimizer.step()
      if batch_idx % log_interval == 0:
        train_losses.append(loss.item())
        train_counter.append(
          (batch_idx*32) + ((epoch-1)*len(train_loader.dataset)))
      save_model(model)
  
  def test():
    model.eval()
    test_loss = 0
    correct = 0
    with torch.no_grad():
      for data, target in val_loader:
        output = model(data.double())
        test_loss += criterion(output, target).item()
        pred = output.data.max(1, keepdim=True)[1]
        correct += pred.eq(target.data.view_as(pred)).sum()
    test_loss /= len(val_loader.dataset)
    test_losses.append(test_loss)
    
  test()
  for epoch in range(1, n_epochs + 1):
    train(epoch)
    test()
  </code></pre>
  <p>Even after doing that, the autograder is still not able to grade my code. I mainly think it's because maybe I am making an error with how I input the data to the model but I am not able to narrow down to what exactly is the problem and how do I correct it. As I'm new to pytorch, I was looking at how to do the preprocessing but all of them involved the Dataset Class so I'm not sure how to go about it.</p>
  <p>My model is as follows:</p>
  <pre><code>class MyModel(nn.Module):
      def __init__(self):
          super(MyModel, self).__init__()
          #self.flatten=nn.Flatten()
  
          self.net_stack=nn.Sequential(
              nn.Conv1d(in_channels=40, out_channels=256, kernel_size=1, stride=2), #applying batch norm
              nn.ReLU(),
              nn.MaxPool1d(kernel_size=1),
              nn.Dropout(p=0.1),
              nn.BatchNorm1d(256, affine=True),
              nn.Conv1d(in_channels=256, out_channels=128, kernel_size=1, stride=2), #applying batch norm
              nn.ReLU(),
              nn.MaxPool1d(kernel_size=1),
              nn.Dropout(p=0.1),
              nn.BatchNorm1d(128, affine=True),
              nn.Conv1d(in_channels=128, out_channels=64, kernel_size=1, stride=2), #applying batch norm
              nn.ReLU(),
              nn.MaxPool1d(kernel_size=1),
              nn.Dropout(p=0.1),
              nn.BatchNorm1d(64, affine=True),
              nn.Conv1d(in_channels=64, out_channels=32, kernel_size=1, stride=2), #applying batch norm
              nn.ReLU(),
              nn.MaxPool1d(kernel_size=1),
              nn.Dropout(p=0.1),
              nn.BatchNorm1d(32, affine=True),
              nn.Flatten(),
              nn.Linear(32, 10),
              nn.Softmax(dim=1)).double()
  
      def forward(self,x):
          # result=self.net_stack(x[None])
          x=x.double()
          result=self.net_stack(x[:, :, None]).double()
          print(result.size())
          return result
  </code></pre>
  <p>One instruction I've got is that they've written:</p>
  <pre><code># Please make sure we can load your model with:
  # model = MyModel()
  # This means you must give default values to all parameters you may wish to set, such as output size.
  </code></pre>