How many layers should I replace in transfer learning CNN
To build on the previous answer:
In transfer learning, the goal is to use a pre-trained model and tweak the model to then specialise it to suit a certain task.
So, what we do is, as SrJ has eluded to, keep the main model's architecture in tact. So this would be the 6 CNN layers (and possibly the three linear layers, if they were also involved in pre-training).
After a model has been pre-trained, what we do is add additional layers to the model so that it is suited for our task. So in this case, the least you would do is have a final output softmax layer, which produces a probability distribution over the authors.
In between the final output layer and the original model's architecture, you can add more layers if it is appropriate.
When training this model with your task-specific data (this stage is called fine-tuning). We freeze our original model's architecture. This essentially means that the parameters within the models original layers will not change to prevent possible loss in generalisation performance. We only allow the additional layer's parameters to change during fine-tuning.
Overall message is to not replace layers, always add onto the existing model to tailor the model more to your classification task.