What is the maximum depth of subchapters one should use in a paper?
Many, if not most, journals specify a maximum of three levels. In a book where the chapter is the top level, four may be ok considering each chapter can be as extensive as a paper. The problem of having too many levels is that the headings disrupt the reading. A good sign of this is when you end up having one heading per paragraph. If that is the case, you can probably remove the lowest level of headings and try to make the resulting segments of text flow by inserting bridges that makes paragraphs into a coherent text.

In my experience having read numerous student reports, theses and articles, three is a good goal. It is rare that a fourth level adds much in terms of structure apart from perhaps helping the writer. In fact, I often recommend students to keep a more detailed list of headings to enable them to see the structure of what they are writing but under the pretext that only a maximum of three should remain in their final version. I stand by that recommendation.