Is it a self plagiarism to reuse a section of my previous paper in a new paper?
Technically, it is not self-plagiarism if you were to cite the relevant material, even if it's your own. It would be advisable if to rephrase the section. If considerable amount of detail is repeated from thee pervious paper, it would be better to have consent from your publisher before hand. 

I notice this many times among authors. Those who make a minor improvement over their own research tend to have major portions of their old papers rephrased (some even repeated, but again, that's not encouraged). As long the overall concept is novel this won't be a problem.