why is the max number of Nodes mod(N/m) + mod(N/m^2) + 1 in R-tree?
Here is the relevant excerpt of that paper.


  The height of an R-tree containing $N$ index records is at most $\lceil\log_m N\rceil$, because the branching factor of each node is at least $m$. The maximum number of
  nodes is $\lceil\frac Nm\rceil + \lceil\frac N{m^2}\rceil + \cdots + 1$. Worst-case space utilization for all nodes except
  the root is $\frac mM$.


The paper you saw is so fuzzy that you mistook the formula as $|\frac Nm| + |\frac N{m^2}| + 1$.

How to explain the correct formula? Since all leaves are on the same level and each leaf-node contains at least $m$ index records, there are at most $\lceil\frac Nm\rceil$ leaf nodes. The number of the parents of all leaf nodes is at most $\lceil\frac{\frac Nm}m\rceil=\lceil\frac N{m^2}\rceil$. And so on.

Here is a copy of that paper that is more legible.