What does mtry < 1 in TuneRF mean?
It's a multiplicative factor, so nonintegers is OK.  

But a value &lt;1 shouldn't be used: while theoretically setting it equal to 0.5 should be the same as setting it equal to 2, the details under the hood may break:

      ...
      mtryCur &lt;- if (direction == "left") {
        max(1, ceiling(mtryCur / stepFactor))
      } else {
        min(ncol(x), floor(mtryCur * stepFactor))
      }
      ...


(https://rdrr.io/cran/randomForest/src/R/tuneRF.R)
when stepFactor &lt; 1, the ceiling(mtryCur / stepFactor) may exceed ncol(x), and the ceiling is the wrong direction to round, etc.

See also https://stackoverflow.com/q/27211765/10495893