Inherent limitation of Switching Lemma for finer lower bounds
I think you can't beat the $\exp(n^{\Omega(1/d-1)})$ lower bound. Here's a construction that should work for computing parities:

In order to compute parity of $n$ variables, partition them into groups of size $t \approx n^{1/d}$, compute parity of each group and recursively compute the parity of parities. That is, start with a parity tree of fan-out $t$. The depth of this tree would be $d$.

Then, replace each parity gate with a CNF or DNF or size $\approx \exp(t)$. This way you get a circuit of depth $2d$. It is possible to alternate between CNFs and DNFs at different levels so the circuit collapses to depth $d$. This way you get a depth $d$ circuit of size $\approx \exp(n^{1/d})$ that computes the parity of $n$ bits.

Now, since switching lemma is tight for the parity function, you shouldn't hope to get a better lower bound for more sophisticated functions just relying on that.