is there a downside to using a wood mortar and pestle?
There are at least three potential downsides:


wood is quite light, so you will not get the benefit of added weight of the pestle for crushing;
many wooden mortars have a smooth texture, which will not aid in the grinding as much as, e.g., a coarse ceramic mortar; and
wood is porous—increasingly so as it dries over time—so it may have the tendency to pick up the flavors of its contents.