Do factors other than concentration gradient and membrane permeability affect the diffusion potential?
You can determine the diffusion potential mathematically with the Nernst equation. If it's not in the Nernst equation, it's not necessary.
The charge of the ion matters (including valance), the temperature matters. Some universal constants are in the equation but these don't vary.
Other factors that potentially matter are found in the other parameters, like the permability.