How to adapt the equation of Gaussian mechanism noise based on number of executions
The formula you mention to get $\sigma$ given $\delta$ and $\varepsilon$ is only correct for $\varepsilon&lt;1$. It's also not tight.
If you use Gaussian noise on multiple statistics, each of sensitivity 1, and want to get the tightest possible privacy guarantees, then you should use the Analytic Gaussian Mechanism instead (Theorem 8), replacing $\Delta$ with the square root of the number of your statistics. It's tight and works for any value of $\varepsilon$ and $\delta$.
The only problem with that approach is that there's no formula giving $\sigma$ as a simple function of $\varepsilon$, $\delta$ and $\Delta$; so you have to do some binary search to find the best possible value (and be careful about how you do the floating-point calculations). Here is an example from our open-source differential privacy library.
Finally, if what you want to do is train a model in a DP way, I would suggest using existing tooling, like TensorFlow Privacy, rather than rolling your own.