Nesterov Accelerated gradient is in correct order?
The standard momentum would have these steps:

straight away, recompute the new momentum:
$$\mu_{t+1} = \mu_{t} \cdot (decayScalar) + (learnRate)\cdot \nabla$$

adjust the weights by this new momentum
$$\theta_{t+1} := \theta_{t} - \mu_{t+1}$$



Nesterov momentum has this:

Make a big jump: correct the weights by any $\mu$ we have so far in our posession:
$$\theta_{t+1} := \theta_{t} - \mu\cdot (decayScalar)$$

Compute the gradient $\nabla$ from the new weights $\theta_{t+1}$

Correct these weights by this gradient (right now without any momentum):
$$\theta_{t+2} := \theta_{t+1} -  (learnRate)\cdot \nabla$$

At the very end, re-compute the momentum as follows:
$$\mu := \theta_{t+2} - \theta_{t}$$


So, the momentum is updated at the very end. It becomes a vector from &quot;weights before the big jump&quot;, pointing towards the &quot;weights after the correction by the fresh gradient&quot;.
reference: Geoffrey Hinton Lecture 6C Corsera

Re-arranging:
To avoid sticking the gradient computation in the middle of our optimizer function (steps 2 and 3), we can instead re-arrange things as follows:

compute gradient for the weights we have thus far.

correct such  weights by the gradient (right now without any momentum), as follows:
$$\theta_{t+1}:=\theta_t - (learnRate)\cdot \nabla$$

update the momentum:
$$\mu := \theta_{t+1} - \theta_{cached}$$
$$\theta_{cached} := \theta_{t+1}$$

big jump
$$\theta_{t+2} := \theta_{t+1} - \mu\cdot (decayScalar)$$


Notice, this way steps 2,3,4 are all inside of our optimizer. We can compute the gradient, outside of our optimizer (during step 1) making our code much more readable :)
    size_t _numApplyCalled = 0;
      
    //Nesterov Accelerated Gradient.
    //Placed at the end of a backprop, should be followed by a usual forward-propagation
    // https://datascience.stackexchange.com/a/26395/43077
    //
    void apply( float *toChange,  float *newGrad,  size_t count ){

        float learnRate = get(OptimizerVar::LEARN_RATE);//scalar
        float momentumCoeff = get(OptimizerVar::MOMENTUM_1);//scalar

        const bool isFirstEver_apply = _numApplyCalled == 0;

        for (int i=0; i&lt;_arraySize; ++i){
                //correction by gradient alone:
                toChange[i]  -= newGrad[i]*learnRate;

                // determine momentum:
                if (isFirstEver_apply){//nothing was cached yet.
                    _momentumVals[i] = 0.0f;
                }
                else {
                    _momentumVals[i] = toChange[i] - _weightsCached[i];
                }
                //caching, AFTER momentum calc, but BEFORE the jump:
                _weightsCached[i] = toChange[i];
            
                //jump:
                toChange[i] -= _momentumVals[i] * momentumCoeff;
        }//end for

        ++_numApplyCalled;//increments by 1
    }