Where is my error in understanding gradient descent calculated two different ways?
Your second function is not an update rule, it is just a re-statement of the approximation between using finite differences to measure a gradient and the infinitesimal ones used in calculus. Most importantly, $s$ is not a learning rate or parameter that you are free to change. It is, by definition, the (negative of the) change observed in $C$ when you alter $w$ to $w'$.

The second equation you have derived is the same as:

$$w + \Delta w \approx w + \Delta C \frac{dw}{dC}$$

In addition, for some reason you decided to drop the approximation symbol partway through and treat your manipulations as if you had created a new weight update rule. Be careful when manipulating expressions on either side of an approximation, they do not always follow the same rules as for an equality, and treating them as such can lead you down some false paths.