Using standard ML models for modeling a derivative when the data set only contains function values I want to model a process that is a function of time, $X(t)$. I have a data set which corresponds to a coarse sampling of the function values $X$ at different time points, $t_1, t_2, ...$. I have a analytic-derived model for the derivative $\frac{dX}{dt}$ = $F(X(t),G(t))$ which has a couple tunable parameters in the (known) functions $F$ and $G$. I've been able to tune these models with the following process:


Chose a set of parameters for the functions $F$ and $G$.
Do a forward-Euler method from the earliest time in my data set to the ending point in my data set, generating the $X(t)$ which corresponds to the above parameters in $F$ and $G$.
Use this $X(t)$, along with my measured data points to generate a model error.
Use the above modeled error to go back to (1), iterating and doing an optimization over the parameters in the expression for $\frac{dX}{dt}$ to minimize the error.


Note that in the above I have to deal with the function itself and not the derivative because my measured data is on a coarser time grid than the derivative needs to be computed. For example, my derivative changes meaningfully on a scale of $\Delta t$, but my data set might only have points every $5\Delta t$ to $10 \Delta t$. I cannot take the function values themselves from the data set and create a derivative myself because of this discrepancy in the time scales.

Now this process works pretty well, and I'm able to achieve a pretty good error. However, I'm wondering if there's a better way to model the derivative $\frac{dX}{dt}$ than this analytic model. For example, would a neural network or a regression tree be able to do a better job?

My question is, how would I do this in a computationally efficient way? Let's take a neural network as an example. I'm used to having inputs and their associated outputs when training a NN with standard packages. But the problem is, if I want to model the derivative, I don't have the derivative as an output to train on. I only know the function values and have only been able to compute a error to minimize on through the forward Euler integration. Of course I could treat the NN parameters as I did with the parameters above, and do a brute force minimization on them, but I obviously lose a lot of the efficiency of backprop in doing so (or, if I was doing this with a decision tree, would lose the logic of how regression trees are normally optimized).

Is there a better way for me to do this? Are there methods that exist to handle this problem? If anyone could point me in a fruitful direction it would be much appreciated.
This is actually really easy to implement in any deep learning framework with automatic differentiation capability. 

You need two neural networks. One to approximate the $X$, another to approximate the $dX/dt$.


The first step is easy. You have samples of input and target for $X$.
Just follow the normal process to train you network to learn $X$
For the second step is that you don't have the target values (i.e. gradient) directly. The trick is that you already got an approximator for $X$ from the first step, so use that to get your target for the second network. 


I am most familiar with pytorch, so I will use it as an example. 



import torch
from torch.nn import Sequential, Linear, ReLU
from torch.autograd import Variable
from torch.optim import Adam
import numpy as np

x = np.linspace(0, 10, 101).reshape(-1, 1)
y = np.sin(x) # Using sin as an example

x = Variable(torch.Tensor(x), requires_grad=True) 
y = Variable(torch.Tensor(y), requires_grad=False)

f = Sequential(....) # Build f approximator
f_optimizer = Adam(f.parameters())

for epoch in range(1000):
    loss= f(x) - y
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

# Build grad approximator
f_grad = Sequential(....) 

# need a new optimizer to train f_grad
f_optimizer = Adam(f_grad.parameters()) 

for epoch in range(1000):
    # Directly backprop from the output to the gradient of the f
    model(x).backward() 
    loss= f_grad(x) - x.grad # x.grad is the grad of f w.r.t x
    f_optimizer.zero_grad()
    loss.backward()
    f_optimizer.step()  

f_grad # this is the gradient function you are looking for


(I don't have a machine with pytorch install at the moment, so don't expect the code to run with some debugging)