Calculating neuron outputs and derivatives
Try to minimaze memory allocations.

Allocate memory for deviceTemp and deviceInputs only once (in the constructor, for example):

cudaCall(cudaMalloc ( (void**)&amp;deviceInputs, some_big_value * sizeof(double) ));
cudaCall(cudaMalloc((void**)&amp;deviceTemp, some_big_value * sizeof(double)));


And in calculateNeuronsOutputsAndDerivatives, reallocate memory only if needed:

if (cur_deviceInputs_size &lt; inputsCount)
{
    cudaCall(cudaFree(deviceInputs));
    cudaCall(cudaMalloc ( (void**)&amp;deviceInputs, inputsCount*sizeof(double) ));
    cur_deviceInputs_size = inputsCount;
}