How to optimize for time correlated hidden function - the magical candy machine
Thought #1

From the short description you give, and assuming you have some data (or can synthesise it), I would have a go at training a Hidden Markov Model. Have a look here for an awesome visual primer. Intuitively, HMMs do what you describe.

There are some observables:


how much candy you put in the machine,
how much comes out, and
which timestep we are in


There are certain states (here just discrete states):


amount of dispensed candy goes up (strictly increasing)
amount of dispensed candy goes down (strictly decreasing)


Lastly, there is a transition: between these state, given the inputs. These are the observables, as far as you are concerned.

With HMMs, we are however assuming that there is more than meets the eye; there are some hiddens states (latent states), which are unobservable to use. The model keeps track of these and uses them in conjunction with the observables list above to decide how to act - it has the full mapping function. Below is a schematic diagram of such a model:



This just shows several states, along with the probabilities of transitioning to the other states, or stating in the same state. Check out the source for a detailed description of the diagram. One can already imagine sketching this drawing for your problem.

For more on the theory of HMMs, I recommend this text based walkthrough by Jeff Bilmes, and this YouTube series by mathematicalmonk (chapter 14 of the playlist). If you go this route, you might consider trying out the Pomegranate library (in Python).

The final logic of deciding what to do, what action to take, is something you could either hard-code or use a model to take it for you. For example, a hard-coded approach would be something like:

if current_state == 'normal':
    take_lots_of_candy()
elif current_state == 'unsure':
    take_some_candy()
elif current_State == 'dangerous':
    pause_taking_candy()
else:
    catch_other_states()


For a model-based approach, you could expand further on the ideas outlines below.



Thought #2

Of course, you are in essence just mapping some input to some outputs. Almost any machine learning algorithm will have a good go at it and - depending on the complexity of the internal workings of your magical machine - will be better or worse. Algorithms such as boosting (e.g. via gradient descent) generally require you to put some kind of information into the model e.g. by specifying a regression equation, mapping inputs to outputs. A good implementation is the mboost package in R. There is a great tutorial to help decide what might be important.



Thought #3

Coming back to the point of just mapping inputs to outputs: a simple Multi-Layer Perceptron (MLP) from Scikit-Learn - a.k.a. feed-forward neural network - should theoretically offer the flexibility to approximate any function you might throw at it (with a few assumptions).
You could also implement this using one of the various deep learning frameworks, should your MLP just not cut the mustard.



Final thoughts

I would lastly like to highlight that your final model will only likely be as good as the data you feed it, and that while it may work well on your synthesised training data, there are no guarantees of generalisation, without putting more thought into the specific problem and specific distributions from which your data are sampled.

If you go the way of Markov models, you could perhaps then use their state predictions as inputs to further models; thus creating an ensemble.

If you want to get really fancy and more towards current state-of-the-art models in sequence analysis, you could venture into the world of Recurrent Neural Networks, e.g. utilising LSTM cells, which maintain an internal state representation, including new information and selectively discarding old information. This might be a good way to take all your points into consideration in one large model. The catch here would be that you generally require vast amounts of data to train such a model.