Pseudo-random sequence prediction Disclaimer: I am a biologist, so sorry for (perhaps) basic question phrased in such crude terms.
I am not sure if I should ask this question here or on DS/SC, but CS
is the largest of three, so here goes. (After I posted, it occurred to me that Cross-Validated might be the better place for it, but alas).

Imagine there is an agent, who makes binary decisions. And an environment, which, for each of the agent's decisions (&quot;trials&quot;), either rewards the agent or not. The criteria for rewarding the agent's decisions are not simple. In general criteria are random, but they have limitation, for example, environment never rewards more than 3 times for the same decision and never alternates rewarded decision more than 4 times in a row.
Sequence of criteria might look something like this then
0 0 0 1 0 1 0 0 1 1 1 0 1 1 0 0 1 0 ...

but never
0 0 0 1 0 1 0 0 1 1 1 1 1 1 0 0 1 0 ...

because reward criterion cannot repeat more than 3 times.
In these conditions it is quite easy to formulate the strategy ideal observer should undertake to maximize the reward. Something along the lines of

decide randomly
if you detect that criteria repeated 3 times -- decide opposite than last criterion
if you detect that criteria alternated 4 times, decide according to the last criterion

Now, the difficult part. Now the criterion on each trial depends not only on the history of previous criteria, but also on the history of agent's decisions, e.g. if agent alternates on more than 8 out of the last 10 trials, reward same decision as agent made last time (as if to discourage the agent from alternating) and if agent repeated same decision on more than 8 of the the last 10 trials, i.e. he is biased, make criterion opposite of the bias. The priority of history of criteria over history of decisions is specified in advance, so there is never ambiguity.
The sequences of decisions (d) and criteria (c) might now look like this
d: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 1 1 0 1 0 1 0 ...
c: 1 0 1 0 0 0 1 1 0 0 1 1 1 1 1 1 1 1 0 1 0 0 1 1 0 0 0 1 0 ...
                       ↑ here criteria counteract bias in decisions  

I do not see any simple way of inventing maximizing strategy for the agent. But I am sure there must be one, and some kind of clever machine learning algorithm should be able to identify it.
My question is not so much about how to solve this problem (although I would be happy if you suggest a solution), but more how these types of problems are called? Where can I read about it? Is there an abstract solution or only simulation can help? In general, how can I, as a biologist, approach this type of problem?
You can approach this problem using Reinforcement Learning.

A classic book for this is Sutton and Barto:

The draft of the second edition is available for free:
https://webdocs.cs.ualberta.ca/~sutton/book/the-book.html

In order to make your problem Markovian, define each state as a vector of the last ten decisions. You actions will be 1 or 0.