How to identify too small network in reinforcement learning? I am using Open AI's code to do a RL task on an environment that I built myself.

I tried some network architectures, and they all converge, faster or slower on CartPole.

On my environment, the reward seems not to converge, and keeps flickering forever.

I suspect the neural network is too small, but I want to confirm my belief before going the route of researching the architecture.

How can I confirm that the architecture is the problem and not anything else in a neural network reinforcement learning task?
Check the function loss.
It might be that your environment is impossible to learn. However, most likely the network simply can't handle it. By measuring the loss during the learning stage, if you find it is always very high and does not decrease, it's a strong indication this might be the issue.
Because the network is too simple, when you optimize for some states, you ruin others. There is not formal way to find out if this is the case, but since the same algorithm works elsewhere, it's either a problem of your environment, or of the network.