Is reinforcement learning using shallow neural networks still deep reinforcement learning?
Even after several years of success of deep learning systems (i.e. neural networks trained with gradient descent and back-propagation), as far as I know, there is not yet a consensus on what constitutes a deep neural network. Some people could use a neural network with 2 hidden layers and call it deep (like in your case), but other people may just dedicate the adjective deep to refer to neural networks with 10, 100, or more hidden layers. In fact, there are some good reasons to associate the term deep only to neural networks that have a significant number of hidden layers (e.g. 100): for example, the exploding (or vanishing) gradient problem does not typically arise if you only have one hidden layer but can easily occur with many (e.g. 100) hidden layers.
Nevertheless, a neural network with at least one hidden layer can approximate any continuous function, given enough (but finite number of) units (or neurons) in the layers. See the universal approximation theorem. For this reason, we could start denoting any such neural network as deep, but, although this rule would exclude perceptrons (which can only approximate linear functions, and nobody would probably call them deep anyway), this rule would be a bit redundant or useless (i.e. we may just not use the adjective deep to start with).
In your case, the rule that the authors are using seems to be the following: if it contains more hidden layers than the bare minimum (i.e. 1) to approximate any continuous function, then let's denote it as deep.