Is there a relation between the size of the neural networks and speed of convergence in deep reinforcement learning?
Speed and size, no. That's because speed is dependent on processor clock periods and the effective parallel nature of the particular deep RL design in the environment, which is also dependent on cores and host clustering. Size is not really quantifiable in a way that is meaningful in this relationship because there is a broader and more complex structure of a network that might be trained to produce a value to use in the RL algorithm chosen.


Width for each layer
Cell type and possibly activation function for each layer
Hyper-parameters


One can say that number of clock cycles times number of effective parallel processors is correlated to all the above, the overhead of the RL algorithm used, and the size of each data tensor.

There are a few inequalities developed in the PAC (probably approximately correct) framework, so it would not be surprising if there were some bounding rules for the relationship between clock cycles, effective parallelism, data widths, activation functions, and RL algorithms for deep RL.

Study of the algorithm, perhaps in conjunction with experimentation, may reveal the primary factors, essentially the processing bottlenecks. Further study of the factors involved in controlling loop iteration counts, which cause the bottleneck, could permit the quantification of computing effort required to maintain a particular maximum allowable response time, but that would be specific to a particular design.

Such might produce a metric that is effectively a count of clock cycles across the system's potentially parallel architecture for a worst case or mean RL action selection. That could then be used to determine the response time for a given system with all of the factors mentioned above fixed, including the priority and scheduling of the processes in each operating system involved.

Here's a guess. Feel free to critique, since such a formulation is a project far beyond the effort that should be put into answering a question online.

$$ t_r \propto e^{k_v} \, n_v \, \eta_v \, t_v \, \mu_v \, \sum c_v + e^{k_d} \, n_d \, \eta_d \, t_d \, \mu_d \, \sum c_d \; \text{,} $$

where $t$ is time, $k$ is tensor complexity, $n$ is number of cores, $\eta$ is the effective efficiency of the parallel processing, $\mu$ is the effective overhead cost of the glue code, $c$ is the cycles require of the significant (and probably repetitive) elements in the algorithms, $t_r$ is the total time to response, and the subscripts $v$ and $d$ designate the variable subscripted as either RL value determination metrics or deep network metrics respectively.