In Deep Deterministic Policy Gradient, are all weights of the policy network updated with the same or different value?
Each Q output is a scalar, so the sum of all those is a scalar. Thus, you're taking a gradient wrt your parameters of a scalar. The result is a vector with one entry per parameter.