using copy module with wildcards
with_fileglog creates a loop with the fileglob lookup. As all other lookups it runs on the controller machine, not on the remote server.

Since I guess you don't already have those backup files on the controller in the path you referenced, your loop is empty.

As you have guessed, this approach will not work for your scenario. I suggest you have a look at the find module to create the list of files to fetch from your primary server.