How to structure git repositories for project?
Drupal.org uses just one repository for module, and branches for each module version (for example, 7.x-1.x for module in version 1.x for Drupal 7, 6.x-1.x for module in version 1.x for Drupal 6, 7.x-2.x for module in version 2 for Durpal 7, and so on).

Also, if your server and client share code you may use just one Durpal module with submodules, so all code is in one repository. Users will just enable client or server (and the module that holds the common code).

So, your actual setup is fine if client and server do not share any code or very little and you want to keep them as separate modules. Just add branches for each Drupal version (6.x, 7.x, 8.x). Or join them into one repository and split code into three modules: common, server and client.

You can check the 'Maintaining a drupal.org project with Git' page.