ansible-pull ignore group_vars vault
Eventually I just got rid of the group_vars all together.
I specified a new role that includes several other roles with 'include_role:' and added the vars directly to those roles with inline-vault variables:

- name: include ipaclient
  include_role:
    name: ipaclient
  vars:
    my_user: user
    my_password: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          123456123456123456123456123456132456...