Query of sys.login_token doesn't include particular windows group while connect within workgroup
The problem solved after I remove logins and associated users in SQL Server then add them back again.
Have no clue what went wrong but a record for people who might run into the same issue.