Use domain username on SQL server reporting services installed in a server which is part of a Workgroup
Well according to MSDN here:


  Group or User
  Type the name of a group or user account in your
  domain. If the report server is running under a local account, you
  must specify local groups or users. If the report server is running
  under a domain account, you must specify domain groups or users. Enter
  the account in this format: &lt;domain&gt;\&lt;account&gt;.


I would take that to mean you will not be able to do what you are wanting. A work around I would try is to create a local group, adding the domain group/user to it, and try to add that local group to the new system role. It has been a while since I have supported a mixed environment so I am not sure if you can add domain accounts to a workgroup computer. You might have to check with your AD admin for assistance on that.