Filtering activities by Source Contact ID
Search Builder seems to be broken for this functionality, you may report that as an issue in the issue tracker: https://issues.civicrm.org/. 

My understanding is that you want to search within the backend of CiviCRM?
Using the API is currently probably the cleanest way of performing the search and adding them at once by using the Activity.get and chaining it with GroupContact.create: 


  cv api Activity.get sequential=1 return="target_contact_id" source_contact_id=11 target_contact_id={"IS NOT NULL":1} options={"limit":0} api.GroupContact.create={"group_id":"5","contact_id":"$value.target_contact_id"} 


Alternatively you can use the advanced search and search for all activities that have your desired contact as "Added by" and display results as Activity.
You can then tag all those activities with a temporary tag that can be used for activities. In a second advanced search you can search for contacts that have an activity "With" and the temporary tag and add those contacts to a group.

There is also the activity detail report that will let you filter those contacts the easiest way, but I cant see how to get those contacts into a group.