Using 都 when refering to both subject and objects
Yes, 都 can refer to either subject or objects.


When subject is plural, 都 refer to "all" of the subject.

For example:


  你们都来嘛？Are all of you coming?
  
  同学们都交按时交了作业。 All of the students in the class handed in their homework in time.

When subject is single, and object is plural, then 都 refers to "all" of the object.

For example:


  你把糖果都吃了？ Did you eat all of the candy?
  
  这些衣服你都买吗 ？ Are you going to buy all of those clothes?
  
  那些岛屿你都会去吗？Will you (single) go to all of the islands?

When both subject and object are plural, if only using 都, it is ambiguous as we don't know if it refers to the subject or object or both. So we either need to draw more info from the context, or if no context, we will have to ask the speaker to clarify on this.

For example:


  那些岛屿你们都会去吗？Will (all of ?)you (plural) go to (all of?) the islands?


When we mean all of the subject AND all of the object, then we will have to use TWO words meaning "all" (所有/全部/都/全/全都) together.

For example:


  你们都交了全部的作业嘛？  Did all of you guys hand in all the homework?
  
  你们都会去所有的那些岛屿吗？Will all of you (plural) go to all of the islands?


But sometimes it is still ambiguous even we use TWO such words.

For exmaple:


  所有的那些岛屿你们都会去吗？ 
  
  In this sentence, 所有 refers to the islands for sure, no doubt about it. But 都 may refer to either object or subject. So we still need to ask the speaker to clarify on it. 
  
  For this sentence, if there is no context, and we are not able to ask the speaker to clarify, then my intuition of interpreting this sentence is that both 所有 and 都 refer to islands. 



Conclusion: The things 都 refers to are somehow impacted by the sentence structure.