1. A method comprising: receiving a voice input from a first user at an input device of a first computing device operated by the first user; recognizing a task request from voice recognition output generated from the voice input; analyzing the task request to identify a geographic constraint imposed by the first user in the voice input; based on the analyzing, determining that the task request relates to a second user and checking an access control list relating to an automated assistant serving the second user to determine whether the first user has appropriate access rights as regards the second user for action to be taken on at least part of the task request; and in response to determining that the first user has appropriate access rights as regards the second user, taking action on at least part of the task request, wherein taking action on at least part of the task request causes the automated assistant serving the second user to engage in a natural language human-to-computer dialog with the second user at a selected time and location via an automated assistant client executing on one or more computing devices operated by the second user; wherein the selected time and location are selected based on satisfaction of the geographic constraint by a position coordinate provided by one or more of the computing devices operated by the second user.