How can I allow the current user to only see their own page?
_permission only allows to give access to the route to the users who have that permission; it doesn't allow to give access basing on more complex conditions, as in the case of this question. As described in Custom route access checking, the route definition should be similar to the following one.
task_notify.user_page.tasks:
  path: '/user/{user}/tasks'
  defaults:
    _controller: '\Drupal\task_notify\Controller\TaskNotifyUserController::tasks'
    _title: 'Tasks list'
  requirements:
    _custom_access:  '\Drupal\task_notify\Controller\TaskNotifyUserController::taskAccess'
    user: \d+

The TaskNotifyUserController class would need to contain at least the following code (and other methods you need from that controller, including tasks()).
class TaskNotifyUserController extends ControllerBase {

  /**
   * Checks access for the users task page.
   *
   * @param \Drupal\user\UserInterface $user
   *   The user being viewed.
   * @param \Drupal\Core\Session\AccountInterface $account
   *   The account viewing the page.
   *
   * @return \Drupal\Core\Access\AccessResultInterface
   *   The access result.
   */
  public function taskAccess(UserInterface $user, AccountInterface $account) {
  public function taskAccess(UserInterface $user) {
    return AccessResult::allowedIf($account-&gt;isAuthenticated() &amp;&amp; $user()-&gt;id() == $account-&gt;id())
      -&gt;cachePerUser();
  }

}

If then the access to the route should be allowed also to users with a specific permission (for example, the permission to administer users), the taskAccess() code would be similar to the following one.
  /**
   * Checks access for the users task page.
   *
  /**
   * Checks access for the tasks tab.
   *
   * @param \Drupal\user\UserInterface $user
   *   The user being viewed.
   * @param \Drupal\Core\Session\AccountInterface $account
   *   The account viewing the page.
   *
   * @return \Drupal\Core\Access\AccessResultInterface
   *   The access result.
   */
  public function taskAccess(UserInterface $user, AccountInterface $account) {
    return AccessResult::allowedIf($account-&gt;isAuthenticated() &amp;&amp; $user-&gt;id() == $account-&gt;id())
      -&gt;orIf(AccessResult::allowedIfHasPermission($account, 'administer users'))
      -&gt;cachePerUser();
  }

Don't forget to import the class used from the controller, adding the following lines at the top of the controller file.
use Drupal\Core\Access\AccessResult;
use Drupal\Core\Session\AccountInterface;
use Drupal\user\UserInterface;

If the controller class uses other classes from different namespaces, similar lines must be added for each class.
I took the code used from Drupal core for the Tracker module as example. The Tracker module shows its tracker page on /activity/{user} using a controller to render it. See TrackerController.php.
  /**
   * Checks access for the users recent content tracker page.
   *
   * @param \Drupal\user\UserInterface $user
   *   The user being viewed.
   * @param \Drupal\Core\Session\AccountInterface $account
   *   The account viewing the page.
   *
   * @return \Drupal\Core\Access\AccessResult
   *   The access result.
   */
  public function checkAccess(UserInterface $user, AccountInterface $account) {
    return AccessResult::allowedIf($account-&gt;isAuthenticated() &amp;&amp; $user
      -&gt;id() == $account-&gt;id())-&gt;cachePerUser();
  }

The custom access callback is defined in tracker.routing.yml.
tracker.users_recent_content:
  path: '/activity/{user}'
  defaults:
    _controller: '\Drupal\tracker\Controller\TrackerController::buildContent'
    _title: 'My recent content'
  requirements:
    _permission: 'access content'
    _custom_access: '\Drupal\tracker\Controller\TrackerController::checkAccess'
    user: \d+