Activity Permission - Prevent User to View Other User Activity - CiviCRM
I think this will require a custom extension to extend the ACLs such that the only folk who can see an Activity are those who are an Activity Contact (or who have View all Activities permission).

AFAIK this sort of permissioning is currently only provided through CiviCase but you have already indicated you want this as a new feature for Activities.

Search existing Extensions in case someone already developed this.

If you want assistance to build such an extension I suggest you try one (but only one) of the following so no one growls about cross-posting


https://chat.civicrm.org/civicrm/channels/dev
https://chat.civicrm.org/civicrm/channels/dev-newcomers
https://chat.civicrm.org/civicrm/channels/extensions


or if you want professional support you can find Partners such as Fuzion listed here.

Another approach, if you were using Drupal, and were interested in keeping most folk outside civicrm and have them interact mostly through Views and Webforms would be to build a View which would limit the Activities displayed to the Current User to those where they are a 'contact' on the Activity. 

We do this quite a lot for clients who have lots of users but few who really need to get deep inside civicrm and bounce folk between using Views to find info and then Webforms to add/update.

Hope that helps.