How to mute a FaceTime iPhone call on OS X through a shortcut?
Yes you can! Here are the steps:


Open Automator and create an Automator Service, with a single Run AppleScript action, containing the following code:

High Sierra and later:

tell application "System Events" to tell process "Notification Center"
    if checkbox "Mute" of window 1 exists then
        click checkbox "Mute" of window 1
    end if
end tell


Sierra and earlier:

tell application "System Events"
    if checkbox "Mute" of window 1 of application process "FaceTime" exists then
        click checkbox "Mute" of window 1 of application process "FaceTime"
    end if
end tell

Set the service to receive "no input" and to work in "any application", save it with a name of your choosing.
Now, go into the Shortcuts tab in the Keyboard preferences in System Preferences, and click on Services. The newly created service should be at the bottom, under the name you chose. Click on the service, then click on "add shortcut" to its right, and specify a shortcut to your liking.
You may have to grant Automator and/or FaceTime access in Accessibility in the Privacy tab of the Security and Privacy preferences.


That's it! You should now have a service set up, tied to the shortcut you chose that can be accessed from anywhere.