What do you call the OSX Window Server, Window Manager, and Desktop Environment?
macOS has implementations of the X11 protocol as you're used to on Linux. This is called XQuartz, it is based on the same Xorg as you'll find on Linux. 

You can use many of the same desktop environments on top of macOS and XQuartz as you can on Linux. It is the exact same programs, just recompiled on macOS.

The terms for these things on macOS are the same as on Linux as it is the same technology (i.e. built on the X11 protocol). 

Wayland is a different display protocol than X111, and is not currently supported on macOS.

However note that "native macOS apps" do not use the X11 protocol, Xorg or anything like that. Instead they use the native Apple supplied systems. 

Here the window server is the process named simply "WindowServer" on a default macOS installation.

The "Desktop Environment" does not have a single counterpart on macOS rather it is a combination of multiple macOS parts - most notably WindowServer, Dock and perhaps Finder.

Regarding the Aqua, Quartz and XQuartz: 


Quartz is a 2D text and graphics rendering library. There's also a Quarts GPU accelerated compositor On Linux, the counterpart 2D library is Cairo or similar.
XQuartz is simply an X11 server built on top of the Quartz rendering library.
Aqua is the name of the Apple graphical user interface. It defines the looks of widgets/controls, menus, etc. The counterpart on Linux would be something like GTK and Gnome, or QT and KDE.