Does programs like After effects and Unity engine use graphics library to render custom guis? I was always interested how programs like: Unity, AE, godot engine and other render their custom guis. So my question is, do they use graphics library to render custom guis or something else. If they don't use graphics library, should I use it? Thanks :).
So:


All GUI are somehow graphics. They don't have to be accelerated by the GPU but these days most are. In the old times, they could be rendered on the CPU (X11 times).
You need to be careful between making a distinction between the frameworks/libraries that are potentially being used for building up UIs and what these libraries use to render the graphics making up these UIs.


A lot of applications like Maya for instance use libraries for creating their UIs (Maya uses Qt so as most products developed by the Foundry: Nuke, Marri, etc.) which is why a lot of these applications look the same ... (sadly).

Qt and most UI library/framework use a graphics API to accelerate the rendering of widgets. 

Some applications like Unity do not use "commercial" libraries like Qt. They develop their own, but in the end they also use the GPU to render their GUIs.