How do I check if an app is going through system proxy?
You can try to run a proxy server locally, and change your Network preferences to point to that proxy, and check in the proxy logs to see if the app sends its request through it.

I would suggest using Charles.
It's quite easy to use: just change our proxy settings in Network preferences to either HTTP proxy localhost:8888 or SOCKS proxy localhost:8889, and you should start seeing outgoing requests if you use your browser.

Once you do, try using your custom app to see if it sends its request through the proxy or not.

Best of luck.