How to run Linux gui apps on macOS (e.g. via XQuartz)
You can't launch the Linux version of Firefox on macOS, because it is a binary for Linux calling libraries from Linux and calling system calls of Linux.

You have to build a macOS version of Firefox on macOS from the source tree you download from here:

https://archive.mozilla.org/pub/firefox/releases/


through the traditionnal steps:

./configure
make
/usr/bin/sudo make install


And during the configure step, you will have to take care to use the standard sets of X11 libraries (disclaimer: I didn't test this).