Can't start Octave 3.8.0 in GUI mode
I had a similar problem with Octave 4.0.0, when I installed it as hadi suggested I got this message:

The graphical user interface is disabled by default since it is still buggy on
OS X; use brew with the option --with-gui to enable it.


Octave was compiled with gnuplot; enable it via graphics_toolkit('gnuplot').
All graphics terminals can be used by setting the environment variable GNUTERM
in ~/.octaverc, and building gnuplot with the corresponding options.

setenv('GNUTERM','qt')    # Requiers QT; install gnuplot --with-qt
setenv('GNUTERM','x11')   # Requires XQuartz; install gnuplot --with-x11
setenv('GNUTERM','wxt')   # Requires wxmac; install gnuplot --with-wxmac
setenv('GNUTERM','aqua')  # Requires AquaTerm; install gnuplot --with-aquaterm

You may also set this variable from within Octave. For printing the cairo backend
is recommended, i.e., install gnuplot with --with-cairo, and use

print -dpdfcairo figure.pdf


When using the the qt or fltk toolkits then invisible figures do not work because
osmesa does currently not work with the Mac's OpenGL implementation. The usage of
gnuplot is recommened.


What worked for me was adding the --with-gui:

brew install octave --with-qt --with-fltk --with-gui


Or reinstalling:

brew reinstall octave --with-qt --with-fltk --with-gui