How to prevent octave from spawning in a new Terminal window?
The actual script that executes when the new Terminal window is opened can be found at /usr/local/octave/3.8.0/bin/octave-3.8.0.  This path will be different for different versions of Octave.  To have this script run when you run the octave command, create a symbolic link to something like /usr/local/bin/octave (assuming /usr/local/bin is on your path).  

The command to run this would be something along the lines of:

sudo ln -sf /usr/local/octave/3.8.0/bin/octave-3.8.0 /usr/local/bin/octave