Installing glassfish on Mountain Lion
The DISPLAY variable tells the X11 libraries where the X11 server is located.

Try

 export DISPLAY=:0


I do not believe the Unix installer supports installing OS X services, so unless you have very explicit needs at this time you can get along just with the zip-distribution of Glassfish.  Unzip it and run asadmin start-domain.