How to evaluate the state of the OS X Server.app from a shell script?
It can be determined by running the following command:

serverinfo --configured


To evaluate the result of the command within a shell script use the -q option and encapsulate it in an if-statement:

 if serverinfo -q --configured;
 then
    echo configured; 
 else
    echo not_configured;
 fi


The serverinfo gives two options to determine if the Server.app is just installed or configured: 

--software [PATH]
    Returns status 0 if the root volume has a server OS installed
    Use optional argument PATH to specify mountpoint of alternate volume
--configured [PATH]
    Returns status 0 if the server is configured
    Use optional argument PATH to specify mountpoint of alternate volume


So if the serveradmin --configured command "fails", you could check if is installed:

 if serverinfo -q --configured;
 then
    echo configured; 
 else
    if serverinfo -q --software;
    then
       echo installed;
    else
       echo not_installed;
    fi
 fi