Sybase server build error, missing library files
It should be enough to install missing libraries on you system. 

libXm is part of openmotif package:

libXm.so.4 (32bit)

libXm.so.4 (64bit)

libXp is part of X.Org X11 runtime library:

libXp.so.6 (32bit)

libXp.so.6 (64bit)

EDIT: I created a support case on SAP site and they responded me with a list of necessary packages. First, there is a site System requirements for Linux (this one is for ASE 16.0 SP03 PL05, but you can change it in the upper right corner) where you can find that you need to setup $LANG environment variable first. Then there is referral to SAP KBA Note 2489781, which is accessible only with valid credentials, so I picked important information below (again valid for ASE 16.0 SP03 PL05):

System: Red Hat

Hardware: x86_64 processor (Intel Xeon with EM64T)  

Libraries:


kernel-2.6.32-71.el6.x86_64 
glibc-2.12-1.7.el6.x86_64
compat-glibc-2.5-46.2.x86_64 
glibc.i686 (Redhat 7 requires 32-bit libraries to execute the isql and dscp utilities)


Additional Linux Libraries:


openmotif-2.3.1-2.el5 
libXp-1.0.0-8.1.el5 
libXt-1.0.2-3.1.fc6
libXtst-1.0.1-3.1 
libXi-1.6.1-1 
libXmu-1.0.2-5 
libXext-1.0.1-2.1
libSM-1.0.1-3.1 
libICE-1.0.1-2.1 
libX11-1.0.3-9.el5


With all of above installed, you should be able to build and run this version of ASE.