We're implementing a new centralized monitoring solution (Zenoss). Incorporating servers, networking, and Java programs is straightforward with SNMP and JMX.The question, however, is what are the best practices for monitoring and managing custom C++ applications in large, heterogenous (Solaris x86, RHEL Linux, Windows) environments?Possibilities I see are:Net SNMPAdvantagessingle, central daemon on each serverwell-known standardeasy integration into monitoring solutionswe run Net SNMP daemons on our servers alreadyDisadvantages:complex implementation (MIBs, Net SNMP library)new technology to introduce for the C++ developersrsyslogAdvantagessingle, central daemon on each serverwell-known standardunknown integration into monitoring solutions (I know they can do alerts based on text, but how well would it work for sending telemetry like memory usage, queue depths, thread capacity, etc)simple implementationDisadvantages:possible integration issuessomewhat new technology for C++ developerspossible porting issues if we switch monitoring vendorsprobably involves coming up with an ad-hoc communication protocol (or using RFC5424 structured data; I don't know if Zenoss supports that without custom Zenpack coding)Embedded JMX (embed a JVM and use JNI)Advantagesconsistent management interface for both Java and C++well-known standardeasy integration into monitoring solutionssomewhat simple implementation (we already do this today for other purposes)Disadvantages:complexity (JNI, thunking layer between native C++ and Java, basically writing the management code twice)possible stability problemsrequires a JVM in each process, using considerably more memoryJMX is new technology for C++ developerseach process has it's own JMX port (we run a lot of processes on each machine)Local JMX daemon, processes connect to itAdvantagessingle, central daemon on each serverconsistent management interface for both Java and C++well-known standardeasy integration into monitoring solutionsDisadvantages:complexity (basically writing the management code twice)need to find or write such a daemonneed a protocol between the JMX daemon and the C++ processJMX is new technology for C++ developersCodeMesh JunC++ionAdvantagesconsistent management interface for both Java and C++well-known standardeasy integration into monitoring solutionssingle, central daemon on each server when run in shared JVM modesomewhat simple implementation (requires code generation)Disadvantages:complexity (code generation, requires a GUI and several rounds of tweaking to produce the proxied code)possible JNI stability problemsrequires a JVM in each process, using considerably more memory (in embedded mode)Does not support Solaris x86 (deal breaker)Even if it did support Solaris x86, there are possible compiler compatibility issues (we use an odd combination of STLPort and Forte on Solariseach process has it's own JMX port when run in embedded mode (we run a lot of processes on each machine)possibly precludes a shared JMX server for non-C++ processes (?)Is there some reasonably standardized, simple solution I'm missing? Given no other reasonable solutions, which of these solutions is typically used for custom C++ programs? My gut feel is that Net SNMP is how people do this, but I'd like other's input and experience before I make a decision.
Monitoring C++ applications
java;c++