Chapter 3. Configuring Java applications
Chapter 3. Configuring Java applications To enable Cryostat to gather, store, and analyze Java Flight Recorder (JFR) data about target applications that run on Java Virtual Machine (JVM)s, you must configure the applications so that Cryostat can detect and connect to them. You can configure the applications in any of the following ways: By using the Cryostat agent component for detection and connectivity, which is implemented as a Java Instrumentation Agent and acts as a plug-in for applications that run on the JVM By configuring the applications to allow Java Management Extensions (JMX) connections and using an OpenShift Service for detection and JMX for connectivity By using the Cryostat agent for detection and JMX for connectivity Cryostat agent From Red Hat build of Cryostat 2.4 onward, the Cryostat agent provides an HTTP API that the Cryostat server can use as an alternative to an application's JMX port. By attaching a properly configured Cryostat agent to the workload applications that you deploy, you can use the full Cryostat feature set without any need for the target applications to expose a JMX port. Note Before Red Hat build of Cryostat 2.4, the Cryostat agent provided a read-only HTTP API that supported a limited set of JFR operations only. The Cryostat agent's HTTP API can offer the following benefits compared to a JMX port: Greater security due to the reduced API surface area Deployment flexibility due to the Cryostat agent's dual role as a Cryostat discovery plug-in If the Cryostat agent detects that JMX is also configured on your application, the agent publishes itself to the Cryostat server with both agent HTTP API definitions and JMX URL definitions. In this situation, you can use whichever configuration option you prefer. To use the Cryostat agent with your workload application, you must configure the application to pass the -javaagent JVM flag with the path to the Cryostat agent's JAR file (for example, -javaagent:/deployment/app/lib/cryostat-agent.jar ). This setting enables your workload application's JVM to load and initialize the Cryostat agent at startup. Once the Cryostat agent's basic initialization is complete, your workload application's normal startup process begins as usual. Options for including the agent's JAR file into your workload application You can include the Cryostat agent's JAR file into the workload application in different ways: The simplest option is to add the JAR file to your application's dependencies in the pom.xml or build.gradle file. Your build tool (Maven or Gradle) downloads the JAR file for inclusion in your application build output. You can use a Maven plug-in, such as the maven-dependency-plugin , to provide more fine-grained control over the download and inclusion of the JAR file in the application build output. You can create a PersistentVolume storage volume that contains the JAR file. Then reconfigure your application's Deployment/DeploymentConfig to mount the PersistentVolume and use -javaagent:/path/to/persistentvolume/cryostat-agent.jar . The exact way to accomplish this task depends on the type of PersistentVolume provider you enabled in your OpenShift cluster. Once the Cryostat agent is successfully added to your application container and loaded, your application's stdout and console logs start displaying log messages from the Cryostat agent. Agent configuration properties You can specify configuration properties for the Cryostat agent in either of two ways: Use JVM system property flags on the application (for example, -Dcryostat.agent.api.writes-enabled=true ). Use environment variables by making all letters upper case and replacing any punctuation with underscores (for example, CRYOSTAT_AGENT_API_WRITES_ENABLED=true ). You must configure the following properties to enable the Cryostat agent to operate successfully: cryostat.agent.baseuri This specifies the URL location of the Cryostat server back end that the Cryostat agent advertises itself to (that is, the internal OpenShift Service object) such as https://my-cryostat.my-namespace.svc.cluster.local . cryostat.agent.callback This specifies the URL location of the Cryostat agent instance or application itself. Cryostat uses this URL to perform health checks and request data from the agent. You can use the OpenShift/Kubernetes Downward API to determine this dynamically. For more information, see the Kubernetes Downward API documentation on status.podIP . Depending on your setup requirements, you can also configure the following agent properties: cryostat.agent.api.writes-enabled This indicates whether the Cryostat agent allows write operations. This is set to false by default. If you want the Cryostat agent to accept requests to start, stop, or delete JFR flight recordings, you must set this property to true . Note Even if this property is set to false , the agent can still fulfill requests to list flight recordings or download individual recording files. cryostat.agent.webserver.port This specifies the HTTP port number that the agent uses to bind its HTTP API (by default, 9977). If this conflicts with an existing port that your application or another tooling agent uses, you must specify a different port number. cryostat.agent.app.name This specifies a label for identifying which application this Cryostat agent instance is attached to (by default, cryostat-agent ). You could use the Downward API metadata.name or metadata.labels['app'] fields for this. For more information, see the Kubernetes Downward API documentation Kubernetes Downward API documentation. Remote Java Management Extensions (JMX) connections JMX is a standard feature on a JVM with which you can monitor and manage target applications that run on the JVM. For Cryostat to use JMX, you must enable and configure JMX when you start the JVM because Cryostat requires the target applications to expose a JMX port. Cryostat communicates with the target applications over this JMX port to start and stop JFR recordings and to pull JFR data over the network, enabling Cryostat to store and analyze this JFR data. Remote monitoring requires security to ensure that unauthorized persons cannot access application. Cryostat prompts you to enter your credentials before Cryostat can access any of the application's JFR recordings. Cryostat agent and JMX hybrid You can configure your target applications to use a hybrid approach where you use both the Cryostat agent and JMX. With this approach, you use the Cryostat agent to detect the target applications and JMX to expose the JFR data to Cryostat, which allows for more flexibility. For example, you can use the agent to detect the applications without needing to depend on specific port numbers and also use the JMX connections to start and stop JFR flight recordings on demand. 3.1. Configuring applications by using the Cryostat agent You can use the Cryostat agent, implemented as a Java Instrumentation Agent, to configure target applications, so that Cryostat can detect the applications, collect data, and send the data to Cryostat for analysis. You can also optionally enable the Cryostat agent to accept requests from the Cryostat server to start, stop, and delete JFR recordings. Red Hat build of Cryostat 2.4 distributes two different variations of the Cryostat agent's JAR file. Depending on your setup requirements, you can use either of the following types of agent JAR file: An all-in-one "shaded" JAR file that is self-contained and includes the agent code and all of its dependencies This "shaded" JAR file provides the most convenient form of Cryostat agent to include in your existing applications, because you need to include only one additional agent JAR file. This is a common distribution pattern for similar agents and tools. A standard JAR file that contains the agent code without any dependencies This type of JAR file is useful if you know that dependency conflicts exist between the agent and your workload applications. If you intend to apply your own strategies to provide the correct versions of each dependency to satisfy both the agent and your applications' requirements, you can use the standalone JAR file. Note releases provided one distribution of the Cryostat agent, which was an all-in-one "shaded" `JAR file. The following procedure describes how to install the "shaded" JAR file distribution of the Cryostat 2.4 agent. As described in Configuring Java applications: Cryostat agent , the Cryostat 2.4 agent supports different options for including the agent's JAR file into your workload applications. The following procedure describes how to add the "shaded" JAR file to your application's dependencies in the pom.xml or build.gradle file. Prerequisites Logged in to your Cryostat web console. Installed JDK version 11 or later. Procedure Install the Cryostat agent. Choose one of the following options, depending on your application build: Using Maven: Update the application pom.xml file with the Cryostat agent JAR file information. Example pom.xml <project> ... <repositories> <repository> <id>redhat-maven-repository</id> <url>https://maven.repository.redhat.com/earlyaccess/all/</url> </repository> </repositories> ... <build> <plugins> <plugin> <artifactId>maven-dependency-plugin</artifactId> <version>3.3.0</version> <executions> <execution> <phase>prepare-package</phase> <goals> <goal>copy</goal> </goals> <configuration> <artifactItems> <artifactItem> <groupId>io.cryostat</groupId> <artifactId>cryostat-agent</artifactId> <version>0.3.0.redhat-00001</version> <classifier>shaded</classifier> </artifactItem> </artifactItems> <stripVersion>true</stripVersion> </configuration> </execution> </executions> </plugin> </plugins> ... </build> ... </project> The time you build your application, the Cryostat agent JAR file is available at target/dependency/cryostat-agent-shaded.jar . Using Gradle: Update the build.gradle file. Example build.gradle file repositories { ... maven { url "https://maven.repository.redhat.com/earlyaccess/all/" credentials { username "myusername" password "mytoken" } } } How you package the agent JAR file into the application depends on the Gradle plug-ins that you use for the build. For example, if you are using the Jib plug-in, update the build.gradle file as follows: Example build.gradle file plugins { id 'java' id 'application' id 'com.google.cloud.tools.jib' version '3.3.1' id 'com.ryandens.javaagent-jib' version '0.5.0' } ... dependencies { ... javaagent 'io.cryostat:cryostat-agent:0.3.0.redhat-00001:shaded' Update the Docker file. The following example uses the JAVA_OPTS environment variable to pass the relevant JVM information. Example ... COPY target/dependency/cryostat-agent.jar /deployments/app/ ... ENV JAVA_OPTS="-javaagent:/deployments/app/cryostat-agent-shaded.jar" Rebuild the container image that is specific to your application. To supply the JVM system properties or environment variables that you need to configure the Cryostat agent, push the updated image and then modify your application deployment. Example apiVersion: apps/v1 kind: Deployment ... spec: ... template: ... spec: containers: - name: sample-app image: docker.io/myorg/myapp:latest env: - name: CRYOSTAT_AGENT_APP_NAME value: "myapp" # Replace this with the Kubernetes DNS record # for the Cryostat Service - name: CRYOSTAT_AGENT_BASEURI value: "http://cryostat.mynamespace.mycluster.svc:8181" - name: POD_IP valueFrom: fieldRef: fieldPath: status.podIP - name: CRYOSTAT_AGENT_CALLBACK value: "http://USD(POD_IP):9977" 1 # Replace "abcd1234" with a base64-encoded authentication token - name: CRYOSTAT_AGENT_AUTHORIZATION 2 value: "Bearer abcd1234" - name: CRYOSTAT_AGENT_API_WRITES_ENABLED 3 value: true ports: - containerPort: 9977 protocol: TCP resources: {} restartPolicy: Always status: {} <1>: Port number 9977 is the default HTTP port that the agent exposes for the internal web server that services Cryostat requests. You can change this port number if it conflicts with your target application into which the agent is installed. <2>: The CRYOSTAT_AGENT_AUTHORIZATION value shows the credentials that the agent includes in the API requests to Cryostat to advertise its own presence or to push JFR data. You can also create a Kubernetes Service Account for this purpose and replace abcd1234 with the base64-encoded authentication token associated with the service account. <3>: The CRYOSTAT_AGENT_API_WRITES_ENABLED variable is set to false by default. If you want the Cryostat agent to accept requests from the Cryostat server to start, stop, or delete JFR flight recordings, you must set this variable to true . 3.2. Configuring applications by using JMX connections For Cryostat to detect and communicate with your target Java applications, you can configure the applications to allow remote Java Management Extensions (JMX) connections. Prerequisites Logged in to your Cryostat web console. Created a Cryostat instance in your project. Procedure To enable remote JMX connections, complete the following steps: On your application, define the following Java system property: Note To add the -Dcom.sun.management.jmxremote.port=<port_num> property without having to rebuild the target application, you can set the JAVA_OPTS_APPEND environment variable on the application. JAVA_OPTS_APPEND is an environment variable that is used by Red Hat Universal Base Images (UBI) only. If you use Red Hat UBI to build application images, set the JAVA_OPTS_APPEND variable at build time in the application Docker file, or at runtime by running the following command: If you do not use Red Hat UBI to build application images, refer to the documentation for your base image for information about how to add the Java system properties at build time or runtime. Specify that the application listens for remote JMX connections by allowing traffic to the application. Use an Red Hat OpenShift Service and specify the following values for its remote JMX port: Example service.yaml apiVersion: v1 kind: Service ... spec: ports: - name: "jfr-jmx" port: 9091 targetPort: 9091 ... Secure the remote JMX connections: Enable and configure authentication and SSL/TLS for the remote JMX connections in your application: Configure Cryostat to trust the application TLS certificate. Create a secret for the application in the same namespace as your Cryostat application and configure Cryostat to refer to the secret. To create a secret for the certificate, run the following command: Note The certificate must be in a .pem file format. When you create your Cryostat instance, add the secret to the list of trusted TLS certificates. For more information, see Configuring TLS certificates . To allow your applications to verify that Cryostat is connecting to them by a means other than password authentication, enable TLS client authentication: Note TLS client authentication requires the cert-manager operator for Red Hat OpenShift. If you use TLS client authentication for remote JMX connections, the application truststore must contain a Cryostat certificate. The Cryostat operator cert-manager integration creates a self-signed certificate for the Cryostat deployment. This certificate is located in the <cryostat>-tls secret, where <cryostat> is the name of the Cryostat instance you created. Note The cert-manager Operator also places a Java keystore truststore in the secret. To mount this truststore in your application deployment, run the following command, replacing "<myapp>" with the name of your application deployment and "<cryostat>" with the name of your Cryostat instance: The Cryostat operator generates the truststore password, which you can find in the <cryostat>-keystore secret. To mount this as an environment variable in your application deployment, run the following command: Configure the Java arguments for the container. Run the following commands: Warning If you deployed Cryostat and your applications in a testing environment, you might want to configure the target applications without any JMX or TLS authentication. You can do so by using the following set of Java system properties, however, this configuration is not secure and not recommended. Additional resources Configuring TLS certificates Storing and managing JMX credentials 3.3. Configuring applications by using the Cryostat agent and JMX connections You can configure target applications that run on Java Virtual Machines (JVM) to use a combination of the Cryostat agent and Java Management Extensions (JMX) connections to detect and communicate with the target applications. You use the Cryostat agent to detect and communicate with the target application and use JMX to expose the Java Flight Recorder (JFR) data. You must configure the Cryostat agent to communicate with Cryostat about itself and that the agent is reachable through JMX rather than through HTTP. Prerequisites Logged in to your Cryostat web console. Created a Cryostat instance in your project. Procedure Install the Cryostat agent. For application builds that use Maven, update the application pom.xml file with the Cryostat agent JAR file information. Example pom.xml file <project> ... <repositories> <repository> <id>redhat-maven-repository</id> <url>https://maven.repository.redhat.com/earlyaccess/all/</url> </repository> </repositories> ... <build> <plugins> <plugin> <artifactId>maven-dependency-plugin</artifactId> <version>3.3.0</version> <executions> <execution> <phase>prepare-package</phase> <goals> <goal>copy</goal> </goals> <configuration> <artifactItems> <artifactItem> <groupId>io.cryostat</groupId> <artifactId>cryostat-agent</artifactId> <version>0.3.0.redhat-00001</version> <classifier>shaded</classifier> </artifactItem> </artifactItems> <stripVersion>true</stripVersion> </configuration> </execution> </executions> </plugin> </plugins> ... </build> ... </project> Modify your application deployment: Example apiVersion: apps/v1 kind: Deployment ... spec: ... template: ... spec: containers: - name: sample-app image: docker.io/myorg/myapp:latest env: - name: CRYOSTAT_AGENT_APP_NAME value: "myapp" # Replace this with the Kubernetes DNS record # for the Cryostat Service - name: CRYOSTAT_AGENT_BASEURI value: "http://cryostat.mynamespace.mycluster.svc:8181" - name: POD_IP valueFrom: fieldRef: fieldPath: status.podIP - name: CRYOSTAT_AGENT_CALLBACK value: "http://USD(POD_IP):9977" - name: CRYOSTAT_AGENT_AUTHORIZATION # Replace "abcd1234" with a base64-encoded authentication token value: "Bearer abcd1234" # This environment variable is key to the Cryostat agent # and JMX "hybrid" setup. # Set the Cryostat agent to register itself with Cryostat # as reachable through JMX, rather than reachable through HTTP. - name: CRYOSTAT_AGENT_REGISTRATION_PREFER_JMX value: "true" # Configure the application to load the agent JAR file and # to enable JMX, so that the Cryostat agent can register # itself as reachable through JMX. # To configure authentication and SSL/TLS for the JMX # connections, see <1>. - name: JAVA_OPTS value: >- -javaagent:/deployments/app/cryostat-agent-shaded.jar -Dcom.sun.management.jmxremote.port=9091 1 ports: - containerPort: 9977 protocol: TCP resources: {} restartPolicy: Always status: {} <1>: To configure authentication and SSL/TLS for the JMX connections and view further configuration options, see Configuring applications by using JMX connections . To enable Cryostat to detect the target application and connect to the Cryostat agent, configure an application Service : Example apiVersion: v1 kind: Service ... spec: ports: - name: "jfr-jmx" port: 9091 targetPort: 9091 - name: "cryostat-agent" port: 9977 targetPort: 9977 ... Additional resources Configuring applications by using the Cryostat agent Configuring applications by using JMX connections
<project> <repositories> <repository> <id>redhat-maven-repository</id> <url>https://maven.repository.redhat.com/earlyaccess/all/</url> </repository> </repositories> <build> <plugins> <plugin> <artifactId>maven-dependency-plugin</artifactId> <version>3.3.0</version> <executions> <execution> <phase>prepare-package</phase> <goals> <goal>copy</goal> </goals> <configuration> <artifactItems> <artifactItem> <groupId>io.cryostat</groupId> <artifactId>cryostat-agent</artifactId> <version>0.3.0.redhat-00001</version> <classifier>shaded</classifier> </artifactItem> </artifactItems> <stripVersion>true</stripVersion> </configuration> </execution> </executions> </plugin> </plugins> </build> </project> repositories { ... maven { url "https://maven.repository.redhat.com/earlyaccess/all/" credentials { username "myusername" password "mytoken" } } } plugins { id 'java' id 'application' id 'com.google.cloud.tools.jib' version '3.3.1' id 'com.ryandens.javaagent-jib' version '0.5.0' } ... dependencies { ... javaagent 'io.cryostat:cryostat-agent:0.3.0.redhat-00001:shaded' COPY target/dependency/cryostat-agent.jar /deployments/app/ ENV JAVA_OPTS="-javaagent:/deployments/app/cryostat-agent-shaded.jar" docker build -t docker.io/myorg/myapp:latest -f src/main/docker/Dockerfile apiVersion: apps/v1 kind: Deployment spec: template: spec: containers: - name: sample-app image: docker.io/myorg/myapp:latest env: - name: CRYOSTAT_AGENT_APP_NAME value: "myapp" # Replace this with the Kubernetes DNS record # for the Cryostat Service - name: CRYOSTAT_AGENT_BASEURI value: "http://cryostat.mynamespace.mycluster.svc:8181" - name: POD_IP valueFrom: fieldRef: fieldPath: status.podIP - name: CRYOSTAT_AGENT_CALLBACK value: "http://USD(POD_IP):9977" 1 # Replace "abcd1234" with a base64-encoded authentication token - name: CRYOSTAT_AGENT_AUTHORIZATION 2 value: "Bearer abcd1234" - name: CRYOSTAT_AGENT_API_WRITES_ENABLED 3 value: true ports: - containerPort: 9977 protocol: TCP resources: {} restartPolicy: Always status: {} -Dcom.sun.management.jmxremote.port=<port_num> set env deployment <name> JAVA_OPTS_APPEND="..." apiVersion: v1 kind: Service spec: ports: - name: "jfr-jmx" port: 9091 targetPort: 9091 -Dcom.sun.management.jmxremote.port=<port_num> # enable JMX authentication -Dcom.sun.management.jmxremote.authenticate=true define users for JMX auth -Dcom.sun.management.jmxremote.password.file=</path/to/jmxremote.password> set permissions for JMX users -Dcom.sun.management.jmxremote.access.file=</path/to/jmxremote.access> # enable JMX SSL -Dcom.sun.management.jmxremote.ssl=true enable JMX registry SSL -Dcom.sun.management.jmxremote.registry.ssl=true set your SSL keystore -Djavax.net.ssl.keyStore=</path/to/keystore> set your SSL keystore password -Djavax.net.ssl.keyStorePassword=<password> create secret generic myapp-cert --from-file=tls.crt=/path/to/cert.pem -Dcom.sun.management.jmxremote.ssl.need.client.auth=true -Djavax.net.ssl.trustStore=</path/to/truststore> -Djavax.net.ssl.trustStorePassword=<password> set volumes deploy <myapp> --add --name=truststore --secret-name=<cryostat>-tls --sub-path=truststore.p12 --mount-path=/var/run/secrets/<myapp>/truststore.p12 set env deploy <myapp> --from='secret/<cryostat>-keystore' -Dcom.sun.management.jmxremote.ssl.need.client.auth=true -Djavax.net.ssl.trustStore=/var/run/secrets/<myapp>/truststore.p12 -Djavax.net.ssl.trustStorePassword="USD(KEYSTORE_PASS)" -Dcom.sun.management.jmxremote.port=<port_num> -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false <project> <repositories> <repository> <id>redhat-maven-repository</id> <url>https://maven.repository.redhat.com/earlyaccess/all/</url> </repository> </repositories> <build> <plugins> <plugin> <artifactId>maven-dependency-plugin</artifactId> <version>3.3.0</version> <executions> <execution> <phase>prepare-package</phase> <goals> <goal>copy</goal> </goals> <configuration> <artifactItems> <artifactItem> <groupId>io.cryostat</groupId> <artifactId>cryostat-agent</artifactId> <version>0.3.0.redhat-00001</version> <classifier>shaded</classifier> </artifactItem> </artifactItems> <stripVersion>true</stripVersion> </configuration> </execution> </executions> </plugin> </plugins> </build> </project> apiVersion: apps/v1 kind: Deployment spec: template: spec: containers: - name: sample-app image: docker.io/myorg/myapp:latest env: - name: CRYOSTAT_AGENT_APP_NAME value: "myapp" # Replace this with the Kubernetes DNS record # for the Cryostat Service - name: CRYOSTAT_AGENT_BASEURI value: "http://cryostat.mynamespace.mycluster.svc:8181" - name: POD_IP valueFrom: fieldRef: fieldPath: status.podIP - name: CRYOSTAT_AGENT_CALLBACK value: "http://USD(POD_IP):9977" - name: CRYOSTAT_AGENT_AUTHORIZATION # Replace "abcd1234" with a base64-encoded authentication token value: "Bearer abcd1234" # This environment variable is key to the Cryostat agent # and JMX "hybrid" setup. # Set the Cryostat agent to register itself with Cryostat # as reachable through JMX, rather than reachable through HTTP. - name: CRYOSTAT_AGENT_REGISTRATION_PREFER_JMX value: "true" # Configure the application to load the agent JAR file and # to enable JMX, so that the Cryostat agent can register # itself as reachable through JMX. # To configure authentication and SSL/TLS for the JMX # connections, see <1>. - name: JAVA_OPTS value: >- -javaagent:/deployments/app/cryostat-agent-shaded.jar -Dcom.sun.management.jmxremote.port=9091 1 ports: - containerPort: 9977 protocol: TCP resources: {} restartPolicy: Always status: {} apiVersion: v1 kind: Service spec: ports: - name: "jfr-jmx" port: 9091 targetPort: 9091 - name: "cryostat-agent" port: 9977 targetPort: 9977