Chapter 11. Configuration of SNMP traps
Chapter 11. Configuration of SNMP traps As a storage administrator, you can deploy and configure the simple network management protocol (SNMP) gateway in a Red Hat Ceph Storage cluster to receive alerts from the Prometheus Alertmanager and route them as SNMP traps to the cluster. 11.1. Simple network management protocol Simple network management protocol (SNMP) is one of the most widely used open protocols, to monitor distributed systems and devices across a variety of hardware and software platforms. Ceph's SNMP integration focuses on forwarding alerts from its Prometheus Alertmanager cluster to a gateway daemon. The gateway daemon transforms the alert into an SNMP Notification and sends it on to a designated SNMP management platform. The gateway daemon is from the snmp_notifier_project , which provides SNMP V2c and V3 support with authentication and encryption. The Red Hat Ceph Storage SNMP gateway service deploys one instance of the gateway by default. You can increase this by providing placement information. However, if you enable multiple SNMP gateway daemons, your SNMP management platform receives multiple notifications for the same event. The SNMP traps are alert messages and the Prometheus Alertmanager sends these alerts to the SNMP notifier which then looks for object identifier (OID) in the given alerts' labels. Each SNMP trap has a unique ID which allows it to send additional traps with updated status to a given SNMP poller. SNMP hooks into the Ceph health checks so that every health warning generates a specific SNMP trap. In order to work correctly and transfer information on device status to the user to monitor, SNMP relies on several components. There are four main components that makeup SNMP: SNMP Manager - The SNMP manager, also called a management station, is a computer that runs network monitoring platforms. A platform that has the job of polling SNMP-enabled devices and retrieving data from them. An SNMP Manager queries agents, receives responses from agents and acknowledges asynchronous events from agents. SNMP Agent - An SNMP agent is a program that runs on a system to be managed and contains the MIB database for the system. These collect data like bandwidth and disk space, aggregates it, and sends it to the management information base (MIB). Management information base (MIB) - These are components contained within the SNMP agents. The SNMP manager uses this as a database and asks the agent for access to particular information. This information is needed for the network management systems (NMS). The NMS polls the agent to take information from these files and then proceeds to translate it into graphs and displays that can be viewed by the user. MIBs contain statistical and control values that are determined by the network device. SNMP Devices The following versions of SNMP are compatible and supported for gateway implementation: V2c - Uses a community string without any authentication and is vulnerable to outside attacks. V3 authNoPriv - Uses the username and password authentication without encryption. V3 authPriv - Uses the username and password authentication with encryption to the SNMP management platform. Important When using SNMP traps, ensure that you have the correct security configuration for your version number to minimize the vulnerabilities that are inherent to SNMP and keep your network protected from unauthorized users. 11.2. Configuring snmptrapd It is important to configure the simple network management protocol (SNMP) target before deploying the snmp-gateway because the snmptrapd daemon contains the auth settings that you need to specify when creating the snmp-gateway service. The SNMP gateway feature provides a means of exposing the alerts that are generated in the Prometheus stack to an SNMP management platform. You can configure the SNMP traps to the destination based on the snmptrapd tool. This tool allows you to establish one or more SNMP trap listeners. The following parameters are important for configuration: The engine-id is a unique identifier for the device, in hex, and required for SNMPV3 gateway. Red Hat recommends using `8000C53F_CLUSTER_FSID_WITHOUT_DASHES_`for this parameter. The snmp-community , which is the SNMP_COMMUNITY_FOR_SNMPV2 parameter, is public for SNMPV2c gateway. The auth-protocol which is the AUTH_PROTOCOL , is mandatory for SNMPV3 gateway and is SHA by default. The privacy-protocol , which is the PRIVACY_PROTOCOL , is mandatory for SNMPV3 gateway. The PRIVACY_PASSWORD is mandatory for SNMPV3 gateway with encryption. The SNMP_V3_AUTH_USER_NAME is the user name and is mandatory for SNMPV3 gateway. The SNMP_V3_AUTH_PASSWORD is the password and is mandatory for SNMPV3 gateway. Prerequisites A running Red Hat Ceph Storage cluster. Root-level access to the nodes. Install firewalld on Red Hat Enterprise Linux system. Procedure On the SNMP management host, install the SNMP packages: Example Open the port 162 for SNMP to receive alerts: Example Implement the management information base (MIB) to make sense of the SNMP notification and enhance SNMP support on the destination host. Copy the raw file from the main repository: https://github.com/ceph/ceph/blob/master/monitoring/snmp/CEPH-MIB.txt Example Create the snmptrapd directory. Example Create the configuration files in snmptrapd directory for each protocol based on the SNMP version: Syntax For SNMPV2c, create the snmptrapd_public.conf file as follows: Example The public setting here must match the snmp_community setting used when deploying the snmp-gateway service. For SNMPV3 with authentication only, create the snmptrapd_auth.conf file as follows: Example The 0x8000C53Ff64f341c655d11eb8778fa163e914bcc string is the engine_id , and myuser and mypassword are the credentials. The password security is defined by the SHA algorithm. This corresponds to the settings for deploying the snmp-gateway daemon. Example For SNMPV3 with authentication and encryption, create the snmptrapd_authpriv.conf file as follows: Example The 0x8000C53Ff64f341c655d11eb8778fa163e914bcc string is the engine_id , and myuser and mypassword are the credentials. The password security is defined by the SHA algorithm and DES is the type of privacy encryption. This corresponds to the settings for deploying the snmp-gateway daemon. Example Run the daemon on the SNMP management host: Syntax Example If any alert is triggered on the storage cluster, you can monitor the output on the SNMP management host. Verify the SNMP traps and also the traps decoded by MIB. Example In the above example, an alert is generated after the Prometheus module is disabled. Additional Resources See the Deploying the SNMP gateway section in the Red Hat Ceph Storage Operations Guide . 11.3. Deploying the SNMP gateway You can deploy the simple network management protocol (SNMP) gateway using either SNMPV2c or SNMPV3. There are two methods to deploy the SNMP gateway: By creating a credentials file. By creating one service configuration yaml file with all the details. You can use the following parameters to deploy the SNMP gateway based on the versions: The service_type is the snmp-gateway . The service_name is any user-defined string. The count is the number of SNMP gateways to be deployed in a storage cluster. The snmp_destination parameter must be of the format hostname:port. The engine-id is a unique identifier for the device, in hex, and required for SNMPV3 gateway. Red Hat recommends to use `8000C53F_CLUSTER_FSID_WITHOUT_DASHES_`for this parameter. The snmp_community parameter is public for SNMPV2c gateway. The auth-protocol is mandatory for SNMPV3 gateway and is SHA by default. The privacy-protocol is mandatory for SNMPV3 gateway with authentication and encryption. The port is 9464 by default. You must provide a -i FILENAME to pass the secrets and passwords to the orchestrator. Once the SNMP gateway service is deployed or updated, the Prometheus Alertmanager configuration is automatically updated to forward any alert that has an objectidentifier to the SNMP gateway daemon for further processing. Prerequisites A running Red Hat Ceph Storage cluster. Root-level access to the nodes. Configuring snmptrapd on the destination host, which is the SNMP management host. Procedure Log into the Cephadm shell: Example Create a label for the host on which SNMP gateway needs to be deployed: Syntax Example Create a credentials file or a service configuration file based on the SNMP version: For SNMPV2c, create the file as follows: Example OR Example For SNMPV3 with authentication only, create the file as follows: Example OR Example For SNMPV3 with authentication and encryption, create the file as follows: Example OR Example Run the ceph orch command: Syntax OR Syntax For SNMPV2c, with the snmp_creds file, run the ceph orch command with the snmp-version as V2c : Example For SNMPV3 with authentication only, with the snmp_creds file, run the ceph orch command with the snmp-version as V3 and engine-id : Example For SNMPV3 with authentication and encryption, with the snmp_creds file, run the ceph orch command with the snmp-version as V3 , privacy-protocol , and engine-id : Example OR For all the SNMP versions, with the snmp-gateway file, run the following command: Example Additional Resources See the Configuring `snmptrapd` section in the Red Hat Ceph Storage Operations Guide .
dnf install -y net-snmp-utils net-snmp firewall-cmd --zone=public --add-port=162/udp firewall-cmd --zone=public --add-port=162/udp --permanent curl -o CEPH_MIB.txt -L https://raw.githubusercontent.com/ceph/ceph/master/monitoring/snmp/CEPH-MIB.txt scp CEPH_MIB.txt root@host02:/usr/share/snmp/mibs mkdir /root/snmptrapd/ format2 %V\n% Agent Address: %A \n Agent Hostname: %B \n Date: %H - %J - %K - %L - %M - %Y \n Enterprise OID: %N \n Trap Type: %W \n Trap Sub-Type: %q \n Community/Infosec Context: %P \n Uptime: %T \n Description: %W \n PDU Attribute/Value Pair Array:\n%v \n -------------- \n createuser -e 0x_ENGINE_ID_ SNMPV3_AUTH_USER_NAME AUTH_PROTOCOL SNMP_V3_AUTH_PASSWORD PRIVACY_PROTOCOL PRIVACY_PASSWORD authuser log,execute SNMP_V3_AUTH_USER_NAME authCommunity log,execute,net SNMP_COMMUNITY_FOR_SNMPV2 format2 %V\n% Agent Address: %A \n Agent Hostname: %B \n Date: %H - %J - %K - %L - %M - %Y \n Enterprise OID: %N \n Trap Type: %W \n Trap Sub-Type: %q \n Community/Infosec Context: %P \n Uptime: %T \n Description: %W \n PDU Attribute/Value Pair Array:\n%v \n -------------- \n authCommunity log,execute,net public format2 %V\n% Agent Address: %A \n Agent Hostname: %B \n Date: %H - %J - %K - %L - %M - %Y \n Enterprise OID: %N \n Trap Type: %W \n Trap Sub-Type: %q \n Community/Infosec Context: %P \n Uptime: %T \n Description: %W \n PDU Attribute/Value Pair Array:\n%v \n -------------- \n createuser -e 0x8000C53Ff64f341c655d11eb8778fa163e914bcc myuser SHA mypassword authuser log,execute myuser snmp_v3_auth_username: myuser snmp_v3_auth_password: mypassword format2 %V\n% Agent Address: %A \n Agent Hostname: %B \n Date: %H - %J - %K - %L - %M - %Y \n Enterprise OID: %N \n Trap Type: %W \n Trap Sub-Type: %q \n Community/Infosec Context: %P \n Uptime: %T \n Description: %W \n PDU Attribute/Value Pair Array:\n%v \n -------------- \n createuser -e 0x8000C53Ff64f341c655d11eb8778fa163e914bcc myuser SHA mypassword DES mysecret authuser log,execute myuser snmp_v3_auth_username: myuser snmp_v3_auth_password: mypassword snmp_v3_priv_password: mysecret /usr/sbin/snmptrapd -M /usr/share/snmp/mibs -m CEPH-MIB.txt -f -C -c /root/snmptrapd/ CONFIGURATION_FILE -Of -Lo :162 /usr/sbin/snmptrapd -M /usr/share/snmp/mibs -m CEPH-MIB.txt -f -C -c /root/snmptrapd/snmptrapd_auth.conf -Of -Lo :162 NET-SNMP version 5.8 Agent Address: 0.0.0.0 Agent Hostname: <UNKNOWN> Date: 15 - 5 - 12 - 8 - 10 - 4461391 Enterprise OID: . Trap Type: Cold Start Trap Sub-Type: 0 Community/Infosec Context: TRAP2, SNMP v3, user myuser, context Uptime: 0 Description: Cold Start PDU Attribute/Value Pair Array: .iso.org.dod.internet.mgmt.mib-2.1.3.0 = Timeticks: (292276100) 3 days, 19:52:41.00 .iso.org.dod.internet.snmpV2.snmpModules.1.1.4.1.0 = OID: .iso.org.dod.internet.private.enterprises.ceph.cephCluster.cephNotifications.prometheus.promMgr.promMgrPrometheusInactive .iso.org.dod.internet.private.enterprises.ceph.cephCluster.cephNotifications.prometheus.promMgr.promMgrPrometheusInactive.1 = STRING: "1.3.6.1.4.1.50495.1.2.1.6.2[alertname=CephMgrPrometheusModuleInactive]" .iso.org.dod.internet.private.enterprises.ceph.cephCluster.cephNotifications.prometheus.promMgr.promMgrPrometheusInactive.2 = STRING: "critical" .iso.org.dod.internet.private.enterprises.ceph.cephCluster.cephNotifications.prometheus.promMgr.promMgrPrometheusInactive.3 = STRING: "Status: critical - Alert: CephMgrPrometheusModuleInactive Summary: Ceph's mgr/prometheus module is not available Description: The mgr/prometheus module at 10.70.39.243:9283 is unreachable. This could mean that the module has been disabled or the mgr itself is down. Without the mgr/prometheus module metrics and alerts will no longer function. Open a shell to ceph and use 'ceph -s' to determine whether the mgr is active. If the mgr is not active, restart it, otherwise you can check the mgr/prometheus module is loaded with 'ceph mgr module ls' and if it's not listed as enabled, enable it with 'ceph mgr module enable prometheus'" cephadm shell ceph orch host label add HOSTNAME snmp-gateway ceph orch host label add host02 snmp-gateway cat snmp_creds.yml snmp_community: public cat snmp-gateway.yml service_type: snmp-gateway service_name: snmp-gateway placement: count: 1 spec: credentials: snmp_community: public port: 9464 snmp_destination: 192.168.122.73:162 snmp_version: V2c cat snmp_creds.yml snmp_v3_auth_username: myuser snmp_v3_auth_password: mypassword cat snmp-gateway.yml service_type: snmp-gateway service_name: snmp-gateway placement: count: 1 spec: credentials: snmp_v3_auth_password: mypassword snmp_v3_auth_username: myuser engine_id: 8000C53Ff64f341c655d11eb8778fa163e914bcc port: 9464 snmp_destination: 192.168.122.1:162 snmp_version: V3 cat snmp_creds.yml snmp_v3_auth_username: myuser snmp_v3_auth_password: mypassword snmp_v3_priv_password: mysecret cat snmp-gateway.yml service_type: snmp-gateway service_name: snmp-gateway placement: count: 1 spec: credentials: snmp_v3_auth_password: mypassword snmp_v3_auth_username: myuser snmp_v3_priv_password: mysecret engine_id: 8000C53Ff64f341c655d11eb8778fa163e914bcc port: 9464 snmp_destination: 192.168.122.1:162 snmp_version: V3 ceph orch apply snmp-gateway --snmp_version= V2c_OR_V3 --destination= SNMP_DESTINATION [--port= PORT_NUMBER ] [--engine-id=8000C53F_CLUSTER_FSID_WITHOUT_DASHES_] [--auth-protocol= MDS_OR_SHA ] [--privacy_protocol= DES_OR_AES ] -i FILENAME ceph orch apply -i FILENAME .yml ceph orch apply snmp-gateway --snmp-version=V2c --destination=192.168.122.73:162 --port=9464 -i snmp_creds.yml ceph orch apply snmp-gateway --snmp-version=V3 --engine-id=8000C53Ff64f341c655d11eb8778fa163e914bcc--destination=192.168.122.73:162 -i snmp_creds.yml ceph orch apply snmp-gateway --snmp-version=V3 --engine-id=8000C53Ff64f341c655d11eb8778fa163e914bcc--destination=192.168.122.73:162 --privacy-protocol=AES -i snmp_creds.yml ceph orch apply -i snmp-gateway.yml