As true lovers of the common alerting throughout the SAP Cloud Platform, we aim to bring the alerts from as much as possible monitoring tools into one and the same shape. To achieve it, we use the SAP Cloud Platform Alert Notification service. As an additional benefit on top of the common alert schema, the service provides a variety of delivery options that we could easily reuse among all the alert sources.
Today, we will integrate Prometheus Alertmanager with SAP Cloud Platform Alert Notification. For the demo purpose, we will use an already configured Alertmanager setup, we'll simulate a troublesome situation that will cause an alert trigger. The alert will be received in shape of an email message coming from Alert Notification.
To complete the scenario below, a couple of prerequisites are needed:
Once all of these are in place, let's start the setup.
In this section, we'll obtain credentials from Alert Notification that are used from Alertmanager. Then, we'll create a subscription to catch all alerts coming from Alertmanager and will forward them to our mailbox.
Note: You could reuse an existing or create another more convenient delivery channel. In the demo, we will use the EMAIL action type due to its simplicity.
Let's start with the credentials. We need a credential pair for BASIC authentication with ALERT_WRITE permission. Follow the steps according to your SAP Cloud Platform environment:
Now, let's create the subscription. Navigate to your Alert Notification instance in SAP Cloud Platform Cockpit, select the Subscriptions menu and choose Create. At the first step of Create Subscription wizard set a name for your and choose Create. At the Select Conditions step, choose Create Condition. Enter a name for the new condition and Condition: eventType Is Equal To PrometheusAlertmanager:
Choose Create and then Assign. At Select Actions step, you have two options:
Now, you're back to the subscription wizard, make sure your action is selected and select Assign.
As stated above, to achieve the scenario you should already have well-established monitoring using Prometheus as well as an alerting on top of it using Prometheus Alertmanager. Now, we will only extend it with the needed setup to forward our alerts to Alert Notification.
For that purpose, we should define a new receiver with webhook_config in our Alertmanager configuration:
Where the placeholders shall be replaced as follows:
As a last step in our Prometheus Alertmanager configuration, we have to add the newly created receiver to an existing route, so that we could easily test our setup in the next section. In our environment, this looks like this:
Finally, we're ready to test how the Prometheus Alertmanager alert will look like in shape of SAP Cloud Platform Alert Notification event. To do this, we need to trigger an alert that will match the route where the alert-notification receiver is added. How to trigger it is up to your particular environment.
However, in the next paragraph I'll explain what is our certain setup and how I'll test the integration. If it's not of interest to you, you could safely skip it and jump directly to the final words and the screenshot to see how the Prometheus Alertmanager alert is represented.
In our case, we have established a monitoring for Kubernetes environment where some of the metrics we observe are kube-state-metrics. We've provided the alert-notification receiver to a route that will match alerts in regards to such metrics. The rule we will break to trigger an alert is based on a couple of deployment metrics:
In order to trigger an alert, we'll "kill" one of our application replicas, so that the expected number won't match. According to our setup, within a couple of minutes an alert will be triggered and send to Alert Notification. The exact timing could differ in your environment but soon you'll receive an email similar to this one:
Would you like to further explore this scenario? See Integrating with Prometheus Alertmanager at SAP Help Portal.
This blog post is part of a series of blog posts related to integration between third-party monitoring tools and SAP Cloud Platform Alert Notification service. We will continuously add newly released integrations, so stay tuned!
We ask you – our customers and partners – to join us in our integration journey! Tell us what you use for monitoring solutions, what you need to integrate with them; tell us your experience, and what your alerting needs are!
Today, we will integrate Prometheus Alertmanager with SAP Cloud Platform Alert Notification. For the demo purpose, we will use an already configured Alertmanager setup, we'll simulate a troublesome situation that will cause an alert trigger. The alert will be received in shape of an email message coming from Alert Notification.
To complete the scenario below, a couple of prerequisites are needed:
- Established monitoring using Prometheus;
- Configured Alertmanager on top of Prometheus;
- Active license or subscription for SAP Cloud Platform Alert Notification.
Once all of these are in place, let's start the setup.
Set up SAP Cloud Platform Alert Notification
In this section, we'll obtain credentials from Alert Notification that are used from Alertmanager. Then, we'll create a subscription to catch all alerts coming from Alertmanager and will forward them to our mailbox.
Note: You could reuse an existing or create another more convenient delivery channel. In the demo, we will use the EMAIL action type due to its simplicity.
Let's start with the credentials. We need a credential pair for BASIC authentication with ALERT_WRITE permission. Follow the steps according to your SAP Cloud Platform environment:
Now, let's create the subscription. Navigate to your Alert Notification instance in SAP Cloud Platform Cockpit, select the Subscriptions menu and choose Create. At the first step of Create Subscription wizard set a name for your and choose Create. At the Select Conditions step, choose Create Condition. Enter a name for the new condition and Condition: eventType Is Equal To PrometheusAlertmanager:
Choose Create and then Assign. At Select Actions step, you have two options:
- if you still don't have an EMAIL action within your Alert Notification instance – select Create, choose the EMAIL action type, provide it with name and refer to your email address, finally select Create. Make sure to confirm the action;
- if you already have an EMAIL action within your Alert Notification instance – you could safely reuse it; make sure the existing action is already confirmed, if not – complete the subscription creation, navigate to the Actions menu, select the not confirmed action's tile, click on Resend Token.
Now, you're back to the subscription wizard, make sure your action is selected and select Assign.
Set up Prometheus Alertmanager
As stated above, to achieve the scenario you should already have well-established monitoring using Prometheus as well as an alerting on top of it using Prometheus Alertmanager. Now, we will only extend it with the needed setup to forward our alerts to Alert Notification.
For that purpose, we should define a new receiver with webhook_config in our Alertmanager configuration:
receivers:
- name: alert-notification
webhook_configs:
- url: https://<<; ALERT_NOTIFICATION_BASE_URL >>/<< SCP_ENVIRONMENT >>/producer/v1/resource-events?schema_definition=prometheus_alert_manager&version=4
http_config:
basic_auth:
username: << client_id >>
password: << client_secret >> Where the placeholders shall be replaced as follows:
- << client_id >> – this is the value of the client_id property you have previously received upon technical client creation in Alert Notification;
- << client_secret >> – this is the value of the client_secret property you have previously received upon technical client creation in Alert Notification;
- << SCP_ENVIRONMENT >> – this is either neo or cf in accordance to your SAP Cloud Platform environment – Neo or Cloud Foundry, respectively;
- << ALERT_NOTIFICATION_BASE_URL >> – this is the base Alert Notification URL that is available in accordance to your SAP Cloud Platform environment:
- for Cloud Foundry environment, you have received it upon creation of your technical client (service key) under the url property. See an example of the final url to be provided in your receiver configuration:
https://clm-sl-ans-live-ans-service-api.cfapps.us10.hana.ondemand.com/cf/producer/v1/resource-events...
- for Neo environment, it is displayed in the Security page of your Alert Notification instance in the Cloud Cockpit. See an example of the final url to be provided in your receiver configuration:
https://clm-sl-ans-live-ans-service-api.cfapps.us10.hana.ondemand.com/neo/producer/v1/resource-event...
- for Cloud Foundry environment, you have received it upon creation of your technical client (service key) under the url property. See an example of the final url to be provided in your receiver configuration:
As a last step in our Prometheus Alertmanager configuration, we have to add the newly created receiver to an existing route, so that we could easily test our setup in the next section. In our environment, this looks like this:
route:
receiver: alert-notification
group_wait: 30s
group_interval: 30s
repeat_interval: 1h
group_by: [alertname]
...Execute the Scenario
Finally, we're ready to test how the Prometheus Alertmanager alert will look like in shape of SAP Cloud Platform Alert Notification event. To do this, we need to trigger an alert that will match the route where the alert-notification receiver is added. How to trigger it is up to your particular environment.
However, in the next paragraph I'll explain what is our certain setup and how I'll test the integration. If it's not of interest to you, you could safely skip it and jump directly to the final words and the screenshot to see how the Prometheus Alertmanager alert is represented.
In our case, we have established a monitoring for Kubernetes environment where some of the metrics we observe are kube-state-metrics. We've provided the alert-notification receiver to a route that will match alerts in regards to such metrics. The rule we will break to trigger an alert is based on a couple of deployment metrics:
- alert: DeploymentReplicasMismatch
expr: kube_deployment_spec_replicas != kube_deployment_status_replicas_available
for: 2m
annotations:
summary: Deployment has replicas mismatch
description: Deployment {{ $labels.namespace }}/{{ $labels.deployment }} has not matched the expected number of replicas for longer than 15 minutes.
labels:
severity: criticalIn order to trigger an alert, we'll "kill" one of our application replicas, so that the expected number won't match. According to our setup, within a couple of minutes an alert will be triggered and send to Alert Notification. The exact timing could differ in your environment but soon you'll receive an email similar to this one:
Would you like to further explore this scenario? See Integrating with Prometheus Alertmanager at SAP Help Portal.
This blog post is part of a series of blog posts related to integration between third-party monitoring tools and SAP Cloud Platform Alert Notification service. We will continuously add newly released integrations, so stay tuned!
We ask you – our customers and partners – to join us in our integration journey! Tell us what you use for monitoring solutions, what you need to integrate with them; tell us your experience, and what your alerting needs are!
- SAP Managed Tags:
| User | Count |
|---|---|
| 14 | |
| 11 | |
| 11 | |
| 10 | |
| 10 | |
| 8 | |
| 7 | |
| 7 | |
| 7 | |
| 7 |
