Alertmanager is a standalone component that focuses on handling alerts generated by Prometheus. It acts as a centralized alerting system that receives, deduplicates, groups, and sends out notifications.
In this post we will utilize this module to send alert to Telegram based on the rule that we create on Prometheus.
Setting up Alerting rule on Prometheus
The way we create our alerting tool is by detecting the parameter “up”, where if the returned data is 1 it means the node is up and if it is 0 then the node is down.
Now we create a new alerting rule
1
sudo nano /etc/prometheus/alert_server_down.yml
Paste in these lines of configuration
12345678910
groups:- name: server_down rules: - alert: server_down expr: up == 0 for: 1m labels: severity: page annotations: summary: Server is down
We give it name “server_down” with the expression “up == 0”, which means if there’s a node with the parameter 0, this rule will be triggered.
We also give it the wait time of 1 minute before firing the rule.
Next we add this new rule to our Prometheus configuration file
1
sudo nano /etc/prometheus/prometheus.yml
Modify the “rule_files” with the yml file that we just created.
12345678910
# Alertmanager configurationalerting: alertmanagers: - static_configs: - targets: - localhost:9093# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.rule_files: - "alert_server_down.yml"
Add the newly created “alert_server_down.yml” file below the rule_files.
After that restart the Prometheus service.
1
sudo systemctl restart prometheus
Now on the alert page, you’ll see the “server_down” rule.
This rule by itself doesn’t do anything, it only shows a visual indicator stating a node is down. If we want to trigger an action, we need Alert Manager.
Setting up Alert Manager
First, download the Alert Manager and extract it.
12
wget https://github.com/prometheus/alertmanager/releases/download/v0.25.0/alertmanager-0.25.0.linux-amd64.tar.gztar xvzf alertmanager-0.25.0.linux-amd64.tar.gz
Next, navigate to the folder and copy the files to its supposed directories
123
cd alertmanager-0.25.0.linux-amd64/sudo cp alertmanager amtool /usr/local/bin/sudo cp alertmanager.yml /etc/alertmanager/
Set the owner
1
sudo chown -R prometheus:prometheus /etc/alertmanager/alertmanager.yml /usr/local/bin/{alertmanager,amtool}
Configure the Alert Manager yaml file
1
sudo nano /etc/alertmanager/alertmanager.yml
Paste in these lines
1234567891011121314
global: resolve_timeout: 5mroute: group_by: ['alertname'] group_wait: 10s group_interval: 10s repeat_interval: 24h receiver: 'telegram'receivers:- name: 'telegram' webhook_configs: - url: "http://localhost:8080/alerts" send_resolved: true
The url is the telegram webhook relay that we will run on a docker later
Run this command to make sure your config file doesn’t have any error
1
amtool check-config /etc/alertmanager/alertmanager.yml
1234567
Checking '/etc/alertmanager/alertmanager.yml' SUCCESSFound: - global config - route - 0 inhibit rules - 1 receivers - 0 templates
Then we create the alertmanager service file
1
sudo nano /etc/systemd/system/alertmanager.service
Paste in these lines
12345678910111213
[Unit]Description=AlertManager Server ServiceWants=network-online.targetAfter=network-online.target[Service]User=prometheusGroup=prometheusType=simpleExecStart=/usr/local/bin/alertmanager --config.file /etc/alertmanager/alertmanager.yml[Install]WantedBy=multi-user.target
Finally start the service
123
sudo systemctl daemon-reloadsudo systemctl start alertmanagersudo systemctl enable alertmanager
Also restart the prometheus service
1
sudo systemctl restart prometheus
Check the status by running this
12
sudo systemctl status alertmanagersudo systemctl status prometheus
123456789
● alertmanager.service - AlertManager Server Service Loaded: loaded (/etc/systemd/system/alertmanager.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2023-07-18 11:01:14 UTC; 11min ago Main PID: 5835 (alertmanager) Tasks: 12 (limit: 9386) Memory: 14.1M CPU: 1.175s CGroup: /system.slice/alertmanager.service └─5835 /usr/local/bin/alertmanager --config.file /etc/alertmanager/alertmanager.yml
You can also access the Web UI on here http://198.18.0.201:9093
This page will not show any rule until the rule is triggered.
Installing Docker
Skip this section if you already have docker installed.
Install docker on ubuntu with this command
12
sudo snap refreshsudo snap install docker
Pulling the Docker image
After docker is installed, pull this Alert Manager Telegram Relay docker image.
1
sudo docker pull ghcr.io/janw/alertmanager-telegram:edge
Run this command to make sure the image is present on your machine
1
sudo docker images
12
REPOSITORY TAG IMAGE ID CREATED SIZEghcr.io/janw/alertmanager-telegram edge 49047627b4f6 7 weeks ago 159MB
Run this command to spin up the docker container.
123456
sudo docker run -d --rm \ --name alertmanager-telegram \ -e TELEGRAM_CHAT_ID="{chat-id}" \ -e TELEGRAM_TOKEN="{bot-id}" \ -p 8080:8080 \ ghcr.io/janw/alertmanager-telegram:edge
Use your own chat-id and bot-id from the telegram bot that you have created
Lastly, run this command to check if the container is running
1
sudo docker ps
12
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES86be69939b3e ghcr.io/janw/alertmanager-telegram:edge "gunicorn alertmanag…" 32 minutes ago Up 32 minutes 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp alertmanager-telegram
Curl localhost:8080 to make sure the web is running
1
curl localhost:8080
1234
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN"><title>404 Not Found</title><h1>Not Found</h1><p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p>
It’ll show an 404 error, That’s normal. All we need to check is the web on that port is up.
And everything is done here. Let’s proceed with testing.
Testing the Alert Manager
Try shutting down one of the monitored node, it’ll be visible on the alert page on Prometheus
We can also see it on Alert Manager
And on the telegram side, we should see our bot sending the alert as well.
