Skip to main content

Prometheus/Grafana Setup

Objective#

The objective of this project is to set up Prometheus and Grafana, install the Prometheus agent on an EC2 instance, and configure an alert in Grafana based on metrics collected by Prometheus.

Prerequisites:

  • AWS account with administrative access
  • EC2 instance(s) running in your AWS account.

IAM Policy#

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"cloudwatch:GetMetricData",
"cloudwatch:GetMetricStatistics",
"cloudwatch:ListMetrics",
"cloudwatch:PutDashboard",
"cloudwatch:PutMetricAlarm",
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents",
"ec2:DescribeInstances",
"ec2:DescribeRegions"
],
"Resource": "*"
}
]
}

Step 1: Set up Prometheus and Grafana#

  1. Launch an EC2 instance or use an existing one to set up Prometheus and Grafana. Ensure that the instance has inbound rules allowing traffic on ports 9090 (Prometheus) and 3000 (Grafana).
  2. Connect to the EC2 instance using SSH.
  3. Update the package repository and install necessary dependencies:
    wget https://github.com/prometheus/prometheus/releases/download/v2.33.0/prometheus-2.33.0.linux-amd64.tar.gz
    tar xvfz prometheus-2.33.0.linux-amd64.tar.gz
    cd prometheus-2.33.0.linux-amd64
    vim prometheus.yml
    ./prometheus --config.file=prometheus.yml
    sudo yum install -y https://dl.grafana.com/oss/release/grafana-8.0.0-1.x86_64.rpm
    sudo systemctl start grafana-server
    sudo systemctl enable grafana-server
  4. Access Grafana using your browser by visiting http://<EC2-Instance-IP>:3000. Login using the default credentials (username: admin, password: admin) and change the password when prompted.

Step 2: Configure Alert in Grafana#

  1. Open Grafana in your browser and log in using your credentials.
  2. Click on the "Configuration" icon in the left sidebar, then select "Data Sources".
  3. Click on "Add data source" and select "Prometheus" as the data source type.
  4. Configure the Prometheus data source with the following details:
    • Name: Prometheus
    • URL: http://<EC2-Instance-IP>:9090
    • Access: Server
  5. Click on "Save & Test" to validate the data source.
  6. Return to the Grafana home dashboard and click on the "+" icon to create a new dashboard.
  7. Click on "Add Query" and select the Prometheus data source you configured earlier.
  8. Enter the desired Prometheus query to fetch the metric you want to set an alert on (e.g., node CPU{instance="<EC2-Instance-IP>:9100", mode="idle"}).
  9. Set the desired alert condition and configure the notification channels (e.g., email, Slack, etc.).
  10. Save the dashboard and enable the alert by toggling the "Alerting" switch.
  11. Test the alert by triggering the condition and ensuring that the notification is sent to the configured channels.

Results#

Grafana Metrics#

Grafana Metrics

Mail Alert#

Mail Alert

Edit this page