Setting Up Monitoring With Grafana

The Grafana Config Drift Trap

We built this skill because setting up Grafana from scratch is a rite of passage that nobody should have to repeat. You pull the official docker-compose.yml, point it at Prometheus, and think you're done. Then you try to provision alert rules and hit the wall. Grafana's unified alerting structure demands exact __dashboardUid__ and __panelId__ bindings. Your team starts hand-editing dashboard JSON, breaking templating variables, and accidentally creating data-source-managed alert rules that silently detach from their panels ^[1]. You end up with a monitoring stack that looks functional on day one but fractures under version upgrades or when a new engineer joins.

The problem isn't Grafana; it's the lack of a standardized, validated provisioning workflow. Most teams treat dashboards as disposable UI artifacts instead of code. They click through the interface, save JSON files to local disks, and version-control them without understanding the schema versioning. When Grafana bumps its API spec, those manually crafted dashboards break. Alert rules created through the UI bypass the provisioning layer entirely, meaning they aren't tracked in Terraform, aren't validated by CI, and disappear when you rebuild the environment. You're left debugging YAML indentation errors, missing Prometheus relabeling configs, and broken OTLP endpoints while your on-call engineers stare at blank panels.

What Broken Monitoring Costs Your Team

Every hour spent debugging provisioning YAML is an hour not spent shipping features. When alert rules aren't linked to panels correctly, your on-call engineers get paged for metrics that don't exist in the dashboard ^[4]. We've seen platform teams burn 15 to 20 hours just untangling broken data source configurations and manual dashboard imports ^[2]. The financial bleed is real: missed P99 latency spikes, uncaught memory leaks in sidecars, and alert fatigue that makes actual incidents feel like noise. If your alerting system requires manual UI clicks to configure notifications, you're already failing SLOs ^[3].

Downstream, broken dashboards mean slower MTTR, frustrated developers, and a culture where "it's monitored" is a lie. You lose visibility into blackbox probes, cAdvisor container metrics, and application traces because your scrape configs are scattered across shell scripts and environment variables. When an incident hits, you're not analyzing signals; you're hunting for the right data source URL. That's not just wasted time—it's lost customer trust and degraded reliability. You can't fix what you can't see, and you can't see what isn't provisioned consistently.

A Platform Team's Provisioning Nightmare

Imagine a fintech team scaling to 40 microservices. They need real-time visibility across Kubernetes nodes, application OTLP traces, and legacy blackbox probes. The senior engineer writes a Prometheus scrape config, provisions a Grafana dashboard via Terraform, and manually creates alert rules in the UI. Two months later, Grafana rolls out a minor version. The dashboard JSON schema shifts. The Terraform-managed data sources break. The alert rules, created through the UI, detach from their panels because the provisioning layer never knew they existed. The team spends three days rebuilding the stack from scratch.

A 2024 infrastructure engineering post ^[5] highlights exactly this pattern: teams that rely on manual dashboard creation and UI-driven alerting quickly drown in configuration drift. They end up with fragmented visibility, where metrics, logs, and traces live in silos, and alerting becomes a game of whack-a-mole. The root cause is always the same: treating monitoring as a manual process instead of a declarative system. When you skip validation, skip schema enforcement, and skip unified alerting bindings, you're not building infrastructure—you're building debt.

If you want to avoid this trap, you need a workflow that treats dashboards and alert rules as code. Pairing this approach with the Monitoring & Observability Pack gives you distributed tracing and log correlation out of the box. Aligning your panels with latency, traffic, errors, and saturation using the SRE Golden Signals Playbook ensures you're measuring what actually impacts your users. You don't need more tools; you need a validated pipeline that ships the right configuration every time.

What Changes Once the Skill Is Installed

You stop guessing. The skill ships with a validated provisioning pipeline that enforces structure before it touches your Grafana instance. The scripts/validate-provisioning.sh script runs against your YAML, catching missing keys and structural mismatches before they break the API. Your dashboard JSON is enforced against a strict schema, guaranteeing that panels, templating, and annotations survive Grafana upgrades. Alert rules are provisioned through alert-rules.yaml using the unified alerting structure, with __dashboardUid__ and __panelId__ bindings baked in, so they never detach.

Prometheus configuration includes blackbox exporter, cAdvisor, and OTLP endpoints out of the box, so you're measuring what actually matters. Terraform manages data sources and notification channels, eliminating UI drift. The embedded references give you canonical knowledge on Grafana API endpoints, dashboard creation, and Prometheus metric exposure, so you never have to guess about relabeling or scrape targets. The test suite runs validation against the schema and exits non-zero if anything fails, meaning broken configs never make it to production.

The result is a monitoring stack that provisions in minutes, validates automatically, and survives team turnover. You get RFC 9457-style error handling baked into your alert payloads, consistent templating across all dashboards, and a single source of truth for your infrastructure visibility. No more manual imports. No more broken JSON schemas. No more alert fatigue from misconfigured thresholds.

What's in the Pack

• skill.md — Orchestrator skill defining workflow, referencing all templates, references, scripts, validators, and examples for Grafana monitoring setup.
• templates/grafana-dashboard.json — Production-grade Grafana dashboard JSON template using the v13 API spec, including panels, templating, and annotations.
• templates/provisioning/alert-rules.yaml — Provisioned alert rules YAML using Grafana's unified alerting structure with __dashboardUid__ and __panelId__ bindings.
• templates/terraform/main.tf — Terraform configuration for Grafana data sources and notification alerts using grafana_asserts resources.
• templates/prometheus/prometheus.yml — Prometheus scrape configuration including blackbox exporter, cAdvisor, and OTLP endpoint setup.
• references/grafana-api-reference.md — Embedded canonical knowledge for Grafana API endpoints, dashboard creation, and alert provisioning schemas.
• references/prometheus-config-guide.md — Embedded canonical knowledge for Prometheus metric exposure, scrape targets, relabeling, and OTLP integration.
• scripts/validate-provisioning.sh — Executable script to validate provisioning YAML files for required keys and structure, exiting non-zero on failure.
• scripts/scaffold-monitoring.sh — Executable script to scaffold a monitoring project structure with directories and copy templates.
• validators/dashboard-schema.json — JSON Schema validator for Grafana dashboard JSON, enforcing spec structure, panels, and templating.
• tests/test-dashboard-validation.sh — Test script that runs dashboard validation against the schema and exits non-zero if validation fails.
• examples/worked-example-dashboard.json — Worked example of a complete, valid Grafana dashboard JSON demonstrating best practices.

Stop Guessing, Start Monitoring

You don't need another tutorial. You need a validated, provisionable monitoring stack that works on day one and survives upgrades. Upgrade to Pro to install the skill. Run the scaffold script, validate your YAML, and ship. Your on-call engineers will thank you, your MTTR will drop, and your dashboards will actually reflect your infrastructure. Stop clicking through UIs and start treating monitoring like code.