Grafana in practice: dashboards as code, not as clicking

I have clicked a dashboard together more than once, then lost it: no export, no version control, a panel someone renamed and nobody remembers why. Grafana is a viewer you point at a data source, and the click-first workflow is the wrong default. This post is the opposite one: a throwaway Grafana plus Prometheus and node_exporter on 127.0.0.1, with a Prometheus datasource, a dashboard, and an alert rule all delivered by Grafana provisioning — dashboards as code, files in git, reproduced with one command. Every block of output below came out of that stack, unedited.

What Grafana is — and what it is not

Grafana stores almost nothing. It is a query front end: you choose a data source (Prometheus, Loki, PostgreSQL, InfluxDB, Elasticsearch, a pile of others), write a query, and Grafana draws the result. It keeps dashboards, variables, users, and alert state in its own SQLite/Postgres database — but it does not keep your metrics.

That single fact settles most arguments:

• Not a time-series database. Sample retention, downsampling, and cardinality belong to Prometheus (or VictoriaMetrics, Mimir, etc.). Grafana reads; it does not own the samples. If your Grafana is slow, look at the query and the datasource before you blame the renderer.
• Not a collector. Nothing scrapes your hosts into Grafana. Prometheus pulls /metrics; Grafana asks Prometheus.
• Not the source of truth. Since everything is a file, the running Grafana is a cache of your repo. Clicking is a temporary convenience, not the artifact.

The data flow is deliberately boring: datasource holds samples, Grafana reads them, and dashboards and alerts are the two readers. Here is the picture for this stack.

Grafana provisioning: the Prometheus datasource as code

The fastest way to make a Grafana claim reproducible is to never configure it by hand. GF_PATHS_PROVISIONING points at a directory (by default /etc/grafana/provisioning) that Grafana scans at startup: datasources/ for datasources, dashboards/ for dashboard providers, alerting/ for alert rules. Mount that directory read-only and the container’s whole configuration is a folder in your repo.

Here is the compose stack — three services, 127.0.0.1 only, no privileged flags:

# docker-compose.yml
services:
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
    ports: ["127.0.0.1:19090:9090"]

  node-exporter:
    image: prom/node-exporter:latest

  grafana:
    image: grafana/grafana:latest
    environment:
      GF_SECURITY_ADMIN_USER: admin
      GF_SECURITY_ADMIN_PASSWORD: admin
    volumes:
      - ./provisioning:/etc/grafana/provisioning:ro
      - ./dashboards:/var/lib/grafana/dashboards:ro
    ports: ["127.0.0.1:3000:3000"]

The Prometheus datasource is one small file. The uid matters: dashboards reference the datasource by that stable id, not by display name, so renaming a datasource never breaks a panel.

# provisioning/datasources/prometheus.yml
apiVersion: 1
datasources:
  - name: Prometheus
    uid: prom-main
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    jsonData:
      httpMethod: POST
      timeInterval: 5s

access: proxy means Grafana’s backend makes the request to Prometheus — the browser never needs to reach it, which is why this works with Prometheus on a private 127.0.0.1. Bring it up and ask Grafana what it loaded:

$ docker compose up -d
$ curl -s localhost:3000/api/health
{
  "database": "ok",
  "version": "13.2.3",
  "commit": "90ffed056f0884267356c12a0eeb72a022af53f1"
}

$ curl -s -u admin:admin localhost:3000/api/datasources | # trimmed
  python3 -c '...'
name=Prometheus uid=prom-main type=prometheus isDefault=True readOnly=True

readOnly: true is the payoff: a provisioned datasource cannot be edited in the UI. The file wins. Grafana says so in its own logs:

logger=provisioning.datasources msg="inserting datasource from configuration" name=Prometheus uid=prom-main
logger=provisioning.dashboard msg="starting to provision dashboards"
logger=provisioning.alerting msg="finished to provision alerting"

Dashboards as code: the JSON is the artifact

A dashboard provider is the bridge between the file tree and Grafana’s database. It says which folder the JSON files live in and whether the UI may override them:

# provisioning/dashboards/dashboards.yml
apiVersion: 1
providers:
  - name: homelab
    orgId: 1
    folder: Homelab
    type: file
    disableDeletion: true
    updateIntervalSeconds: 30
    allowUiUpdates: false        # the file is authoritative
    options:
      path: /var/lib/grafana/dashboards

allowUiUpdates: false and disableDeletion: true are the two lines that make this as code. The dashboard is a plain JSON document — uid, title, a templating block, and a panels array. A panel is a data source, a query, and a display type:

{
  "uid": "node-overview",
  "title": "Node Overview (provisioned)",
  "templating": { "list": [ { "name": "instance",
    "type": "query",
    "query": "label_values(up{job=\"node\"}, instance)",
    "includeAll": true, "multi": true } ] },
  "panels": [
    { "type": "stat",  "title": "Targets up",
      "targets": [ { "expr": "sum(up{job=\"node\", instance=~\"$instance\"})" } ] },
    { "type": "gauge", "title": "Memory available %",
      "targets": [ { "expr":
        "100 * node_memory_MemAvailable_bytes{instance=~\"$instance\"} / node_memory_MemTotal_bytes{instance=~\"$instance\"}" } ] }
  ]
}

Search the API and the dashboard is there, tagged and folder-scoped, and the metadata confirms it came from a file rather than from a click:

$ curl -s -u admin:admin 'localhost:3000/api/search?type=dash-db' | # trimmed
  python3 -c '...'
uid=node-overview title="Node Overview (provisioned)" tags=[node, provisioned] folder=Homelab

$ curl -s -u admin:admin localhost:3000/api/dashboards/uid/node-overview | # trimmed
  python3 -c '...'
provisioned=True provisionedExternalId="node-overview.json"
panels=4 vars=[instance]

That is the whole idea of dashboards as code: a git diff shows what changed, a review catches the bad panel, and a fresh Grafana rebuilds itself. Give each dashboard a file; give each datasource a uid; never let the UI be the only copy.

The query editor vs raw PromQL

Grafana’s query editor is a PromQL builder: a metric picker, a label filter, a function menu, an “explain” panel. It is good for discovery — find the metric name, see the labels. It is a lossy translation of the query you actually run, and some of what it produces is ugly on purpose (auto-generated regex filters, a chain of functions you did not write).

Every panel ultimately sends a PromQL string to the datasource. You can send the same string by hand, through Grafana’s own proxy, and see exactly what the panel sees:

$ curl -s -u admin:admin \
    'localhost:3000/api/datasources/proxy/uid/prom-main/api/v1/query' \
    --data-urlencode 'query=up'
{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      { "metric": { "__name__": "up", "env": "demo", "instance": "node-exporter:9100",
                    "job": "node", "role": "host" }, "value": [1791798871.408, "1"] },
      { "metric": { "__name__": "up", "instance": "localhost:9090",
                    "job": "prometheus" }, "value": [1791798871.408, "1"] }
    ]
  }
}

Two rules of thumb. Write the query in code mode (editorMode: "code") once it stops being trivial — the visual editor is for the first draft, not for a rate over a histogram. And the API is the ground truth: when a panel shows nothing, hit the proxy URL with the same expression and see whether Prometheus returns an empty vector (your query is wrong) or an error (your labels are wrong). The panel is a rendering; the proxy is the answer.

Panels that matter — and when each lies to you

Four panel types carry almost every useful dashboard, and each has a way of telling a comfortable lie.

• Time series — the line chart. The default and usually the right one: rate, latency, queue depth over time. It lies when the Y axis is auto-scaled from zero to maximum and a 3% wobble looks like a cliff; pin the axis, and use rate() on counters rather than plotting raw totals that only ever climb.
• Stat — one big number. Perfect for “is it up”, “how many”, a value with thresholds. It lies by averaging away the shape: a single avg() of a bimodal latency looks calm while half your requests time out. If the distribution matters, do not reduce it to a stat.
• Gauge — a stat with a known range. Memory %, disk %, a saturation ratio. It lies when the range is wrong: a memory gauge set 0..100 when the host has 64 GB hides a run to 90% behind a half-full dial. Set min/max from the thing being measured.
• Table — many series, side by side. Best for “which instance is the outlier”: top-N by error rate, per-host disk free, a label-value breakdown. It lies by becoming a wall of numbers nobody reads — cap it with topk(10, …) and sort by the column that matters.

The choice is not cosmetic. A stat answers “what is it now”; a time series answers “what is happening”; a table answers “which one”. Pick by the question, and set units, thresholds, and axis bounds explicitly, because a panel that lies is worse than no panel.

Templating: variables for reuse

A dashboard hard-coded to one instance is a dashboard you copy for each host. Templating fixes that: a dashboard variable that becomes a dropdown, interpolated into every query with $name. The variable above is a query variable — its options come from Prometheus itself:

label_values(up{job="node"}, instance)   →  node-exporter:9100

includeAll adds an All choice (expanding to a regex over every value), multi allows several at once, and instance=~"$instance" is the interpolation in a panel query. Two habits worth forming: use =~ with variables even when you expect one value (the All case needs regex), and scope the variable query with a label ({job="node"}) so it does not enumerate every series in the store. One templated dashboard then covers the fleet, and a new host appears in the dropdown without a single click.

Alerting in Grafana vs in Prometheus

Both can alert, and the honest answer is that they overlap enough to cause confusion. The distinction that matters:

• Prometheus alerting rules evaluate PromQL inside Prometheus on evaluation_interval. The rule is a file, the evaluation is cheap (the data is already there), and firing alerts go to Alertmanager for routing, grouping, and silences. This is the right home for simple, expression-based conditions over metrics.
• Grafana alerting evaluates in Grafana, can query any data source (not just Prometheus) in one rule, and gives you a UI for contact points, notification policies, and silences. It is the right home for “alert when the log count from Loki spikes”, or when the condition spans data sources.

Because this stack provisions everything as code, the Grafana rule is a file too:

# provisioning/alerting/rules.yml
apiVersion: 1
groups:
  - orgId: 1
    name: homelab
    folder: Homelab
    interval: 1m
    rules:
      - uid: target-down
        title: TargetDown
        condition: C
        for: 30s
        labels: { severity: page }
        data:
          - refId: A
            datasourceUid: prom-main
            model: { expr: "up == 0", instant: true, editorMode: code }
          - refId: C
            datasourceUid: __expr__
            model: { type: threshold, expression: A, conditions: [ { evaluator: { type: gt, params: [0] } } ] }

Read the two datasourceUid values: A runs PromQL against prom-main, and C is a threshold expression over A — Grafana’s alerting is a small expression pipeline, not a single PromQL string. The API returns the rule it built, which is how you check a provisioned alert actually loaded:

$ curl -s -u admin:admin localhost:3000/api/v1/provisioning/alert-rules | # trimmed
  python3 -c '...'
uid=target-down title=TargetDown condition=C for=30s labels={severity: page}
expr="up == 0" datasourceUid=prom-main

The advice (also in my Prometheus write-up) is not to run both for the same condition. Pick the home by where the data lives, keep for: long enough to survive a blip, and alert on things a human would act on. A green light that lies is worse than none — the same discipline a small-fleet setup lives by.

Anti-patterns worth naming

• The dashboard nobody opens. If a dashboard has not answered a question in a month, it is a maintenance burden pretending to be monitoring. Delete it; git remembers.
• The useless average. avg() across rooms, hosts, or request types hides exactly the outlier you care about. Average the thing that is homogeneous, and use max, topk, or a percentile everywhere else.
• The hundred-panel wall. A dashboard is a story, not a data dump. Six panels that each answer a question beat sixty that answer none; split the rest into a second dashboard behind a link.
• The clicked-but-uncommitted change. A UI edit that never became a file is a mystery for the next person. With allowUiUpdates: false, the UI simply refuses — and the fix is a commit.

Clean up

The whole stack was one compose file, three provisioning YAMLs, and one dashboard JSON — deleted the moment the output was captured:

$ docker compose down -v && cd ..

The promise of Grafana provisioning is small and real: the install is never the hard part, the hard part is deciding which question a panel answers, and then committing it. Point Grafana at a data source, keep the datasource and dashboards as code, and the dashboard you rebuild on a new box is the dashboard you reviewed.