Metrics

Every backd process can tell Prometheus how it is doing: requests, errors, database latency and, as the next steps land, functions, jobs and email. It is off by default.

Turning it on#

backd serve, backd worker, backd executor and backd egress each serve their own /metrics (the executor and egress read the same two variables); a process that runs both (serve --with-worker) serves one. The listener answers GET /metrics and nothing else (a request without the token gets 401 whatever the path), serves at most four scrapes at once, and is a separate server: a slow or flooding scraper never slows the API. If the port can’t be bound, the process refuses to start, so a mistake shows at once.

# prometheus.yml
scrape_configs:
  - job_name: backd
    static_configs:
      - targets: ["backd:9090"]
    authorization:            # only with METRICS_TOKEN
      credentials: "<the token>"

The port must be private#

The metrics hold no user ids, emails, addresses, tokens, key names, documents or request paths: labels come from fixed lists and from your configuration (route patterns, realm, database and function names). Still, they tell an attacker which realms and routes exist, how busy and how healthy the service is, and when it is slow. Keep the port on a private network: never publish it, never route it through the public proxy (the production reference answers 404 for /metrics at the edge, and its test checks it), and set METRICS_TOKEN when anything but Prometheus can reach the network it is on.

What is measured#

All names start with backd_; durations are in seconds.

The Go runtime (go_*) and process (process_*) metrics are included.

The production reference and the local stack#

Both run a Prometheus that scrapes every backd process on start.

• The production reference sets METRICS_ADDR and a generated METRICS_TOKEN (BACKD_METRICS_TOKEN in .env, also written to secrets/metrics-token for Prometheus) on backd, worker, executor and egress. Prometheus sits on the func network, which has no route out, and its own web port listens only on its container’s loopback: no other container can reach its UI or query API, and nginx has no route to any metrics port. Look at it with docker compose exec prometheus wget -qO- http://127.0.0.1:9090/api/v1/targets, or change --web.listen-address and give it a network of its own when you add Grafana. The reference’s test checks that the edge answers 404 for /metrics, that a request without the token is refused, that every process is scraped and that the key series exist.
• The local stack (make example) runs the same Prometheus at http://localhost:9090, with a fixed dev-only token. Its metrics ports aren’t published.

Alert rules#

deploy/production/prometheus/alerts.yml has sample rules for Prometheus (the local stack loads the same file): a process that can’t be scraped, MongoDB down or slow, a high rate of 5xx answers, jobs that wait too long, an erase that needs an administrator, emails that fail, and an executor at its process limit. They only evaluate; connect an Alertmanager to be told. Adjust the thresholds to your traffic.

Dashboards#

Five Grafana dashboards are in deploy/grafana/dashboards/, ready to import:

Each alert rule has a panel that shows what it watches: its description says Alert <name>, and the panels that have a threshold draw it as a line.

In the local stack (make example) Grafana runs at http://localhost:3000, with no login (it only listens on your machine), the Prometheus data source and these dashboards already loaded, in the backd folder.

In your own Grafana: add a Prometheus data source with the uid prometheus (the dashboards refer to it), then import the files, or point a file provider at the folder (deploy/grafana/provisioning/ has both). The panels use the role label that the scrape configuration in deploy/production/prometheus/prometheus.yml adds to each target; keep it. The production reference doesn’t run Grafana, and its Prometheus’ web port listens only on its container’s loopback. To connect Grafana, change --web.listen-address and put Prometheus and Grafana on a network of their own that nothing else (and no function process) shares.

scripts/check-dashboards.py checks the files in CI: valid JSON, every metric they use is in this page, and every alert has its panel; in the local stack’s test it also runs every query against Prometheus.