kubectl — context & namespace

kubectl — inspect

kubectl — logs

kubectl — act

Watch out for short-name collisions: on clusters with extra CRDs, kubectl get backup may resolve to a different resource than you expect — prefer the fully-qualified form (backups.velero.io) in scripts.

k9s — navigation

Everything starts with : (command mode) or / (filter).

k9s — on a selected pod

k9s — wider views

The triage loop

My default incident sequence, entirely in k9s:

1. :pods then / on the app name — anything not Running?
2. d on the suspect — read the Events section first, not the spec.
3. l then p — the previous container’s logs hold the crash reason; the current one may be too young to have logged anything.
4. :events sorted by time — anything cluster-wide (evictions, image pulls, OOM kills) that the pod view doesn’t show.

Ctrl-b is the prefix. Notation C-b d = press Ctrl-b, release, then press d.

Survive a dropped SSH connection

tmux new -s work          # start a named session
# ... launch your long-running command inside ...
# C-b d                   # detach — command keeps running on the server
# (connection can now die safely)

tmux attach -t work       # reattach later
tmux attach -d -t work    # force-attach, detaching any stale client

Sessions

Windows (like tabs)

Panes (splits)

Scrolling & copy mode

Handy

Watch a long job after reattaching

tail -f my-long-job-*.log     # Ctrl-c to stop tailing (job keeps running)

Minimal ~/.tmux.conf niceties (optional)

set -g history-limit 100000      # bigger scrollback
set -g mouse on                  # mouse scroll/select panes
setw -g mode-keys vi             # vi keys in copy mode

Two options: read the upstream live at deploy time, or materialize it into git — fetch it on a schedule, write it to a committed file, let a bot commit the diff. This post is about the second one: when it’s the right call, and a reusable way to build it in GitLab CI. The concrete example is the public Tezos test-network registry at teztnets.com/teztnets.json, but nothing here is specific to it.

Why commit it instead of reading it live

Reading upstream live is simpler and always current. But materializing into git buys you things a live read can’t:

• Auditable history. Every change to the list is a commit with a diff and a timestamp. “When did foonet appear?” is git log, not a guess.
• Review and pinning. The committed file can carry your metadata per row — a pinned image tag, a release channel, a rollout policy — that upstream doesn’t know about. A live read would flatten all of that away.
• A stable interface for other automation. A dependency bot (Renovate/Dependabot) can bump the pinned versions in the file; renderers can read it deterministically; nothing at deploy time depends on the upstream being reachable.
• Decoupled failure. If upstream has a bad day, your last-known-good file is still in git. A live read fails your reconcile.

The cost is a moving part — a scheduled job — and a few footguns. Here’s the whole thing.

A reusable, parameterized job

The core insight that keeps this from sprawling: make the job take the list of files to regenerate as an input, so one job template covers every such file in the repo. GitLab’s spec.inputs does this — a typed array parameter:

# .gitlab/generate-networks.yml
spec:
  inputs:
    files:
      type: array
      description: >-
        Repo-root paths to regenerate from the upstream registry.
        Add a path here to bring another file under scheduled regeneration.
---
generate-networks:
  stage: generate
  image: alpine:3.22.1
  rules:
    - if: '$CI_PIPELINE_SOURCE == "schedule"'   # only on a schedule
    - when: never
  before_script:
    - apk add --no-cache curl jq git bash
    - git config --global user.email "ci-bot@example.com"
    - git config --global user.name  "CI Bot"
  script:
    - |
      set -euo pipefail
      # inputs.files renders as a JSON array; regenerate each, stage all,
      # commit once so concurrent pushes never race.
      echo '$[[ inputs.files ]]' | jq -r '.[]' | while IFS= read -r target; do
        echo "=== Regenerating $target ==="
        bash .gitlab/generate-networks.sh "$target"
        git add "$target"
      done
      if git diff --cached --quiet; then
        echo "No changes"
      else
        git commit -m "chore: update networks files [ci skip]"
        git push "https://ci-bot:${GITLAB_TOKEN}@${CI_SERVER_HOST}/${CI_PROJECT_PATH}.git" \
          HEAD:${CI_COMMIT_REF_NAME}
      fi

and you include it, passing the files, from your main pipeline:

# .gitlab-ci.yml
include:
  - local: .gitlab/generate-networks.yml
    inputs:
      files:
        - networks-teztale.json
        - networks-test.json      # add a path -> it's now on the schedule

Three details in that job matter:

• rules pins it to schedule. It never runs on push or MR — only from a GitLab scheduled pipeline (say, nightly). Everything else is when: never.
• [ci skip] on the commit stops the bot’s own push from triggering another pipeline — no loops.
• One commit for all files. Regenerating each and committing together means a second scheduled run (or a human push) can’t interleave with a half-written batch.

The regeneration script

The job delegates per-file logic to a script. The non-obvious requirements: don’t clobber the metadata you keep locally, give new rows a sensible default, and drop rows that vanished upstream.

Say each row in the file carries an “upgrade-stream” schema you maintain — a pinned tag, a release channel, a policy — that upstream doesn’t provide:

[
  { "name": "alpha", "octezTag": "octez-v25.0", "octezChannel": "stable", "octezPolicy": "manual" }
]

The script fetches upstream, then merges: keep existing rows verbatim (preserving your pins), add newcomers inheriting defaults from the first existing row, and let disappearances fall off:

#!/usr/bin/env bash
set -euo pipefail
target_path="${1:?usage: $0 <target_path>}"
url="${TEZTNETS_URL:-https://teztnets.com/teztnets.json}"

# Derive defaults for new rows from the first existing entry's schema.
default_tag=$(jq -r '.[0].octezTag // empty'     "$target_path")
default_channel=$(jq -r '.[0].octezChannel // empty' "$target_path")
default_policy=$(jq -r '.[0].octezPolicy // empty'   "$target_path")

# Build into a temp file and mv into place: the target is BOTH read (for the
# existing schema) and written, and a `> "$target"` redirect would truncate it
# before jq could slurp it. This is the footgun.
tmp="$(mktemp)"; trap 'rm -f "$tmp"' EXIT

curl -fsS "$url" \
  | jq --slurpfile existing "$target_path" \
       --arg dt "$default_tag" --arg dc "$default_channel" --arg dp "$default_policy" \
    '($existing[0] | map({(.name): .}) | add) as $by_name
     | to_entries
     | map(select(.value.aliasOf == null) | .key)
     | map(. as $n | $by_name[$n]
           // {name: $n, octezTag: $dt, octezChannel: $dc, octezPolicy: $dp})' \
  > "$tmp"

mv "$tmp" "$target_path"; trap - EXIT

Reading it top to bottom: build a name→row lookup of what’s already committed; take upstream’s canonical entries (aliasOf == null); for each, reuse the committed row if it exists (keeping your pins untouched), otherwise synthesize a new row on the fleet’s default tag/channel/policy. Rows not in upstream simply aren’t emitted — they drop out, and that deletion shows up as a reviewable diff on the next run.

That “reuse if present, default if new” merge is why a dependency bot can bump octezTag on one row and the next regeneration won’t stomp it — the row already exists, so it’s preserved verbatim.

The two footguns

1. Truncation. The file is read and written in the same step. jq … > "$target" truncates it to empty before jq opens it via --slurpfile, so you lose your existing schema and every new row gets null defaults. Build in a temp file, mv into place.
2. New-row defaults must come from somewhere. If the file is empty, there’s no first row to inherit from, and a new network lands with null tag/channel/policy. Guard for it: refuse to run (or seed one canonical row by hand) rather than emit half-populated entries.

When to use which

Materialize-into-git and read-live are two ends of a spectrum:

If the list is disposable and you want to track upstream instantly, read it live. If rows carry state you care about — pins you review, policies you enforce, a history you audit — regenerate on a schedule and let the diff speak. The pattern above is about forty lines of YAML and jq, reusable across every such file in the repo.

The annoying part is that the list already exists, upstream, and is already kept current. Tezos publishes the canonical registry of live test networks at https://teztnets.com/teztnets.json — every network, its category, and the exact octez image it expects. Copying that into my repo by hand is busywork. I wanted the ApplicationSet to read it directly.

This is a concrete application of the ApplicationSet plugin generator from the previous post — read that first for how the plugin works and how to deploy it (raw manifests, in the ArgoCD controller’s namespace). Here I’ll just wire it to teztnets.json.

What teztnets.json looks like

It’s an object keyed by network name. Trimmed:

{
  "ghostnet": {
    "aliasOf": null,
    "category": "Long-running Teztnets",
    "docker_build": "tezos/tezos:octez-v23.1"
  },
  "ushuaianet": {
    "aliasOf": null,
    "category": "Protocol Teztnets",
    "docker_build": "tezos/tezos:octez-v25.0"
  },
  "ushuaianet-20260701": {
    "aliasOf": "ushuaianet",
    "category": "Protocol Teztnets"
  }
}

Three things matter for my use case:

• aliasOf — dated snapshots alias the rolling name; I only want the canonical entries (aliasOf == null).
• category — I only want the short-lived protocol-test networks, not the long-running public ones (which I run differently).
• docker_build — tezos/tezos:octez-v25.0; I need the tag part, octez-v25.0, to pin the node image per network.

The filter

The plugin’s job is to turn that object into a flat list of {name, octezTag}. As a jq filter (JSON_FILTER):

to_entries
| map(select(.value.aliasOf == null
             and (.value.category | test("Protocol"))))
| map({
    name: .key,
    octezTag: (.value.docker_build | split(":")[1])
  })

Fed the live payload, today that returns exactly:

[ { "name": "ushuaianet", "octezTag": "octez-v25.0" } ]

One network now; more when upstream announces the next protocol test net. JSON_URL is https://teztnets.com/teztnets.json, and that filter is the only network-specific configuration the plugin needs.

The ApplicationSet

The generator is a matrix of the plugin (which networks?) against a single-element list (which cluster do the nodes run on?). The template stamps one octez-node per network, using the network name and the image tag the plugin extracted:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: octez-test
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - matrix:
        generators:
          - plugin:
              configMapRef:
                name: json-plugin      # the teztnets-configured plugin
              input:
                parameters: {}
              requeueAfterSeconds: 300
          - list:
              elements:
                - cluster: my-workload-cluster
                  server: https://my-workload-cluster.example   # where nodes run
  template:
    metadata:
      name: "octez--test"
      labels:
        network: ""
    spec:
      project: default
      destination:
        server: ""
        namespace: "octez--test"
      source:
        # octez-node packaged as a Helm chart — point repoURL/chart at wherever
        # you host it. Only the values below are network-specific.
        repoURL: <your-octez-node-chart-repo>
        chart: octez-node
        targetRevision: 0.9.1
        helm:
          releaseName: "octez--test"
          values: |
            network: ""
            history: "rolling"
            image:
              repository: tezos/tezos
              tag: ""
            persistence:
              enabled: true
              size: 250Gi
      syncPolicy:
        automated: { prune: true, selfHeal: true }
        syncOptions: [ CreateNamespace=true ]

Note the split of responsibilities that makes this robust: the plugin (running next to the ArgoCD controller) answers which networks exist and on what image; the template’s destination decides where the nodes actually run. The two are independent — the plugin can be on your control-plane cluster while every node lands on a separate workload cluster.

The payoff

With prune: true, the loop is fully self-maintaining on a five-minute requeue:

• a new protocol-test network announced on teztnets.com becomes a running octez-{name}-test node with no MR;
• a decommissioned network drops out of teztnets.json and its node is pruned the same way;
• the image tag always matches what upstream says the network expects, because it comes straight from docker_build.

The hand-maintained network file is out of the critical path entirely.

The trade-off

Reading the registry live means you follow upstream’s image tag exactly and immediately — which is the point, but also means you’ve handed your rollout timing to docker_build. If you need to pin each network to a reviewed image, gate upgrades behind a policy, or keep an auditable git diff of “what changed and when,” you want a committed file you regenerate on a schedule instead — see the companion post.

For a disposable, follow-upstream test fleet, live is exactly right: nobody has to touch it when the network list changes.

This post is a from-scratch, no-Helm walkthrough of standing one up: the protocol, the raw manifests, how to write the filter, and a few production pitfalls. I’ll use a small open-source plugin — argocd-applicationset-json-plugin — that does one generic job: fetch JSON from a URL, filter it with jq or JSONPath, and hand the result back to ArgoCD as parameters. The protocol described here applies to any plugin, though.

What a plugin generator actually is

A plugin generator does not run your code inside the controller. It makes an authenticated HTTP call to a service you deploy, and turns the JSON that comes back into generator parameters. The argocd-applicationset-controller is the client; your plugin is a tiny HTTP server.

The contract is a single route:

POST /api/v1/getparams.execute
Authorization: Bearer <token>
Content-Type: application/json

{ "applicationSetName": "my-appset", "input": { "parameters": {} } }

and the response is a list of parameter objects:

{ "output": { "parameters": [
  { "name": "alpha", "region": "eu" },
  { "name": "beta",  "region": "us" }
] } }

Each object becomes one set of template variables.

Deploying the plugin backend (raw manifests)

Four objects, all in the same namespace as the ApplicationSet controller (usually argocd). We’ll come back to why that matters.

Secret — the shared token the controller authenticates with:

apiVersion: v1
kind: Secret
metadata:
  name: json-plugin
  namespace: argocd
stringData:
  token: "replace-me-with-a-real-secret"

Deployment — the plugin server. It reads its config from environment variables and the token from a mounted file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: json-plugin
  namespace: argocd
spec:
  replicas: 1
  selector:
    matchLabels: { app: json-plugin }
  template:
    metadata:
      labels: { app: json-plugin }
    spec:
      containers:
        - name: plugin
          image: ghcr.io/cmehat/argocd-applicationset-json-plugin:jq-latest
          env:
            - name: JSON_URL
              value: "https://api.github.com/users/kubernetes/repos?per_page=100"
            - name: JSON_FILTER          # jq expression
              value: 'map(select(.archived == false) | {name: .name, stars: (.stargazers_count|tostring)})'
          ports:
            - { name: http, containerPort: 4355 }
          volumeMounts:
            - { name: token, mountPath: /var/run/argo, readOnly: true }
          # The only route is POST-only; a GET returns 501. Probe the socket,
          # not the endpoint. (More on this below.)
          readinessProbe:
            tcpSocket: { port: 4355 }
          livenessProbe:
            tcpSocket: { port: 4355 }
      volumes:
        - name: token
          secret: { secretName: json-plugin }

Service — how the controller reaches it, in-cluster:

apiVersion: v1
kind: Service
metadata:
  name: json-plugin
  namespace: argocd
spec:
  selector: { app: json-plugin }
  ports:
    - { port: 4355, targetPort: 4355 }

ConfigMap — this is what the ApplicationSet references. ArgoCD discovers it by the app.kubernetes.io/part-of: argocd label, and reads the plugin’s URL and token from it:

apiVersion: v1
kind: ConfigMap
metadata:
  name: json-plugin
  namespace: argocd
  labels:
    app.kubernetes.io/part-of: argocd   # required for discovery
data:
  baseUrl: "http://json-plugin.argocd.svc.cluster.local:4355"   # include the port!
  token: "$json-plugin:token"           # $<secret-name>:<key>

Wiring the ApplicationSet

Now the generator just points at that ConfigMap by name:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: repos
  namespace: argocd
spec:
  goTemplate: true
  generators:
    - plugin:
        configMapRef:
          name: json-plugin
        input:
          parameters: {}
        requeueAfterSeconds: 300   # re-fetch the source every 5 min
  template:
    metadata:
      name: "repo-"
    spec:
      project: default
      source:
        repoURL: https://github.com/my-org/app-of-apps.git
        targetRevision: main
        path: "charts/generic"
        helm:
          values: |
            displayName: ""
            stars: ""
      destination:
        server: https://kubernetes.default.svc
        namespace: "repo-"
      syncPolicy:
        automated: { prune: true, selfHeal: true }
        syncOptions: [ CreateNamespace=true ]

Every object the plugin returns becomes one Application, refreshed on the requeue interval. New entries in the upstream JSON appear on their own; entries that disappear are pruned.

Writing the filter: jq or JSONPath

The plugin ships two flavours — a jq image (jq-* tags) and a JSONPath image (jsonpath-* tags, the default). Both take the fetched JSON and must emit a flat array of objects with string values (ArgoCD parameters are strings).

A few generic shapes:

Array of objects → pick fields (e.g. the GitHub repos list above):

map({name: .name, url: .html_url})

Object-of-objects → keys as items, dropping entries that are aliases of another (a common pattern in registry-style files):

to_entries
| map(select(.value.aliasOf == null))
| map({name: .key})

Filter by a property — only public, only in a region, only “active”:

map(select(.status == "active") | {name: .id, region: .region})

The JSONPath variant covers the simple cases without a jq binary — JSON_PATH=$.*, JSON_PATH_KEYS_ONLY=true, JSON_PATH_EXCLUDE_IF_EXISTS=aliasOf gives you the “keys, minus aliases” shape declaratively. Reach for jq when you need to reshape or compute fields.

Whichever you use, test the filter against the real payload before you deploy it — pipe the live URL through jq locally. Most plugin-generator “it produces nothing” incidents are actually a filter that returns [] or the wrong shape.

Deployment notes

Three lessons from running this for real:

1. The plugin must live where the controller runs — not where the Applications deploy. This is the counterintuitive one. The controller resolves configMapRef in its own namespace and dials baseUrl over its own in-cluster network. If your ArgoCD control plane and your workloads are on different clusters (a very common topology), the plugin belongs on the control-plane cluster, next to the controller. Put it on the workload cluster and the controller just reports:

error getting plugin from generator: error fetching ConfigMap "json-plugin" not found

Where the generated Applications ultimately deploy is decided entirely by the template’s destination — independently of where the plugin lives.

2. Health-check the socket, not the endpoint. The plugin’s only route is POST /api/v1/getparams.execute, and it’s token-authenticated — a plain GET returns 501. An httpGet probe against that path therefore always fails: liveness kills the container in a crash loop (exitCode 137), and readiness keeps the pod out of the Service’s endpoints, so even a healthy process is unreachable. Use tcpSocket probes (as above). For a stateless request/response server, “is the port accepting connections?” is the right liveness signal.

3. Two small ones. Pin an image tag your registry actually has (an unbuilt tag gives ImagePullBackOff with a misleading not found), and make sure the ConfigMap’s baseUrl carries the service port — no port means port 80, and the Service listens on 4355.

When to use it

Use a plugin generator when the source of truth is external JSON you don’t want to copy into git and you want Applications to track it live. It’s a running service you have to operate, so it’s overkill for a static list. For “template one Application per row of this API,” it turns an MR-per-change chore into a five-minute reconcile loop.

The plugin used here is open source: argocd-applicationset-json-plugin. In the next post I put it to work against a real, public feed — the Tezos test-network registry.

What makes PID 1 special

On Linux, the kernel assigns PID 1 to the first process in a new PID namespace. Two things make it different from every other process:

Signal handling. The kernel never delivers SIGTERM to PID 1 unless it has explicitly installed a signal handler for it. Most applications don’t — they rely on the default handler the C runtime sets up, which the kernel bypasses for PID 1. The practical result: docker stop sends SIGTERM to your container’s PID 1, nothing happens, and after the timeout (default 10 seconds) Docker sends SIGKILL. Every graceful shutdown attempt silently fails.

Zombie reaping. When a process exits, it stays in the process table as a zombie until its parent calls wait() on it. If the parent exits first, the orphan is reparented to PID 1, which becomes responsible for reaping it. The operating system init daemon knows this and handles it. Your application almost certainly doesn’t.

What Tini does

Tini (Tiny Init) is a minimal init process designed to solve exactly these two problems. It sits as PID 1, registers signal handlers that forward signals to its child process, and reaps zombie processes. Nothing else.

You launch your application as a child of Tini rather than as PID 1 directly:

RUN apk add --no-cache tini
ENTRYPOINT ["/sbin/tini", "--"]
CMD ["your-app"]

docker stop now sends SIGTERM to Tini, which forwards it to your application. Your application’s default SIGTERM handler runs, it shuts down cleanly, and the container exits in milliseconds rather than ten seconds.

The Docker Compose shortcut

If you control the Compose file and don’t need the init embedded in the image, there’s a one-liner:

services:
  web:
    image: your-image
    init: true

init: true injects the Tini binary that ships with Docker Engine itself as PID 1. No separate download, no image change, no supply chain surface beyond what you’ve already trusted by running Docker. This is the lowest-effort correct solution for most Compose-based workloads.

Why not just handle signals in the application?

You can. But it doesn’t solve zombie reaping unless you also implement waitpid() loops, every language runtime handles signals slightly differently, and getting all shutdown paths right (graceful, panicking, subprocess spawning) is non-trivial. Tini is 200 lines of C that has been doing exactly this since 2015.

Checking whether you need it

A quick test: run your container and check what’s at PID 1.

docker exec <container> cat /proc/1/cmdline | tr '\0' ' '

If it prints your application binary directly, you’re exposed to both problems. If it prints tini or dumb-init, you’re covered.

The alternative: dumb-init

dumb-init (from Yelp) solves the same problem with a similar approach. It’s more actively maintained than Tini and also available in Alpine and Debian package repos:

RUN apk add --no-cache dumb-init
ENTRYPOINT ["dumb-init", "--"]
CMD ["your-app"]

The functional difference between the two is small enough not to matter for most workloads. If you’re starting from scratch: dumb-init from the package manager, or init: true in Compose if that fits your deployment model. If you’re already using Tini: stay on it, there’s no reason to switch.

If you’re starting from scratch: dumb-init from the package manager, or init: true in Compose. If you’re already using Tini: no reason to switch.

Grafana Cloud, Mimir, Thanos Receive, Amazon Managed Prometheus — they all ingest via remote_write. The transport difference is real, but it comes with a less-obvious consequence: the pull model isn’t just a transport choice, it’s a health-checking model. When you switch to push, some of your monitoring assumptions silently invert, and your alerting rules need to change with them.

What the pull model gives you for free

In classic Prometheus, the server initiates every scrape. That single design decision buys you three things:

1. up is a real health check. Every scrape attempt produces an up{job, instance} sample: 1 if the target answered, 0 if it didn’t. This is active verification, performed by the same component that evaluates your alerts. up == 0 means “I tried to reach this thing just now and it did not respond.” There is no ambiguity about whose fault it is.

2. Absence is loud. A dead target doesn’t disappear from your metrics — it shows up as up == 0. The failure mode is a signal, not a gap.

3. Service discovery and scrape config live in one place. The Prometheus server knows the full inventory of what it’s supposed to be monitoring. If a target exists in SD but stops answering, that’s detectable by construction.

The canonical alert in this world is trivially simple:

- alert: TargetDown
  expr: up == 0
  for: 5m
  annotations:
    summary: "{{ $labels.job }} on {{ $labels.instance }} is unreachable"

What the push model takes away

In an agent-based architecture, the topology inverts. An agent sits next to (or inside) each node or cluster, scrapes its local targets, and ships samples over remote_write to a central store — Mimir, Thanos Receive, a managed service. Rules are evaluated centrally, against whatever data arrived.

The central store never tries to reach your targets. It only knows what it receives. So the question your alerting can answer changes from “did the target respond when I probed it?” to “has data about this target shown up recently?” — two questions that sound similar but have very different failure modes.

Consider the failure modes:

Four out of five failure modes in the push column produce the same symptom: nothing. And “nothing” is the one thing a naïve up == 0 alert cannot fire on, because rule evaluation needs samples to evaluate. Your loudest failure signal just became your quietest.

Rewriting the alerting layer

The migration is therefore not “port the alert rules and change the datasource.” Three categories of rules need rethinking.

1. Presence alerts replace liveness alerts

up == 0 still works for the case where the agent is healthy but its local target is down — keep those rules. But every one of them needs a companion absence rule that fires when the time series stops arriving entirely:

- alert: MetricsAbsent
  expr: absent_over_time(up{job="octez-node"}[10m])
  for: 5m
  labels:
    severity: p1
  annotations:
    summary: "No samples received for job octez-node in 10m — target, agent, or pipeline is down"

Two practical constraints. First, absent_over_time() can only take a fixed selector — it can’t enumerate instances it has never seen, so you lose the per-instance granularity up gave you. You either maintain one absence rule per critical job (my choice), or you generate them from your inventory (we generate ours with Jsonnet, which keeps the rule set in lockstep with the deployment config). Second, absence alerts fire during intentional removals too — decommission a node and forget to remove the rule, and you page yourself at 3 a.m. for a machine that no longer exists. The inventory and the rules must come from the same source of truth. GitOps helps enormously here: the same ArgoCD application that removes the scrape config removes the absence rule.

2. The pipeline itself becomes a monitored system

With pull, the path from target to alert evaluation was one hop. With push, it’s target → agent scrape → WAL → remote_write queue → network → ingester → store → ruler. Each hop can back up, and a backed-up pipeline delays every alert downstream — your for: 5m is now for: 5m + ingestion lag.

The agent exposes everything you need; you just have to actually alert on it:

• prometheus_remote_storage_highest_timestamp_in_seconds minus prometheus_remote_storage_queue_highest_sent_timestamp_seconds — your end-to-end write lag. Alert when it exceeds, say, 2 minutes.
• prometheus_remote_storage_samples_failed_total and samples_dropped_total — non-zero rates here mean you are losing data, not just delaying it.
• prometheus_remote_storage_shards pinned at max_shards — the queue can’t keep up.
• WAL disk usage on the agent — a long partition fills the buffer, and recovery replays a flood of out-of-order-adjacent samples at the store.

None of these alerts existed in the old stack because none of these components existed. Budget for it: in our migration, roughly a third of the “new” alert rules monitor the monitoring pipeline itself.

3. The dead man’s switch is no longer optional

In the pull world, a fully dead Prometheus was already a known blind spot, and the standard mitigation was a Watchdog/DeadMansSwitch alert — a rule that always fires, routed to an external service that pages you when the heartbeat stops. In the push world this pattern graduates from “nice to have” to structural requirement, because entire-pipeline death is now indistinguishable from “everything is fine” at every layer you control.

The heartbeat must terminate outside the failure domain of the stack it monitors: healthchecks.io, Dead Man’s Snitch, PagerDuty’s dead-man integration, or a second minimal Prometheus in another provider doing nothing but probing the first. If your monitoring runs in-cluster — as ours now does — this external witness is also your answer to the awkward question “what tells us the cluster hosting the monitoring is down?” Whatever you pick, the property that matters is independence: different network path, different provider, different credentials.

What you get in exchange

What you get in exchange:

• Topology fit. No inbound connectivity to edge networks, NATed VMs, or short-lived workloads. The agent only needs one outbound HTTPS path. This is the original motivation and it’s real — our old stack needed carefully curated firewall rules and a VPN mesh for the central server to reach every target.
• One query surface. All regions and clusters land in one store with consistent external_labels. Cross-fleet queries and global dashboards stop being a federation science project.
• HA without gaps. Run two agents per site with identical configs and distinct replica labels; the store (Mimir, or Grafana Cloud’s Adaptive dedup) deduplicates. Compare that to HA pull-Prometheus pairs, where deduplication is the query layer’s problem forever.
• Centralized rule management. Rules live in the ruler, deployed via GitOps, versioned with everything else — no more Ansible-templated rules.d drift across regional servers.

Migration checklist

1. Inventory every up-based alert and write its absent_over_time() companion. Generate them from the same source of truth as the scrape configs.
2. Add pipeline alerts on remote_write lag, failed/dropped samples, shard saturation and agent WAL disk before cutover, not after the first silent outage.
3. Stand up the external dead man’s switch first. It’s the only component that can tell you the cutover itself went wrong.
4. Re-tune for: durations. Add expected ingestion lag headroom to time-critical rules, or alerts that fired reliably at for: 2m under pull will flap under push.
5. Set external_labels deliberately (cluster, region, replica) before the first sample ships — relabeling history after the fact is miserable.
6. Run both models in parallel for at least one full incident cycle. The gaps you find will be in the absence-detection layer, and you want to find them while the old up == 0 safety net still exists.

The one-line summary: pull-based monitoring fails loud, push-based monitoring fails silent — and the entire migration effort, beyond plumbing, is about re-introducing loudness on purpose. If you only carry your alert rules across unchanged, you haven’t migrated your monitoring; you’ve migrated your dashboards and quietly deleted your smoke detector.

This is a how-to for wiring that up on GitLab: every merge request gets its own preview page, with a “View app” button right on the MR. No review server to babysit, no extra infrastructure — and it works on the Free plan, because it doesn’t rely on publishing multiple GitLab Pages sites at once.

The example is lifted from a real Terraform + Ansible infrastructure pipeline I maintain, where each MR renders an HTML “infrastructure report” so reviewers can see what a change produces instead of reading raw plan output.

Why not parallel deployments

The obvious way to do per-MR previews is GitLab Pages parallel deployments (pages.path_prefix), which host each branch under its own URL prefix. It’s clean — and on many setups it’s either gated behind a paid tier or a recent GitLab version. If you’re on the Free plan, you typically get one Pages deployment: your default branch.

So we use a different lever that exists on every plan:

GitLab serves a job’s artifacts over the web. If a CI job uploads an HTML file as an artifact, GitLab gives you a URL that renders it — sandboxed on the Pages domain. Point a merge-request environment at that URL and you have a per-MR preview without ever publishing a second Pages site.

The build artifact is the preview. No parallel deployment required.

How the pieces fit

1. A build job produces the page into public/ and uploads it as an artifact (with a short expire_in, since previews are disposable).
2. A review job declares an environment whose url points at that artifact, served on the Pages domain so the HTML renders inline.
3. auto_stop_in + an on_stop job make the environment self-clean.

The environment block is the magic: attaching one to a job is exactly what makes the View app button appear on the merge request.

Step 1 — build the page into an artifact

Nothing special; produce your HTML into public/ and upload it. Keep the default-branch publish and the MR preview as two jobs sharing the same build — here I’ll show a single build that both consume:

build-page:
  stage: build
  script:
    - mkdir -p public
    - ./generate_report.sh > public/index.html   # whatever produces your HTML
  artifacts:
    paths:
      - public
    expire_in: 1 day        # previews are disposable
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    - if: $CI_MERGE_REQUEST_IID

expire_in: 1 day matters: the preview lives only as long as the artifact does, so you’re not accumulating stale renders.

Step 2 — publish the real site on the default branch

On the default branch, do a normal Pages publish. This is your “production” page and the baseline reviewers compare previews against:

pages:
  stage: deploy
  script:
    - echo "Publishing to ${CI_PAGES_URL}"
  needs:
    - job: build-page
      optional: true
  artifacts:
    paths:
      - public
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  environment:
    name: "Production page - $CI_DEFAULT_BRANCH"
    url: "$CI_PAGES_URL"

(The pages job name is special — it’s what triggers GitLab’s built-in Pages deployment. The script is a formality; the artifact is what gets published.)

Step 3 — the per-MR preview (the actual trick)

Here’s the review job. It runs only on merge requests and points the environment URL at the build artifact, served on the Pages domain:

pages:review:
  stage: deploy
  script:
    - echo "Creating review environment for ${CI_MERGE_REQUEST_SOURCE_BRANCH_NAME}"
  needs:
    - build-page
  artifacts:
    paths:
      - public
  rules:
    - if: $CI_MERGE_REQUEST_IID
  environment:
    name: "Review page - ${CI_MERGE_REQUEST_SOURCE_BRANCH_NAME}"
    url: "${CI_PAGES_URL}/-/jobs/${CI_JOB_ID}/artifacts/public/index.html"
    auto_stop_in: 1 hour
    on_stop: stop_review

The url line is the whole point, so it’s worth dissecting:

• ${CI_PAGES_URL} is your project’s Pages root — e.g. https://group.gitlab.io/-/subgroup/project. Using the variable instead of hardcoding the host keeps it correct across renames and custom domains.
• /-/jobs/${CI_JOB_ID}/artifacts/public/index.html is GitLab’s artifact-browsing path. CI_JOB_ID ties the URL to this job’s artifacts, so each MR pipeline gets its own preview.
• Serving it through the Pages domain (the group.gitlab.io host) is what makes the browser render the HTML. That detail is the difference between a working preview and a downloaded file — see the gotcha below.

auto_stop_in: 1 hour tells GitLab to retire the environment automatically, and on_stop: stop_review names the teardown job.

If you’ve ever hardcoded that artifact URL with the literal hostname and left yourself a “replace with variables” TODO — ${CI_PAGES_URL} is the variable. For a project at group/subgroup/project it expands to exactly https://group.gitlab.io/-/subgroup/project, so the composed URL matches the hand-written one character-for-character.

Step 4 — the stop job

on_stop needs a real job to point at. It does almost nothing — its existence and action: stop are what let GitLab (and the manual “Stop environment” button) close the environment:

stop_review:
  stage: .post
  script:
    - echo "Stopping review environment for ${CI_MERGE_REQUEST_SOURCE_BRANCH_NAME}"
  rules:
    - if: $CI_MERGE_REQUEST_IID
      when: manual
  environment:
    name: "Review page - ${CI_MERGE_REQUEST_SOURCE_BRANCH_NAME}"
    action: stop

The environment.name must match the review job’s exactly, or GitLab won’t know which environment this job stops.

The full thing

stages:
  - build
  - deploy

build-page:
  stage: build
  script:
    - mkdir -p public
    - ./generate_report.sh > public/index.html
  artifacts:
    paths:
      - public
    expire_in: 1 day
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    - if: $CI_MERGE_REQUEST_IID

# Real Pages site on the default branch
pages:
  stage: deploy
  script:
    - echo "Publishing to ${CI_PAGES_URL}"
  needs:
    - job: build-page
      optional: true
  artifacts:
    paths:
      - public
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  environment:
    name: "Production page - $CI_DEFAULT_BRANCH"
    url: "$CI_PAGES_URL"

# Per-MR preview served from job artifacts
pages:review:
  stage: deploy
  script:
    - echo "Creating review environment for ${CI_MERGE_REQUEST_SOURCE_BRANCH_NAME}"
  needs:
    - build-page
  artifacts:
    paths:
      - public
  rules:
    - if: $CI_MERGE_REQUEST_IID
  environment:
    name: "Review page - ${CI_MERGE_REQUEST_SOURCE_BRANCH_NAME}"
    url: "${CI_PAGES_URL}/-/jobs/${CI_JOB_ID}/artifacts/public/index.html"
    auto_stop_in: 1 hour
    on_stop: stop_review

stop_review:
  stage: .post
  script:
    - echo "Stopping review environment for ${CI_MERGE_REQUEST_SOURCE_BRANCH_NAME}"
  rules:
    - if: $CI_MERGE_REQUEST_IID
      when: manual
  environment:
    name: "Review page - ${CI_MERGE_REQUEST_SOURCE_BRANCH_NAME}"
    action: stop

Open an MR and the pipeline produces a View app button linking to that MR’s freshly built page. The default branch keeps publishing the real site at the Pages root, so reviewers can compare the two side by side.

Gotchas

• Pages must be enabled for the project, even though the preview rides on artifacts rather than a published Pages site — the artifact-serving URL lives on the Pages domain. No pages job has to have run yet; the domain just needs to exist.
• It only renders inline on the Pages domain. The same artifact reached via $CI_JOB_URL/artifacts/... (the main gitlab.com host) is served with a download disposition for HTML, so the browser saves the file instead of showing it. The ${CI_PAGES_URL}/-/jobs/... form is sandboxed on the Pages domain and renders inline. This is the reason the URL is built the way it is.
• environment.name must match between the review job and its stop job, character for character — including the branch-name interpolation. A mismatch silently breaks teardown.
• Link to a file, not a directory. Point at .../public/index.html, not .../public/; the artifact browser doesn’t do directory-index redirects.
• expire_in is your cleanup. When the artifact expires the preview 404s, which is fine — auto_stop_in closes the environment in parallel. Keep the two roughly aligned so the button doesn’t outlive the page it points to.
• Free-plan reality check. This exists precisely because you can’t publish a second Pages site on Free. If you’re on a tier/version with parallel deployments (pages.path_prefix), that’s the cleaner route — but this one works everywhere and costs nothing.

Recap

The setup: self-hosted Loki receiving logs from a handful of Kubernetes clusters, rules in a ConfigMap synced by a sidecar, alertmanager from a sub-chart of the same Helm umbrella. Names (my-loki, my-alert, my-channel) are anonymized.

TL;DR

A LogQL-based alert sat in a ConfigMap, watched by a sidecar, mounted into the ruler container — and silently produced nothing for ten weeks. Four serial bugs, each masking the next:

1. The sidecar wrote the file to /etc/loki/rules/ but the ruler expects /etc/loki/rules/<tenant>/.
2. The line filter used {label=~".*"} which Loki rejects (“empty-compatible regex”), so the rule never parsed.
3. The line filter caught one phrasing of the error log but not the other.
4. The alertmanager_url resolved to a service that didn’t exist — the chart’s sub-chart had a longer name than the umbrella’s release name.

Each layer failed silently. The metrics endpoint told the whole story; the logs told none of it.

The setup

ConfigMap (rules)                            alertmanager service
   ↑                                              ▲
   │ (kustomize configMapGenerator)               │
   │                                              │
+-----------------------------------+   alertmanager_url  +--------------------+
| loki-backend (SimpleScalable)     | ───────────────────►| alertmanager       |
|                                   |                     | (from a sub-chart) |
|   sidecar ──writes──► emptyDir    |                     +--------------------+
|              /etc/loki/rules/...                                ▼
|   ruler ──reads──► same emptyDir                          Slack route
|              /etc/loki/rules/...
+-----------------------------------+

Standard “ConfigMap → emptyDir → ruler” pattern. The chart values point the ruler at the alertmanager’s in-cluster DNS name.

The rule

A simple log-based alert: when a particular fatal log line appears, page the team.

- alert: my-alert
  expr: |
    sum by (instance) (rate({app=~".*"} |= "fatal: store corruption" [5m])) > 0
  for: 0s
  labels:
    severity: warning
    maturity: wip
  annotations:
    summary: "Fatal store corruption on {{ $labels.instance }}"

for: 0s means it should fire on the first eval after a match. Couldn’t be simpler.

The discovery

I was investigating an unrelated regression and noticed something off: a known-bad pod had been crash-looping for hours, log lines were piling up in Loki containing exactly the substring the alert was supposed to catch, and the team’s WIP-alerts channel was completely silent.

Slack search across the whole instance, bots included, for the alert name: zero hits since the rule had been merged ten weeks earlier.

That’s a long time for “the alert never fired”. Time to find out why.

Layer 1 — The rule on disk

The first thing to check is whether the rule file actually lands where the ruler reads from. The container is distroless (no shell, no ls), but the sidecar is a Python container:

kubectl -n monitoring exec my-loki-backend-0 -c sidecar -- python3 -c \
  'import os; [print(os.path.join(r,f)) for r,_,fs in os.walk("/etc/loki/rules") for f in fs]'

/etc/loki/rules/my-alert.yaml

Sidecar did its job. File is there.

Now ask the ruler what it has loaded:

kubectl exec … -- python3 -c \
  'import urllib.request, json;
   r=urllib.request.Request("http://127.0.0.1:3100/prometheus/api/v1/rules",
                            headers={"X-Scope-OrgID":"anonymous"});
   print(json.dumps(json.loads(urllib.request.urlopen(r).read()), indent=2))'

{"data": {"groups": []}, "status": "success"}

Zero groups. The ruler can see no rules at all, even though the file is right there on disk.

The fix: Loki’s local rule store doesn’t scan a flat directory; it scans <directory>/<tenant_id>/<rule_file>. With auth_enabled: false, the ruler’s implicit tenant is anonymous. Files placed directly under the configured directory, with no tenant subdirectory, are silently ignored. The chart’s sidecar.rules.folder needed to be /etc/loki/rules/anonymous, not /etc/loki/rules.

(Note: Loki uses fake as the tenant on the query path and anonymous on the ruler path. The two are not interchangeable.)

Move the file under the tenant subdir and check again.

Layer 2 — The ruler refuses to parse the file

{"data": {"groups": []}, "status": "success"}

Still empty. Now to the logs:

kubectl logs my-loki-backend-0 -c loki --since=30m | grep -E 'rule|ruler' | head

level=error caller=ruler.go:579 msg="unable to list rules"
  err="failed to list rule groups for user anonymous:
       error parsing /etc/loki/rules/anonymous/my-alert.yaml:
       parse error : queries require at least one regexp or equality matcher
       that does not have an empty-compatible value. For instance, app=~\".*\"
       does not meet this requirement, but app=~\".+\" will"

Exactly the regex used in the rule: {app=~".*"}. Loki refuses to load any rule file with an empty-compatible regex matcher, because such a query would have to scan every stream the instance knows about — billions of bytes, with no way to fingerprint a useful index.

The fix is to make the selector specific. The right answer here was switching to a label that’s actually meaningful ({app="the-actual-app"}), which has the added benefit of dropping the candidate-stream count from ~3700 to ~30.

That was the bug the ruler complained about, in plain English, in the logs — but only after layer 1 was fixed. As long as the file was at the wrong path, the ruler wasn’t even attempting to parse it, so the parse-error log line never showed up.

Layer 3 — The line filter is too narrow

Now the rule loads. The ruler logs show it evaluating every 15 seconds. But the alert still doesn’t fire.

Sanity-check the expression directly against Loki. Pull a sample of recent matches:

TOKEN=…   # service account token
DS=https://grafana.example.com/api/datasources/proxy/uid/<loki-uid>

curl -fsS --get \
  --data-urlencode 'query={app="my-app"} |= "fatal: store corruption"' \
  --data-urlencode "start=$(date -u -d '7 days ago' +%s)000000000" \
  --data-urlencode "end=$(date -u +%s)000000000" \
  --data-urlencode "limit=10" \
  -H "Authorization: Bearer $TOKEN" "$DS/loki/api/v1/query_range" | jq '.data.result[]'

Three matches in the last week. None on the day I was looking at the known-bad pod.

A look at the crash-looping pod’s actual log lines shows why: the application emits two different phrasings of the same logical error, only one of which contains the substring the alert filters on. The first phrasing comes from one code path (fatal: store corruption), the second from another (Store_corruption — capitalised, underscored — bubbled up as part of an exception type name).

|= is a case-sensitive substring match. Of course it didn’t catch the second phrasing. Switch to a regex alternation:

|~ "fatal: store corruption|Store_corruption"

…and the count over the last week jumps from “the few I found” to “thousands” — matching almost exactly the duration of the crash-loop event.

Layer 4 — The notifier dispatches into the void

Now the rule is loaded, the selector is correct, and the expression evaluates to a non-zero rate during the crash-loop window — verified directly:

# At the time the pod was crash-looping
curl --data-urlencode 'query=<expr> > 0' --data-urlencode "time=<that timestamp>" …

[{"metric":{...},"value":[<ts>,"0.0033333333"]}]

0.003 > 0 → alert should be firing.

Slack: still zero. The wip channel receives plenty of other alerts; it’s not the channel.

The answer is in the ruler’s metrics. There’s a family of loki_prometheus_notifications_* counters that count attempts the ruler makes to dispatch to alertmanager:

kubectl exec … -c sidecar -- python3 -c \
  'import urllib.request;
   [print(l) for l in urllib.request.urlopen("http://127.0.0.1:3100/metrics").read().decode().split("\n")
    if "loki_prometheus_notifications_" in l and not l.startswith("#")]'

loki_prometheus_notifications_alertmanagers_discovered{user="anonymous"}              1
loki_prometheus_notifications_sent_total{alertmanager="http://…/api/v2/alerts"}       2590
loki_prometheus_notifications_errors_total{alertmanager="http://…/api/v2/alerts"}     2590
loki_prometheus_notifications_dropped_total{user="anonymous"}                         2590
loki_prometheus_notifications_latency_seconds{quantile="0.5"}                         NaN
loki_prometheus_notifications_latency_seconds_count                                   2590

Every attempted dispatch — 2,590 of them — errored out, was dropped, and produced a NaN latency quantile because no request ever succeeded long enough to record a duration. And alertmanagers_discovered=1 only means the URL parsed, not that it pointed at anything real.

Why? DNS:

kubectl exec … -c sidecar -- python3 -c \
  'import socket; print(socket.gethostbyname_ex("the-configured-alertmanager-service.monitoring.svc.cluster.local"))'

socket.gaierror: [Errno -2] Name does not resolve

The chart we use packages alertmanager as a sub-chart of a larger observability umbrella. When the umbrella is installed under a given release name, the sub-chart synthesises its service name from the umbrella’s name plus its own component tag — not just the umbrella’s release name. So kubectl get svc -n monitoring | grep alert shows a service name with an extra segment in the middle, several characters longer than what the original author of the rule had hand-typed into the values file.

The notifier never logged the DNS failure (DNS errors are info-level in the upstream notifier code, and the bundled binary doesn’t surface them). The only place the failure appeared was in the metrics endpoint — and even there, the counters initially read as “lots of activity, all happening”, which is the wrong intuition until you look at errors_total and dropped_total side by side.

One-character fix to the values file (well, one segment) and we wait for the pods to roll.

Verification

loki_prometheus_notifications_errors_total{alertmanager="…right-name…"}  0
loki_prometheus_notifications_dropped_total                              0
loki_prometheus_notifications_sent_total                                 0  (no firings yet — the pod stopped crash-looping)
loki_prometheus_notifications_latency_seconds_count                      0

A finite latency quantile would have been even better evidence, but we’d need a firing to populate it. The right read on a freshly-rolled pod is: errors_total and dropped_total stop incrementing, the alertmanager target label in the metric carries the new URL, and alertmanagers_discovered is 1.

Why all four hid behind each other

Each layer fails silently in its own particular way, and each masks the next:

1. Wrong path → the ruler can’t find the file. There’s no error log because the ruler isn’t even trying to read this file — it’s scanning a directory the file isn’t in. You can’t see the parse error from layer 2 until you fix layer 1.
2. Bad regex → the ruler tries to parse the file, fails, and logs the error. But you only notice the parse-error logs after you’ve fixed layer 1 and started looking. From the outside, layer 1 and layer 2 produce identical symptoms (“the alert isn’t loaded”).
3. Narrow line filter → the rule loads and evaluates, but the query returns zero. You can only tell whether that’s “everything is fine, no real event” or “your regex is too narrow” by running the query directly and visually inspecting actual log lines for variant phrasings.
4. Wrong alertmanager URL → the notifier dispatches, increments sent_total (which counts attempts, not successes), and silently fails. The DNS error is buried at info-level. Only the errors_total / sent_total / dropped_total triplet exposes the truth.

None of the four layers emits a warning at the level the operator would see by accident. Every one needs a specific assertion when the rule is first wired up. The metrics endpoint, not the logs, is the source of truth at the dispatch layer.

What to do differently

Bench-test new rules before merging. Drop a vector(1) > 0 test rule in for ten minutes after every ruler-related change and watch it land on Slack. A passing test there proves all four layers at once. Remove it before merging the real rule.

Monitor the notifier metrics. A Grafana panel of loki_prometheus_notifications_errors_total per alertmanager target is five lines of PromQL. Without it, 2,590 failed dispatches look like 2,590 successful attempts until you check errors_total and dropped_total side by side.

The whole chain — ConfigMap, sidecar, ruler, notifier, alertmanager, Slack — looks like a single declarative pipeline. It isn’t: each link is a separate process making independent decisions, and each fails without surfacing an error at the level an operator watches. Write the diagnostic chain down somewhere durable; you’ll need it again.

[EROR] failure (cid=N, error="resolution failed: unknown scheme")
[EROR] response (cid=N, code=500)

The path from that error to a one-line fix involved being publicly wrong about the cause first.

I’ll use generic names: my-service (the OCaml HTTP server), my-service:latest (its image), https://hooks.example.com/... (an outbound URL it tries to POST to). Stack: OCaml 5.2 + cohttp-lwt-unix 6.2 + conduit-lwt-unix 8.0 + a multi-stage Dockerfile that copies the built binary onto debian:bookworm-slim.

Symptoms

The service has a /metrics Prometheus endpoint. After the first outbound failure, its counters looked like this:

my_service_http_requests_total{method="POST",path="/inbound/slash",code="200"} 14
my_service_http_requests_total{method="POST",path="/inbound/interactivity",code="500"} 1

Server-side, every successful slash-command POST handled itself — read, decode, build JSON response, write — without ever touching the network. The 500 was the first request that needed to make an outbound HTTPS call (post a webhook back to the platform’s response_url). The bot logged the URL it was about to call:

[DBUG] send HTTP request (url="https://slack.com/api/chat.postMessage", verb="POST")
[EROR] failure (cid=N, error="resolution failed: unknown scheme")

The URL string is plainly well-formed: https://, host, path. So why “unknown scheme”?

Wrong diagnosis #1: “TLS isn’t linked”

My first hypothesis was that the binary lacks TLS support. Plausible: cohttp-lwt-unix alone is just an HTTP-over-TCP library; for https:// you need a TLS implementation (tls-lwt for pure OCaml, or lwt_ssl for OpenSSL). If you forget to depend on one, Conduit_lwt_unix has no resolver registered for the https scheme.

I checked the published binary for TLS symbols by running:

docker run --rm --entrypoint /bin/sh my-service:latest \
  -c 'strings /usr/local/bin/my-service | grep -iE "tls|x509|ssl|cert"'

The output was empty. I took that as proof that no TLS library was linked, opened a PR adding tls-lwt and ca-certs to the dune (libraries …) stanza, and wrote a confident description that traced the symptom to missing TLS linkage.

The check was wrong. The runtime image is debian:bookworm-slim, which doesn’t ship strings — so the command silently produced no matches because binutils wasn’t installed, not because the symbols were absent. I’d run the check inside the wrong container.

The reviewer’s pushback

A reviewer caught it:

The PR description claims tls-lwt wasn’t being linked into the executable, but the project already lists tls-lwt in the (executables …) stanza. If outbound HTTPS is still failing with unknown scheme, it likely means the specific Conduit HTTPS resolver/initializer module isn’t getting linked or initialized.

I re-checked, this time extracting the binary onto the host (docker cp out of a container created with docker create, then running strings on the host where it actually exists):

$ cid=$(docker create my-service:latest)
$ docker cp "$cid":/usr/local/bin/my-service /tmp/my-service-bin
$ docker rm $cid

$ strings /tmp/my-service-bin | grep -cE "^(Tls|X509|Mirage_crypto|Tls_lwt|X509_lwt)(__|$)"
1871

$ strings /tmp/my-service-bin | grep -iE "^Tls_lwt|^X509|^Mirage_crypto_rng_unix" | sort -u | head
Mirage_crypto_rng_unix
Tls
Tls_lwt
Tls__Core
Tls__State
Tls__Utils
X509
X509__
X509__Crl
X509__P12

Plenty of TLS code. So the binary was fine. Conduit_lwt_unix 8.0 also already lists tls-lwt and ca-certs in its META requires, so they get pulled in transitively from cohttp-lwt-unix. The first diagnosis was wrong; the binary has had HTTPS support all along.

Which meant my open PR was wrong too, and I had to figure out the actual cause.

Reading Conduit’s source

The error string “resolution failed: unknown scheme” isn’t in my code or my dependencies’ *.ml files outside conduit. The path through conduit looks like this:

conduit-8.0.0/lib/conduit/resolver.ml:

let resolve_uri ?rewrites ~uri t =
  match Uri.scheme uri with
  | None -> return (`Unknown "no scheme")
  | Some scheme -> (
      t.service scheme >>= function
      | None -> return (`Unknown "unknown scheme")
      ...

So unknown scheme fires when t.service "https" returns None. The default t.service is system_service, defined in conduit-lwt-unix-8.0.0/lib/resolver_lwt_unix.ml:

let is_tls_service =
  function
  | "https" | "imaps" -> true
  | _ -> false

let system_service name =
  Lwt.catch
    (fun () ->
      Lwt_unix.getservbyname name "tcp" >>= fun s ->
      let tls = is_tls_service name in
      let svc = { Resolver.name; port = s.Lwt_unix.s_port; tls } in
      Lwt.return (Some svc))
    (function Not_found -> Lwt.return_none | e -> Lwt.reraise e)

Lwt_unix.getservbyname "https" "tcp". That’s the libc getservbyname(3) syscall, which reads /etc/services:

$ man 5 services
DESCRIPTION
       services is a plain ASCII file providing a mapping between human-friendly
       textual names for internet services, and their underlying assigned port
       numbers...

If the file is absent, getservbyname returns NULL, OCaml raises Not_found, system_service returns None, conduit returns Unknown "unknown scheme". Independently of TLS. Independently of the URI scheme too, if the OS lookup for that scheme fails.

Confirming the actual cause

Check the file in the running container:

$ docker run --rm --entrypoint /bin/sh my-service:latest -c 'ls -la /etc/services'
ls: cannot access '/etc/services': No such file or directory

And on a stock debian:bookworm-slim:

$ docker run --rm --entrypoint /bin/sh debian:bookworm-slim -c 'ls -la /etc/services'
ls: cannot access '/etc/services': No such file or directory

debian:bookworm-slim doesn’t ship /etc/services. The Debian package that provides it is netbase, which is not in the slim base. Same story with iana-etc on Alpine, and most distroless variants.

The fix

One line in the runtime stage of the Dockerfile:

 RUN apt-get update && apt-get install -y --no-install-recommends \
     libgmp10 \
     libssl3 \
     zlib1g \
     ca-certificates \
+    netbase \
  && rm -rf /var/lib/apt/lists/*

That’s it. No dune changes, no opam changes, no code changes. The Dockerfile had been omitting one OS-level data file the OCaml runtime depends on, and cohttp-lwt-unix translated the absence into a misleading-looking library-level error.

Verifying it actually works

I went back and proved the fix end-to-end with a controlled reproducer. Built both the broken and fixed images, ran each one, and sent a properly HMAC-signed POST to the interactivity endpoint:

import hmac, hashlib, time, urllib.parse, urllib.request, json

secret = b"signing-secret"
payload = {
    "type": "block_actions",
    "user": {"id": "U1", "team_id": "T", "username": "test", "name": "test"},
    "team": {"id": "T", "domain": "d"},
    "channel": {"id": "C", "name": "c"},
    "container": {"type":"message","message_ts":"1","channel_id":"C","is_ephemeral":True},
    "trigger_id": "T",
    "is_enterprise_install": False,
    "response_url": "https://hooks.example.com/actions/T/1/x",
    "actions": [{"type":"users_select","action_id":"a","block_id":"b",
                 "selected_user":"U2","action_ts":"1"}],
}
body = "payload=" + urllib.parse.quote(json.dumps(payload), safe="")
ts = str(int(time.time()))
sig = "v0=" + hmac.new(secret, f"v0:{ts}:{body}".encode(), hashlib.sha256).hexdigest()

req = urllib.request.Request(
    "http://localhost:8080/inbound/interactivity",
    data=body.encode(),
    headers={"Content-Type": "application/x-www-form-urlencoded",
             "X-Slack-Signature": sig,
             "X-Slack-Request-Timestamp": ts},
    method="POST",
)
try:
    print("HTTP", urllib.request.urlopen(req, timeout=10).status)
except urllib.error.HTTPError as e:
    print("HTTP", e.code)

The fake response_url (https://hooks.example.com/actions/T/1/x) doesn’t exist on the platform, so the platform’s edge will return 404 — but that’s a success indicator for the experiment. If the outbound HTTPS call goes through at the TLS layer, we’ll see a 4xx body coming back from the platform. If it fails at conduit’s resolver, we’ll see the original unknown scheme and HTTP 500.

Broken image:

HTTP 500
[INFO] request (cid=0, verb="POST", path="/inbound/interactivity")
[WARN] add assignee (cid=0, ..., result="no such alert")
[EROR] failure (cid=0, error="resolution failed: unknown scheme")
[EROR] response (cid=0, code=500)

Fixed image:

HTTP 200
[INFO] request (cid=0, verb="POST", path="/inbound/interactivity")
[WARN] add assignee (cid=0, ..., result="no such alert")
[EROR] platform error (cid=0, error="HTTP 404 Not Found")
[INFO] response (cid=0, code=200)

The HTTP 404 Not Found is the progress signal: TLS handshake succeeded, DNS resolved, the platform’s edge returned the expected error for the fake URL. The transport layer works.

Lessons

There’s one bug here and two takeaways.

1. Slim base images strip OS data files that runtime libraries read.

netbase (provides /etc/services, /etc/protocols, /etc/rpc), tzdata (/usr/share/zoneinfo/*), iana-etc on Alpine — none of these are in the minimal images, and lots of libraries quietly need them. The breakage looks like a library bug because the error originates inside the library; the root cause is the missing data file the library expected.

A small audit list before deploying any slim/distroless image with HTTP/TLS/timezone/protocol-name behaviour:

• Outbound HTTPS or HTTP with named ports → /etc/services → netbase
• Timezone-aware logging or scheduling → /usr/share/zoneinfo → tzdata
• DNS via name resolution → /etc/nsswitch.conf may be needed → libc-bin
• TLS verification → CA bundle → ca-certificates

Each of these is one Debian/Alpine package. Each is one line in the Dockerfile.

2. “I checked” is not the same as “I checked correctly.”

My first check — strings | grep for TLS symbols — was the right idea executed wrong. The runtime container didn’t have strings, so an empty output meant “couldn’t run the check,” not “no symbols found.” I read the result as proof and shipped a wrong diagnosis on the back of it.

What I should have done before claiming the binary lacked TLS: extract the binary out of the container to a host that has strings, or use apk add binutils / apt install binutils inside, or use a different tool entirely (nm, readelf). When a verification check returns the answer you expected, that’s exactly when to double-check that the check actually ran.

The reviewer’s “but the project already lists tls-lwt” was the cheap nudge that re-opened the diagnosis. Without it I would have shipped the wrong PR. Cheap nudges from reviewers who are reading the same code are worth more than confidence in your own reasoning.

TL;DR

If you ship an OCaml binary using cohttp-lwt-unix or anything else that reaches for Lwt_unix.getservbyname and put it on debian:bookworm-slim, your outbound calls will die with resolution failed: unknown scheme. Add netbase to the runtime stage. One line.

And if you’re staring at a binary trying to prove a negative (“symbol X isn’t here”), make sure the tool you’re using to look exists in the same image you’re looking inside.