# K8s Monitoring Widget — Design Spec

**Date:** 2026-04-22
**Status:** Approved

---

## Context

The user hosts several applications (joplin, n8n, grocy, dnddrugs, shared-postgres) on a Kubernetes cluster (ITSH Cloud, namespace `tenant-5`). There is no always-open terminal or dashboard for at-a-glance cluster health. The goal is a Quickshell bar widget that surfaces pod health and resource pressure without leaving the desktop.

**Constraints discovered during design:**
- `kubectl get nodes` is forbidden — the service account only has namespace scope
- `kubectl top pods` works — metrics-server is available
- `kubectl get resourcequota` works — quota has both requests and limits tracked
- Pods absent from `kubectl top` (e.g. dnddrugs) should show `—`, not an error

**Dependencies:** `kubectl` and `jq` must be in `$PATH` on the host.

---

## Files

### Create

| File | Purpose |
|---|---|
| `scripts/executable_k8s-status.sh` | Fetches pod list + ready status → JSON (30s poll) |
| `scripts/executable_k8s-metrics.sh` | Fetches kubectl top + resourcequota → JSON (15s poll) |
| `shared/Kubernetes.qml` | Singleton: owns all k8s state, runs both scripts, exposes properties |
| `bar/KubernetesPill.qml` | Bar pill: k8s icon + ready/total pod ratio, color-coded by health |
| `bar/popouts/KubernetesPopout.qml` | Popout: per-pod table + quota bars |

> The `executable_` prefix is the chezmoi convention (see `scripts/executable_gpu.sh`). Chezmoi maps these to plain filenames at apply time with the executable bit set.

### Modify

| File | Change |
|---|---|
| `Bar.qml` | Import `KubernetesPill`, add `PopoutSlot`, add `kubernetesSlot.animating` to popout `visible` guard |
| `shared/Config.qml.tmpl` | Add `kubeNamespace`, `kubeStatusRefreshMs`, `kubeMetricsRefreshMs` |

---

## Scripts

Both scripts accept the namespace as `$1` and are invoked from QML as:
```qml
command: ["bash", Config.scriptsDir + "/k8s-status.sh", Config.kubeNamespace]
```

### `scripts/executable_k8s-status.sh`

Runs: `kubectl get pods -n "$1" -o json`

Extracts per pod:
- `app`: from `metadata.labels["app.kubernetes.io/instance"]`, fallback to `metadata.name`
- `ready`: `true` if `status.conditions` contains `{type: "Ready", status: "True"}`

Outputs JSON:
```json
{
  "pods": [{"app": "joplin", "ready": true}, ...],
  "readyCount": 4,
  "totalCount": 4
}
```

On any error: exits non-zero, outputs nothing. The QML handler checks `this.text.trim() === ""` to detect this.

### `scripts/executable_k8s-metrics.sh`

Runs two commands sequentially:
1. `kubectl get pods -n "$1" -o json` — builds a `podName → app` label map
2. `kubectl top pods -n "$1" --no-headers` — parses into `{podName → {cpuM, memMi}}`
3. `kubectl get resourcequota -n "$1" -o json`

**Pod name → app mapping:** `kubectl top` returns raw pod names (e.g. `joplin-7bfd5bc954-jq47w`). The metrics script resolves each top pod name to its `app.kubernetes.io/instance` label using the pod list from step 1, producing `podMetrics` keyed by `app` label. Pods in the top output that have no matching pod object are skipped.

**Unit normalization for quota values:** All CPU values are normalized to millicores (e.g. `"1"` → `1000`, `"500m"` → `500`). All memory values are normalized to MiB (e.g. `"4Gi"` → `4096`, `"512Mi"` → `512`, raw bytes → `/ 1048576`).

From resourcequota `status.used` and `status.hard` (after normalization), computes:
- `cpuActualM`: sum of CPU millicores from `kubectl top` output (actual live usage, not from resourcequota)
- `memActualMi`: sum of MEM MiB from `kubectl top` output (same — actual, not requests)
- `cpuReqPct`, `cpuLimPct`: `used / hard` as 0.0–1.0 fractions
- `memReqPct`, `memLimPct`: same for memory
- Human-readable labels (e.g. `"78m / 2000m"`, `"1.01Gi / 4Gi"`) for display

Outputs JSON:
```json
{
  "podMetrics": [{"app": "joplin", "cpuM": 13, "memMi": 343}, ...],
  "quota": {
    "cpuActualM": 29,
    "cpuReqPct": 0.04, "cpuLimPct": 0.73,
    "memActualMi": 921,
    "memReqPct": 0.25, "memLimPct": 0.54,
    "cpuReqLabel": "78m / 2000m",
    "cpuLimLabel": "1450m / 2000m",
    "memReqLabel": "1.01Gi / 4Gi",
    "memLimLabel": "2.14Gi / 4Gi"
  }
}
```

Pods not present in `kubectl top` output get `cpuM: -1, memMi: -1` (displayed as `—`).

On any error: exits non-zero, outputs nothing.

---

## Singleton: `shared/Kubernetes.qml`

Root element must be `Singleton` (the Quickshell type), matching every other file in `shared/`:

```qml
pragma Singleton
import Quickshell
import Quickshell.Io
import QtQuick

Singleton {
    id: root
    // properties, Processes, Timers here
}
```

### Properties

| Property | Type | Description |
|---|---|---|
| `status` | string | `"loading"` \| `"ok"` \| `"degraded"` \| `"error"` \| `"stale"` |
| `pods` | var | Array of `{app, ready, cpuM, memMi}` — merged from both scripts |
| `readyCount` | int | Number of ready pods |
| `totalCount` | int | Total pod count |
| `quota` | var | Quota object from metrics script |
| `lastUpdatedSecs` | int | Seconds since last successful **status** fetch |

### Health logic

- `"loading"` (blue/accent): no data yet — initial state before first fetch completes
- `"ok"` (green): all pods ready, last status fetch < 60s ago
- `"degraded"` (yellow): any pod not ready
- `"error"` (red): status script failed and no prior data
- `"stale"` (overlay0): last successful status fetch > 60s ago (connectivity lost). Transitions back to `"ok"` or `"degraded"` on the next successful parse.

Metrics staleness (15s cadence) is not tracked separately — if metrics are stale, pods show `cpuM: -1` (displayed as `—`), which is sufficient signal.

### Polling

Two independent `Process` + `StdioCollector` pairs:

| Process | Script | Interval | Timer start |
|---|---|---|---|
| `statusProc` | `k8s-status.sh` | 30 000ms | `Component.onCompleted` |
| `metricsProc` | `k8s-metrics.sh` | 15 000ms | Staggered 500ms after status (via single-shot Timer) |

**Merge:** After each metrics fetch, the Singleton iterates `podMetrics` and patches `cpuM`/`memMi` into the existing `pods` array by matching on `app`. Pods in `pods` with no match in `podMetrics` retain `cpuM: -1, memMi: -1`.

**Error handling in `onStreamFinished`:**
```js
if (this.text.trim() === "") {
    // script failed
    root.status = (root.status === "ok" || root.status === "degraded") ? "stale" : "error";
    return;
}
try {
    let data = JSON.parse(this.text);
    // update properties
    root.status = "ok"; // or "degraded" if any pod not ready
} catch(e) {
    root.status = (root.status === "ok" || root.status === "degraded") ? "stale" : "error";
}
```

---

## UI Components

### `bar/KubernetesPill.qml`

Follows the `BarPill` pattern exactly (see `WeatherPill.qml`):

```
groupName: "kubernetes"
accentColor: status → green(ok) / yellow(degraded) / red(error) / overlay0(stale) / blue(loading)
content: [
  Text { icon glyph 󱃾 (U+F03BE, Nerd Font k8s wheel), font.family: iconFont },
  Text {
    text: status === "loading" ? "…" : readyCount + "/" + totalCount
    color: accentColor
  }
]
```

### `bar/popouts/KubernetesPopout.qml`

Two sections:

**Pods table** — one row per pod:
- Columns: `app name | CPU | MEM | ● Ready/NotReady`
- CPU/MEM shown as `13m` / `343Mi`; shown as `—` if `cpuM === -1`
- Ready: green dot + "Ready"; red dot + "NotReady"

**Quota section** — two cards (CPU, MEM), each card contains:
- Header: label + actual value (`29m actual` / `921Mi actual`)
- `%REQ` bar: `cpuReqLabel` + percentage + colored bar (green / `Theme.green`)
- `%LIM` bar: `cpuLimLabel` + percentage + colored bar (yellow dimmed / `Theme.yellow` at 60% opacity)

Footer: `Updated Xs ago` from `lastUpdatedSecs`.

### `Bar.qml` wiring

```qml
// In pill column:
BarComponents.KubernetesPill {}

// PopoutSlot (add alongside existing slots):
PopoutSlot {
    id: kubernetesSlot
    name: "kubernetes"
    verticalAnchor: "bottom"
    sourceComponent: Popouts.KubernetesPopout {}
}

// Update the popout PanelWindow visible guard (existing line, add the new condition):
visible: notifSlot.animating || mediaSlot.animating || weatherSlot.animating
      || datetimeSlot.animating || systemSlot.animating || kubernetesSlot.animating
```

---

## Config additions (`shared/Config.qml.tmpl`)

```qml
readonly property string kubeNamespace: "tenant-5"
readonly property int kubeStatusRefreshMs: 30000
readonly property int kubeMetricsRefreshMs: 15000
```

---

## Error Handling Summary

| Scenario | Pill | Popout |
|---|---|---|
| First load | blue `…` | empty / loading indicator |
| All pods ready | green `4/4` | normal |
| Any pod NotReady | yellow `3/4` | affected row shows red dot |
| Status script fails (first time) | red `0/0` | error message |
| Status script fails (had data) | overlay `4/4` | stale indicator + last known data |
| Metrics script fails | unchanged | CPU/MEM show `—` |

---

## Verification

```bash
# 1. Test scripts standalone (after chezmoi apply)
~/.config/quickshell/scripts/k8s-status.sh tenant-5 | jq .
~/.config/quickshell/scripts/k8s-metrics.sh tenant-5 | jq .

# 2. Apply and restart quickshell
chezmoi apply ~/.config/quickshell
# kill and relaunch quickshell

# 3. Visual checks:
# - Pill appears with correct ready/total count and green color
# - Click pill → popout opens with pod table (joplin/n8n/shared-postgres show CPU/MEM, dnddrugs shows —)
# - Quota bars match: kubectl top pods -n tenant-5 and kubectl get resourcequota -n tenant-5 -o json

# 4. Degraded state (transient, brief outage):
kubectl rollout restart deployment/joplin -n tenant-5
# → pill turns yellow, joplin row shows NotReady during rollout
# → returns green once pod is Ready again

# 5. Stale state (simulate connectivity loss):
# Temporarily rename kubeconfig or revoke access, wait >60s → pill turns overlay color
```
