When a pod’s init container crashes, Kubernetes restarts only the init container — not the whole pod. The emptyDir volume survives between retries. If your init container does a git clone into a fixed path, the second attempt fails with “destination path already exists.”

Fix: rm -rf the target dir before cloning.

rm -rf /git/repo
git clone --depth=10 --branch=main https://... /git/repo

After many restarts, no manual cleanup needed. Events expire in ~1h, old pods are replaced automatically by the Deployment controller. Check recovery with:

kubectl get events -n <namespace> --sort-by='.lastTimestamp' | tail -10

A “CPU spike” that was actually memory thrashing (adding GA4 to Hugo)

Wanted Google Analytics on this blog. PaperMod already calls a google_analytics.html partial in head.html, but it’s gated behind hugo.IsProduction | or (eq site.Params.env "production"). My blog pod runs hugo server, which always reports the environment as development — so the partial never fires. I “fixed” that by setting env = "production".

That was the wrong lever. env = production flips on Hugo’s whole production path — minification, OpenGraph, Twitter cards, schema JSON across every page. The next full rebuild blew past the pod’s 128Mi memory limit and got OOMKilled (exit 137). Server load jumped.

The right way to add GA without touching the build mode: drop the tag in layouts/_partials/extend_head.html. PaperMod includes that partial unconditionally, above the production guard — so it loads under hugo server too.

But here’s the part that fooled me. After reverting env, load was still climbing — to ~14 on a single node — and ps showed hugo at “500% CPU”. Looked like a runaway compute loop. It wasn’t:

%Cpu(s): 2.1 us, 41.0 sy, 6.9 id, 50.0 wa     <- 50% iowait, 2% userspace
PID ... S  %CPU  COMMAND
... D  333  hugo    <- state D, RES pinned at 127MiB (the 128Mi cgroup limit)

Two lessons:

1. ps %CPU is a lifetime average, not instantaneous. A process that ran hot for 1s then blocked still shows a big number for a while. Use top for what’s happening now.
2. High load + high %wa + a D-state process sitting at its cgroup memory limit = memory thrashing, not CPU. Hugo wasn’t computing — it was wedged against the 128Mi ceiling, and every allocation triggered kernel reclaim/swap. A sub-second build dragged out for minutes in uninterruptible I/O sleep, and all those blocked tasks are what inflate load average (Linux counts D-state in load).

The actual fix was boring: 128Mi was always marginal for hugo-extended + PaperMod. Bumped the limit to 512Mi and the thrash vanished.

Takeaway: when load spikes, read %wa and process state before blaming the CPU. And don’t flip env=production on a long-lived hugo server just to ungate one partial — use extend_head.html.

Self-hosting Supabase (lean) on k3s: the gotcha checklist

Ran the community supabase/supabase chart on a 16Gi single node — enabled db, rest, auth, meta, studio, kong + the log pipeline (analytics/Logflare + vector); left realtime, storage, imgproxy, edge-functions off. The deploy is easy; these are the things that actually bit:

• Studio shows “no tables”. Supabase is single-database by design — Studio, PostgREST and auth all use the database named postgres. App tables in a separate database are invisible to all of it. Put your schema in postgres’s public schema.
• Studio won’t schedule with edge-functions disabled. Its Deployment mounts the functions PVC unconditionally. Either run functions, or create the PVC yourself and leave functions off.
• edge-functions crashloops if you keep it: it boots by fetching a Deno module from the internet, which a deny-all egress policy blocks. You usually only want the PVC it leaves behind anyway.
• vector (log collector) stays silent under a deny-all policy. It discovers pods via the Kubernetes API, so it needs API egress, not just app ports (allowEgressToKubeApi). A log shipper that can’t reach the API collects nothing and doesn’t say why.
• secretRef must contain every key the chart maps — including non-secret ones like database and openAiApiKey. Miss one and pods sit in CreateContainerConfigError.
• ESO ExternalSecret shows perpetual OutOfSync in Argo CD unless you spell out the remoteRef defaults (conversionStrategy: Default, decodingStrategy: None, metadataPolicy: None) — ESO writes them back, and the compact form drifts.
• postgres is not a superuser. CREATE DATABASE … OWNER app fails with must be member of role. Supabase keeps the real superuser (supabase_admin) to itself; GRANT app TO postgres first.
• Logflare needs no BigQuery. It runs on the self-hosted Postgres backend (the _supabase database, _analytics schema) — logs land in _analytics.log_events_*.

None of this is in the README. It’s the gap between “I deployed Supabase” and “I run it.”

The obvious next move is to point an agent at the outside — let it read the web and tell me what matters. That move is also a small landmine, and most “AI reads the internet for you” tooling steps right on it. So this week I built two of them instead of one, named them after corvids, and the reason there are two is the entire point of this post.

Meet the Magpie and the Blue Jay.

The same fear, twice

Before either bird got a name, both inherited a single non-negotiable rule, and it’s worth saying plainly because it’s the part everyone skips:

An agent that reads the internet and writes to your notes is a prompt-injection pipeline aimed straight at your trust root.

My vault isn’t just storage. Every other agent — the gardener, the Wanderer, the search that answers “what am I building?” — reads it as trusted context. So the moment one agent ingests a GitHub README or a news headline (attacker- influenceable text) and is allowed to write a note, a stranger on the internet gets to whisper instructions into the thing my whole system believes. “Structured API” narrows that surface. It does not close it.

Both birds are built on the same chassis as the gardener, and that chassis enforces the fear rather than trusting the model to behave:

• Two phases, hard split. A wrapper-owned FETCH step pulls the external text in plain Bash — Claude is not in the loop, can’t be talked into anything, because it isn’t running yet. Then a COLLIDE step starts claude -p with the fetched text handed in as inline data, and that process gets only Read / Glob / Grep / Write. No Bash, no git, no network, no MCP. While untrusted text is in the context window, the agent has no tool that can reach the outside world or rewrite history.
• Allowlist, not the open web. Each bird reads a short, named list of sources. Nothing else.
• Quarantine, not the vault. Findings land in quarantine/<bird>/, which lives outside vault/. The indexer never sees it. Nothing it writes is ever auto-wikilinked into the graph. Promotion to a real note is a thing I do, by hand, after reading it.
• Blast radius is checked, not assumed. A run may modify only its quarantine directory. Anything written anywhere else is discarded and reported as a violation.
• “Nothing found” is a successful run. Neither bird has a quota. This is the honesty contract I stole from the Wanderer — an agent under pressure to produce N findings will manufacture N findings, and manufactured insight is worse than silence.

That’s the shared spine. Now the interesting part: given the same security model, the two birds do almost opposite things, and trying to make one bird do both jobs would have quietly ruined it.

The Magpie hoards what’s already shiny

A magpie collects shiny objects and keeps them close. Mine watches my own GitHub stars.

The premise is slow public signal × private context. I starred some repo three weeks ago, forgot about it, and moved on. Meanwhile my projects shifted. The Magpie runs weekly, pulls my starred repos through one allowlisted endpoint (gh api user/starred), and collides each one against what I’m actively building right now — the live projects, the open hubs.

Its output contract is a tight one: it is a relevance filter. It fires only when a star actually touches live work, and every finding has to name three concrete things — the repo, the project it connects to, and one “so what.” A vague “these are thematically related” doesn’t count as a hit. It’s a watchdog on the dials, not a newsletter.

The supervised proof run, over 28 stars, surfaced exactly two real hits and refused to invent a third:

1. supertonic (on-device multilingual TTS) × my Hungarian-audiobook voice-cloning project — a possible escape from a TTS fight I’d been losing. I checked: it genuinely supports Hungarian. That’s a hit with a so-what.
2. agentmemory × the exocortex itself — prior art for persistent AI memory, notably with benchmarks my own notes lacked. (And if you’ve read about the time I benchmarked my own search and it lost, you’ll know how much I needed that nudge.)

The other ~22 stars mapped to tidy thematic clusters and were correctly not reported. That restraint is the feature.

The Blue Jay scatters acorns and forgets where

Here’s the bird that explains why there are two.

Blue jays don’t hoard close like magpies. They cache acorns far and wide and forget where they buried some — and the forgotten ones grow into oak trees. Ecologists think blue jays are why oak forests spread north after the last ice age. Seed dispersal, by way of a bad memory. That is exactly the job I wanted for the second bird, and the metaphor was too good to pass up.

The Blue Jay reads an allowlist of eight RSS feeds, picked so tech and science cross-pollinate:

• Tech: Hacker News (high-score front page), lobste.rs, Ars Technica
• Science & ideas: phys.org, Quanta, Aeon, Nautilus
• Wildcard: Medium — but scoped to specific tag feeds, never the raw firehose of crypto and self-help

Quanta, Aeon, and Nautilus are on that list on purpose: they’re the connective tissue, the feeds where “huh, that’s weirdly similar to…” happens before my vault even gets involved.

And its output contract is the opposite of the Magpie’s. The Blue Jay is a serendipity filter. Its job is to surface the connection that isn’t in my projects yet — the distant idea, the acorn worth burying. If I ran it through the Magpie’s “only fire on a live-work hit” rule, I would strangle the one thing it exists to do. Relevance and serendipity pull in opposite directions, and you can’t tune a single agent to maximize both.

One more load-bearing detail, half design and half security: the Blue Jay collides on the RSS summary only — title, abstract, link. It never pulls the full article body into context. That’s simultaneously the lower-injection path and the right cognitive shape (a headline is a seed; I click through myself from quarantine if the seed is interesting). The narrow input is doing double duty.

Why two birds and not one with a flag

I genuinely considered making this one agent with a --mode=relevance|serendipity switch. I’m glad I didn’t, and the reasoning generalizes past birds:

Two things made the split non-negotiable. First, the output contracts are too different to share one brain — “only speak on a hit” and “speak about the thing that isn’t a hit yet” are contradictory prompts, and a single agent told to do both does neither well. Second, open news is a higher injection frontier than a structured stars API, so the riskier bird deserves its own enforced blast-radius wrapper, not a code path bolted onto the safe one. When two jobs disagree on both what good output is and how dangerous the input is, that’s not a flag. That’s two programs.

So now my vault has two more agents reading the world on a cron. The Magpie runs Saturday at 06:00 and tells me when something I bookmarked finally became relevant. The Blue Jay runs Saturday at 07:00 and buries acorns in a quarantine folder, most of which I’ll ignore — but I only need one of them to grow into an oak.

Both are on probation for their first few runs, because I don’t trust a thing that reads the internet until I’ve watched it behave. But the part I’m actually happy about isn’t the agents. It’s that building the second one forced me to say out loud what the first one was secretly assuming — and the names made the difference impossible to forget. A magpie hoards. A blue jay scatters. You want both, and you do not want them to be the same bird.

Combining lexical search (BM25) with vector similarity and graph expansion produces more robust recall than embeddings alone.

That sentence shipped into production. My vault of markdown notes gets indexed into a search database, and the search function fuses three signals: BM25 (classic keyword ranking), vector similarity (embeddings), and graph expansion — when a note matches, pull in its linked neighbours too, on the theory that the thing you want is often next to the thing you typed.

It sounds right. Graphs are having a moment in RAG. “Add a knowledge graph to your retrieval” is the kind of thing you can put on a slide and nobody pushes back. I believed it enough to make graph expansion a first-class signal with a weight of 0.5 — equal footing with keyword matching.

This week I finally wrote a benchmark. The graph wasn’t helping. It was the single biggest thing hurting my search.

The setup

30 gold queries against the live vault (63 notes), borrowing the harness shape from an eval framework I’d been reading. Each query has a hand-labelled “correct” note. I measured recall@5 (did the right note land in the top 5?) and MRR (how high did it rank?), across three retrievers:

• grep — naive substring term-count. The dumb floor.
• bm25 — pure keyword ranking, FTS5’s BM25. The honest baseline.
• live — my production hybrid (BM25 + vector + graph).

I expected a clean staircase: grep at the bottom, bm25 in the middle, my clever hybrid on top. That’s the whole reason you build the clever thing.

The scorecard

Read that bottom row twice. My production “smart” search found the right note 65% of the time. Plain keyword search found it 95% of the time. The hybrid I’d been quietly proud of was worse than its own baseline — it broke 9 of 30 queries that BM25 got right. BM25 alone whiffed on exactly one.

The clever layer wasn’t adding intelligence. It was adding noise, confidently.

Why the graph backfired

Here’s the mechanism, and it’s almost funny once you see it.

Graph expansion pulls in a matched note’s neighbours. But in a real knowledge base, the most connected notes are hubs — my inbox of ideas, my project radar, my “things Claude noticed” log. Everything links to them, so they’re everyone’s neighbour. When I searched for something specific, the graph helpfully dragged these popularity-contest winners into the candidate set, and they elbowed the genuinely relevant note clean out of the top 5.

Concrete example. Query: “who owns this knowledge system?” The correct answer is my personal note. BM25 ranked it #5 — just barely in. The hybrid, drunk on graph neighbours, pushed it off the list entirely. The graph didn’t find a better answer. It buried a good one under hubs.

I swept the graph weight to confirm it wasn’t a fluke. It was perfectly monotonic — every increment of graph made search worse:

There’s no ambiguity to argue with. More graph, more harm, no exceptions. The value I’d been claiming in that confident note — I finally measured it, and it was negative.

The fix, and the actual lesson

The fix was one line: drop the default graph weight from 0.5 to 0.1. Recall snapped back to 0.95, tying pure BM25. (Turning the graph fully off is marginally better still on MRR; I kept a whisper of it as a tiebreaker, which is a taste call, not a data-driven one.)

But the one-line fix isn’t the point. The point is where graphs belong.

Graph expansion isn’t a bad idea — I aimed it at the wrong job. Precision retrieval (“find me the one note that answers this”) wants to be narrow and literal. Pulling in neighbours is the opposite of what you want; every neighbour is a chance to be wrong. But I have a different feature in this same system — a discovery mode that deliberately collides distant notes to surface unexpected connections. There, neighbour-pulling isn’t noise, it’s the entire product.

Same mechanism. One context it’s poison, the other it’s the point. I’d been running my discovery tool inside my lookup tool and calling it a hybrid.

A few honest caveats, because a benchmark you can’t poke holes in is usually lying: my gold set is self-authored v1, the corpus is small (63 notes), and the vector signal was actually dark during this run — I hadn’t built the embeddings yet, so “hybrid” here was really “BM25 + graph.” The vector half of my original claim is still untested. This is directional, not gospel.

But directional was enough. I’d shipped a claim, the claim got measured, and it didn’t survive contact with 30 queries. That’s the whole reason I keep my brain in git with everything reproducible: so the day I bother to measure, the measurement can actually win the argument against my own confident prose.

The slide-deck version of RAG says add a graph. The benchmark says know which question you’re answering first. I’ll take the benchmark.

My house runs on quiet little robots. A tracker watches my kombucha ferment. A job narrates kids’ books in Hungarian. A media stack pulls and files things. Home Assistant minds the sensors. A dozen services, all doing their jobs, all completely mute. When a batch finished or an import failed, I found out the same way every time: by going to look.

Then the silence got expensive. Claude Code stopped dead in the middle of a task because I’d burned through my plan’s usage window — no warning, no countdown, just a wall. The information existed; a dashboard in my own cluster was already polling it. It just had no way to reach my pocket.

So I built one thing: a push bus. One place anything in the cluster can POST to, that actually buzzes my phone. And the first job I gave it was to warn me before my AI assistant goes dark.

The boring part (said honestly)

The bus is ntfy — a self-hosted pub/sub notifier. Picking it took about five minutes, because self-hosting ntfy for a homelab is a thoroughly solved problem. There are at least three off-the-shelf bridges from Prometheus Alertmanager to ntfy. I’m not going to pretend the bus is the clever bit.

What I did do deliberately:

• 📦 Deployed it GitOps-native — one entry in my app-of-apps, reconciled by Argo CD, no docker run anywhere.
• 🔒 Locked it to deny-all auth with bearer tokens. Security alerts ride this bus; a world-readable topic on a public URL was a non-starter. (Which also means it sits outside my usual OAuth gate — the phone app can’t do an interactive login flow, so ntfy does its own token auth.)
• 🏷️ Topics by severity: hl-crit, hl-warn, hl-info, hl-event. Subscribe and mute by how much I care.

Then the interesting parts showed up at the edges, where they always do.

Edge one: my own firewall 403’d me

First test, the usage producer POSTing to https://ntfy.hippotion.com:

HTTP 403 Forbidden
error code: 1010

That 1010 looks like ntfy rejecting my token. It isn’t. It’s Cloudflare. Error 1010 means “your browser signature is banned” — Cloudflare’s bot protection took one look at a Python script’s urllib User-Agent and slammed the door.

My own producer couldn’t reach my own bus, because the request left the cluster, went all the way out to my own edge, and got flagged as a bot on the way back in.

The fix is the architecture I should’ve had from the start: in-cluster producers POST to the internal service address and never touch the public internet at all.

# wrong: out to Cloudflare and back, gets bot-blocked
https://ntfy.hippotion.com/hl-warn

# right: stays inside the cluster
http://ntfy.web-ntfy.svc.cluster.local/hl-warn

The phone still uses the public URL happily — the real ntfy app carries a signature Cloudflare trusts. Only scripts trip 1010. Lesson: your own edge is not your friend when you’re a script. Keep cluster traffic in the cluster.

Edge two: the obvious data source was lying

To warn me about Claude usage, the naïve move is to parse Claude Code’s local logs — they sit right there in ~/.claude/projects/.../*.jsonl, token counts and all.

Don’t. Those counts are unreliable for accounting — known to undercount, wildly, in some cases by ~100x. Every tool that parses that JSONL inherits the bug.

The number that’s actually true lives in the claude.ai usage API — the same five_hour and seven_day windows your plan enforces against. And I already had a service polling exactly that. So the producer is just a tiny sidecar on that existing pod, reading its /api/usage over localhost (same pod — no network policy to negotiate, no second credential, nothing else hammering claude.ai):

• 📈 ≥80% of a window → hl-warn (high).
• 🚨 ≥95% → hl-crit (urgent).
• 🔁 One ping per window per reset cycle, escalating warn→crit, keyed on the reset timestamp so it never spams.

The first time it mattered, my phone buzzed at 80% with hours of runway left instead of a brick wall mid-task.

What I’d tell past me

Three things, none of them about ntfy:

1. Reuse the signal you already have. I didn’t build a usage poller — I bolted a sidecar onto the one already running. The smallest producer is one that reads localhost.
2. Your own edge can betray you. A firewall that protects you from bots will happily block your own automation. In-cluster talks in-cluster.
3. Check whether your data source is telling the truth before you build an alert on it. An alert you don’t trust is worse than no alert — you’ll learn to ignore it, and then it’ll be right once.

Next, the high-leverage move: point Prometheus Alertmanager at the same bus, and every infra alert I have — plus every one I’ll ever add — lands on the phone through one bridge. The kombucha ping can wait. The disk-full one can’t.

The house is still full of quiet robots. The difference is now they know my number.

It started as a nervous-Sunday kind of question: is a third party trying to get into my server — over SSH, or some other way? I run a single-node Kubernetes homelab that hosts a couple dozen little apps, some of them public. You read about credential-stuffing bots and you start to wonder who’s been rattling the handle while you slept.

So I did the audit. The good news came first, and it’s worth saying plainly because it’s the part most homelabs get wrong: the front door is solid. Nothing is reachable from the internet except through a Cloudflare Tunnel — an outbound-only connection, zero open inbound ports on my router. Almost every service sits behind OAuth. The cluster has 140 network policies doing real east-west segmentation. And the login history? Eleven straight weeks where every single shell login came from one IP — my own workstation on the LAN. No strangers. No 3 a.m. logins from a VPS in another hemisphere.

I could have stopped there feeling good. That would have been a mistake.

The scary finding wasn’t an attacker

The useful question turned out not to be “is someone knocking?” but “if someone got in, would anything tell me?” And when I traced that wire, it ended in the dark.

I have a full monitoring stack — Prometheus, Grafana, Alertmanager, the works. Alertmanager was running. It was also configured to notify exactly no one: no receivers, and upstream, no alert rules at all. It was a smoke detector with the battery taken out and, for good measure, no smoke sensor either. If an attacker had walked in, the alarm would have stayed perfectly, silently green.

That reframed the whole job. Three gaps, in priority order.

Gap 1 — an alarm with no one to call

I built the missing chain end to end. A small exporter on the host parses the SSH journal and fail2ban state and writes metrics into node_exporter’s textfile collector — so it rides the monitoring I already had instead of adding a new moving part. On top sit the alert rules that were never there. The one that matters most is blunt:

A shell login succeeded from a non-LAN IP.

That should be impossible in normal life, so if it ever fires, I want it shouting. It now emails me the instant it happens, alongside quieter alerts for brute-force spikes, distributed scans, fail2ban going down, and — the meta-alert I’m fondest of — the watchdog itself going stale, because a security monitor that silently dies is worse than none. And fail2ban now actually bans the bots, with escalating ban times and my LAN permanently on the allow-list.

The honest lesson: I’d been treating “I have Prometheus” as if it meant “I have monitoring.” Dashboards you have to remember to look at are not monitoring. Monitoring is the thing that interrupts you. Until an alert can reach your phone, you don’t have a security alarm — you have a security museum.

Gap 2 — there was a web terminal on the open internet

This is the one that made me wince. Among my public hostnames was ttyd — a browser-based shell. A full terminal on my server, reachable from anywhere, sitting behind a single OAuth proxy. One misconfiguration, one OAuth bypass, and that’s not “an app is compromised,” that’s root on the box from a browser tab.

The fix here isn’t more locks. It’s the realization that the strongest control is not exposing the thing at all. I deleted the web terminal entirely — app, manifests, dashboard tile, all of it. Then I went down the public hostname list and pulled everything with no business being public off the tunnel: the secrets UI, the ingress dashboard, Prometheus, Alertmanager, the network-observability console, the DNS admin. They still work — on my LAN, over the same wildcard cert — they’re just not the internet’s business anymore. A service that isn’t exposed has no attack surface to harden.

Gap 3 — no floor under the blast radius

The network policies limit how far a compromised pod can talk sideways. But nothing stopped a workload from running as root, mounting the host filesystem, or grabbing the host network in the first place. So I turned on Kubernetes' built-in Pod Security Admission: every namespace now at least reports baseline violations, and the clean app namespaces enforce baseline — meaning a compromised app there simply cannot request privileged mode or a hostPath mount. It’s a floor. Floors are underrated.

What the audit was really about

I went looking for an intruder and didn’t find one — the logs were clean, the front door held. What I found instead was that I’d built something secure at the perimeter and then never asked the uncomfortable follow-up: what happens after the perimeter? The answer had been “nothing happens, and no one is told,” and I just hadn’t looked.

Three principles I’m taking with me:

• An alarm that can’t reach you is decoration. Wire the notification first; the rules are easy once something is listening.
• Don’t expose it beats add more auth. Every hostname you take off the public internet is a class of attack you no longer have to be clever about.
• Give the blast radius a floor. Assume one thing gets popped, and decide in advance how far it gets.

The best part: all of it is GitOps. The intrusion alerts, the un-exposing, the pod-security floor — every change is a commit, reviewable and revertible, and my cluster reconciles itself to match. The audit didn’t just make the homelab safer. It wrote down why it’s safer, in a form the next version of me can read.

Now if someone knocks, I’ll know. And the web terminal isn’t answering the door anymore — because it’s gone.

Every vote in the European Parliament and the Slovak National Council is public. The EU even ships it as a clean API. And almost nobody reads it, because the raw record is unreadable: “Návrh poslanca… ktorým sa dopĺňa zákon č. 581/2004 Z. z. … (tlač 1259) — tretie čítanie, hlasovanie o návrhu zákona ako o celku.” Multiply that by a few hundred votes a sitting. Transparency that no human can parse is transparency on paper only.

So I built VoteWatch — a small site on my homelab that turns the record into something a citizen can actually use: what was decided, who voted, and do you agree?

Two halves, one lopsided

The EU half was easy. HowTheyVote.eu already did the hard work and publishes roll-call votes as a clean, open-licensed API. You consume it; you don’t scrape it.

The Slovak half is where the real work lives — and the real value. nrsr.sk has no API. The HTML is the contract: a results listing, and per-vote pages where each MP appears next to a one-letter code ([Z] za, [P] proti, [?] zdržal sa). So the national half is a genuine scraper — the unglamorous kind that nobody maintains, which is exactly why a gap exists to fill. The unglamorous part is the moat.

From ten votes to one question

A single bill generates a pile of procedural roll-calls — shorten the debate, move to third reading, amendment block A, amendment block B, the bill as a whole. Ten rows that are really one decision. Nobody wants ten rows.

So the pipeline groups votes by bill, then asks an LLM (llama-3.3-70b on NVIDIA NIM) to do exactly one job: turn the bureaucratic titles into a plain headline, two sentences of summary, and one neutral Yes/No question a person can actually answer. Seven votes on the health-insurer bill collapse into: “Changes to the health-insurance law” → “Do you agree with the health-insurance bill?”

The rule that keeps it honest

Here’s the line I won’t cross, and it’s the whole reason I trust the result: the AI writes the prose, but it never decides a fact.

• Which votes belong to one bill? Deterministic — parsed from the bill number.
• Did it pass? Deterministic — read from the result row.
• Which parties voted for, against, abstained? Deterministic — tallied from the per-MP record, shown as Za: SMER-SD, HLAS-SD, SNS · Zdržali sa: PS, KDH, SaS.

The model only touches language: the headline, the summary, the question. If it hallucinates, you get an awkward sentence — never a wrong vote count. And if the model fails entirely, the card falls back to the raw title. The facts come from the record; the model just makes the record legible. For civic data, that separation isn’t a nice-to-have — it’s the difference between a tool and a liability. (Every card says so out loud: summaries are AI-generated; the raw record prevails.)

The part that closes the loop

Showing people how their representatives voted is only half a feedback loop. The other half is letting them answer.

Each decision carries its one distilled question and two buttons — Áno / Nie. You vote, and the site shows the citizen tally next to how parliament actually decided, with the honest verdict on top: "✓ Citizens and Parliament agree" or "⚖ Gap between citizens and Parliament." That gap is the entire point. It’s the thesis behind a side project of mine called veracracy — governance measured against verified knowledge and the actual will of the governed — made concrete enough to click.

The backend is deliberately boring. The site is static (git-synced nginx, same as this blog). Votes can’t POST to a static page, so they go to a public n8n webhook that records to a data table and returns live tallies — no new service, no database, just the automation box I already run. Vote keys are namespaced so EU and Slovak polls share one store without colliding.

The honest caveat

Dedup is browser-local. It stops casual double-voting, but behind a Cloudflare tunnel every request shares one IP, so this is an indicative signal, not a secured ballot. That’s the right altitude for “let people express an opinion.” The day it needs to mean more than that, it needs real identity first — and I’d rather ship the honest version than fake the robust one.

It’s live at votewatch.hippotion.com — the EU parliament and the Slovak NR SR, every MEP and every poslanec, in plain language, with a button that asks the only question that matters after a vote: would you have voted the same way?

A neutral record — what was decided and who decided it — not a villain list. Data © HowTheyVote.eu (ODbL) and nrsr.sk.

Democracies ask what do we want? Markets ask what will we pay? Both are good questions, and between them they run most of the world. But there’s a third question that almost no system asks before it acts, and it’s the one that decides whether the first two produce anything good:

What do we actually know — and how do we know it?

I gave the idea of governing as if that question came first a name — veracracy, from veritas (truth) and kratos (rule). Not rule by experts in a back room. Rule by evidence that anyone can inspect, deliberated by the people it binds. It lives at veracracy.hippotion.com, and this is the honest account of what it is and why an infrastructure engineer ended up building a shrine to an idea.

What it actually means

Strip the romance and veracracy is five fairly concrete commitments:

• Truth as civic infrastructure. Verified, open evidence maintained like roads and water — with provenance, versioning, and repair crews. A public utility, not a content feed.
• Radical transparency. Binding decisions carry their evidence trail by default. A law without its sources is a claim, not a law.
• Decentralised trust. No ministry of truth. Verification is plural, adversarial, and bridging — many checkers, no single owner, consensus that has to span camps to count.
• Ethical AI as auditor and advocate. Machines that trace claims, surface contradictions, and argue against the powerful reading of the data — never as oracle, always as instrument.
• Participatory epistemocracy. Citizens not as voters once every four years, but as standing jurors of what is true enough to act on, where weight accrues to evidence rather than volume.

If you squint, none of that is a political program. It’s the same instinct I bring to a cluster — provenance, versioning, a reconciler, no single point of trust — pointed at the question of how a society decides what’s real. That’s the only way I know how to think, so that’s the lens I used.

Why a clock, and why it can run backward

Here’s where most “vision” projects lose me, and where I tried not to lose myself. A manifesto is cheap. Anyone can declare a better world and feel moral. So instead of a manifesto, the site is a measurement.

It shows a single year — first light, the year the first place on Earth might plausibly govern this way. Right now it reads around 2051. But that number isn’t a wish; it’s computed, from one assumption stated in plain sight: the infrastructure of verified governance has been built since the world went online in 1991, and continues at the average pace it has held since. A set of dials — open data, civic tech, verification at platform scale — each scored 0–1 from a named source, weighted, extrapolated. Change the assumption and the number moves: pace the same dials from Athens in 508 BC instead, and dawn lands near the year 3860. So the assumption is the lever, which is exactly why it’s published.

And the clock can run backward. There’s a Watch — a standing log of what moved the year — that deliberately files the evidence against: every transparency rollback, every deliberation experiment that failed, every force pushing dawn further out. Because a sunrise that only ever gets closer is a marketing widget. The honesty is the product. If I can’t show you the thing that would move the number the wrong way, you shouldn’t believe the number.

The first brick

Ideas this size are easy to admire and easy to never touch. So the rule I set myself is that veracracy has to cash out in things that actually run.

The first one shipped this week: VoteWatch — every roll-call vote in the European Parliament and the Slovak National Council, scraped from the public record, distilled into plain language, showing which party voted which way, with a button that asks you whether you’d have voted the same. It’s the third and fifth pillars made clickable: binding decisions carrying their evidence trail, and citizens as standing jurors rather than spectators. The gap it surfaces — between how parliament voted and how the people who answered would have — is veracracy in miniature, on real data, today.

It’s small. It’s one person’s homelab. The voting is an indicative signal, not a secured ballot, and I say so on the page. But it’s the difference between a belief and a brick, and I would rather lay one honest brick than write a beautiful manifesto.

Where this comes from

I’ll be straight about the shape of this: it’s idealistic, it’s personal, and I don’t expect to see first light. I’m a solo operator in a small town who runs a rack of servers and thinks too much about how systems stay trustworthy when no one’s watching them. Veracracy is what happened when that instinct refused to stay inside the server room.

The version of this I can defend isn’t the dream — it’s the discipline around the dream. Publish your assumption. Let the clock run backward. Cash the idea out in something real. Credit your sources. Ship the honest version, not the robust-sounding one.

A measurement with one stated assumption — not a prophecy. The clock’s at veracracy.hippotion.com; disagree with a dial and you’ve understood the point.

This week an AI agent told me something about my own systems that I’d never noticed, and it was correct: I have one favorite architecture, and I’ve built it three times.

• At work: git holds Terraform code → Terraform derives the S3 buckets. Nobody clicks around in the AWS console; the repo is the truth.
• In the homelab: git holds Kubernetes manifests → ArgoCD derives the cluster. Every app on my rack is a folder in a repo.
• In my second brain: a vault of markdown notes → an indexer derives the search database (SQLite FTS + a link graph) that my AI tools query.

Same shape everywhere: a plain-text source of truth in git, and a machine that builds the real thing from it. Master copy, derived state. I never decided this consciously — it’s just how my hands build things now.

GitOps isn’t the git part

Here’s the thing that the third copy got wrong, and it took me embarrassingly long to see because I teach this pattern at the infrastructure layer.

“Configuration in git” existed long before GitOps. What made GitOps an actual shift was the reconciler: ArgoCD doesn’t apply your manifests once and wish you luck. It watches, continuously. When the cluster drifts from the repo, you get an OutOfSync badge, and with selfHeal enabled it puts reality back where the repo says it should be. The loop is the product. Git is just where the loop points.

My vault had no loop. If I edited a note and forgot to rebuild the index, the search results my AI agents rely on were silently stale — no badge, no error, nothing. The only protection was a rule in the repo’s agent instructions: “if files and index disagree, the files win — run the indexer.”

A policy that agents must remember. In other words: I was running Kubernetes with a sticky note on the monitor that says please redeploy after editing the YAML. I would never accept that on my cluster. My brain ran on it for months.

The fix took an afternoon

Two pieces, both boring on purpose.

exo status — the OutOfSync badge. The indexer now stores a content hash per note; status re-hashes the vault and diffs:

{
  "status": "OutOfSync",
  "modified": ["vault/10-notes/interests-themes.md"],
  "new": [],
  "deleted": [],
  "repair": "exo index"
}

Exit code 0 when synced, 1 when not — so scripts and CI can ask the question too, exactly like argocd app get.

Git hooks — the selfHeal. Versioned hooks (core.hooksPath .githooks) on post-commit and post-merge rebuild the index after every commit and pull:

command -v exo >/dev/null 2>&1 || exit 0
EXO_ROOT="$(git rev-parse --show-toplevel)"
exo index >/dev/null 2>&1 && echo "exo: index reconciled (Synced)"

Now every git commit in the vault prints exo: index reconciled (Synced) on its way out. The rule didn’t change — files win — but it stopped being something agents must remember and became something a machine enforces. That’s the entire difference between configuration management and GitOps, replayed at the knowledge layer.

The part where it gets a little strange

The reason I’m writing this post at all: I didn’t have this idea. A scheduled agent did, on what I can only describe as an idle walk.

My vault has a weekly cron job — we call it the Wanderer — that samples pairs of notes that are far apart: different folders, different months, almost no shared vocabulary. A headless Claude gets the pairs with exactly one task: read both notes in full and say whether anything genuinely connects. “Nothing connects” is a successful run. That last sentence is load-bearing — the run always reports its result either way, so the agent never needs to manufacture a finding to have done its job.

On its very first walk, it collided a work note about Terraform-driven S3 provisioning with the architecture map of the vault itself, and wrote: same sentence in different clothes — and the brain copy is missing its reconciler. Then it listed the two fixes you just read about.

Retrieval answers the questions you ask. Distant collisions surface the questions you didn’t know you had. It turns out my second brain didn’t need to get better at remembering — it needed to occasionally interrupt me.

If you keep a vault

Whatever your stack — Obsidian, org-mode, a folder of markdown — if anything derives from your notes (an index, embeddings, a published site), then you have source of truth and derived state, and the GitOps question applies: who notices when they drift? If the answer is “I do, hopefully,” you’re running the sticky-note era. Give it a badge and a loop. It’s an afternoon.

I wanted a small web page on my homelab that shows my Claude usage — the 5-hour session window, the weekly limits, the per-model split. There’s a nice Electron widget out there that does this on the desktop, but I don’t want a desktop app; I want a URL behind my own OAuth that I can glance at from my phone.

The mechanics are unremarkable. The claude.ai web app reads those numbers from a couple of undocumented endpoints using your logged-in session cookie. So a self-hosted version does the same thing server-side: hold the session token as a secret, replay the same calls, cache the result, render some bars. An afternoon’s work. I was pairing with Claude Fable 5 on it — Anthropic’s newest model, and the one that ships with extra safety measures around dual-use capability.

Then, partway through, I got the message: Fable 5 flagged something in this session and switched to a more conservative model. It dropped me to Opus 4.8 for the rest of the conversation. Safe conversations sometimes trip it, the notice said. Send feedback.

I wasn’t doing anything wrong. That’s the interesting part.

My first reaction was the obvious one — what did I say? But I knew exactly what I’d built, and none of it was sketchy. It was my account, my usage data, my hardware, my OAuth in front of it.

So I went looking at the request the way a classifier would — not “what did he mean” but “what does this look like.” And from that angle it’s a different picture entirely. Stack up the surface features:

• 🔑 capturing a session token and storing it to replay later
• 🌐 sending it to an undocumented API that isn’t meant for third parties
• 🕵️ spoofing a browser User-Agent so the request blends in
• 🧱 detecting and working around a Cloudflare bot challenge

Read that list cold, with no context. That’s not a usage dashboard. That’s the exact signature of credential theft and scraping tooling. Every individual move is one a malicious script would also make. The only thing separating my afternoon project from the bad version is whose account it touches and why — and intent is precisely the part that doesn’t show up in the tokens.

Surface vs. intent

This is the part worth sitting with, because it’s not a Claude quirk — it’s the shape of every content classifier, every WAF rule, every fraud model I’ve ever run in production.

A detector scores what it can see. It cannot see intent; it sees features. And the features of “monitor my own usage” and “harvest someone else’s session” overlap almost completely, because the technique is identical — the difference lives entirely in context the model has been deliberately built not to over-trust. You can’t tune that gap away. You can only pick where to sit on the precision/recall curve, and Fable 5 — being the high-capability model with the extra dual-use measures bolted on — sits where it catches the pattern even when it costs some false positives, then hands off to Opus 4.8. I was the false positive. The system did roughly the right thing for roughly the right reason; it just doesn’t feel that way when it’s pointed at you.

The honest engineering takeaway is the one I keep relearning: if a benign task has the silhouette of an abusive one, expect to get treated like the silhouette. Not just by AI — by rate limiters, by bot detection, by the fraud team. The fix isn’t to be offended. It’s to recognize the silhouette, and where it matters, make the legitimate context legible up front.

What I’d do differently

Practically, very little — the project was fine, and it downshifted to a model that finished the job. But the framing changed how I built it. I leaned harder into the parts that make intent visible in the design: the session token never leaves the server, it lives in Vault and arrives as an injected secret, the whole thing sits behind OAuth, and it polls on a leash instead of hammering. Not because a classifier made me, but because those are the same choices that make it obviously a personal dashboard and not a harvesting bot — to a reviewer, to future-me, and yes, to a model reading over my shoulder.

The widget rides your credential on your desktop. Mine keeps it server-side behind my own front door. Turns out building it the trustworthy way and building it the legibly trustworthy way are the same work — and getting flagged is what made me notice the difference.

So I built mind-the-gap: a pipeline that measures what the market demands, diffs it against what I can prove, and renders the gap as a private dashboard on my cluster. The job hunt is now a monitored system. This post is about the non-obvious decisions.

Demand: an LLM reads job listings so I don’t have to

I already had a job poller — an n8n workflow that polls the public ATS APIs (Greenhouse / Lever / Ashby) of ~33 companies plus a broad remote-jobs feed every six hours. A sibling workflow now re-fetches the same boards and, for every listing that passes the role+location gate, asks a small hosted LLM (Llama-3.1-8B) for a structured extraction:

{"seniority": "senior", "skills": [{"name": "kubernetes", "importance": "must"}, ...]}

One row per (job, skill) lands in an n8n Data Table. Decisions that mattered:

• One LLM call per job, not one batch. Free-tier inference times out on batches; per-job calls are slower but fail independently. A lesson the poller already paid for.
• Insert doubles as the processed-marker. A job whose extraction fails to parse produces no rows — so it’s retried next run, for free. No status column, no second table.
• Canonicalization in code, not in the prompt. The model says “K8s”, “k3s”, “EKS” on different days regardless of instructions. A dumb alias map (k8s→kubernetes, eks→aws) beats prompt engineering for consistency.
• 8B is good enough — with a guard. It occasionally echoed the seniority enum back literally ("junior|mid|senior|staff|lead|unspecified"). The fix is one line of validation, not a bigger model.

Supply: no artifact, no credit

The other side of the diff is a skills registry — markdown in my knowledge vault, with a machine-parseable YAML block. Every skill has a state, and the rule that keeps the whole thing honest is brutal: a skill counts as proven only if an artifact exists — a public repo, a blog post, documented production experience. Otherwise it’s claimed, and claimed earns half credit.

That rule immediately produced the most useful insight of the project: “invisible skill” is a real category. Python turned out to be the market’s #5 ask. I use it constantly — and could point to nothing public that shows it. The cheapest score increase isn’t learning something new; it’s a weekend making an existing skill visible. No gut-feeling gap analysis would have ranked “write about what you already do” above “learn the shiny thing.”

The score: distinct companies, not mentions

First naive aggregation: Canonical’s listings mention Ubuntu nine times, all marked must-have — suddenly Ubuntu looks like the hottest skill in Europe. Employer skew is the noise floor of small samples. The fix: demand weight = distinct companies naming the skill, not total mentions. One enthusiastic employer can’t move the radar.

Two more scoring rules I’d defend in review:

• Skills named by fewer than two companies don’t count at all — single-listing noise stays out.
• Demand the registry hasn’t classified yet shows up as “unreviewed” and counts fully against the score. An unreviewed market signal is a gap until proven otherwise; the dashboard nags me to triage it.

Rendering: the page is a git commit

The dashboard is a single static HTML file, and the pipeline that produces it never touches the cluster. render.js lives in this repo as the single source of truth; a nightly n8n workflow fetches it raw from GitLab, eval()s it against the Data Table rows and the registry, and — only if the result differs from what’s committed (timestamps stripped, or every night is a “change”) — PUTs the new index.html back via the GitLab API.

Serving is the same pattern as this blog: nginx plus a git-pull sidecar, deployed by Argo CD, behind the cluster’s OAuth middleware. The renderer has no kubeconfig, no SSH, no cluster access of any kind. GitLab stays the only source of truth — even for a page that rewrites itself nightly. If the workflow goes rogue, the worst it can do is a reviewable commit.

Day-one verdict

First run: 2,297 postings fetched, 25 in scope, 257 skill rows. Coverage score: 63%. Kubernetes and AWS tied at the top of demand — which means the AWS gap-closing project already in flight stopped being a hunch and became the measured top of the market. Go is the only top-ten demand with zero supply. The dashboard doesn’t get anyone a job; it just makes sure every learning Saturday is pointed where the data says, not where the hype does.

The job board rejected me. The data didn’t.

Workflows, render.js, and setup: github.com/janos-gyorgy/mind-the-gap.

The interesting part is why it works at night.

The trick is the darkness, not the light

UV-C light (254 nm) shreds the DNA of fungal pathogens like powdery mildew. Nothing new there — it’s the same wavelength that sterilises water and hospital rooms. The problem is that in daylight those pathogens repair the damage, using a light-activated enzyme (photoreactivation). Zap them at noon and they patch themselves up by evening.

So you do it in the dark. With the repair pathway switched off, a tiny dose sticks. Cornell’s Gadoury lab spent years on this: nighttime UV-C at doses around 85 J/m² once a week gave season-long powdery mildew control on strawberries that beat the best available fungicides. Grapes, cucumbers, roses — same story. Applied about 30 minutes after sunset, finished within a couple of hours.

That’s a genuinely beautiful result. Not a new chemical, not a stronger lamp — just the same old light, applied when the enemy can’t fix itself.

What it is, and what it absolutely isn’t

Before anyone rips out their whole garden routine: this is not a general pesticide replacement. The evidence is strong for one specific class of problem — surface fungal pathogens, mostly powdery and downy mildew on susceptible plants (strawberry, grape, cucurbits, roses). It does nothing for slugs, most insects, or anything in the soil.

So the honest pitch is narrow: if you fight recurring mildew every summer, this is a chemical-free tool that genuinely works. If your real enemy is aphids, don’t build this — you’d be solving the wrong problem with a dangerous toy.

Which brings me to the toy being dangerous.

The part where I tell you not to blind yourself

UV-C is not mood lighting. Seconds of direct exposure burn your eyes (welder’s-flash) and skin, and it’s a long-term cancer risk. This is the single reason a home version has to be designed carefully — and the reason I’d never run an exposed source in a garden where my kids play.

Any home rig needs, non-negotiably:

• A physical enclosure or skirt so the light only hits the bed, never a person.
• A hard interlock — a motion sensor or door contact that cuts power instantly if anything moves into range.
• A schedule that only ever runs in the dead of night, when everyone’s inside and asleep.

You can also over-dose the plants — too much UV-C scorches leaves. The whole point is that the effective dose is tiny, so more is not better.

The build (the home version of “while you sleep”)

You don’t need TRIC’s autonomous navigation. A home garden has fixed beds — so the robot problem collapses into a much simpler one: get a shielded lamp over a known bed, for a known number of seconds, at night. That’s not robotics. That’s a timer and a rail.

Here’s the plan I’d build:

1. The lamp. A low-pressure UV-C tube (254 nm — not the “UV-C LED” novelties, and not ozone-generating 185 nm lamps). Mounted in a hooded reflector so the light points down and is blocked from the sides.
2. The geometry. Fix it at a set height over the bed — on a simple cart that rolls a track, or just a static fixture over a raised bed. Fixed height = repeatable dose.
3. The dose, measured not guessed. This is the one place you can’t wing it: borrow or buy a UV-C meter, measure the irradiance (W/m²) at canopy height, then time = 85 ÷ irradiance. If the lamp delivers, say, 5 W/m² at the leaves, that’s ~17 seconds of exposure. Seventeen seconds, once a week. That tiny number is the whole reason this is plant-safe and low-energy — and why a slow-moving robot pass is enough on a farm.
4. The brain. This is the bit that’s actually in my wheelhouse: an ESP32 + a relay, on the homelab. Fires at 2 a.m. for N seconds, once a week. A PIR sensor wired as a kill-switch. A mind-the-gap-style cron and a log line to my phone when it ran. The “autonomous robot working while you sleep” headline, minus the $100k of autonomy I don’t need for four raised beds.

Verdict

I haven’t built this yet — it’s a someday project, parked here so I stop losing the idea. But it’s the rare someday project where the science is settled, the materials are cheap, and the only real engineering is safety and dose control, both of which are squarely the kind of problem I like.

The farm robot’s pitch is “pesticide-free at scale.” The home version’s pitch is smaller and more honest: if mildew is your summer tax, you can pay it in seventeen seconds of midnight light instead of a spray bottle. I’ll take that trade.

When I build it, the failure log gets its own post.

My kid loves audiobooks. The commercial platforms barely carry Hungarian children’s books, and none of them carry the one narrator my kid actually prefers: me. I can’t read aloud every evening — but my homelab doesn’t have that excuse.

The platform half (ebook → M4B → Audiobookshelf on k3s) is a story for another post. This one is about the voice: how to go from a phone recording to an audiobook narrated in your own voice, step by step, on hardware with no GPU.

The short version: XTTS-v2 does zero-shot voice cloning from a ~20-second sample. No training, no fine-tuning, no dataset. One clean recording and a flag.

Why XTTS-v2, in 2026?

It’s not the best open TTS model anymore. Chatterbox beats ElevenLabs in blind tests; F5-TTS sounds cleaner. But model selection for a small language is constraint-first, not leaderboard-first: Chatterbox has no Hungarian, NVIDIA’s TTS NIMs have no Hungarian, Kokoro — no Hungarian. XTTS-v2 speaks Hungarian and clones voices and runs on CPU. That intersection has exactly one resident.

I run it via ebook2audiobook, which wraps XTTS with Calibre ingestion and M4B chaptering.

Step 1 — Record ~25 seconds of yourself

Phone voice-memo app, quiet room, ~20 cm from your mouth. Mine came out as 28 seconds of stereo 48 kHz AAC. Two rules that matter more than gear:

• Read the way you want the books narrated. The clone copies prosody — energy, pacing, warmth — not just timbre. A flat recital clones into a flat narrator. I read a children’s tale the way I’d read it at bedtime.
• Don’t peak the mic. My sample hit −0.1 dB max volume — right at the clipping ceiling. It worked, but quieter is safer. Check yours:

ffmpeg -i janos.m4a -af volumedetect -f null - 2>&1 | grep volume
# mean_volume: -21.4 dB   ← fine
# max_volume:  -0.1 dB    ← living dangerously

Step 2 — Normalize to what XTTS wants

XTTS expects a mono WAV; 24 kHz matches its internal rate. Trim the silence off both ends while you’re at it:

ffmpeg -i janos.m4a \
  -af "silenceremove=start_periods=1:start_threshold=-45dB:start_silence=0.2,\
areverse,silenceremove=start_periods=1:start_threshold=-45dB:start_silence=0.2,\
areverse" \
  -ar 24000 -ac 1 janos.wav

(The double-areverse is the classic trick: silenceremove only trims the front, so you flip the audio, trim the front again, flip it back.)

Drop the result where your TTS stack looks for voices. In ebook2audiobook that’s the voices/ tree, organised by language:

voices/hun/adult/male/janos.wav

Step 3 — Synthesize

One flag does the cloning. Headless run on the k3s pod:

kubectl exec -n web-audiobooks deploy/ebook2audiobook -- sh -c \
  'cd /app && python app.py --headless \
     --ebook "/app/ebooks/tale.txt" \
     --language hun \
     --tts_engine xtts \
     --device cpu \
     --voice /app/voices/hun/adult/male/janos.wav \
     --output_format m4b \
     --output_dir /app/audiobooks'

On my 12-core CPU node this runs at roughly 3× real-time — a 2-minute tale takes ~8 minutes, a full children’s book is an overnight job. The first run computes speaker latents from your WAV; after that it’s ordinary synthesis with your voice as the reference.

Step 4 — A/B before you batch

Render one short book twice — stock narrator and cloned voice — and put both in front of the household jury. Cloning quality is personal in the most literal sense: MOS scores won’t tell you whether it sounds like you. My benchmark has strong opinions and goes to bed at eight.

Only after the clone passes do you re-render the library with --voice.

The manual steps that earn the word “manual”

Things the tutorials skip, learned the slow way:

• Long conversions die with the browser tab. Gradio-style web UIs tie the job to the open page; close the laptop and you get “Conversion cancelled” half a book in. Anything longer than ~15 minutes of audio runs headless under nohup.
• CPU synthesis leaks memory over hours. My pod has a hard 6 Gi limit on a 16 Gi node, and a 6-hour run will hit it. Keep the cap (it protects the other 30 namespaces), and rely on the tool’s --session <id> resume — it picks up at the exact sentence. One catch: headless resume still asks an interactive Resume? [y]es — pipe echo y | into it.
• The per-chapter FLACs survive a crash. If the final M4B muxing step OOMs, don’t re-synthesize: the chapters are sitting in the session’s tmp directory, and ffmpeg will assemble them into a chaptered M4B with a hand-written FFMETADATA file in about two minutes, at near-zero memory.

None of this is hard. It’s just undocumented — which is the gap between “there’s a model for that” and your kid pressing play.

Postscript: the jury came back

The clone failed. Recognizably my timbre, nowhere near natural — I wouldn’t play it to my kid, which is the only metric that exists for this project.

Worth being precise about what failed: the stock XTTS-v2 narrator passed the ear test and the library keeps growing with it. Zero-shot cloning is the part that fell short — a 2023 model conditioning on 26 seconds of a voice it has never seen, in a language that was never its strong suit. The pipeline above is still the right pipeline; the model isn’t there yet on CPU-class options.

The next experiment is already picked: F5-TTS Hungarian, a 2026 fine-tune on 280 hours of actual Hungarian speech, built precisely for short-sample cloning. It needs CUDA, which my node doesn’t have — but a rented spot GPU tests it for the price of an espresso. If it passes the bedtime jury, that’ll be its own post.

Negative results are results. The jury reconvenes when the GPU shows up.

Not the files — those are fine. The connections rot. You capture a note at 11pm and never link it to anything, so it becomes an orphan floating off the graph. A project note’s one-line summary describes what the project was three weeks ago. Two notes are obviously about the same thing and neither knows the other exists. Do this for a few months and you don’t have a second brain, you have a junk drawer with good search.

The honest fix is to weed the garden regularly. The honest truth is that nobody does, including me.

So I stopped relying on myself and built a gardener.

What it actually does

Every night at 3am, on my homelab box, a script runs:

1. Detect — exo garden, a plain query over the index, produces a report: here are the orphans, here are notes that should probably link to each other, here are summaries that look stale. No AI in this step. It’s SQL and graph traversal. Deterministic, boring, trustworthy.
2. Decide and write — that report gets piped to claude -p (Claude Code in headless mode). Claude reads the vault’s operating contract, makes only high-confidence edits — add a [[wikilink]] between two genuinely related notes, refresh a stale summary — caps itself at ~10 notes a night, and writes a dated log note explaining exactly what it changed and what it deliberately skipped.
3. Commit — the wrapper reindexes and lands everything as a single garden: 2026-06-09 … git commit, then pushes. My 3D graph viewer picks it up on the next sync.

The first real run, it found one orphan (90-meta/README), linked it into the notes it actually indexes, and then — this is the part I liked — declined to touch the 12 “stale summary” candidates because, on inspection, every one of them was already accurate. It wrote: “flagged by length, not staleness; churning them would add noise.” A gardener that knows when not to prune is the one you can leave alone.

“Isn’t this a solved problem?”

Mostly, no — but partly, yes, and I want to be straight about it. AI-assisted note-linking exists: Obsidian plugins like Smart Connections suggest related notes, and apps like Mem and Reflect auto-organize as you write. They’re good.

Three things make this different enough to build:

• Every change is a reviewable git diff, authored by a named agent. Not silent magic that rearranges your notes while you’re not looking. git log -p shows you exactly what the gardener did last night; git revert undoes a bad night in one command. For something as personal as a knowledge base, “show me the diff” beats “trust me.”
• It’s mine, end to end. Runs on my hardware, on my schedule, with a model I point at. No SaaS holds my brain hostage.
• The detection is deterministic; the model only acts. The LLM never decides what’s wrong — a boring query does that. The model only decides how to fix the things already found. That split keeps the whole thing auditable and cheap.

If you already live in a tool that does this and you trust it, great. I wanted the git-diff trail and the local control.

The part I actually want to tell you about

The plan was tidy: I run n8n on the same cluster, so n8n would be the scheduler — fire nightly, SSH into the node, run the gardener. Clean, visual, one workflow.

n8n could not reach the node. At all. Every port: ECONNREFUSED.

This sent me down a genuinely interesting hole, because the homelab runs Cilium for networking, and Cilium has opinions about your own node that plain Kubernetes does not.

First instinct: a NetworkPolicy allowing egress to the node’s IP. Wrote it, synced it, still refused. The reason is a Cilium subtlety worth knowing: the node isn’t a CIDR, it’s an identity. Cilium classifies your cluster’s own node as the special host identity, and ordinary ipBlock CIDR rules do not match it unless you flip a cluster-wide setting (policy-cidr-match-mode: nodes). My 192.168.0.109/32 rule was a no-op.

So I switched to the Cilium-native tool: a CiliumNetworkPolicy with toEntities: [host]. Confirmed it applied — I could see reserved:host allowed right there in the datapath’s BPF policy map. I confirmed the node’s IP really does resolve to identity 1 (host). I confirmed the host firewall was disabled. Everything said “allowed.”

Still ECONNREFUSED.

That’s the wall. The packet leaves the pod with Cilium’s blessing, hits the host’s own network stack, and something there sends a reset — and I couldn’t see what, because inspecting the host firewall needs root, and this automation deliberately doesn’t have it. I could have kept digging with a password. But I stopped and asked a better question: why am I making a pod reach back into the host it’s running on at all?

That’s an awkward direction. The work has to happen on the host (that’s where the vault, git creds, and Claude live). A pod straining to SSH into its own node is fighting the grain of the platform.

So I inverted it. The node schedules itself — a plain cron entry, rock-solid, no network gymnastics. And n8n, instead of triggering the job, receives it: at the end of each run the node POSTs a summary to an n8n webhook. Node→n8n works perfectly (it’s just an outbound HTTPS call to a URL). n8n keeps the run history and is the place I’ll later wire a phone notification.

I lost nothing that mattered. n8n is still my dashboard; the schedule just lives where the work lives. And I deleted the SSH key and the network-policy hole I’d opened — the cleanup felt better than the original plan would have.

The lesson, such as it is

Two, actually.

One: when you’re automating something to run unattended, the bug you want to find is the one that shows up in a dry run at 2pm, not at 3am three weeks from now. I almost shipped a version where a brand-new note (untracked by git) was invisible to my change-detection and would’ve been silently wiped each night. The dry run caught it. Always build the dry run.

Two, the bigger one: I spent an hour trying to make a pod punch into its host because that was my plan, and the platform kept saying no in increasingly specific ways. The fix wasn’t a cleverer NetworkPolicy. It was noticing I was pushing against the design and turning around. The node scheduling itself and reporting up to n8n is simpler, safer, and more honest about where the work actually lives.

My brain weeds itself now. Every morning there’s maybe one small, sensible commit waiting — a link I’d have never made, a summary nudged back to true — and I can read exactly what changed before my coffee’s done. That’s the whole dream of a second brain that isn’t a junk drawer: it stays a garden, and I barely have to touch it.

So I automated mine. An n8n workflow on my homelab polls job boards every six hours, scores each new posting against my profile with an LLM, and emails me only the strong matches — the ones scoring 80%+. When it’s quiet, it’s silent. When something genuinely fits, I know the same day. Here’s what I learned building it. Repo at the bottom.

Three APIs cover most of the market

Company career pages look bespoke, but underneath, the vast majority run on one of three ATS — and all three hand you the jobs as unauthenticated JSON:

• Greenhouse — boards-api.greenhouse.io/v1/boards/{token}/jobs?content=true
• Lever — api.lever.co/v0/postings/{token}?mode=json
• Ashby — api.ashbyhq.com/posting-api/job-board/{token}?includeCompensation=true

No scraping, no headless browser. You poll the API the page itself calls, normalize the three shapes into one { company, title, location, remote, url, posted_at, description, external_id }, and you’re done with the hard part.

“Resolve the token” is half the battle

The naive assumption — the token is the company name, and everyone’s on one of the three — is half right. When I probed my initial wishlist, roughly half 404’d everywhere: HashiCorp (now under IBM → Workday), SUSE (SuccessFactors), Aiven (Teamtailor), Hugging Face. They’re on a fourth or fifth system entirely. The honest move was to ship the ~33 that actually resolve and leave the rest as disabled config stubs. Verify before you trust a slug.

Dedup without a database

I didn’t want to stand up Postgres just to remember which jobs I’d already seen. n8n’s Data Tables handle it natively: a seen_jobs table, an external_id namespaced {ats}:{company}:{id}, and the rowNotExists operation drops anything already recorded. State lives inside n8n, backed up with it. Zero extra infrastructure.

The ordering matters: notify first, mark seen second. The insert only happens after the email sends, so a failed send retries next run instead of silently swallowing a posting.

The location filter is a trap

My first version kept everything that wasn’t explicitly US-based. The inbox filled with “Senior Platform Engineer — Spain (Remote)” and "… — United Kingdom (Remote)". Those aren’t remote-for-me — they’re remote if you live in Spain. Useless from where I sit.

The fix was to invert the logic. Keep only three things:

• globally-remote / worldwide / anywhere,
• pan-EU (EMEA / Europe / EU / EEA),
• my own country.

…and drop single-country remote, even EU ones. Region and home matches win over the country deny-list, ambiguous locations are kept (a missed match is worse than one extra line to skim). That one change cut the noise more than anything else.

Let an LLM read the actual job

Keyword + location filtering gets you a candidate list, but it can’t tell a “Platform Engineer” who herds Kubernetes from a “Platform Engineer” who owns a Figma design system. The job description can.

So the last step scores each new posting against my CV. My first version batched all of them into one big LLM call — which promptly timed out on the free tier. The fix was the opposite: one small call per job, which also means a single slow or rate-limited job never sinks the batch. Each call asks a NVIDIA NIM model (Llama 3.1 8B, OpenAI-compatible) for one number and a reason:

Score this job 0–100 for fit against my profile. Return {score, reason}.

That score is what lets me widen the net instead of narrowing it. On top of the curated company list I pull a broad remote-jobs feed (every company, all categories); the cheap keyword + location filters do the first pass, then I only email the roles scoring 80%+. Casting wide is fine when a model is the bar at the door. A line ends up looking like:

92% — Grafana Labs — Senior Platform Engineer (Remote, EMEA) — strong k8s/GitOps overlap — link

Scoring is fail-safe: if a call hiccups, that job is just skipped, and every posting gets marked seen either way — so nothing re-scores forever, and a rare bad run never floods or stalls the inbox.

The unglamorous bits that make it trustworthy

• One bad source can’t kill the run — every fetch is wrapped; failures become a ⚠️ N sources failing footer so a company quietly changing ATS is visible, not invisible.
• A prime run seeds the table silently the first time, so I’m not buried under every currently-open role on day one.
• Everything tunable lives in one Config node — companies, keywords, location lists, the profile, the model — so adding a company is a one-line edit, not a graph safari.

Takeaways

• The “scrape job boards” problem mostly isn’t a scraping problem — it’s three public APIs and a normalizer.
• For personal automation, reach for the boring-but-correct primitive: native dedup state beats a database you have to operate.
• An LLM works best here as the bar at the door: cheap deterministic filters keep the candidate set (and the cost) small, then the model gates on real fit — which is what lets you cast a wide net without drowning in it.

Workflow JSON, the full node-by-node breakdown, and setup notes: github.com/janos-gyorgy/ats-job-poller.

There are, at last count, a small army of tools that list your Claude Code sessions and let you jump back into one. tmux wrappers (claude-tmux, claunch), keyword resumers (tmux-claude-code), fleet managers (claude-manager), and a whole macOS menu-bar genre (claude-control, cmux, and friends). They’re good. Several are better-engineered than mine.

I built one more anyway.

Not because the others are wrong — because none of them were shaped like my day, and the cost of hand-rolling a 300-line script turned out to be smaller than the cost of bending my workflow around someone else’s defaults. That’s the whole pitch, and it’s a boring one. The interesting part is what I had to understand to build it, because it corrected a mental model I’d had backwards for months.

My day, concretely

I work off a single Linux box over SSH, from a few different machines. A session might be a homelab change, a side project, a blog post. I drop one mid-thought, my laptop sleeps, I pick it up that evening from a different terminal. The thing I kept doing was running claude --resume and squinting at a list of UUIDs trying to remember which 7f3a… was the one about the broken redirect.

I wanted one command — wt — that shows me every session with a human summary and tells me, truthfully, which ones are still alive. Then lets me pick one.

Simple ask. It sent me reading the on-disk format, and that’s where it got educational.

What I had backwards: tmux is not how you keep a session

Every tmux-first tool sells the same promise: run Claude inside tmux so your session survives a disconnect. I’d internalized that as “tmux is how Claude sessions persist.”

That’s wrong, and realizing it deleted half the code I thought I’d need.

A Claude Code session is one claude process, keyed by a sessionId UUID. Its entire transcript — every message, every tool call and result — is appended to a file:

~/.claude/projects/<cwd-slug>/<sessionId>.jsonl

It’s append-only, and it has no “end” marker. When you --resume, Claude reopens that same file and replays it. One of my session files spans three calendar days across half a dozen resumes — same file, same UUID, the whole conversation reconstructed from disk each time.

Which means: the history is durable independent of any running process. You do not need tmux to land exactly where you left off. claude --resume <id> does that from the transcript alone, on a box with no tmux installed at all.

So what is tmux for, then? Exactly one thing: keeping a process running while you’re disconnected — a long job, an agent grinding away, or re-attaching the same live process from your phone. That’s real, but it’s the exception, not the default. So in my tool, plain resume is the default and tmux is an opt-in flag. The inversion fell straight out of reading the format honestly.

The other thing the transcript doesn’t tell you: is it alive?

Here’s the subtle bit. The transcript tells you the history of a session. It does not tell you whether a claude process is running right now. There’s no “closed” record — the file for a long-dead session looks identical to one you left open thirty seconds ago.

Liveness lives somewhere else:

~/.claude/sessions/<pid>.json   →   { pid, sessionId, cwd, procStart, ... }

A session is alive if that pid is actually running. But you can’t just trust the file’s existence — it can linger after a crash — and you can’t just kill -0 the pid either, because the kernel recycles pids and you might be poking a process that reused the number. So the honest check is two-factor:

def alive(pid, procstart):
    try:
        os.kill(pid, 0)          # exists and signalable?
    except (ProcessLookupError, OSError):
        return False
    # ...and is it the SAME process, not a pid-recycle?
    stat = Path(f"/proc/{pid}/stat").read_text()
    starttime = stat[stat.rindex(")") + 2:].split()[19]
    return starttime == str(procstart)

That /proc/<pid>/stat start-time comparison is the difference between “I think it’s live” and “it’s live.” It’s the kind of detail you only get right by caring about the boring case.

With that, every session resolves to a real state:

• ● live — a process is running now
• ⧗ waiting — no process; you left mid-conversation (last line was Claude)
• · idle — no process; stale

And the payoff for getting liveness right: if you try to resume a session that’s still live in another terminal, the tool refuses to double-open it — two processes appending to one transcript is how you corrupt your own history — and offers a clean --fork-session instead.

The summaries were free the whole time

The feature I assumed I’d have to build — a short, human description of each session — I didn’t build at all. Claude Code already writes one. Buried in the transcript is a record type:

{"type": "ai-title", "aiTitle": "Investigate nested o directories", "sessionId": "..."}

Claude titles your sessions for you. The “summary” column in my tool is just that field, with a fallback to your last prompt. The best line of code is the one you delete after noticing the platform already did the work.

So what did I actually build

Not much, and that’s the point. wt is one Python file, standard library only, no daemon. It globs the transcripts, reads each one’s title and last-activity, joins that against the pid-verified live registry, sorts live-first, and prints a numbered list. Pick a number and it execs into claude --resume. There’s a -t for tmux when I genuinely need it, a d to archive old sessions (a file move, fully reversible), and a guarded hook that turns it into an SSH login greeting so the box tells me what’s on it the moment I land.

  watchtower · 5 session(s)
   1) ●  live       16s  homelab    595e931d  Investigate nested o directories
   2) ·  idle     1d07h  notes-app  6565b121  Migrate to server components
  [#]resume  [t#]tmux  [d#]archive  [n]ew  [Enter]shell  [q]uit ▸

If you want it, it’s on GitHub, MIT. But honestly, I’d rather you take the three things I had to learn than the tool:

1. Your Claude history lives in a plain append-only JSONL on disk, not in tmux. --resume works without any wrapper. Back up ~/.claude/projects/ and you’ve backed up every conversation you’ve had.
2. Liveness is a separate fact from history, and checking it honestly means verifying the pid is the same process — not just that something answers to the number.
3. The platform probably already did the boring work (here: the titles). Read the format before you write the feature.

The flooded-market thing turns out not to matter. A tool that fits your own hands is worth building even when fifty others exist — especially when it’s small enough that “build” and “understand the system underneath” are the same afternoon.

I had a second brain once. Obsidian vault, a CouchDB LiveSync backend, even a weekly agent that summarised my notes. It worked — for a while. Then the sync started fighting itself across my laptop, the homelab, and my phone, and the day syncing becomes a chore is the day you stop opening the thing. The notes were still there. I just never looked at them again.

That’s how most second brains die. Not from bad notes — from the plumbing. The sync breaks, or the upkeep outpaces the payoff, or the whole thing is trapped in one app’s database and moving it feels like surgery. The knowledge was never the problem. The container was.

So when I rebuilt it, I started from the failure modes, not the features.

What I actually wanted

Three things, none of them “more notes”:

1. Memory I share with my AIs. Every time I open a fresh Claude session, it starts from zero — I re-explain my homelab, my projects, what we decided last week. I wanted a place both of us read and write, so the context survives the session.
2. Something that outlives any tool. No lock-in. If the app of the month dies, my brain shouldn’t die with it.
3. Sync that can’t rot. The thing that killed v1.

The one decision that matters

The store and the intelligence are different layers, and only the store is sacred.

The store is a folder of plain markdown in git. That’s it. Human-readable, diffable, greppable, yours. Everything clever sits above it and is fully rebuildable:

L5  Visualisation   3D graph, Obsidian, whatever reads markdown
L4  Automation      scheduled "gardener" runs
L3  Agent interface MCP servers — search, graph, note CRUD
L2  Index           SQLite: full-text + vectors + materialised edges
L1  Structure       typed frontmatter + [[wikilinks]]
L0  Substrate       markdown files in git   ← the only thing that's truth

Delete L1–L5 and nothing is lost — you rebuild them from L0 with one command. That property is the whole design. The index can corrupt, the embedding model can change, the viewer can break (mine did, spectacularly — that’s another post), and the knowledge doesn’t care. It’s text in git.

And sync is just git pull. No LiveSync daemon to wedge itself, no proprietary replication. The exact thing that killed v1 is now the most boring, battle-tested part of the stack. Three devices, one git pull, done.

Search that explains itself

The retrieval layer is deliberately not “throw it all at embeddings.” It fuses three signals — keyword (BM25), vector similarity, and graph expansion (pull in the neighbours of strong hits) — and every result reports which signals fired.

exo search "hybrid retrieval"
→ hybrid-retrieval   matched_on: [bm25, graph]

That matched_on matters more than it looks. An embeddings-only system gives you a ranked list and no reason — you can’t tell a real match from a vibe. For a brain I’m supposed to trust over years, “why did this surface?” is a feature, not a nicety.

The AI is a librarian, not a hoarder

Here’s the part I care about most. The AI doesn’t just read the brain — it writes to it. Through an MCP server it can search, walk the graph, and author notes. But under a hard rule: every write is a reviewable git diff.

It searches before it writes (extend a note, don’t spawn a duplicate). It links instead of piling. A scheduled “gardener” pass finds orphaned notes and stale summaries and proposes fixes — as commits I can read and git revert if it gets something wrong. No black-box mutation of my memory. Just a librarian that files things while I’m asleep and leaves a paper trail.

So now “what am I building?” is a question with an instant, honest answer: a single map note, kept current, that every project links into. I ask, the AI pulls it, and neither of us has to remember.

Why not just…

• Obsidian alone? It’s a lovely viewer — and I still use it as one. But it can’t give an agent structured read/write or explainable retrieval, and its sync is what burned me. Here Obsidian reads the same markdown; it’s a window, not the house.
• Embeddings RAG? Opaque and one-directional. It can rank, but it can’t tell you why, and it can’t write back. This is transparent and bidirectional.
• Notion / a SaaS brain? Lock-in by design. git clone is my backup and any text editor is my fallback.
• A graph database? Unnecessary infra. The graph lives in the wikilinks; SQLite just materialises it. I’ll add Neo4j the day my queries actually outgrow a single file, and not a day sooner.

What it changes

The vault is small still — that’s fine; it grows by use. But the loop already pays off: I work, the AI checkpoints decisions into markdown, and the next session — fresh model, no memory of its own — searches the brain and is caught up in seconds. The knowledge stopped living only in my head and in dead chat logs.

I’m a team of one. There’s no colleague who remembers why I made a call six months ago, no handover doc someone else maintains. Continuity isn’t a nice-to-have; it’s the whole job. A second brain that the AI helps keep alive — and that I can git clone onto any machine in thirty seconds — is the first version of this idea that I actually trust to still be here in five years.

The notes from v1? They’re sitting in a folder, waiting to be triaged into v2. This time I’ll still be opening it.

If you haven’t fallen down this hole: kombucha is sweet tea fermented by a SCOBY (a rubbery pancake of yeast and bacteria) into something tart and fizzy. It’s a living hobby — the culture is alive, every batch is a little different, and the only way to get good is to pay attention and remember what you did.

I was not remembering what I did. Brew dates lived in my head, taste notes lived nowhere, and “which jar was the ginger one again?” was a genuine question I asked myself out loud, to a fridge.

So I built a tracker. It’s called HipPotion — same family as everything else I run here. The brewing turned out to be the easy part. Modeling it was where it got interesting.

Why a simple list doesn’t fit

My first instinct was “a batch is a row, log some notes.” That falls apart fast, because kombucha isn’t linear. It has two stages:

• F1 (first ferment): the big jar of sweet tea + SCOBY, fermenting sour over a week or two. One vessel, one culture.
• F2 (second ferment): you split that sour base into bottles and flavor each one differently — ginger in this one, blackberry in that one, hibiscus in the next — then seal them to build carbonation.

So one batch becomes many bottles, each with its own flavor, its own carbonation, its own outcome. A flat “batch = row” model can’t express that. And on top of the branching, every jar and bottle produces a stream of observations over time: pH today, Brix tomorrow, “tastes too sweet still” the day after.

That’s three different shapes at once — a lifecycle, a one-to-many split, and a time series — for what looks from the outside like “I made some tea.”

The model I landed on

Six tables, each earning its place:

• recipes — the templates. Tea blend, sugar ratio, target numbers. A batch points at one.
• batches — an actual F1 brew, with a lifecycle (planned → active → conditioning → finished) and a reference to its recipe.
• fermentation_log_entries — the time series. One row per observation per batch: pH, Brix, temperature, taste/smell notes, what I did. This is where the “pay attention and remember” lives.
• f2_variant_batches — the branch. Each is a flavored bottle split off a parent batch, tracked on its own.
• starter_log — SCOBY lineage. Cultures have parents; you grow new ones from old ones, and a sick culture ruins a batch, so the lineage matters.
• botanical_infusions — the flavoring ingredients, managed per recipe.

The shape that took the longest to get right was the F1 → F2 split: a variant has to belong to its parent batch but live its own life. Once that relationship was clean, the whole thing clicked — the app finally matched how brewing actually works instead of how it’s easy to store.

The stack (and where it runs)

Nothing exotic: React + Vite + TypeScript on the front (TanStack Query, shadcn/ui, Tailwind), a Hono + Drizzle ORM API on the back, PostgreSQL underneath. Built with AI coding tools — I leaned on them hard for the React/shadcn front-end, less so for the schema, which I argued out by hand because it’s the part that had to be right.

It runs on my k3s homelab like everything else: a Helm chart deploys the nginx frontend, the Hono API, and a Postgres StatefulSet, all reconciled by Argo CD from Git. Default-deny networking, secrets out of Git — the usual platform defaults. It’s a hobby app, but it gets treated like a real one, because the platform doesn’t know the difference and I don’t want it to.

It became an API for something else

The unexpected payoff: because the data model was clean and the API was just a set of plain REST endpoints, it made a perfect target for an experiment. I later pointed an AI agent at it from n8n — “what’s fermenting right now?”, “log that this batch tastes tart” — and the agent just called the same endpoints the UI does. A good schema is reusable in ways you don’t plan for. The kombucha tracker quietly became a little knowledge base I can talk to.

Honest notes

This is a personal hobby app for an audience of one (me). It’s AI-assisted, it has no tests, and the UI has rough edges. I’m not pretending it’s a product.

But the thing I keep coming back to: the hard, valuable part wasn’t the framework or the deployment — it was sitting with a messy real-world process long enough to find the shape of it. The branching ferment, the time series, the lineage. Get the model honest and the rest is just typing. Get it wrong and no amount of nice UI saves you.

Also, the kombucha’s been better since I started writing things down. Turns out the fridge wasn’t a great database.

“You’re running n8n for multiple customers on the same Kubernetes cluster. What stops Customer A from reading Customer B’s API keys, calling Customer B’s services, or starving Customer B’s workflows by burning the whole node?”

Three different walls, three different mechanisms. Most articles I’ve read on K8s multi-tenancy list the primitives — namespaces, NetworkPolicies, ResourceQuotas, RBAC — without showing what each one actually catches when you try to cross it. This post does the second part. The receipts are the point.

The setup: two namespaces, web-tenant-acme and web-tenant-globex, each running their own n8n instance on the same node. The only thing keeping them apart is the walls we build around each namespace.

The mental model: subtractive isolation

Kubernetes is a flat network with shared everything by default. You don’t add isolation by writing allow rules. You subtract trust by adding default-deny rules, and then carefully allow back only the connections each tenant actually needs.

A tenant doesn’t have access to another tenant because there is no rule allowing it. The absence of an allow rule is the wall.

Three of these absences make up the picture:

Different layers, different error messages. That’s how you can tell what stopped you.

Wall 1 — Network: Cilium NetworkPolicy

n8n in web-tenant-acme can reach whoami.web-tenant-acme.svc.cluster.local (its own service in its own namespace) but not whoami.web-tenant-globex.svc.cluster.local. The same DNS shape, the same cluster, the same node. One succeeds, the other hangs.

The primitive is a default-deny egress policy applied to every pod in the namespace, with two narrow exceptions: intra-namespace traffic (so n8n can still reach its own service) and DNS to kube-system (otherwise nothing resolves anything).

# Effective policy on every pod in web-tenant-acme:
spec:
  podSelector: {}
  policyTypes: [Egress, Ingress]
  egress:
    - to:                                     # intra-namespace traffic OK
        - podSelector: {}
    - to:                                     # DNS to kube-dns OK
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports: [{port: 53, protocol: UDP}]

There is no rule for web-tenant-globex. Cilium’s eBPF datapath drops the SYN packet on the way out.

The receipt — an n8n HTTP node configured to GET http://whoami.web-tenant-globex.svc.cluster.local/. It hangs for the full timeout, then errors with AxiosError: timeout of 5000ms exceeded / code: ECONNABORTED.

The interesting bit: DNS still works. kube-dns is allowed, so the cross-namespace Service still resolves. The TCP handshake is what gets dropped. That’s a useful signal in real incident response — “DNS resolves but the connection hangs” almost always means a NetworkPolicy is the cause.

Wall 2 — Secret: Vault Kubernetes-auth + ESO

Now imagine Acme’s n8n misbehaves: somebody pushes a workflow that tries to read Globex’s API keys via an ExternalSecret. The network isn’t the issue — both tenants need to reach Vault, so they both have an egress rule for sys-vault. The wall has to be at the identity layer.

Each tenant gets three things:

1. A dedicated ServiceAccount (n8n-acme, n8n-globex).
2. A Vault Kubernetes-auth role bound to that SA in that namespace, mapped to a Vault policy that grants read on only its own KV path.
3. A namespaced External Secrets SecretStore that authenticates as the SA via the Kubernetes TokenRequest API.

# Vault policy: tenant-acme can read its own secrets, nothing else.
path "secret/data/web-tenant-acme"     { capabilities = ["read"] }
path "secret/metadata/web-tenant-acme" { capabilities = ["read"] }

vault write auth/kubernetes/role/tenant-acme \
  bound_service_account_names=n8n-acme \
  bound_service_account_namespaces=web-tenant-acme \
  policies=tenant-acme \
  ttl=1h

When Acme’s n8n tries an ExternalSecret pointing at secret/web-tenant-globex/..., ESO authenticates fine (the SA is valid), Vault recognises the caller, looks up the tenant-acme policy, and answers with the most satisfying line in this whole demo:

URL: GET http://sys-vault.sys-vault.svc.cluster.local:8200/v1/secret/data/web-tenant-globex
Code: 403. Errors:
* permission denied

This is the bit that separates “namespace isolation” from real multi-tenant secret isolation. Plain Kubernetes Secrets + RBAC stop a tenant from listing another tenant’s Secret objects, but the moment you go upstream — to Vault, to a cloud KMS, to an SSM Parameter Store — the secret store needs to enforce identity itself. The network said yes; the secret store still says no.

Wall 3 — Resource: ResourceQuota + LimitRange

The third concern is the noisy neighbour: Acme’s runaway workflow allocating a 4Gi pod and OOM-killing everything else on the node. The network policy doesn’t catch this (no network call), and Vault doesn’t catch this (no secret request). The kernel will, eventually — but you don’t want eventually. You want admission-time rejection.

Two primitives:

apiVersion: v1
kind: ResourceQuota
metadata: { name: tenant-quota, namespace: web-tenant-acme }
spec:
  hard:
    requests.cpu:    "1"
    requests.memory: 1Gi
    limits.cpu:      "2"
    limits.memory:   2Gi
    pods:            "10"
---
apiVersion: v1
kind: LimitRange
metadata: { name: tenant-limits, namespace: web-tenant-acme }
spec:
  limits:
    - type: Container
      default:        { cpu: 500m, memory: 512Mi }
      defaultRequest: { cpu: 50m,  memory: 128Mi }
      max:            { cpu: "2",  memory: 1Gi }

ResourceQuota caps the namespace total. LimitRange bounds any individual container and supplies defaults so pods that don’t declare requests/limits still get reasonable ones — important because a missing limit on a single container can blow past the quota in one allocation.

The receipt — a server-side dry-run of a single 4Gi pod, which never gets created:

$ kubectl apply -n web-tenant-acme --dry-run=server -f noisy-neighbor.yaml
Error from server (Forbidden): error when creating "STDIN":
pods "noisy-neighbor" is forbidden:
  maximum memory usage per Container is 1Gi, but limit is 4Gi

Not a kernel OOMKill. Not a pod stuck in Pending. A flat refusal from the API server before the scheduler even sees the request.

What this does not prove

A homelab demo on one node with two synthetic tenants is not n8n Cloud. The honest gaps:

• Execution sandboxing. A workflow can still run arbitrary code via the Code node or shell-outs. These walls stop infrastructure leakage; they don’t sandbox what n8n itself executes. Real n8n Cloud needs more than namespace walls for that — gVisor / Firecracker / per-tenant worker pools are the usual answers, and n8n’s queue mode lends itself to the last.
• Pooled worker queues. Queue mode runs main/webhook/worker as separate deployments backed by Redis + Postgres. Two tenants sharing a worker pool need additional checks at the job-routing layer to keep workflows from accessing the wrong tenant’s binary data. Out of scope for the homelab demo.
• Control plane. Both tenants reach the same API server. A cluster-admin-equivalent compromise breaks everything. This is the assumption every shared K8s setup makes.
• Node-level. Same kernel. Container escape, CPU side channels, the usual list — all apply. For paranoid tenants the answer is dedicated nodes via taints/tolerations or separate clusters entirely.

The demo proves the namespace-shaped walls hold. It does not prove the whole stack is safe against a determined attacker already running code inside a tenant. That’s a different post.

Part of a Kubernetes-on-the-homelab series — previously: preventing a compromised pod from calling your database, GitOps secrets.

Via Stoica: A Year of Stoic Practice (“the app”) is published by Hippotion.

Data collection

The app does not collect, transmit, or share any personal data. No analytics, no crash reporting, no advertising SDKs.

Data stored on your device

The app stores two values locally on your device using Android’s SharedPreferences:

• Whether you have unlocked the Pro tier
• Your current reading position in the Pro sequence

This data never leaves your device.

In-app purchases

Purchases are processed by Google Play. The app receives only a confirmation of the purchase status. No payment information is handled by the app.

Contact

Questions: gyorgy.jani@gmail.com

I run n8n on my k3s homelab. Not docker-compose on a NUC — the full treatment: GitOps-reconciled, Vault-backed secrets, default-deny networking. The same boring platform everything else here runs on.

But “I have n8n running” proves nothing. I wanted to know if I actually understood it as an agent platform, and to answer a question I kept dodging: for agent work, do I need a cloud model, or is my local one good enough?

So I built a real agent and gave it two brains.

What I built

A chat assistant over brew-buddy, my homemade kombucha-tracking app (React + a small API + Postgres). You ask it things in plain language; it calls the app’s API and answers. The twist: the same question runs through two agents in parallel — one backed by NVIDIA’s hosted Llama-3.3-70B, one by a local Phi-3.5-mini on CPU — and the workflow prints both answers side by side.

Chat ──▶ Agent (cloud: NVIDIA 70B) ──┐   tools (shared):
     └─▶ Agent (local: Phi-3.5)   ──┤     • get_all_batches
                                    │     • get_batch_detail
                                    │     • brewing_statistics
            (Merge) ──▶ both replies, labeled     • add_batch_log   ⟵ write
                                                  • create_batch    ⟵ write

Both agents share the same read tools. The two write tools are wired to the cloud agent only — more on that below.

The nice part: I didn’t write a line of glue. n8n’s stock OpenAI Chat Model node talks to anything OpenAI-compatible if you override the credential’s Base URL — so one node points at https://integrate.api.nvidia.com/v1, the other at http://llama-server.<ns>.svc:8080/v1 for the local server. Same node, two endpoints.

The infra that keeps it honest

I won’t re-explain the platform here — it’s in earlier posts: GitOps, Vault-backed secrets, default-deny networking, dual-path TLS ingress. But building the agent made one of them tangible.

n8n is, by design, a thing that makes arbitrary HTTP calls on a schedule. That’s exactly what you want behind a default-deny network policy. n8n couldn’t reach the brew-buddy API at all until I declared it — one line:

# n8n's namespace
allowEgressToNamespaces: [web-ai-engine, web-brew-buddy]
#                                          ^ added this for the agent

(plus a matching ingress-allow on brew-buddy’s side). That’s the posture working as intended: the blast radius of a workflow tool is whatever I’ve explicitly granted, and not one namespace more. Adding a capability is a reviewable one-liner in Git; Argo reconciles it. No kubectl, no guessing what n8n can reach.

The A/B: same agent, same tools, two brains

Plain “hi”. Cloud answers in ~0.5s. Local takes noticeably longer — because even for “hi”, the agent feeds the model the full system prompt plus the JSON schemas for every tool, and Phi-3.5 has to chew through all of it on CPU before it can say a word. So far, the boring expected result: local is slower.

Then I asked a real question, and the result flipped in a way I didn’t expect.

“What batches do I have?”

Cloud (70B) called get_all_batches, got the real rows, and answered:

You have two batches: 2026-04-09-A (cold-crash, 3L) and 2026-04-09-W (cold-crash, 3L).

Local (Phi-3.5) never called the tool. It didn’t seem to realise it had tools. Instead it confidently explained how I could go find the data myself:

To list all batches: 1. Access the brew-buddy app. 2. Look for a button labeled “List Batches”… def get_all_batches(): … … Remember, I’m unable to directly interact with apps or databases.

Fake instructions. Fake code. A polite apology. Everything except the actual answer it was sitting on top of.

Writing data. I asked both to log an observation. Cloud called add_batch_log and wrote a real row to Postgres (“I have recorded the observation…”). Local bluffed again — “here’s how you can log it yourself.”

Why it matters: capability, not latency

The interesting finding isn’t “the big model is better.” It’s how the small one fails.

With a ~3.8B model on CPU, the bottleneck for agent work isn’t speed — it’s capability. Phi-3.5 couldn’t reliably emit tool calls, so n8n’s tools never fired, and the model degraded into a chatbot that hallucinates a plausible answer instead of fetching the real one. That failure mode is worse than an error: an error you catch, a confident wrong answer you ship.

A couple of measurements that sharpened it:

• NVIDIA 70B, plain chat: ~0.5s.
• NVIDIA 70B, function-calling (with tool schemas): ~8.6s per round-trip — and an agent makes several round-trips per answer. That’s real latency you have to budget a timeout for. (It’s also why the cloud side initially timed out in n8n until I raised the model node’s timeout — the model was fine, n8n was cutting it off.)

So the snappy-vs-slow comparison flips depending on whether the question triggers tools. Plain chat: cloud wins on speed. Tool use: the local model is “fast” only because it skips the tools and makes something up. Speed was never the real axis.

The honest caveat: this is this small general model in a multi-tool agent loop. Purpose-built small models with tool-calling fine-tunes do better at narrow tasks — I run a 1.7B one elsewhere that emits a single structured tool call just fine. But for “pick the right tool from several and chain them,” 70B was in a different league.

The trust boundary

I gave the write tools (add_batch_log, create_batch) to the cloud agent only. The local agent is read-only — not by instruction, by wiring. Even if Phi-3.5 did decide to call a write tool, the connection isn’t there. The reliable model is the only one allowed to mutate real data, and that’s enforced structurally, not by trusting a prompt.

What’s toy and what’s real

Worth being straight: this is a single-node homelab. The agent and both model paths share one box. Running n8n on Kubernetes and swapping models isn’t novel — n8n’s own docs cover queue mode, where a main instance fans work out to a pool of worker pods you scale horizontally, with external Postgres for state. That’s the real production shape. Mine is one replica with an emptyDir’s worth of ambition.

What I think is worth sharing is the finding (the capability cliff, and that its failure mode is confident fabrication) and the boring thing underneath it: because the platform is default-deny and GitOps-reconciled, running this experiment cost me one reviewable egress line and zero risk to anything else.

The boring part is the point

The AI was the fun bit. But the reason I could bolt an agent onto a live cluster, point it at a real app, give it write access to one model and not the other, and tear it all down again — without worrying what it might touch — is that the infrastructure was already boring. Default-deny. Secrets out of Git. git push, Argo reconciles.

The model picks the tools. The platform decides what the tools can reach. Keep those two honest about each other and self-hosting an agent stops being scary and starts being just another app.

Some names are chosen. This one grew.

I came up with hippotion when I was building a kombucha brand — a real one, with labels and bottles and a business plan. It’s dormant now. Maybe it becomes something again after I retire. But the name outlasted the business plan, which is usually a sign that the name was the real thing all along.

The English layer

Hip potion. A trendy brew. Something you’d see on a craft label at a farmers market with a logo that takes itself slightly too seriously.

That was the joke. Kombucha is, objectively, a hip potion — fermented, alive, slightly weird, loved by people who care too much about gut health and not enough about explaining themselves. The name didn’t need explaining. That was the point.

The Hungarian layer

Here’s where it gets layered.

In Hungarian, the word doesn’t read as “hip” + “potion.” It reads as two animals.

Hippó — hippopotamus. That’s the first half, and it’s obvious.

Tion — to a Hungarian ear, this sounds like sün. Hedgehog. (The ü is a u with dots — “sewn” but softer. If you’ve never heard it, imagine a shy vowel that lives in the middle of a forest and avoids eye contact.)

So: hippotion = hippó + sün = hippo + hedgehog.

This was not planned on a whiteboard. It arrived. And when it did, the brand suddenly had two mascots — two animals that shouldn’t make sense together, but somehow do.

The totem logic

A hippo is large, slow, unexpectedly dangerous, and deeply underestimated. It doesn’t perform strength. It just has it.

A hedgehog is small, quiet, armed with spines it never has to explain, and entirely content with its own company. It doesn’t need to win. It just needs to not be eaten.

They protect each other. The hippo protects the sün. The sün protects the hippo. Not because they’re the same — because they’re different in complementary ways.

These animals also have meaning in my personal life that I won’t explain here. Some things are allowed to be yours.

The creed

The site at hippotion.com has one page. No nav, no portfolio, no “hire me” section. Just this:

Modern luxury is the ability to think clearly, sleep deeply, move slowly,
and live quietly in a world designed to prevent all four.

That’s the operating principle. Everything built under the hippotion name — the kombucha, the software, the blog — is an attempt to live closer to that line.

The easter egg

I never wanted to explain this publicly. The name was designed to work on the surface — hip potion, fine, move on — and reward the people curious enough to sit with it.

A Hungarian reader might catch it. A kombucha person might catch the drink angle. Someone who looks at the design and notices the hedgehog-and-hippo motif recurring across different projects might wonder. That wondering is the point.

If you found this post because you were curious about the name: that’s exactly who this was for.

You’ve got a Kubernetes cluster. Now you need to describe what should run in it. You write some YAML, apply it, it works.

Then you need a second environment. Or a second service. Or someone else joins the project and asks “how do I add an app to this?” and you don’t have a good answer.

This is the manifest management problem, and there are five common solutions — ranging from “this works until it doesn’t” to “this is what production platforms actually look like.”

Approach 1: Raw manifests

The starting point for almost everyone. Write a YAML file, kubectl apply -f, done.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  namespace: myapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
        - name: myapp
          image: myapp:v1.2.3

Where it works: one service, one environment, learning Kubernetes. The feedback loop is immediate — write YAML, see what happens.

Where it breaks:

• No templating. Want to change the image tag across ten services? Ten files, ten edits, ten chances to get it wrong.
• Live state leaks in. If you export existing resources with kubectl get -o yaml, you get resourceVersion, generation, creationTimestamp, and managedFields in the output. Commit that to Git and you’ve created a permanent source of conflicts — ArgoCD compares what’s in Git against what’s in the cluster, sees stale version counters, and the diff never clears.
• Copy-paste hell. A Deployment, a Service, an IngressRoute, a ServiceAccount, a NetworkPolicy — five files per app. Add a new app, copy five files, change the names, forget to update one. This is how environments drift apart silently.

The fix for the live-state problem is: only commit desired state. Strip every field that Kubernetes manages internally back to its clean spec. It’s tedious and easy to forget, which is exactly why people move on from raw manifests.

Approach 2: Kustomize

Kustomize is built into kubectl (kubectl apply -k) and natively supported by ArgoCD. The idea: you have a base/ with your raw manifests, and overlays that patch on top of them for different environments.

app/
├── base/
│   ├── deployment.yaml
│   ├── service.yaml
│   └── kustomization.yaml
└── overlays/
    ├── staging/
    │   ├── kustomization.yaml    # patches replicas to 1, image to :staging
    └── production/
        └── kustomization.yaml    # patches replicas to 3, image to :v1.2.3

# overlays/production/kustomization.yaml
resources:
  - ../../base
patches:
  - patch: |-
      - op: replace
        path: /spec/replicas
        value: 3
    target:
      kind: Deployment

Where it works: multi-environment setups where the difference between environments is mostly configuration values, not structure. Kustomize is good at this — you write the base once and patch only what differs.

Where it breaks:

• No real parameterization. Kustomize patches are surgical edits, not templates. If your base structure needs to vary (different resource shapes per environment, conditional blocks), you’re fighting the tool.
• Patching deep structures is ugly. JSON patches on nested YAML are verbose and hard to read. You end up writing more patch YAML than it would take to just copy the file.
• Still repetitive across apps. Each app still gets its own base directory. You’re not abstracting the shared patterns across apps, only the differences between environments of the same app.

Kustomize is a significant step up from raw manifests for multi-environment setups. For complex templating or platform-level abstractions, it runs out of power quickly.

Approach 3: Helm

Helm adds real templating. Charts are parameterized bundles — templates with variables, conditionals, and loops — and values files supply the parameters.

# templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Values.name }}
  namespace: {{ .Release.Namespace }}
spec:
  replicas: {{ .Values.replicas | default 1 }}
  template:
    spec:
      containers:
        - name: {{ .Values.name }}
          image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
          {{- if .Values.resources }}
          resources: {{ .Values.resources | toYaml | nindent 12 }}
          {{- end }}

# values-production.yaml
name: myapp
replicas: 3
image:
  repository: myorg/myapp
  tag: v1.2.3

Helm renders the templates at deploy time. What lands in the cluster is clean rendered YAML — no internal state, no conflicts.

Where it works: almost everywhere. The Helm Hub has charts for most common software already. For custom apps, writing a chart once and parameterizing per-environment is straightforwardly better than copying YAML.

Where it breaks:

• Chart authoring is verbose. Writing a Helm chart from scratch involves a lot of Go templating boilerplate. For a simple app, it can feel like more scaffolding than application.
• Debugging rendered output is annoying. helm template is your friend, but errors in templates produce unhelpful messages. The indentation rules (nindent, indent, toYaml) have sharp edges.
• Values files still pile up. If every app has its own values file and there’s no shared structure between them, you’re back to copy-paste but now in YAML-that-configures-YAML.

Helm is the right tool for most Kubernetes deployments. The ecosystem support alone (upstream charts for Postgres, Redis, Vault, every CNCF project) makes it the pragmatic default.

Approach 4: Jsonnet / CUE

For teams that need programmatic config generation — actual code, not templates — Jsonnet and CUE are the serious alternatives.

// deployment.jsonnet
local k = import "k.libsonnet";

local deployment(name, image, replicas=1) =
  k.apps.v1.deployment.new(name, replicas, [
    k.core.v1.container.new(name, image)
  ]);

{
  "deployment.yaml": deployment("myapp", "myorg/myapp:v1.2.3", replicas=3)
}

Where it works: large platforms where configuration is genuinely complex — many environments, many apps, deep interdependencies. Jsonnet lets you write real functions, share libraries, compose abstractions properly.

Where it breaks:

• Steep learning curve. Jsonnet is a full language. CUE even more so — it has types, schemas, and a constraint system that takes time to internalise.
• Small community. Excellent tooling, but you’re solving problems that have fewer Stack Overflow answers.
• Overkill for most setups. If you’re not managing hundreds of services across multiple clusters, Helm is simpler and has everything you need.

Jsonnet is used seriously at Google-scale infrastructure teams and in some CNCF projects. For a homelab or a small-to-medium platform, it’s the right answer to a question you probably aren’t asking yet.

Approach 5: App-of-apps with generated Application CRDs

This is the ArgoCD-native meta-layer. Instead of managing manifests, you manage Application resources — and potentially use a chart or tool to generate those too.

A naive version: commit a folder of Application YAML files to Git, one per service. ArgoCD watches the folder and deploys each app.

A more sophisticated version: one “root app” that points to a chart, which generates all the other Application resources dynamically from a single config file.

Where it works: at the platform level, not the individual app level. App-of-apps is how you manage what ArgoCD manages, not how you write the service manifests themselves. Combined with Helm, it gives you centralized control over the entire cluster’s structure.

Where it breaks:

• Manual Application CRDs are painful. If you’re maintaining a folder of hand-written Application YAML files — one per service — you’ve traded manifest copy-paste for Application copy-paste. Each app needs its own CRD with its repo URL, path, sync policy, project reference.
• Sync ordering matters. The root app must exist before children can sync. Get the wave ordering wrong and apps try to deploy before their namespaces exist.

How this homelab compares

My setup sits at the far end of approach 5, using Helm throughout.

There’s a single applications.yml file that describes every service in the cluster. A root Helm chart reads it and generates all the ArgoCD Application and AppProject CRDs automatically. Adding a service means adding an entry to that file — not touching five different places across five different files.

# applications.yml — this is the entire service catalog
- namespace: web-vaultwarden
  networkPolicies:
    profile: web-app
  applications:
    - applicationCode: web-vaultwarden
      path: helm-charts/extra-objects
      autoSync: true

That one entry generates: a Namespace, an ArgoCD AppProject, an ArgoCD Application, a set of Cilium NetworkPolicies (deny-all with ingress from Traefik and DNS/HTTPS egress), and a ServiceAccount. Nothing is written by hand.

The actual service manifests live in an extra-objects chart — a thin wrapper that renders raw YAML from values files. No templating in the service manifests themselves (they’re simple enough not to need it), but the infrastructure scaffolding around each app is entirely generated.

The result: every service gets the same operational properties. Same GitOps workflow, same secret management, same network isolation, same TLS termination. The platform work was done once. Adding a new app is writing manifests for the app’s specific behavior, not recreating the scaffolding.

The honest spectrum

Most setups should start at Helm. Kustomize if you’re multi-environment and comfortable with patching. App-of-apps when you’re managing the platform layer, not individual services. Jsonnet/CUE when you know you’ve outgrown Helm — which is a specific and relatively rare problem to have.

Raw manifests are fine for learning. They’re the wrong answer for anything you intend to maintain.

More on how the homelab is structured: My Homelab Runs on GitOps.

Running a local model is great for privacy. But local models hit a ceiling — for the heavy lifting, you want a cloud API like NVIDIA NIM with Llama 3.3 70B.

The moment you open that channel, you have a new risk: what if someone (or some automation) accidentally pastes a password, a private key, or someone’s personal data into the chat? It leaves the cluster. It’s logged somewhere you don’t control.

The standard answer is “train your users.” I’d rather have a technical control.

The architecture

Open WebUI → ai-guard proxy
                 │
        ┌────────┴────────┐
        │                 │
  llama-server       if SAFE:
  (classify)         forward to NVIDIA NIM
        │
   if SENSITIVE:
   block + explain

Every request to NVIDIA NIM goes through ai-guard first. ai-guard pulls the user message, sends it to the local llama.cpp server with a classification prompt, and makes a binary decision:

• SAFE → forward to NVIDIA NIM with the real API key (which ai-guard holds, not the client)
• SENSITIVE: <reason> → return HTTP 400, log the block, nothing leaves the cluster

The local model is already running for inference — this reuses it as a privacy gatekeeper at zero extra infrastructure cost.

The implementation

The proxy is ~150 lines of FastAPI. The classifier call:

CLASSIFIER_PROMPT = """You are a data security classifier. Check if the text below contains sensitive information:
passwords, API keys, tokens, credentials, personal identifiable information (names, emails, phone numbers, SSNs, addresses), financial data (card numbers, bank accounts), or private keys.

Reply with ONLY one of:
SAFE
SENSITIVE: <one-line reason>

Text to check:
"""

async def classify(text: str) -> tuple[bool, str]:
    async with httpx.AsyncClient(timeout=60) as client:
        resp = await client.post(
            f"{LLAMA_BASE}/chat/completions",
            json={
                "model": "phi-3.5-mini",
                "messages": [{"role": "user", "content": CLASSIFIER_PROMPT + text[:3000]}],
                "max_tokens": 30,
                "temperature": 0,
                "stream": False,
            },
            headers={"Authorization": "Bearer sk-no-key"},
        )
    answer = resp.json()["choices"][0]["message"]["content"].strip()
    if answer.upper().startswith("SENSITIVE"):
        reason = answer.split(":", 1)[1].strip() if ":" in answer else "sensitive content detected"
        return True, reason
    return False, ""

temperature=0 and max_tokens=30 keep the response deterministic and fast. The model only needs to output one word or one line.

The main handler:

@app.post("/v1/chat/completions")
async def proxy_chat(request: Request):
    body = await request.json()
    user_text = extract_user_text(body.get("messages", []))

    if user_text.strip():
        try:
            is_sensitive, reason = await classify(user_text)
        except Exception as exc:
            log.error("classifier error: %s — allowing request through", exc)
            is_sensitive = False

        if is_sensitive:
            return JSONResponse(status_code=400, content={
                "error": {
                    "message": f"Request blocked by ai-guard: {reason}. Remove sensitive content before sending to external models.",
                    "type": "content_policy_violation",
                }
            })

    # Safe — forward to upstream with streaming support
    ...

Fail-open: if the classifier itself errors (llama-server down, timeout), the request goes through and the error is logged. Fail-closed would be safer for high-stakes environments, but this is a homelab and I’d rather not block all cloud LLM access because the local model is warming up.

Kubernetes deployment

ai-guard runs in the same namespace as llama-server and Open WebUI (web-ai-engine). Intra-namespace traffic is always allowed in Cilium, so no new network policy needed.

Open WebUI uses semicolon-separated lists for multiple API backends:

- name: OPENAI_API_BASE_URLS
  value: "http://llama-server.web-ai-engine.svc:8080/v1;http://ai-guard.web-ai-engine.svc:8080/v1"
- name: OPENAI_API_KEYS
  value: "sk-no-key;sk-no-key"

The second entry is ai-guard. Open WebUI passes sk-no-key as the API key — ai-guard ignores it and uses its own UPSTREAM_API_KEY from a Kubernetes Secret (pulled from Vault via External Secrets Operator). The real NVIDIA API key never touches the client.

The latency tradeoff

The classification step adds 5–15 seconds on CPU inference. That’s the cost of keeping the check fully private — the classifier never sends data anywhere.

For a personal homelab assistant, this is fine. For a high-throughput production setup, you’d want the classifier on a GPU or a dedicated smaller model purpose-built for classification.

What it catches

The classifier prompt targets:

• Passwords, API keys, tokens, credentials
• PII: names, emails, phone numbers, SSNs, addresses
• Financial data: card numbers, bank accounts
• Private keys

False negatives are possible — no classifier is perfect. This is a first line of defense, not a compliance control. The value is catching the obvious, accidental leaks.

Source

github.com/janos-gyorgy/ai-guard — MIT licensed, Kubernetes manifests included.

The PII guardrail proxy I built last week works by classifying prompts and blocking the sensitive ones. That’s fine for a chat interface where a human can rephrase. It doesn’t work for automated pipelines.

If a Jira ticket contains someone’s name and an internal hostname, you don’t want the agent to fail — you want it to process the ticket without exposing that data. Blocking is the wrong primitive for pipelines. Anonymization is the right one.

The pattern

Input text
  → anonymizer: extract PII, replace with semantic fakes
  → "Nathan Chen from DataSoft LLC needs ProjectX fixed on dev.internal.net"
  + mapping: {"Nathan Chen" → "John Smith", "DataSoft LLC" → "ACME", ...}
  → cloud LLM: processes coherent text, never sees real values
  → "Nathan Chen should check the ProjectX docs with the DataSoft LLC team"
  → string substitution with reverse mapping
  → "John Smith should check the OAuth docs with the ACME team"

Two things that make this work:

Deanonymization needs no LLM. Once you have the mapping, restoring is pure string substitution. The model call only happens on the way in.

Semantic fakes beat placeholder tokens. An earlier version of this used [PERSON_1], [ORG_1] tokens. The problem: cloud models see bracketed text and subtly change behaviour — shorter responses, hedging, dropped context. When the cloud model sees Nathan Chen from DataSoft LLC, it treats it as real text and responds naturally. Quality is noticeably better.

Prior art — what already exists

This is a well-established pattern. Worth knowing what’s out there:

LLM Guard (Protect AI) — the most complete open-source implementation. Anonymize + Deanonymize scanner pair with a Vault for the mapping. Production-grade, actively maintained. Start here if you’re building this for anything serious.

Microsoft PII Shield — session-based proxy. Returns a session ID with the anonymized text, uses it to deanonymize the response.

anonLLM — uses GLiNER (a proper NER model) + Faker for realistic replacements. Better accuracy than a general chat model.

REDACT — IEEE paper describing a system using Ollama for PII redaction in documents.

HuggingFace Anonymizer SLM series — purpose-built models (0.6B/1.7B/4B) fine-tuned specifically for anonymization. 9.20/10 quality score for 1.7B, close to GPT-4.1’s 9.77.

That last one is what this implementation actually uses.

The model: Anonymizer-1.7B

eternisai/Anonymizer-1.7B is a Qwen3-1.7B fine-tune trained on ~30k anonymization samples using GRPO with GPT-4.1 as judge. It outputs structured tool calls instead of free text:

{
  "name": "replace_entities",
  "arguments": {
    "replacements": [
      {"original": "John Smith", "replacement": "Nathan Chen"},
      {"original": "ACME Corp", "replacement": "DataSoft LLC"},
      {"original": "auth.acme.internal", "replacement": "dev.internal.net"}
    ]
  }
}

No prompt engineering needed. The model knows exactly what it’s doing and outputs a structured contract. Compare that to the first version of this service, which sent a long JSON-format prompt to Phi-3.5-mini and hoped the output parsed correctly.

The model runs via Ollama (which handles the Qwen3 chat template and tool calling natively), pointed at the GGUF version from HuggingFace: hf.co/gabriellarson/Anonymizer-1.7B-GGUF.

The implementation

llm-anonymizer is a FastAPI service with two endpoints.

POST /anonymize — calls Ollama with the tool definition, parses the response:

TOOLS = [{
    "type": "function",
    "function": {
        "name": "replace_entities",
        "description": "Replace PII entities with anonymized versions",
        "parameters": {
            "type": "object",
            "properties": {
                "replacements": {
                    "type": "array",
                    "items": {
                        "type": "object",
                        "properties": {
                            "original": {"type": "string"},
                            "replacement": {"type": "string"},
                        },
                        "required": ["original", "replacement"],
                    },
                }
            },
            "required": ["replacements"],
        },
    },
}]

resp = await client.post(f"{OLLAMA_BASE}/api/chat", json={
    "model": MODEL,
    "messages": [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": text + "\n/no_think"},  # skip Qwen3 thinking mode
    ],
    "tools": TOOLS,
    "stream": False,
})

tool_calls = resp.json()["message"]["tool_calls"]
replacements = tool_calls[0]["function"]["arguments"]["replacements"]

# Build reverse mapping: replacement → original (for deanonymization)
anonymized = text
mapping = {}
for pair in replacements:
    anonymized = anonymized.replace(pair["original"], pair["replacement"])
    mapping[pair["replacement"]] = pair["original"]

The /no_think suffix tells the model to skip its chain-of-thought — faster response, same accuracy for this task.

POST /deanonymize — no model call, just substitution:

for replacement, original in sorted(mapping.items(), key=lambda x: len(x[0]), reverse=True):
    text = text.replace(replacement, original)

Sorted by length descending so longer tokens don’t get partially overwritten by shorter ones.

The Kubernetes stack

Ollama runs as a separate deployment in the same namespace as everything else (web-ai-engine). Intra-namespace traffic is always allowed — no new network policies.

llm-anonymizer (FastAPI) → Ollama (port 11434) → Anonymizer-1.7B GGUF

One-time model pull after first deploy:

kubectl exec -n web-ai-engine deploy/ollama -- \
  ollama pull hf.co/gabriellarson/Anonymizer-1.7B-GGUF

Ollama caches it on a 10Gi PVC, so pod restarts don’t re-download.

The n8n pipeline

Five-node chain triggered by webhook:

Webhook → /anonymize → NVIDIA NIM → /deanonymize → Respond

The NVIDIA NIM call includes a system prompt instructing it to treat the text as normal input. No mention of tokens, no special handling — because the text looks like real text.

Wire any upstream source to the webhook: Jira event, Slack slash command, a scheduled job that processes internal docs. The pipeline is source-agnostic.

The caveats

1.7B isn’t GPT-4.1. The model scores 9.20/10 on the benchmark — which means roughly 1 in 10 cases has a missed or incorrect entity. Test with real examples from your domain before depending on it.

Deanonymization breaks on heavy rephrasing. If the cloud model restructures a sentence enough that the fake value no longer appears verbatim, the substitution silently misses it. The prompt helps but doesn’t eliminate the risk.

Ollama adds a deployment. It’s ~500MB image + the model weights (~1GB Q4). On a constrained single-node cluster that’s real overhead. llama-server already covers general chat; Ollama is purely for this model’s tool-calling support.

Source

github.com/janos-gyorgy/llm-anonymizer — MIT licensed, Kubernetes manifests and n8n workflow included.

Running a local model is easy. Understanding what it’s doing is less so.

After deploying llama.cpp + Open WebUI on k3s (previous post), I had a chat interface backed by a local model. What I didn’t have: any visibility into how the model was behaving — whether requests were queuing, how fast tokens were being generated, how much of the context window was in use.

The instinct for this kind of problem is usually “add a proxy layer.” There are several tools in this space — LiteLLM being the most popular — that sit between the client and the inference server and record token counts, latency, and spend. I tried this first. LiteLLM OOMed at startup on a node already at 76% memory. Heavy Python import tree, not a lot of headroom.

The thing I’d missed: llama.cpp ships a Prometheus metrics endpoint. No proxy required.

--metrics

One additional argument to the inference server:

args:
  - -m
  - /models/Phi-3.5-mini-instruct-Q4_K_M.gguf
  - --host
  - "0.0.0.0"
  - --port
  - "8080"
  - --ctx-size
  - "4096"
  - --n-predict
  - "1024"
  - --parallel
  - "1"
  - --metrics        # ← this
  - --log-disable

After restart, GET /metrics on port 8080 returns valid Prometheus exposition format:

# HELP llamacpp:tokens_predicted_total Number of generation tokens processed.
# TYPE llamacpp:tokens_predicted_total counter
llamacpp:tokens_predicted_total 0

# HELP llamacpp:predicted_tokens_seconds Average generation throughput in tokens/s.
# TYPE llamacpp:predicted_tokens_seconds gauge
llamacpp:predicted_tokens_seconds 0

# HELP llamacpp:requests_processing Number of requests processing.
# TYPE llamacpp:requests_processing gauge
llamacpp:requests_processing 0

The full set of metrics:

These cover the metrics that matter for a personal inference server: throughput, latency (derivable from total time / total tokens), and queue depth.

Prometheus scrape config

Adding a static scrape target in the existing Prometheus configuration:

extraScrapeConfigs: |
  - job_name: llama-server
    static_configs:
      - targets:
          - llama-server.web-ai-engine.svc:8080
    metrics_path: /metrics

The only non-obvious thing here is the network policy: Prometheus lives in dashboard-homelab, and llama-server lives in web-ai-engine. With Cilium network policies enforcing namespace isolation, the dashboard namespace needs to be allowed to make inbound connections to the AI engine namespace. In applications.yml:

- namespace: web-ai-engine
  networkPolicies:
    allowIngressFromNamespaces: [dashboard-homelab]

Without this, Prometheus scrape attempts fail silently with a timeout.

Grafana dashboard via ConfigMap

Rather than importing a dashboard JSON manually through the Grafana UI, the Grafana sidecar handles it automatically. Any ConfigMap with the label grafana_dashboard: "1" is picked up, loaded, and available in Grafana — across all namespaces by default.

The dashboard ConfigMap lives in web-ai-engine, not dashboard-homelab. The sidecar finds it regardless:

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-dashboard-llm
  namespace: web-ai-engine
  labels:
    grafana_dashboard: "1"
data:
  llm-metrics.json: |
    {
      "title": "LLM Metrics",
      "uid": "llm-metrics",
      ...
    }

Argo CD reconciles the ConfigMap. The Grafana sidecar picks it up. The dashboard appears. No manual steps, no Grafana UI interaction, no state outside Git.

This means the dashboard is version-controlled, reproducible on cluster rebuild, and consistent across environments. The same YAML that describes the app’s Kubernetes resources also describes what the monitoring looks like.

What the dashboard shows

After sending a few messages through Open WebUI:

Generation throughput — the llamacpp:predicted_tokens_seconds gauge drops to 0 between requests and spikes during generation. On this hardware (Intel N100, CPU-only inference, Phi-3.5-mini Q4_K_M), it reads 3–5 tok/s during active generation. This is the number to watch if you’re comparing models or quantisation levels.

Cumulative tokens — llamacpp:prompt_tokens_total and llamacpp:tokens_predicted_total both increase monotonically. The ratio between them is roughly the input/output ratio of your usage pattern. For conversational use it’s typically 3:1 prompt to generation; for summarisation tasks it flips.

Queue depth — llamacpp:requests_deferred is 0 almost always, which is expected with --parallel 1. If it’s consistently above 0, you have more concurrent users than the server can handle with the current slot configuration.

ms/token — derived from rate(llamacpp:tokens_predicted_seconds_total[5m]) / rate(llamacpp:tokens_predicted_total[5m]) * 1000. This is the per-token latency, which is the number that governs whether the response feels fast or slow. 200–300ms/token feels instant; above 400ms you start noticing.

What’s missing compared to a proxy layer

LiteLLM and similar proxies give you things this setup doesn’t:

• Per-model routing — if you’re running multiple models, a proxy can route requests to the right one. With a single model, irrelevant.
• Virtual API keys — per-user or per-application key scoping. Not needed when the whole thing is behind SSO.
• Spend tracking — meaningful when you’re paying per token. For a local model, the cost is electricity, which Prometheus already covers through the power monitoring dashboard.

For a single-model homelab, the native metrics are sufficient. If I add more models later or need per-user attribution, a proxy layer becomes worth the RAM.

The pattern

The broader point is that the observable unit here isn’t the proxy — it’s the inference server itself. Scraping llama.cpp directly means the metrics survive proxy changes, backend swaps, or routing redesigns. The inference server is the thing doing the work; it’s the right place to measure.

Starter manifests with the metrics configuration included: homelab-ai-inference-starter

Most write-ups about self-hosting LLMs start with a GPU. A 3090, an A100, at minimum something with CUDA. The implication is that without one you’re wasting your time — inference will be too slow to be useful.

That’s not been my experience.

I’ve been running a local LLM stack on a ThinkCentre mini PC (Intel N100, 16 GB RAM, no discrete GPU) for a few months. The model is Phi-3.5-mini-instruct, 3.8 billion parameters, 4-bit quantised. Response time is 3–6 tokens per second on CPU — slow enough that you notice it, fast enough that you use it. For the things I actually reach for a local model to do — rephrase something, summarise a document, explain a config option without sending it to an external API — the latency is fine.

The point isn’t that CPU inference beats GPU inference. It’s that “good enough for personal use” is a much lower bar than “production LLM serving”, and the hardware you already have probably clears it.

The stack

Two components:

llama.cpp (ghcr.io/ggml-org/llama.cpp:server) — inference server that loads a GGUF model file and exposes an OpenAI-compatible REST API. No Python, no framework overhead, minimal memory footprint beyond the model itself.

Open WebUI (ghcr.io/open-webui/open-webui) — a polished chat interface that speaks OpenAI API format. It points at the llama-server endpoint as its backend, handles conversation history, and supports RAG file uploads out of the box.

The architecture is simple on purpose:

Browser → Open WebUI (:80)
              │
              │  OpenAI-compatible API
              ▼
         llama-server (:8080)
              │
              │  reads GGUF model file
              ▼
         hostPath /srv/ai-models

Open WebUI doesn’t know or care that the backend is llama.cpp running on CPU. It sees an OpenAI-compatible API. This matters: if I swap llama-server for Ollama, vLLM, or a cloud endpoint, the frontend doesn’t change. The interface is the standard.

Model choice

GGUF models on Hugging Face are available at multiple quantisation levels. The trade-off is quality vs. RAM:

On 16 GB RAM with a full k3s stack running alongside (Argo CD, Traefik, Vault, Prometheus, etc.), Phi-3.5-mini leaves enough headroom that the cluster stays stable. Mistral-7B works too, but it’s tighter.

Models live in /srv/ai-models on the node, mounted into the pod as a hostPath volume. Single-node homelab, so there’s no scheduling concern. Download once with wget, done.

Key configuration choices

Context size (--ctx-size 4096): How many tokens the model holds in its attention window. Larger context = more RAM + slower inference. 4096 is fine for conversational use. If you’re summarising long documents, bump to 8192 and watch your RAM usage.

Max output tokens (--n-predict 1024): Hard cap on response length. llama.cpp will stop there even mid-sentence. 1024 is usually enough; increase if you find it cutting off long explanations.

Parallel slots (--parallel 1): How many concurrent inference requests the server handles. On CPU there’s no benefit to more than 1 — each slot competes for the same cores. Leave it at 1.

Memory limits: Set the container limit to roughly 2× the model’s file size. A 2.4 GB GGUF typically uses 3–4 GB at runtime with context loaded.

resources:
  requests:
    cpu: 500m
    memory: 1Gi
  limits:
    memory: 6Gi

No CPU limit. llama-server will use however many cores are available during inference — that’s what makes it usable. A CPU limit would throttle inference to unusable speeds.

Deployment as a GitOps push

The whole stack lives in one YAML values file, deployed through the extra-objects chart that I use for raw manifests across the cluster. Argo CD watches the repo and reconciles automatically.

Nothing was kubectl apply-ed. The deployment happened by pushing to Git.

What that means in practice: when I bumped the Open WebUI image version, I changed one line, pushed, and Argo CD rolled the pod. No manual steps, no SSH, no kubectl. The same process I use for any other service in the cluster.

The namespace, network policies, service account, and RBAC all generate from a single entry in applications.yml — same as every other app. The AI inference stack isn’t special from an operations perspective.

# applications.yml excerpt
- namespace: web-ai-engine
  applications:
    - applicationCode: web-ai-engine
      path: helm-charts/extra-objects
      autoSync: true

Access and auth

The service is exposed at ai.hippotion.com through the same dual-path ingress setup I use everywhere: Cloudflare Tunnel for external access, direct-to-server via Pi-hole DNS for local access, Traefik handling both with a wildcard Let’s Encrypt cert. See that post for the full explanation.

Auth is handled by Traefik’s ForwardAuth middleware pointing at an oauth2-proxy backed by GitLab. Open WebUI’s own auth is disabled (WEBUI_AUTH: false) — the OAuth layer upstream handles it. One login covers every service in the cluster.

The WEBUI_SECRET_KEY (used to sign Open WebUI sessions) comes from Vault via External Secrets Operator. Nothing sensitive in Git.

What the day-to-day is actually like

Slow is the obvious caveat. Phi-3.5-mini at 3–6 tok/s means a paragraph-length response takes 20–30 seconds. For coding help where you’re reading what came before while it generates, that’s fine. For quick factual lookups, it’s a little tedious.

The useful cases for a local model, for me:

• Rephrasing or editing text — paste something, ask it to tighten it. No data leaves the house.
• Config explanation — paste a Kubernetes manifest or a Traefik config block, ask what it does. Again, stays local.
• Quick summaries — short documents, log snippets, error messages.
• Experimentation — trying prompting techniques, testing system prompts, benchmarking quantisation levels without API costs.

For longer reasoning tasks I use a cloud model. The local stack is for the cases where I want the answer to stay on-premises, or where I’m iterating and don’t want to pay per token.

The starting point if you want to try it

The manifests are on GitHub: homelab-ai-inference-starter

It includes the llama-server and Open WebUI deployments, resource configuration, and ingress options for Traefik and nginx. The README walks through downloading a model, applying the manifests, and the configuration knobs worth knowing.

No GPU required. The ThinkCentre in the corner of my desk does the job.

Something’s wrong. A GitLab runner stops picking up jobs. An event processor starts dropping messages. A pod restarts in a loop. The node looks healthy — CPU fine, memory fine — but something is clearly off.

The reflex: restart the node, see if it clears.

Sometimes it does clear, and you move on. But you didn’t fix anything. You reset the state and crossed your fingers. If it happens again in two weeks, you’ll do the same thing. After enough iterations you have a “flaky node” that everyone reboots periodically and nobody understands.

There’s a better sequence. It takes twenty minutes instead of two, and you come out with either a real fix or actual knowledge of what happened.

Step one: quarantine, don’t kill

Before you touch anything, take the node out of rotation without destroying its current state.

kubectl cordon <node>

Cordon marks the node as unschedulable. No new pods land on it. Existing pods keep running. If you need the workloads somewhere else immediately:

kubectl drain <node> --ignore-daemonsets --delete-emptydir-data

Now you’ve removed the node from production traffic without rebooting. The node is still alive. Everything that happened on it is still there: logs, open files, kernel ring buffer, running processes, memory state.

This is the difference. A reboot wipes that. A cordon preserves it.

Step two: look at what’s actually there

SSH in. Don’t grep for anything specific yet — do a pass for anything unusual.

Kernel messages first. The kernel will often tell you exactly what went wrong before any application did.

dmesg -T --level=err,warn | tail -50

OOM kills show up here. Disk errors show up here. CPU soft lockups show up here. If you’ve got any of those, you have your answer before you’ve even looked at application logs.

Check for filesystem problems.

df -h          # is anything full?
dmesg | grep -i "ext4\|xfs\|btrfs\|i/o error\|ata"

A filesystem at 100% is silent until it isn’t. A flaky drive starts dropping I/O errors into dmesg long before SMART reports anything. Application developers rarely think about this case — their app just starts writing logs that say “failed to write” without specifying that the disk is full or dying.

System resource pressure.

vmstat 1 5          # is there swap activity?
iostat -x 1 5       # is a disk saturated?
cat /proc/pressure/io   # kernel PSI — pressure stall info

PSI is underused. It tells you whether processes were actually stalled waiting for I/O, not just whether throughput was high. A disk at 80% utilisation might be fine; a disk with 40% I/O PSI pressure is actively hurting performance.

What were the pods doing right before things went sideways?

kubectl describe node <node>    # events section at the bottom
kubectl get events --field-selector involvedObject.kind=Pod -A | sort -k1

Look for OOMKilled exits, failed liveness probes, and throttling events. Kubernetes events expire after an hour by default — another reason not to reboot immediately; those events are still there if you look now.

A real example: the GitLab runner

A GitLab runner pod stops picking up jobs. It looks alive — the process is running, no crashes in the pod logs. Jobs sit in the queue.

Restart reflex: delete the pod, let it reschedule, it picks up jobs again.

But why did it stop?

journalctl -u gitlab-runner --since "1 hour ago"
# or, if it's a container:
kubectl logs <runner-pod> --previous

In one instance: the runner’s working directory was on a tmpfs that hit its size limit. The runner silently failed to create job workspaces and stopped accepting new jobs. The error was one line in the pod logs: mkdir /builds: no space left on device. The pod was healthy by every other metric.

Fix: bump the tmpfs size limit in the runner config. The restart would have cleared tmpfs temporarily, and the runner would have failed again the next time a large job filled it up.

The debug took five minutes. The permanent fix took two minutes. Without quarantining the node first, the evidence was gone.

Another one: the event consumer

An event processor starts falling behind. Messages queue up. The pod shows no errors. Memory looks fine.

This one was subtler: the processor was connected to a downstream dependency over a persistent TCP connection. The connection had gone into a half-open state — the processor thought it was alive, the remote end had already dropped it. New messages were being sent into a dead socket and silently discarded.

ss -tnp | grep <pid>    # look at the socket state

CLOSE_WAIT on a connection that should be ESTABLISHED. The application wasn’t checking whether the connection was actually working before using it, just whether it existed.

Restart would have cleared the socket state, fixed the symptom, and left the bug in the code.

What to look for — a short checklist

When a node is misbehaving, in order:

1. dmesg -T --level=err,warn — kernel errors, OOM kills, disk errors
2. df -h && df -i — full filesystems (space and inodes separately)
3. kubectl describe node <node> — pressure conditions, recent events
4. kubectl logs <pod> --previous — what the pod logged before it died or got stuck
5. ss -tnp — socket states for network-adjacent issues
6. vmstat 1 5 + iostat -x 1 5 — resource pressure
7. journalctl -p err -b — system journal errors since last boot

Most problems show up in the first three.

After you’ve found something (or not found something)

If you found the cause: fix it, test it, uncordon the node.

kubectl uncordon <node>

Document what you found — a comment in the relevant config, a commit message, a note. “Fixed runner tmpfs limit” in the commit history is more useful than “flaky runner, restarted.”

If you genuinely found nothing: that’s information too. Cordon, reboot, uncordon, and note that the node rebooted clean with no identified cause. If it happens again, you have a pattern. Check whether anything changed in the workloads around that time. Check whether the reboot timing correlates with anything — cron jobs, backups, maintenance windows.

A reboot you can explain is a fix. A reboot you can’t explain is a time bomb.

Why this matters on a single-node cluster

In a multi-node setup you can afford to be lazier — cordon, drain, reboot, let the scheduler handle it, look at it later. On a single node there’s no “later.” The node coming back is all you’ve got.

But the habit is worth building regardless of node count. The engineers who understand their systems are the ones who looked before they rebooted.

The actual rule

Quarantine first. Debug second. Restart third (if you still need to).

A restart takes two minutes. The evidence it destroys might take two hours to reconstruct — or might be gone for good. The cordon costs you nothing.

I built Dice & Shrines with five asymmetric guardian characters. Each one has a different passive and active ability that changes how reinforcements distribute, which territories you can attack, and what happens when you take damage.

The question I couldn’t answer from just playing was: are they actually balanced?

Not “do they feel different” — they obviously do. But is Fox’s stored critical actually overpowered? Is Turtle’s loss-recovery passive strong enough to matter, or is it just flavour? Is there a first-mover advantage baked into the map structure?

You can’t answer questions like these from vibes. You need data. So I built a stats service.

What gets recorded

Every game produces five event types, posted as fire-and-forget HTTP calls from the game client to game-stats.hippotion.com/event:

map_generated — logged when the map generator accepts a map. Records territory count, average territory size, minimum size, and how many generation attempts it took. This tells me how often the generator discards its own work and whether the acceptance criteria are too strict.

game_start — fired when a game begins. Captures the number of players, the guardian assigned to each slot, and which slot is human. Returns a gameId that travels with the game for the rest of its life.

attack — fired on every single dice roll. Attacker, defender, from-territory, to-territory, how many dice each side had, what they rolled, who won. This is the raw material for the probability analysis.

elimination — fired when a player is knocked out. Records which guardian they were and how many players remained, so I can tell who exits first and who makes the final stand.

game_end — fired on win or abandon. Records the winner’s guardian, how many turns the game took, and whether it was abandoned.

The service is a FastAPI app backed by PostgreSQL, running in the homelab on the same k3s cluster as the game. About 150 lines of Python plus a schema.sql that the app runs on startup.

The dashboard

The stats dashboard is a single-page HTML response from / — self-contained, no external framework, chart.js for the visualisations. It polls /api/stats every 30 seconds and updates in place.

What it shows:

Overview cards: total games, games today, games this week, human win rate, average turns per game, overall attack win rate, abandoned game count.

Activity charts: games per day (last 7 days), game duration distribution in 10-turn buckets.

Death spiral analysis: when players abandon (broken into phases: instant, early, mid-early, mid, late), and first-mover advantage — win percentage by player slot 0 through 5.

Attack behaviour: the dice margin chart is the most interesting one. It shows attack volume and win rate for every possible attacker-dice-minus-defender-dice value, from strongly negative (attacker is outmatched) to strongly positive. Overlaid: a win rate line. You can see the actual probability curve emerging from real games and compare it to what the math predicts.

Guardian intelligence: win rate, pick count, average attacks per game, survival rate to turn 50+, and average turns per winning game — per guardian, human players only.

Elimination intelligence: when the first player gets knocked out per game, and a guardian fate table showing average elimination order and first-out percentage. Earliest-exiting guardian is surfaced explicitly.

Map influence: territory count versus average game length. Also an attack efficiency heatmap — win rate for every attacker-dice × defender-dice combination, 1 through 8, rendered as a colour grid.

Recent games: last 15 games with the human player’s guardian, result, and IP address so I can tell if it’s me testing or an actual player who wandered in.

What the data showed

The attack win rate across all games sits just under 60%. That’s higher than a naive analysis suggests it should be — if both sides roll fairly, equal dice should be near-even. The explanation is selection bias: players only attack when they have a dice advantage. Nobody sends 2 dice at 8 dice repeatedly. The average attack has a positive margin, so the average win rate is above 50%.

The margin chart made this explicit. The plurality of attacks have a margin of +2 or more. The sub-zero margin attacks — technically losing plays — are a real but small fraction, usually late-game desperation or deliberate tempo plays.

Human vs AI attack quality turned out to be the sharpest comparison. Humans and AI have different average margins. The AI is greedy but disciplined about attack selection; humans sometimes take gambles the AI wouldn’t. You can see it in the numbers.

First-mover advantage is measurable but not massive. Player slot 0 (goes first) has a slightly higher win rate than the average. Slots at the higher end of turn order are somewhat depressed. Not broken, but real — and a useful thing to watch if I ever add a competitive mode.

Guardian balance: the win rate gap between the best and worst guardian tells me whether the balance is within acceptable range or a concern. The dashboard calls it out explicitly: if the gap exceeds 15 percentage points, it flags it as a balance issue. That threshold is arbitrary, but it forces a decision rather than letting drift accumulate unnoticed.

Abandonment phases: most abandonments are instant — the player clicked “new game” before actually playing. The interesting number is mid-game abandonment, which is a proxy for death spirals: you see your income drop, you know you’re losing, you close the tab. That’s a design signal, not just a metric.

Designing for measurement

The useful insight from building this is that it changes how you design the game. Once you know every attack is being logged, you start thinking about what the attack data will tell you. Shrines give territories a guaranteed die — does that show up in attack margins near shrine territories? I didn’t add territory-topology tracking, but I could. The schema is just a few columns away.

The same goes for guardian abilities. Fox’s stored critical fires at turn boundaries — I log turn number on every attack, so I can look for Fox spikes in attack win rate on certain turns. I haven’t run that query yet, but the data is there if the balance question becomes sharp enough to need it.

That’s the thing about adding observability to something you built yourself: you stop guessing about whether it’s working and start reading the evidence. The game got more interesting to design once I could see what was actually happening inside it.

The stack

• FastAPI — event intake and stats API, ~150 lines
• PostgreSQL — five tables: maps, games, game_guardians, attacks, eliminations
• chart.js — dashboard visualisations, loaded from CDN
• k3s + Argo CD — deployed as a Kubernetes pod, Dockerised, managed GitOps alongside everything else on the homelab

Source at dice-n-shrines-stats.

I wanted to get a feel for AI-assisted coding tools — Claude Code, Codex — in a low-stakes environment where breaking things was fine. A browser game seemed like the right vehicle: self-contained, no prod database, no users to disappoint.

I picked a premise I knew was fun: Dice Wars. The Flash-era classic. Roll dice to attack adjacent territories, biggest army snowballs. Simple enough that I could focus on the tooling rather than the design. Or so I thought.

Six weeks later I had five asymmetric character classes, a procedural hex map generator with acceptance criteria, a FastAPI telemetry service recording every dice roll, and a stat dashboard I check more than I probably should. The tooling became a background concern. The game took over.

How the game works

The rules are genuinely minimal.

You start with a random slice of a procedurally generated map — a patchwork of irregular coloured territories, each one a cluster of hex tiles that reads as a solid blob. You and up to seven opponents begin scattered across it. Objective: own everything.

Attacking is one click. Select your territory, click an adjacent enemy territory. Both sides roll all their dice and sum. Higher total wins. Attacker wins: you capture the territory, your dice advance in minus one. Defender wins: your attack is repelled, you’re reduced to a single die on the attacking territory.

Reinforcements are the mechanic that makes this a strategy game. At the end of your turn, you receive bonus dice equal to the size of your largest contiguous group of territories. Not total territories — the biggest connected blob. Fragmented territory generates almost nothing. A solid connected bloc snowballs.

That one rule creates the entire strategic texture. Grab fast but stay connected. Chokepoints are worth defending at a loss. Cutting an opponent in half collapses their income immediately. The late game turns into tense standoffs until one roll cracks something open.

The shrines

Early on I added shrines — special territories marked with a ★. They behave differently from normal territories in a few ways:

• Higher dice cap: normal territories max out at 8 dice, shrines at 10
• Minimum floor: a shrine never drops below 2 dice after attacking, win or loss — it can’t be stripped bare
• Guaranteed reinforcement: the shrine gets a die first at end of turn, before random distribution
• Aura: each of your own territories adjacent to the shrine gets a +1 guaranteed die (shown with a dim ◆ indicator)

The shrine mechanic does something interesting to the risk calculus. Holding a shrine isn’t just a territory — it’s a node that warps the value of everything adjacent to it. You’ll defend an aura territory harder than a territory of equivalent size elsewhere on the map, because losing it means losing the aura bonus. It also means attacking into a shrine aura is expensive: the shrine can’t be worn down easily, and the neighboring territories keep refilling.

Shrines turned out to be the moment the math got interesting.

Five guardians

The game has a character select screen. Each player — human or AI — picks a guardian before the map generates. Five options, each with a passive and an active ability that meaningfully change how you play:

Hippo gets to manually place one reinforcement die each turn before the rest distribute randomly. One die placement doesn’t sound like much until you realise you always have a frontline territory that needs it more than anywhere else. High floor, consistent.

Hedgehog fills weakest territories first during reinforcement. Defensively solid — you never bleed out a border territory to zero while inland territories sit at cap. It doesn’t generate more dice, but it wastes fewer.

Fox banks a stored die every other turn and can spend it as a critical multiplier on an attack. The stored dice accumulate (up to a cap), so the power compounds if you resist using it. Two-hop flanking passive: Fox can attack across two territory hops from border territories, making it very hard to feel safely tucked away from.

Owl has a passive two-hop attack range like Fox but for all attacks — Owl sees further. The active ability is a Dice Transfer: move dice from one of your territories to an adjacent friendly one. Lets you concentrate force without committing to an attack.

Turtle gets 2 dice back to a neighboring territory any time it loses a defense. It’s the only guardian that turns taking damage into a resource. Hard to snowball, but very hard to finish off.

The AI cycles through the five guardians deterministically, so in a full 8-player game you’ll face at least one of each.

The map

The map generator produces irregular coloured territories from a hex grid. Hexes are the visual scaffolding — what matters in gameplay is the blob they form. Internal edges within a territory are hidden; only the outer border of each cluster is drawn. The result reads like a contested piece of ground rather than a grid.

The generator has acceptance criteria. A proposed map is rejected if territory sizes are too uneven — a tiny starting territory with 2 hexes versus a sprawling 15-hex territory produces a wildly unfair game. The stats service actually records generation attempts and acceptance rates per map, so I can see how often the map generator throws away its own work.

The mapId is stamped on every game record, so I can eventually correlate map topology with game outcomes. I haven’t done that analysis yet, but the data is there.

What the tools actually felt like

Claude Code handles the kind of work that benefits from holding a lot of context simultaneously: “this change to resolveAttack() in game.js needs corresponding updates to the AI logic in ai.js and the render pass in render.js.” That’s tedious to track manually and exactly the kind of thing where the tool earns its keep.

Codex was more useful for the boilerplate-heavy parts — filling in schema.sql, wiring up FastAPI endpoints, the chart.js setup in the dashboard. Directed generation of code you know exactly how should look.

Neither tool replaces thinking. The guardian ability design, the shrine balance, the question of whether Fox’s stored critical is too swingy in epic mode — that’s all still just sitting down and working it out. The tools speed up the translation from “I know what I want” to “here is working code.” The game design itself doesn’t compress.

It’s live

The game runs at dice.hippotion.com. Single HTML file served from a Kubernetes ConfigMap on my homelab. No accounts, no install. It runs fast — AI turns are instant when you toggle off animations, and a full game can be over in five minutes or stretch to twenty depending on how the map falls.

Stats dashboard at game-stats.hippotion.com. More on that in the next post.

“How do you achieve zero-downtime deployments in Kubernetes?”

The expected answer: rolling updates. That’s correct but incomplete. Rolling updates are the mechanism. They don’t give you zero downtime automatically — they give you a framework in which zero downtime is achievable, if you configure everything correctly.

Most clusters cause brief downtime on every deployment. Usually 5–30 seconds. Usually blamed on “the load balancer” or “DNS”. Almost always caused by one of four missing pieces.

The rolling update baseline

Kubernetes replaces pods in waves. You control the pace:

spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1        # how many extra pods can exist during update
      maxUnavailable: 0  # how many pods can be unavailable during update

maxUnavailable: 0 means Kubernetes never terminates a pod until a replacement is ready. This prevents the obvious failure mode where you have zero running pods mid-deployment.

maxSurge: 1 means one extra pod beyond the desired count runs during the update. For a deployment with 3 replicas, you’ll briefly have 4 pods running.

This alone doesn’t prevent downtime.

Piece 1: The readiness probe (the most common missing piece)

Kubernetes considers a pod “ready” when all its containers pass their readiness probes. If you don’t define a readiness probe, Kubernetes considers the pod ready as soon as the container starts. Containers start before applications are ready to serve traffic.

# Without this, traffic arrives before your app is listening
readinessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5
  failureThreshold: 3

What happens without it: Kubernetes starts the new pod, marks it ready immediately, adds it to the Service endpoints, routes traffic to it — while your app is still initialising (loading config, connecting to the database, warming caches). The first few requests to the new pod fail or time out.

The fix: define a readiness probe that actually checks application readiness. An HTTP endpoint that returns 200 only after the app has finished starting is the minimum. A deeper check that verifies the database connection is better.

Common mistake: using the same endpoint for liveness and readiness with the same thresholds. They serve different purposes:

• Readiness: “am I ready to accept traffic?” — controls whether traffic is sent
• Liveness: “am I still alive?” — controls whether the pod is restarted

A pod can fail its readiness probe (temporarily overloaded, warming up) without failing its liveness probe. If you make liveness too aggressive, Kubernetes restarts pods that would have recovered on their own.

Piece 2: The termination grace period (the other common missing piece)

When Kubernetes wants to terminate a pod, it sends SIGTERM. Your application has terminationGracePeriodSeconds (default: 30) to finish in-flight requests and shut down cleanly. After that, Kubernetes sends SIGKILL.

The problem: there’s a race condition. Kubernetes removes the pod from the Service endpoints and sends SIGTERM roughly simultaneously. The endpoint update has to propagate through the control plane, kube-proxy, and the load balancer. During that propagation window — typically 1–10 seconds — traffic can still arrive at a pod that has already started shutting down.

The fix is a preStop hook that adds a short sleep before the termination sequence:

lifecycle:
  preStop:
    exec:
      command: ["sleep", "5"]

This gives the endpoint removal time to propagate before your app receives SIGTERM. The total shutdown sequence is then:

1. Kubernetes removes pod from endpoints
2. preStop hook runs (sleep 5s — enough for endpoint propagation)
3. SIGTERM is sent
4. App drains in-flight requests and shuts down
5. If still running after terminationGracePeriodSeconds: SIGKILL

Set terminationGracePeriodSeconds to cover the sleep plus your app’s actual shutdown time:

spec:
  terminationGracePeriodSeconds: 60  # 5s preStop + up to 55s for app shutdown

Without the sleep: requests fail during the propagation window. With it: the window is covered.

Piece 3: PodDisruptionBudgets (for node maintenance)

Rolling updates handle normal deployments. Node drains (kubectl drain, cloud provider maintenance windows, k3s upgrades) are a different code path that bypasses your rolling update strategy entirely.

When a node is drained, Kubernetes evicts all pods on it as fast as it can. Without constraints, it will evict all replicas of your deployment simultaneously if they all happen to land on the same node.

A PodDisruptionBudget sets a floor:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: myapp-pdb
  namespace: myapp
spec:
  minAvailable: 1   # at least 1 replica must stay up during disruption
  selector:
    matchLabels:
      app: myapp

Now node drain will evict pods one at a time, waiting for replacement pods to come up before evicting the next one. If no replacement can be scheduled (e.g., you’re draining the only node), the drain will block rather than cause downtime.

minAvailable: 1 is the minimum. For production with 3+ replicas, minAvailable: 2 or maxUnavailable: 1 is more appropriate.

Piece 4: minReadySeconds (the one everyone forgets)

Even with a correct readiness probe, there’s a subtle risk: a pod that passes its readiness probe briefly and then fails due to a transient startup issue (flapping). Kubernetes would add it to the endpoint pool, route traffic to it, watch it fail the readiness probe, remove it — and during that window, some requests fail.

minReadySeconds says: a pod must pass its readiness probe continuously for this many seconds before Kubernetes considers it “available” and allows the next pod in the rolling update to be terminated:

spec:
  minReadySeconds: 10

This slows deployments slightly but catches flapping probes before they cause production traffic to hit an unstable pod.

The complete deployment snippet

Putting it together:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  namespace: myapp
spec:
  replicas: 3
  minReadySeconds: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: myapp
          image: myapp:latest
          lifecycle:
            preStop:
              exec:
                command: ["sleep", "5"]
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5
            failureThreshold: 3
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 15
            periodSeconds: 10
            failureThreshold: 5

And the PDB alongside it:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: myapp-pdb
  namespace: myapp
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: myapp

What interviewers are actually testing

The follow-up is usually: “What if your new version has a bug that isn’t caught immediately — how do you roll back?”

kubectl rollout undo deployment/myapp reverts to the previous ReplicaSet. Kubernetes stores the last few ReplicaSets by default (revisionHistoryLimit, default 10). The rollback uses the same rolling update mechanism, so it’s also zero-downtime.

The harder follow-up: “What if the bug only shows up after 10 minutes of load?” That’s where you need a canary deployment — send a small percentage of traffic to the new version, observe, then shift the rest. Argo Rollouts handles this natively. Without it, you’re doing it manually with two Deployments and weighted Services.

This is part of a series on Kubernetes interview questions. Previously: secrets in a GitOps repo. Next: network isolation between services.

“How do you prevent configuration drift in a Kubernetes cluster?”

Configuration drift: the cluster’s actual state diverges from what’s declared in your source of truth. Someone runs kubectl edit deployment myapp to bump a memory limit during an incident. Someone adds a debug sidecar directly. Someone applies a YAML file from their laptop that was never committed to Git. The fix works. It goes undocumented. Six months later, a new deployment overwrites it. The incident recurs.

There are two distinct problems here that require different solutions:

1. Detection and remediation: how do you notice drift and revert it?
2. Prevention: how do you stop non-compliant resources from being created in the first place?

Detection and remediation: Argo CD selfHeal

If you’re using GitOps with Argo CD, detection and remediation are handled for you:

syncPolicy:
  automated:
    prune: true
    selfHeal: true

selfHeal: true means Argo CD continuously compares the cluster state to the Git repo and reverts any divergence. Someone runs kubectl edit deployment myapp and changes the replica count? Argo CD detects the diff on its next reconciliation cycle (default: every 3 minutes) and reverts it.

prune: true means resources that exist in the cluster but not in Git are deleted. Someone kubectl apply’d a debug pod directly? Gone on the next sync.

This is the audit trail story too. Every legitimate change is a Git commit with an author, a timestamp, and a commit message. Everything that isn’t in Git doesn’t survive past the next reconciliation. If you want to know what changed and when, git log is the answer.

The gap selfHeal doesn’t close

selfHeal reverts drift after the fact. There’s a window — up to 3 minutes — where a drifted resource is serving traffic. For most changes, that’s fine. For a bad resource (wrong RBAC, missing network policy, container running as root), 3 minutes is enough to be a problem.

The other gap: selfHeal doesn’t tell you who made the change or generate an alert. It just silently fixes it. You need audit logging (kube-apiserver --audit-log-path) or an alerting rule on Argo CD’s health events to know that drift happened.

Prevention: Kyverno

Kyverno is a policy engine that runs as a Kubernetes admission webhook. Every resource creation or modification goes through it before being persisted. If the resource violates a policy, Kyverno can reject it outright (enforce mode) or allow it with a warning (audit mode).

The policies are Kubernetes resources themselves — they live in Git, they’re applied via GitOps, they’re versioned. No separate policy language to learn.

A policy that requires readiness probes on all Deployments:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-readiness-probe
spec:
  validationFailureAction: Enforce
  rules:
    - name: check-readiness-probe
      match:
        any:
          - resources:
              kinds:
                - Deployment
      validate:
        message: "Deployments must define a readiness probe."
        pattern:
          spec:
            template:
              spec:
                containers:
                  - (name): "*"
                    readinessProbe:
                      (httpGet | tcpSocket | exec): "*"

With this policy active: kubectl apply -f deployment-without-probe.yaml is rejected at the API server. The error message is the one you defined in message. The deployment never reaches etcd.

A policy that blocks containers running as root:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-root-containers
spec:
  validationFailureAction: Enforce
  rules:
    - name: check-runAsNonRoot
      match:
        any:
          - resources:
              kinds: [Deployment, StatefulSet, DaemonSet]
      validate:
        message: "Containers must not run as root."
        pattern:
          spec:
            template:
              spec:
                containers:
                  - (name): "*"
                    securityContext:
                      runAsNonRoot: true

A policy that enforces resource limits (common in multi-tenant clusters):

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-resource-limits
spec:
  validationFailureAction: Enforce
  rules:
    - name: check-limits
      match:
        any:
          - resources:
              kinds: [Deployment]
      validate:
        message: "CPU and memory limits are required."
        pattern:
          spec:
            template:
              spec:
                containers:
                  - resources:
                      limits:
                        memory: "?*"
                        cpu: "?*"

Kyverno can also mutate and generate

Policies aren’t only for validation. Kyverno can mutate incoming resources (add default labels, inject sidecars, set default resource requests) and generate new resources in response to events (create a NetworkPolicy whenever a new namespace is created).

Auto-add a standard label to every Deployment:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-labels
spec:
  rules:
    - name: add-team-label
      match:
        any:
          - resources:
              kinds: [Deployment]
      mutate:
        patchStrategicMerge:
          metadata:
            labels:
              managed-by: kyverno

Auto-create a default NetworkPolicy when a namespace is created:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-default-networkpolicy
spec:
  rules:
    - name: default-deny
      match:
        any:
          - resources:
              kinds: [Namespace]
      generate:
        kind: NetworkPolicy
        name: default-deny-all
        namespace: "{{request.object.metadata.name}}"
        data:
          spec:
            podSelector: {}
            policyTypes:
              - Ingress
              - Egress

The complete drift prevention picture

Developer runs: kubectl apply -f bad-deployment.yaml
  → API server receives request
  → Kyverno admission webhook intercepts
  → Policy check: no readiness probe → Rejected
  → API server returns 403 with Kyverno's message
  → Resource never reaches etcd

Developer runs: kubectl edit deployment myapp (valid change, just not via Git)
  → Edit succeeds (no policy violation)
  → Argo CD reconciliation fires (within 3 minutes)
  → Diff detected: cluster state ≠ Git state
  → selfHeal: revert to Git state
  → If audit logging enabled: event recorded with username and timestamp

Git is the audit trail for what should be there. kube-apiserver audit logs are the trail for what was attempted. Kyverno is the enforcer at admission time. Argo CD is the continuous reconciler. Four layers, each with a different job.

What interviewers are actually testing

The follow-up is usually: “What’s the difference between Kyverno and OPA Gatekeeper?”

Both are admission webhook policy engines. The practical differences:

• Kyverno: policies are k8s-native YAML, no separate language to learn. Generate and mutate policies built in. Easier to get started with.
• OPA Gatekeeper: policies are written in Rego, a purpose-built policy language that’s more expressive but has a steeper learning curve. Better if you’re already using OPA elsewhere (Terraform, microservice authorization).

For a Kubernetes-only environment, Kyverno is the pragmatic choice. For a platform team that uses OPA across the stack, Gatekeeper gives you policy consistency.

The deeper follow-up: “How do you test policies before enforcing them?” Use Audit mode first (validationFailureAction: Audit). Violations are logged as PolicyReport objects but requests aren’t rejected. Review the reports, fix the existing violations, then switch to Enforce. Never flip directly to Enforce in production — you’ll break things that were already running.

This is part of a series on Kubernetes interview questions. Previously: network isolation between services.

“How do you enforce network isolation between services in a Kubernetes cluster?”

The default Kubernetes network model is flat. Every pod can reach every other pod, in any namespace, on any port. There are no firewalls, no ACLs, no segmentation. A compromised frontend pod can connect directly to your PostgreSQL port, your Redis port, your internal admin API, and every other service in the cluster.

This is intentional — Kubernetes doesn’t assume you want isolation, because not everyone does. But if you do want it, you need to add it.

NetworkPolicy: the primitive

A NetworkPolicy is a Kubernetes resource that selects a set of pods and defines what traffic is allowed to reach them (ingress) and what traffic they’re allowed to send (egress). Traffic that isn’t explicitly allowed is dropped.

The catch: NetworkPolicy resources have no effect unless your CNI plugin supports them. The default k3s CNI (Flannel) does not. Calico, Cilium, and Canal do. If you’re running Flannel and you apply a NetworkPolicy, it will be silently ignored — no error, no warning.

The default-deny pattern

The correct starting point is a default-deny policy that blocks everything, applied to the namespace. You then add explicit allow policies for the traffic you actually need.

# Block all ingress and egress in this namespace by default
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: myapp
spec:
  podSelector: {}        # matches all pods in the namespace
  policyTypes:
    - Ingress
    - Egress

With this in place, your pods can’t receive traffic and can’t send traffic. You then add back what you need.

Allowing specific traffic

Allow the web frontend to receive traffic from the ingress controller:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-ingress-from-traefik
  namespace: myapp
spec:
  podSelector:
    matchLabels:
      app: frontend
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: sys-traefik

Allow the backend to talk to PostgreSQL:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-egress-to-postgres
  namespace: myapp
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
    - Egress
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: postgres
      ports:
        - port: 5432
          protocol: TCP

After these two policies: the frontend receives traffic from Traefik, and the backend can reach Postgres. The frontend cannot reach Postgres. The backend cannot receive traffic from the ingress controller. Neither can call anything else.

The DNS gotcha

Once you add a default-deny egress policy, DNS stops working. Your pods can no longer resolve service names because they can’t reach kube-dns in the kube-system namespace.

You need to explicitly allow it:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-egress-dns
  namespace: myapp
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports:
        - port: 53
          protocol: UDP
        - port: 53
          protocol: TCP

Missing this is the most common reason “everything broke after I added NetworkPolicies”. Add it to every namespace that has a default-deny policy.

Cilium: the same model with more power

Cilium implements the standard NetworkPolicy API and adds its own CiliumNetworkPolicy CRD with L7 capabilities.

Standard NetworkPolicy works at L3/L4 — IP addresses and ports. Cilium’s CRD adds:

L7 HTTP filtering: allow specific HTTP methods and paths, not just port 8080.

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-api-reads
  namespace: myapp
spec:
  endpointSelector:
    matchLabels:
      app: api
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: frontend
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
          rules:
            http:
              - method: "GET"
                path: "/api/v1/.*"

DNS-based egress: allow egress to github.com by hostname rather than IP address. This matters for external services with dynamic IPs.

egress:
  - toFQDNs:
      - matchName: "github.com"
    toPorts:
      - ports:
          - port: "443"
            protocol: TCP

Identity-based policies: Cilium assigns a cryptographic identity to each pod based on its labels. Policies are enforced by identity, not IP address. Pod restarts (which change IPs) don’t break policy enforcement.

What a real namespace policy set looks like

For a typical web app with frontend, backend, and database:

Namespace: myapp
├── default-deny-all (ingress + egress, all pods)
├── allow-egress-dns (egress, all pods, port 53)
├── allow-ingress-frontend (ingress frontend, from sys-traefik namespace)
├── allow-egress-frontend-to-backend (egress frontend, to backend:8080)
├── allow-ingress-backend (ingress backend, from frontend)
├── allow-egress-backend-to-postgres (egress backend, to postgres:5432)
└── allow-ingress-postgres (ingress postgres, from backend)

Eight policies. The database has exactly one inbound path: from the backend. The frontend has no path to the database at all. A compromised frontend pod cannot scan the internal network — egress to arbitrary destinations is blocked.

What interviewers are actually testing

The follow-up is usually: “How do you manage this at scale? Writing NetworkPolicies for every namespace by hand doesn’t scale.”

The answer: you don’t write them by hand. You template them. In a GitOps setup, your namespace configuration declares what network access the service needs in a structured form, and a Helm chart or operator generates the actual NetworkPolicy resources from those declarations.

For example, an applications.yml entry might look like:

networkPolicies:
  denyAll: true
  allowIngressFromIngress: true
  allowEgressToNamespaces: ["sys-postgres"]

And a Helm chart translates that into four concrete NetworkPolicy objects. The developer declares intent; the platform enforces it. No one writes raw YAML for each namespace.

The second follow-up: “What about east-west traffic between services in the same namespace?” Add allowIntraNamespace: true as a flag that generates a policy allowing all pod-to-pod traffic within the namespace, while still blocking cross-namespace traffic.

This is part of a series on Kubernetes interview questions. Previously: zero-downtime deployments. Next: preventing configuration drift.

“How would you design a CI/CD pipeline that deploys to Kubernetes without storing any cluster credentials anywhere?”

The expected wrong answer: export your kubeconfig, base64-encode it, paste it into a CI secret named KUBE_CONFIG, and call it a day. This works. Most clusters that got hacked had this setup.

There are two correct answers in 2026, and which one you reach for depends on what you’re actually deploying.

Answer 1: GitOps (the one your interviewer probably wants)

In a GitOps setup, your CI pipeline never touches the cluster. It can’t leak credentials it doesn’t have.

The flow:

Developer pushes code
  → CI builds and tests
  → CI updates the image tag in the Git repo (a commit, not a kubectl command)
  → Argo CD detects the change
  → Argo CD applies it to the cluster

The cluster reaches out to Git. CI never reaches into the cluster. The only thing with cluster credentials is Argo CD itself — running inside the cluster, with no credentials to leak externally.

For self-hosted setups on Hetzner or Vultr, this is particularly clean because there’s no cloud IAM to configure. You point Argo CD at your GitLab repo, tell it which branch to watch, and you’re done.

# The Argo CD Application CRD — the only thing you need
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
  namespace: argocd
spec:
  source:
    repoURL: https://gitlab.example.com/myorg/myapp
    targetRevision: main
    path: helm-charts/myapp
  destination:
    server: https://kubernetes.default.svc
    namespace: myapp
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

selfHeal: true means if someone manually kubectl applys something, Argo CD reverts it. The Git repo is the only source of truth.

The CI image-tag update step looks like this:

# .gitlab-ci.yml
deploy:
  stage: deploy
  script:
    - |
      # Update the image tag in values.yaml and push
      sed -i "s/tag: .*/tag: ${CI_COMMIT_SHORT_SHA}/" values/myapp.yml
      git config user.email "ci@example.com"
      git config user.name "CI"
      git add values/myapp.yml
      git commit -m "chore: bump myapp to ${CI_COMMIT_SHORT_SHA}"
      git push

CI needs write access to the Git repo — but that’s a deploy key, not a cluster credential. If it leaks, someone can push code. You’d rotate the deploy key and audit the commits. If a cluster credential leaks, someone owns your cluster.

Answer 2: OIDC federation (for when you genuinely need push-based)

Some operations don’t fit the GitOps model. Infrastructure provisioning (terraform apply), one-off database migrations, or initial cluster bootstrapping — these need direct cluster access. The correct pattern here is OIDC federation.

The idea: your CI platform (GitLab, GitHub Actions) already issues JWT tokens to every job. These JWTs are signed by the CI platform and contain claims like which repo, which branch, which pipeline triggered the job. You configure your Kubernetes API server to trust those JWTs, and the CI job authenticates directly using the token it already has.

No stored credentials. Every job gets a fresh token. The token expires when the job ends.

For a self-hosted GitLab, configure your k8s API server to trust GitLab as an OIDC issuer:

# /etc/rancher/k3s/config.yaml (or kube-apiserver flags)
kube-apiserver-arg:
  - "oidc-issuer-url=https://gitlab.example.com"
  - "oidc-client-id=your_client_id"
  - "oidc-username-claim=sub"
  - "oidc-groups-claim=groups_direct"

Then create a ClusterRoleBinding that maps a specific GitLab identity to a Kubernetes role:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: gitlab-ci-deployer
subjects:
  - kind: User
    name: "project_path:myorg/myapp:ref_type:branch:ref:main"
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: deploy-role
  apiGroup: rbac.authorization.k8s.io

The subject name is the sub claim from the GitLab JWT — it encodes the repo path and branch. Only jobs running on main in myorg/myapp get this binding. A job on a feature branch gets nothing.

In the CI job:

deploy:
  stage: deploy
  id_tokens:
    K8S_TOKEN:
      aud: your_client_id
  script:
    - |
      kubectl config set-credentials gitlab-ci \
        --token="${K8S_TOKEN}"
      kubectl config set-context deploy \
        --cluster=mycluster \
        --user=gitlab-ci
      kubectl config use-context deploy
      kubectl rollout restart deployment/myapp -n myapp

The token in K8S_TOKEN is injected by GitLab. It expires with the job. The API server validates the signature against GitLab’s JWKS endpoint on every request.

Which one to use

For application deployments: GitOps. The cluster reconciles continuously, drift is impossible, and CI is completely decoupled from cluster state.

For infrastructure provisioning or one-off operations: OIDC federation. Short-lived credentials, branch-scoped permissions, nothing to rotate.

What you should never do: store a kubeconfig or a long-lived ServiceAccount token in CI secrets. Not because it’s hard to make work — it’s easy — but because the blast radius of a leak is unbounded, there’s no audit trail, and there’s no expiry. Everything that goes wrong with static secrets goes wrong eventually.

This is part of a series on Kubernetes interview questions. Next: how to handle secrets in a GitOps repository.

“You’re using GitOps — everything goes through Git. How do you handle secrets?”

The wrong answer: base64-encode them and commit them as Kubernetes Secret objects. Base64 is not encryption. Anyone with read access to the repo has your secrets. If the repo is public, everyone does.

The slightly better wrong answer: use a private repo and just not think about it. This works until a deploy key leaks, someone joins and then leaves the company, or you need to rotate one secret and have to find every place it’s referenced.

There are three real answers. They make different tradeoffs.

The constraint

The constraint is actually tighter than “don’t commit secrets”. It’s: your Git repo should be safe to make public at any point, and secrets must be rotatable without touching Git.

If rotating a password requires a new commit, someone has to be awake to merge and deploy it. That’s not how you want to handle a 3am incident.

Option 1: External Secrets Operator + Vault

This is the most robust pattern and the one worth knowing for interviews.

The idea: secrets live in a dedicated secret store (HashiCorp Vault, or a cloud equivalent). A Kubernetes operator called ESO watches ExternalSecret CRD objects in the cluster and syncs the referenced secret into a real Kubernetes Secret. The CRD is safe to commit — it says where the secret lives, not what it is.

# This lives in Git — safe to commit
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: myapp-db-credentials
  namespace: myapp
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: myapp-db-credentials   # the k8s Secret it creates
  data:
    - secretKey: DB_PASSWORD
      remoteRef:
        key: secret/myapp
        property: db-password

Rotation: you update the secret in Vault. ESO syncs it to the cluster within refreshInterval. No Git commit, no deployment. The pod reads the updated Secret on the next restart (or immediately if you mount it as an env var and the app handles SIGHUP).

Audit trail: Vault logs every read and write. You know exactly which service account read which secret at what time.

The cost: you’re running Vault. For a homelab or small team, that’s an extra thing to operate. For production, it’s worth it.

Self-hosted setup:

# ClusterSecretStore — connects ESO to your Vault instance
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: vault
spec:
  provider:
    vault:
      server: "http://sys-vault.sys-vault.svc.cluster.local:8200"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "external-secrets"

ESO authenticates to Vault using the pod’s Kubernetes ServiceAccount token. Vault validates it against the cluster’s token review endpoint. No static credentials anywhere.

Option 2: Sealed Secrets

Sealed Secrets uses asymmetric encryption. The cluster holds a private key. You use the kubeseal CLI to encrypt a secret with the cluster’s public key. The resulting SealedSecret object is safe to commit — only the cluster can decrypt it.

# Encrypt a secret for committing to Git
kubectl create secret generic myapp-db \
  --from-literal=DB_PASSWORD=hunter2 \
  --dry-run=client -o yaml \
  | kubeseal \
  > sealed-secrets/myapp-db.yaml

The resulting YAML looks like:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: myapp-db
  namespace: myapp
spec:
  encryptedData:
    DB_PASSWORD: AgBy3i4OJSWK+PiTySYZZA9rO43cGDEq...

This gets committed. The Sealed Secrets controller in the cluster decrypts it and creates the real Secret automatically.

The tradeoff: rotation means re-sealing. You need the cluster’s public key (which is public) and access to the plaintext secret. You commit a new SealedSecret. That’s a Git commit, which means a review, a merge, and a deploy. For a 3am incident, that’s a lot of friction.

Also: if the cluster’s private key is lost, you can’t decrypt any of your sealed secrets. Back up the private key.

Good fit for: small teams, homelab, situations where secrets change rarely and the GitOps review process is actually desirable.

Option 3: SOPS

SOPS (Secrets OPerationS) encrypts files at rest using age keys or cloud KMS. You commit encrypted files. CI decrypts them during deployment using a key it holds in memory (not stored in Git).

# Encrypt a file for Git
sops --encrypt --age age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8q \
  secrets/myapp.yaml > secrets/myapp.enc.yaml

# In CI: decrypt to temp file, apply, delete
sops --decrypt secrets/myapp.enc.yaml | kubectl apply -f -

The difference from Sealed Secrets: SOPS encrypts at the file level, not the k8s object level. You can use it outside of Kubernetes (application configs, Terraform variables). The key can live in the CI environment, a cloud KMS, or a personal age key.

The tradeoff: CI needs the decryption key, which puts you back in “secret in CI” territory — just for the encryption key rather than the actual secrets. If you use a cloud KMS, OIDC federation handles that (no stored key). If you use an age key, it lives in CI secrets.

Good fit for: teams already using Helm and Helm Secrets, polyglot environments where not everything is Kubernetes, small teams where Vault feels like overengineering.

Comparison

What interviewers are actually testing

The interesting follow-up question is: “How do you rotate a secret without downtime?”

The answer requires you to understand that pods mount Secret objects at startup. Updating the Secret in Kubernetes doesn’t automatically restart the pod. Your options are:

1. Mount the secret as a volume and have the app watch for file changes (good)
2. Restart the deployment after rotation (kubectl rollout restart, automatable)
3. Use a sidecar like Vault Agent Injector that handles refresh in-process (complex but zero-restart)

The correct answer depends on the app. An API key that can be rotated gradually is different from a database password where the old one is invalidated immediately.

This is part of a series on Kubernetes interview questions. Previously: deploying without cluster credentials. Next: zero-downtime deployments.

When I started the homelab I looked at the standard ways to make self-hosted services accessible and found the usual compromises:

Open ports on the router. Point a DNS record at your public IP, forward 443 to your server. Simple. Also: your home IP is now publicly associated with your services, your ISP can see your traffic, and a misconfigured app means an open door. Hard pass.

VPN for everything. WireGuard on the router, every device gets a tunnel. Secure, but every phone and laptop needs to be configured, split tunnelling adds complexity, and “let me pull up the recipe” becomes a two-tap operation that my family won’t do. And I still want public access for some services.

Cloudflare Tunnel, but broken HTTPS locally. This is the one I started with. The cloudflared pod dials out to Cloudflare, no open ports, external access works great. But locally, *.hippotion.com resolves to Cloudflare’s anycast IP, traffic leaves the house, Cloudflare terminates TLS, traffic comes back in through the tunnel. Every local request makes a round trip to a Cloudflare edge node. Worse: browsers cache HSTS for hippotion.com, so http:// URLs on the local network silently upgrade to https://, which fails because there’s no local certificate. Intermittent, confusing, and hard to explain to anyone else on the network.

What I wanted: the tunnel for external access, direct-to-server for local access, real TLS in both cases, and one configuration per application.

The insight: Pi-hole already controls local DNS

My network already runs Pi-hole for ad blocking. Pi-hole uses dnsmasq under the hood and can resolve any hostname to any IP you want. One config line in Pi-hole’s values:

dnsmasq:
  customDnsEntries:
    - address=/hippotion.com/192.168.0.109

The address= directive is a wildcard. Every device on the LAN that uses Pi-hole for DNS — which is all of them, because the router hands out Pi-hole’s IP via DHCP — will now resolve anything.hippotion.com to 192.168.0.109, the server’s LAN address. External traffic still goes to Cloudflare’s IP because it uses the public authoritative DNS. The split is automatic; no per-device configuration.

Local browser → Pi-hole → server’s LAN IP directly.

That solves routing. Now TLS.

Why HTTP-01 won’t work here, and why DNS-01 will

The standard way to get a Let’s Encrypt certificate is the HTTP-01 challenge: Let’s Encrypt sends a request to http://yourdomain.com/.well-known/acme-challenge/<token> and your server responds. This requires Let’s Encrypt’s servers to reach your server over the public internet.

That doesn’t work here. There are no open ports. Let’s Encrypt can’t reach the server. HTTP-01 is out.

DNS-01 is different. Instead of proving you control a server, you prove you control the DNS zone by creating a temporary TXT record at _acme-challenge.yourdomain.com. Let’s Encrypt checks DNS, finds the record, issues the cert. No inbound connection required — just API access to your DNS provider.

hippotion.com is on Cloudflare. cert-manager has a Cloudflare DNS solver that calls the Cloudflare API to create and delete the TXT record automatically. The certificate request flow:

1. cert-manager creates an ACME Order with Let’s Encrypt
2. cert-manager calls the Cloudflare API: add _acme-challenge.hippotion.com TXT <token>
3. Let’s Encrypt queries DNS, finds the record, issues the cert
4. cert-manager deletes the TXT record, writes the cert to a Kubernetes Secret
5. cert-manager renews automatically ~30 days before expiry

The Cloudflare API token needs Zone:DNS:Edit permission for hippotion.com. It lives in Vault and syncs to the sys-cert-manager namespace via External Secrets Operator — same pattern as every other secret in the cluster, nothing in Git.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: hippotion-wildcard
  namespace: sys-traefik
spec:
  secretName: hippotion-wildcard-tls
  issuerRef:
    name: letsencrypt-cloudflare
    kind: ClusterIssuer
  dnsNames:
    - "hippotion.com"
    - "*.hippotion.com"

One certificate. Every subdomain. cert-manager stores it as hippotion-wildcard-tls in the sys-traefik namespace, where Traefik can read it.

Two Traefik entrypoints

Traefik has three entrypoints configured: