Homelab as production

There is a machine in my apartment in Ho Chi Minh City that talks to a VPS I rent somewhere else, and neither of them knows or cares that there is an ocean in between. A WireGuard mesh handles that. Every node has a private address on the same network, so a container running at home and a container running on rented hardware are neighbours as far as anything above the network layer is concerned. Portainer sits on top and manages the containers wherever they happen to live. AdGuard Home does DNS, and it discovers the services on the mesh by itself, so when something new comes up it gets a name without me typing one.

That is the whole setup. It is not large. The point of it is not the hardware.

I started it because I wanted to know what production feels like when nobody is paid to fix it at 2am. At work there is a rotation, and there are people whose job it is to be awake. At home there is me, and I have a day job that starts in the morning and a P&L that does not care why I was up. So the only sustainable way to run the thing is to make it not need me. Zero touch, because I refuse to babysit it.

Saying that is easy. What it actually forces is narrower and more annoying than it sounds.

What "no babysitting" costs in practice

First, no manual step I cannot repeat. Not "documented", repeatable. If the recovery path for a service is a sequence of things I did once on a Tuesday and half remember, that service is already broken and I just have not found out yet. The test is whether I can wipe a node and get back to the same state without thinking. Anything that fails that test either gets written down as something that runs, or it goes.

Second, DNS has to find things on its own. This sounds like a small convenience and it is not. The moment names are maintained by hand, every change has a second step that lives only in my head, and the head is the part that is asleep at 2am. Service discovery is not there because typing hostnames is tedious. It is there because a config file I edit manually is a config file that drifts.

Third, and this is the one I underestimated, I have to be willing to delete things. I have run services in the mesh that were genuinely useful and needed a nudge every couple of weeks to keep working. A restart here, a token refreshed there. Each nudge is thirty seconds. That is the trap. Thirty seconds is cheap enough that you never fix it, and expensive enough that it is the reason you stop trusting the whole system. I have removed things that worked because they only worked when I was watching.

The cost lands in evenings, and it lands unevenly. Doing something properly the first time takes an evening. Doing it in the way that would work by tomorrow takes twenty minutes. Since I do not allow myself the twenty minute version, there are things in the mesh that have stayed broken for weeks, not because they are hard but because I have not had an evening to fix them the only way I have left myself. That is the honest trade. The system is reliable and my backlog is not.

Why the constraint is the part that teaches

None of what I learned came from the boxes. You can buy better boxes. What taught me was the rule I imposed, which is that I am not allowed to be part of the runtime.

That rule surfaces things a work environment hides from you. At work, a service that needs occasional human attention still looks healthy, because there is a team and the attention gets absorbed into somebody's afternoon. Alone, there is nowhere for it to hide. You feel every piece of hand-holding directly, in your own week, and you start seeing which designs quietly assume a person is available. Most of them do.

It also changed what I build for myself. I self-host my own tools on this, including wiki-viewer, which reads and edits local files through a web UI with an HTTP API behind it. Because it runs on the mesh, it is reachable from anywhere I am, and because it is a thing I run rather than a thing I demo, it has to come back up on its own after a reboot without me remembering how. Writing software you have to operate yourself is a different activity from writing software you hand over. You stop adding steps.

I am aware of how this sounds from outside. A few machines, a private network, some containers, a DNS server. It is a hobby. But the constraint on it is the same constraint that decides whether real systems survive their second year, and it is much cheaper to learn on a mesh nobody depends on than on something with customers behind it. The hardware is the excuse. The rule is the thing.