Over the past weeks I built a Kubernetes environment based on K3s on Ubuntu. Not as a demo, but as a serious learning journey (Ongoing!). The starting point was clear from day one: GitOps only. Everything declarative in Git, deploys via GitLab CI CD and ArgoCD, no structural kubectl actions.
Architecture and principles
The cluster runs single node, but with production-like choices:
- K3s without Traefik and servicelb
- Cilium as CNI
- Gateway API as the standard for external traffic
- ArgoCD self managed
- DNS managed externally, the cluster is not leading for DNS
- Everything separated per namespace
- No external manifests via URLs, everything in local version control
The goal was not speed, but control and insight.
Networking turned out to be the real complexity
In the first iteration Cilium ran in host networking mode with elevated privileges to get Gateway API working. This solved reachability, but immediately introduced a number of risks:
- Pods share the host network, less isolation
- Bigger impact of configuration mistakes
- Harder to secure in a multi tenant scenario
- Debugging becomes harder because node and cluster traffic mix
On top of that I ran into a bleeding edge bug in Cilium in combination with host networking. No crash, but unpredictable behaviour. That is exactly the kind of signal you do not want to work around with workarounds that weaken security even further.
Deliberate security and network decisions
This led to a clear choice: do not optimise for today, but for the desired end state. The homelab was therefore extended with:
- An extra NIC
- A second WAN IP
- Cilium loadbalancing instead of host networking
With this, host networking could disappear completely and the network path is now much more explicit:
- External traffic enters via Gateway API
- Cilium LB handles IP allocation
- Services are internal by default
- Only explicitly exposed workloads are reachable
- Node network and cluster network are logically separated
The result is a setup that is more predictable, easier to secure and closer to how this is set up in larger environments.
GitOps as a learning tool
GitOps turned out to be not just a deployment model, but also a learning tool. Because everything is recorded declaratively:
- Mistakes are reproducible
- Configuration can be reviewed
- Rollbacks are trivial
- Drift is visible
Every misconfiguration became a data point instead of a frustration. What went wrong here and why.
Current status and next phase
The foundation is in place. Networking is stable, security choices are explicit and the platform behaves predictably. ArgoCD guards the desired state and the cluster follows.
The next phase is where it gets fun. Building and deploying applications:
- Building images via CI CD
- Configuration fully via Git
- Exposing services correctly via Gateway API
- Adding observability where needed
The foundation is there. From here on it is no longer about getting the cluster to work, but about running actual value on it.
This journey has sharpened my picture of Kubernetes considerably. Addictively fun and interesting once you see what it can do.
#kubernetes #k3s #gitops #argocd #cilium #gatewayapi #homelab #devops #infrastructureascode
