Managing production servers and self-hosted homelabs by manually SSHing into hosts, running ad-hoc docker run commands, and tweaking live configuration files leads directly to configuration drift, undocumented states, and catastrophic outages when a node must be rebuilt.
The modern enterprise standard is GitOps: treating your Git repository as the single, immutable source of truth for all infrastructure. When an engineer commits a change to a repository, automated CI pipelines validate the configuration, and deployment orchestrators synchronize that state to target hosts via secure webhooks.
⚡ Quick Runbook: 60-Second Implementation
TL;DR: Copy and deploy the baseline production definition directly onto your host:
git checkout -b feat/add-monitoring-stack
This guide documents the complete end-to-end GitOps workflow utilizing Forgejo (self-hosted Git), Komodo (Docker orchestration), Automated Secrets Management, and Renovate (dependency automation), built for long-term maintainability.
1. The GitOps Architecture Blueprint
Instead of exposing host Docker daemons over the public internet or giving continuous integration runners root SSH keys to your servers, modern GitOps separates the workflow into clear operational boundaries:
1. Version Control (Forgejo)
- All Docker Compose files, reverse proxy configurations, and systemd units live in dedicated Git repositories.
- The
mainbranch is protected: direct commits are blocked, and all changes require pull requests with passing automated validation checks.
2. Orchestration (Komodo)
- A centralized Komodo Core controller communicates with lightweight Komodo Periphery agents running on each node over encrypted internal networks.
- Komodo acts as the deployment controller, pulling updated configurations only when triggered by authenticated webhooks.
2. The Development Workflow: From Branch to Production
Every operational change follows a strictly repeatable, disciplined lifecycle:
- Feature Branching: Developers create an isolated branch from a freshly updated
main:git checkout -b feat/add-monitoring-stack - Local Validation & Linting: Before pushing, automated validation scripts parse compose files, verify image tags, and validate syntax:
# Validate compose definitions and syntax docker compose config --quiet - Pull Request & Automated CI: Pushing the branch opens a Pull Request in Forgejo. Automated CI workflows execute:
- Compose syntax linting across all modified stacks.
- Validation that no plaintext secrets or unpinned image tags are introduced.
- Branch protection prevents merging until all checks pass.
- Merge & Webhook Trigger: Merging the PR triggers an authenticated webhook to Komodo:
POST https://komodo.example.com/listener/webhook/deploy-changed-stacks Headers: X-Komodo-Signature: sha256=<HMAC_SECRET> - Zero-Downtime Rollout: Komodo instructs the node agent to pull the new container image layer cache, execute
docker compose up -d, verify container health checks, and retire the stale container.
3. Secrets Management: Zero-Knowledge Git Repositories
Golden Rule: Never commit passwords, API tokens, or private keys to Git—even in private repositories. Git history is permanent, and encrypted secrets (like Ansible Vault or git-crypt) introduce key rotation nightmares.
The modern production standard is Host-Managed Secret Mounting via Docker Compose native secrets:
# compose.yaml
services:
db:
image: dhi.io/mariadb:11.4-debian
restart: unless-stopped
environment:
- MYSQL_DATABASE=appdb
- MYSQL_USER=appuser
- MYSQL_PASSWORD_FILE=/run/secrets/DB_PASS
secrets:
- DB_PASS
secrets:
DB_PASS:
file: ../0.secrets/app/DB_PASS
Operational Architecture:
- Secrets live inside a gitignored directory (
0.secrets/) stored strictly on the server’s encrypted host storage pool (e.g. ZFS with native encryption). - Each secret is stored as a raw file with no trailing newlines (
chmod 0600). - Rotations occur on the host disk without requiring Git commits or repository rebuilds.
4. Automated Dependency Maintenance with Renovate
Allowing container images to remain static for years leads to security vulnerabilities and breaking upgrade cliffs. Modern homelabs automate image maintenance using Renovate:
- Automated Minor & Patch Updates: Renovate scans repositories weekly, checks upstream registries for new patch versions, and opens automated pull requests.
- Automerge on Green CI: When a patch release passes automated compose linting, Forgejo automerges the PR, triggering Komodo to deploy the updated container without human intervention.
- Major Version Guardrails: Major upgrades (e.g. PostgreSQL 16 → 17) are pinned to a Dependency Dashboard, requiring manual review, database dump verification, and scheduled execution.
5. Troubleshooting & Disaster Recovery Protocol
Because the Git repository is the complete declarative record of your infrastructure, rebuilding a destroyed host requires only three steps:
- Node Bootstrap: Execute an idempotent provisioning script to install Docker, UFW, and fail2ban.
- Restore Secrets: Restore the encrypted
0.secrets/bundle from your offsite backup vault. - Execute Git Sync: Clone the repository and trigger Komodo’s Resource Sync. Within 5 minutes, every container stack, network, and reverse proxy is fully recreated.
Related Runbooks in This Series
- The Complete GitOps Deployment Lifecycle: Forgejo, Webhooks, Komodo & Docker Compose
- Enterprise Secrets Management: Docker Native File Secrets and GitOps Security Hygiene
- Modern Git Workflows, SSH Signing & Ed25519 Best Practices
- Declarative Server Administration: Why We Avoid Web GUIs in Favor of GitOps, Tmux & Pure CLI