GitOps and Platform Engineering

In the previous chapter, you mastered GKE cluster operations. But every administrator faces a critical question: how do you manage what goes into the cluster? kubectl apply from your laptop works for experiments, but production demands every change be tracked, reviewed, and automatically reconciled. This is GitOps.

Analogy: Blueprint-Driven Construction

Imagine a construction company building developments across a region. Every detail lives as approved blueprints in a central planning office. Once revised blueprints are approved, they reach every site automatically. Workers build exactly what the blueprint specifies. An inspector on each site compares what was built against the blueprint daily, flagging and correcting unauthorized changes. The blueprint is the single source of truth.

This is GitOps. Git is the planning office — storing every approved manifest. The sync agent is the delivery truck. The cluster is the construction site. The controller is the inspector — checking whether the live cluster matches Git and correcting drift.

8.2.1 GitOps Principles

GitOps applies version-controlled software development practices to infrastructure management. It rests on four pillars.

Git as the Single Source of Truth. Git repositories hold the declarative descriptions of all infrastructure and applications. Every Deployment, Service, ConfigMap, and RBAC policy exists as YAML in Git. Git provides complete version history, immutable commits, pull request workflows, and signed audit trails. When a compliance auditor asks "Who changed this and when?" you point to the Git log.

Automated Synchronization. A GitOps tool continuously monitors Git and the live cluster. When it detects a difference, it reconciles. Changes flow from Git to the cluster, never the reverse. This eliminates configuration drift, where undocumented kubectl edit commands accumulate until your cluster becomes an irreproducible snowflake.

Drift Detection and Correction. The controller periodically compares cluster state against Git state. If someone manually patches a Deployment replica count, the controller detects this drift and reverts it. The cluster always converges toward the desired state in version control.

Pull-Based vs. Push-Based Patterns. In push-based deployment, external CI/CD pushes changes via kubectl apply, requiring cluster credentials in the pipeline. In pull-based deployment, an agent inside the cluster (like ArgoCD) watches Git and pulls changes in. The cluster only needs outbound Git access — no externally exposed credentials. Pull-based is more secure.

⚠️ Common Misconception: "GitOps means I never use kubectl again." GitOps eliminates kubectl apply and kubectl edit for routine changes, not kubectl for debugging or emergency troubleshooting. The blueprint office does not ban hammers on the construction site, but it requires structural changes go through the blueprint approval process.

🛑 PAUSE & RECALL — 2 minutes

Without looking back:

1. Why is Git chosen as the source of truth rather than a regular database?
2. In the construction analogy, what role does the inspector play? What is its technical equivalent?
3. What is the security advantage of pull-based over push-based deployment?
4. What happens if someone manually edits a resource with kubectl edit under GitOps?

Rate your confidence (0-4), then continue.

8.2.2 ArgoCD

ArgoCD is the most widely adopted GitOps tool for Kubernetes. It implements pull-based deployment as a Kubernetes-native application, running inside your cluster and using Custom Resources to define what it manages.

Application Definitions as Custom Resources. An ArgoCD Application CRD specifies the Git repository URL, manifest path, target cluster, and namespace. ArgoCD watches this resource, fetches manifests from Git, and ensures they exist in the cluster.

Automated Sync and Prune. With auto-sync, ArgoCD applies changes detected in Git. With auto-prune, deleting a manifest from Git removes the corresponding cluster resource. The cluster remains an exact mirror of the repository.

Application Sets for Multi-Cluster Management. The ApplicationSet controller generates multiple Applications from a single template, deploying the same app to staging and production clusters from one source of truth.

Sync Waves and Resource Hooks. Sync waves assign numeric weights so databases deploy before applications. Pre-sync and post-sync hooks run Jobs for database migrations or cache warming.

The ArgoCD web UI presents each app as a card with sync status — green for synchronized, yellow for progressing, red for out of sync. Click an app to see a live topology: Deployment → ReplicaSet → Pods → Service. When drift occurs, the resource glows red and the diff view shows the exact changes.

8.2.3 Config Sync and Config Connector

Google Cloud provides native GitOps tooling for GKE that integrates deeply with the GCP ecosystem.

Config Sync is GKE's native GitOps engine, synchronizing resources from Git into your cluster. It supports multiple sync sources, hierarchical repositories, and policy controller integration.

Config Connector is GKE's secret weapon. It extends the Kubernetes API to manage GCP resources — Cloud SQL, Pub/Sub, Cloud Storage, IAM — as Kubernetes objects. Define a SQLInstance manifest, apply it with kubectl apply, and Config Connector creates the actual Cloud SQL instance. Your entire infrastructure lives in one Git repository, managed through one pipeline.

apiVersion: sql.cnrm.cloud.google.com/v1beta1
kind: SQLInstance
metadata:
  name: production-postgres
spec:
  databaseVersion: POSTGRES_15
  region: us-central1
  settings:
    tier: db-custom-2-4096
    ipConfiguration:
      ipv4Enabled: true

GKE Note: Config Sync and Config Connector require GKE Enterprise for full managed support, or can be installed manually. Config Connector needs Workload Identity enabled for GCP API authentication.

🛑 PAUSE & RECALL — 2 minutes

Close your eyes and picture the flow:

1. What is the difference between Config Sync and Config Connector?
2. Why is Config Connector called a "secret weapon" for GKE?
3. In the construction analogy, what would Config Connector represent?
4. What GCP IAM mechanism does Config Connector use to authenticate?

Rate your confidence (0-4), then continue.

8.2.4 Helm and Package Management

Helm is Kubernetes' package manager, providing templating for reusable, configurable applications.

Helm Charts as Templated Manifests. A Helm chart is a collection of YAML templates parameterized with Go template syntax. Instead of separate YAML files per environment, you maintain one set of templates with different values.yaml files.

mychart/
  Chart.yaml          # Metadata
  values.yaml         # Defaults
  values-prod.yaml    # Prod overrides
  templates/
    deployment.yaml
    service.yaml

Helm renders templates with merged values to produce plain Kubernetes YAML. Chart repositories (including Artifact Registry) store versioned charts. Helm hooks attach lifecycle Jobs for schema migrations or cleanup. Subcharts compose complex applications from modular components.

Helm vs. Kustomize. Helm uses templating — inject variables into parameterized YAML. Kustomize uses overlays — patch complete YAML without templates. Helm excels for third-party packages (installing Prometheus from a chart repo). Kustomize excels for your own applications. Many teams use both, and ArgoCD supports both natively.

8.2.5 Operators and Custom Resource Definitions

The Kubernetes API is extensible. You can define your own resource types, pair them with custom controllers, and create Operators that automate complex application lifecycles.

A Custom Resource Definition (CRD) registers a new resource type with the API server. Once created, you use kubectl to manage instances like built-in resources. The cert-manager project defines a Certificate CRD — creating a Certificate triggers cert-manager to provision a TLS certificate automatically.

The Operator pattern pairs a CRD with a controller that watches instances and reconciles desired state with reality. Create a PostgreSQL resource with replicas: 3, and the operator spins up three StatefulSet pods, configures replication, manages backups, and handles failover. It encodes a database administrator's expertise into software.

Common operators include Prometheus Operator, cert-manager, and CloudNativePG. The Operator Lifecycle Manager (OLM) installs and manages operators like a package manager.

🤔 TRY BEFORE YOU SEE

You need to deploy a production PostgreSQL database on GKE with automated daily backups, a connection pooler (PgBouncer), and a read replica. You also need a Cloud Storage bucket for backup archival.

Before reading the solution, sketch your approach: What Kubernetes objects would you create manually? What custom resources might an operator provide? How would Config Connector fit? Write 3-5 bullet points.

Reveal: A PostgreSQL operator provides a PostgreSQL custom resource specifying replicas, backups, and connection pooling. The operator creates the StatefulSet, Service, Secrets, and PgBouncer automatically. A Config Connector StorageBucket in the same Git repository lets Config Sync apply both through one pipeline. The operator handles day-2 operations without manual intervention.

8.2.6 CI/CD Integration

GitOps handles the "deploy" portion; Google Cloud provides managed services for the "build" portion.

Cloud Build compiles source code, runs tests, builds container images, and pushes them to Artifact Registry. Define your pipeline in cloudbuild.yaml; a Git push triggers it automatically.

Cloud Deploy orchestrates progressive rollouts on GKE through dev, staging, and production — supporting canary deployments, blue-green cutovers, and automated rollback.

Buildpacks analyze source code and produce optimized container images without a Dockerfile, supporting Node.js, Python, Java, Go, and more.

Webhook triggers from GitHub, GitLab, or Cloud Source Repositories initiate Cloud Build, completing the loop from commit to deployment.

GKE in Practice

For GKE users, the native GitOps stack combines Config Sync for cluster resources, Config Connector for GCP provisioning, and Cloud Build plus Cloud Deploy for CI — all fully managed by Google.

Config Sync supports unstructured repositories, hierarchical repositories with namespace-scoped inheritance, and OCI artifacts. The policy controller, built on Open Policy Agent Gatekeeper, enforces security rules before Config Sync applies resources.

Config Connector shines in multi-project environments. A single GKE cluster manages resources across multiple GCP projects through namespace-based project mapping — your production namespace creates resources in your production project, staging in your staging project — all through one GitOps workflow.

Visual Description: The Construction Blueprint Pipeline

Git holds Kubernetes manifests, Config Connector manifests, and Helm charts. The GitOps agent transports blueprints to GKE clusters, which apply Kubernetes resources directly. Config Connector files permit requests with GCP for external resources. The controller loops back, comparing live state against Git and correcting drift.

Lab: LAB-8.2 — GitOps on GKE (90 minutes)

Prerequisites: A running GKE cluster with Workload Identity enabled, kubectl configured, and a Git repository ready.

Part 1: Install ArgoCD (20 min)

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=argocd-server -n argocd --timeout=120s
kubectl port-forward svc/argocd-server -n argocd 8080:443

Retrieve the password: kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d. Open https://localhost:8080, log in as admin.

Part 2: Configure Git Sync (25 min)

In your Git repo, create argocd-apps/ with a deployment.yaml and service.yaml for nginx. Commit and push. Create an ArgoCD Application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: nginx-demo
  namespace: argocd
spec:
  project: default
  source:
    repoURL: "https://github.com/YOUR-USER/YOUR-REPO.git"
    targetRevision: HEAD
    path: argocd-apps
  destination:
    server: https://kubernetes.default.svc
    namespace: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Apply with kubectl apply -f application.yaml. In the ArgoCD UI, the nginx-demo tile appears — click it to see the topology: Deployment → ReplicaSet → Pods → Service.

Part 3: Git Push Trigger (10 min)

Edit deployment.yaml — change the nginx image tag. Commit and push. Watch the ArgoCD UI: the app shows "OutOfSync," then auto-synchronizes. Pod icons cycle through terminating and creating. You deployed by pushing to Git.

Part 4: Config Connector and Cloud SQL (20 min)

Enable Config Connector on your cluster. Verify: kubectl get pods -n cnrm-system. Apply:

apiVersion: sql.cnrm.cloud.google.com/v1beta1
kind: SQLInstance
metadata:
  name: lab-postgres
  annotations:
    cnrm.cloud.google.com/project-id: YOUR_PROJECT_ID
spec:
  databaseVersion: POSTGRES_15
  region: us-central1
  settings:
    tier: db-f1-micro

Verify: gcloud sql instances list and kubectl get sqlinstance lab-postgres. You created a GCP database with kubectl apply.

Part 5: Drift Detection (15 min)

Manually scale: kubectl scale deployment nginx --replicas=5. In the ArgoCD UI, the app shows "OutOfSync." Click the Diff tab — live state shows replicas: 5, desired shows the Git value. Auto-sync scales it back. You witnessed drift detection and self-healing.

Chapter Summary

GitOps transforms Kubernetes operations into a version-controlled workflow where Git is the single source of truth. Pull-based tools like ArgoCD and GKE Config Sync place the sync agent inside the cluster for improved security. Config Connector extends GitOps beyond the cluster, managing Cloud SQL, Storage, and other GCP resources through the same manifests. Helm provides templating for reusable packages, while Operators encode operational expertise into self-managing controllers. Together with Cloud Build and Cloud Deploy, these tools form a complete platform engineering stack.

📇 KEY CONCEPT CARDS

1. Q: What are the four pillars of GitOps?
   A: (1) Git as the single source of truth; (2) automated synchronization from Git to cluster; (3) drift detection and correction; (4) pull-based deployment agents running inside the cluster.

1. Q: What is the fundamental difference between push-based and pull-based GitOps?
   A: Push-based uses external CI/CD pipelines with cluster credentials to kubectl apply. Pull-based runs an agent inside the cluster that fetches from Git — cluster credentials never leave the cluster, only outbound Git access is required.

1. Q: What unique capability does Config Connector provide on GKE?
   A: It lets you manage GCP resources (Cloud SQL, Cloud Storage, Pub/Sub, IAM) as Kubernetes objects through YAML manifests and kubectl apply, unifying Kubernetes and cloud resource management in one GitOps workflow.

1. Q: What is the Operator pattern?
   A: The Operator pattern combines a Custom Resource Definition (CRD) that defines a new API resource type with a controller that watches instances and automates their lifecycle. It encodes human operational knowledge into software running inside the cluster.