The Problem of Manual Kubernetes Management

Managing Kubernetes clusters represents one of the most complex challenges in the modern cloud-native ecosystem. As the number of nodes and clusters grows, operational complexity increases exponentially, quickly making operations like provisioning new workers, coordinated control plane upgrades, network configuration management, and underlying infrastructure maintenance unmanageable.

Limitations of Traditional Methods

Traditional methods for managing Kubernetes clusters typically rely on:

  • Custom scripts for node provisioning and configuration
  • Manual procedures documented, hopefully, for upgrades and maintenance
  • Static configurations difficult to version and replicate
  • Imperative approaches that describe “how to do” rather than “what to achieve”

Concrete Operational Problems

According to CNCF surveys, operational complexity represents one of the main challenges in enterprise Kubernetes adoption.

Error-Prone Operations

Every manual intervention introduces potential failure points. Consider for example a possible script for adding a worker node:

#!/bin/bash
ssh worker-node-03
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" > /etc/apt/sources.list.d/kubernetes.list
apt-get update && apt-get install -y kubelet kubeadm kubectl
systemctl enable kubelet
swapoff -a
# ... container runtime configuration
# ... networking configuration
# ... cluster join

This approach has significant issues:

  • Error-prone: every manual step can fail
  • Time-consuming: repetitive operations that require supervision
  • Not reproducible: difficulty in replicating identical configurations
  • Limited scalability: operational load grows linearly with the number of clusters

Configuration Drift

Manually managed clusters tend to diverge over time ("configuration drift"). Ad-hoc modifications, hotfixes applied directly to nodes, and inconsistent upgrade procedures lead to “unique snowflakes” clusters that are difficult to debug and maintain.

Scaling Complexity

The same issues that concern initial provisioning also appear when we need to scale our infrastructure:

  • Infrastructure provisioning (VMs, networking, storage)
  • Operating system installation and configuration
  • Kubernetes components setup
  • Cluster join and status verification

Cluster API: Infrastructure as Code for Kubernetes

Cluster API (CAPI) is an official Kubernetes sub-project designed to solve these problems through declarative APIs and automated tooling for managing the entire lifecycle of Kubernetes clusters.

Architectural Principles

Declarative Configuration

CAPI embraces Kubernetes’s declarative paradigm, where users define the desired state of their clusters using standard Kubernetes manifests:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: production-cluster
spec:
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1beta1
    kind: KubeadmControlPlane
    name: production-control-plane
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: ProxmoxCluster
    name: production-proxmox

Eventual Consistency

Like Kubernetes itself, CAPI operates on an eventual consistency model. Controllers continuously observe the current state of resources and work to reconcile differences between the observed state and the desired state.

Infrastructure Provider Pattern

CAPI uses a modular architecture based on providers that allow abstracting the specifics of the underlying infrastructure. The Cluster API Provider Ecosystem includes:

  • Core Controller: manages Cluster and Machine resources
  • Bootstrap Provider: generates configurations to transform machines into Kubernetes nodes
  • Control Plane Provider: manages control plane components
  • Infrastructure Provider: interfaces with specific infrastructure (AWS, Proxmox, vSphere, etc.)

Management/Workload Cluster Architecture

CAPI introduces a fundamental separation between two types of clusters:

Management Cluster

  • Kubernetes cluster that hosts CAPI controllers and providers
  • Contains Custom Resources that represent the desired state of workload clusters
  • Manages the complete lifecycle of other clusters
  • Can be a lightweight cluster (even local with kind)

Workload Cluster

  • Target Kubernetes cluster where applications are deployed
  • Completely managed by the Management Cluster
  • Declarative lifecycle (creation, update, deletion)

Operational Advantages

Idempotency and Reproducibility

CAPI operations are idempotent by design, following Kubernetes controller principles. The same configuration applied multiple times always produces the same result, eliminating configuration drift problems.

Native Version Control

Configurations are YAML manifests that can be versioned in Git, allowing:

  • Complete change tracking
  • Deterministic rollbacks
  • Code review for infrastructure changes
  • Integration with GitOps pipelines

Self-Healing Infrastructure

CAPI controllers continuously monitor infrastructure state and apply automatic corrections when they detect discrepancies from the desired state.

Implementation with Proxmox

Why Proxmox for Homelab

Proxmox Virtual Environment represents an ideal platform for experimenting with CAPI in a fully virtualized environment, suitable for both experimentation and real workloads:

  • Complete control of virtualized infrastructure
  • REST API for automation (Proxmox VE API)
  • Contained costs compared to cloud solutions
  • Operational realism comparable to enterprise environments

Target Architecture

The implementation includes:

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│  Management     │    │   Proxmox VE     │    │  Workload       │
│  Cluster        │───▶│   Infrastructure │───▶│  Cluster        │
│  (Kind)         │    │                  │    │  (Talos)        │
└─────────────────┘    └──────────────────┘    └─────────────────┘

Main components:

  • Management Cluster: Kind cluster local with CAPI controllers
  • Infrastructure Provider: Proxmox provider for VM management
  • Bootstrap/Control Plane Provider: Talos provider for immutable OS
  • Workload Cluster: Production-ready Kubernetes cluster

Integration with Talos Linux

The implementation uses Talos Linux as the operating system for Kubernetes nodes:

  • Immutability: read-only filesystem prevents configuration drift
  • API-driven: complete management via gRPC API, eliminating SSH
  • Minimalism: includes only essential components for Kubernetes
  • Security: reduced attack surface

End-to-End Operational Flow

Deployment Process

Broadly speaking, the deployment process works this way:

  1. Declarative definition: creation of YAML manifest for the desired cluster
  2. Apply to Management Cluster: kubectl apply -f cluster.yaml
  3. Controller Reconciliation: CAPI controllers process resources
  4. Infrastructure Provisioning: VM creation on Proxmox
  5. Bootstrap Process: Kubernetes installation and configuration
  6. Cluster Ready: operational cluster ready for workloads

Scaling Operations

At the end of the deployment, we’ll have a functional k8s cluster (workload cluster) managed by the management cluster, just like any other resource typically managed by k8s. Precisely for this reason, we can operate on it simply by editing the yaml file that defines the cluster structure, for example to increase the number of replicas it’s sufficient to specify the new value:

# Scale control plane from 1 to 3 nodes
spec:
  replicas: 3  # Modified from 1

The controller automatically:

  • Provisions 2 new VMs
  • Installs Talos Linux
  • Configures control plane components
  • Updates the load balancer
  • Verifies cluster health

Series Structure

Part 2: Anatomy of Cluster API

  • Core components and their interactions
  • Detailed Custom Resource Definitions
  • Reconciliation loop and state management
  • Complete flow from manifest to cluster

Part 3: Talos Linux Integration

  • Architecture and principles of Talos
  • TalosControlPlane and TalosConfig CRDs
  • Bootstrap process and configuration management
  • Advantages of immutable approach

Part 4: Practical Setup

  • Proxmox configuration and prerequisites
  • CAPI and provider installation
  • Python generator for parametric configurations
  • Deploying the first workload cluster

Part 5: Advanced Management

  • Worker node management and scaling
  • Upgrade procedures and maintenance
  • Troubleshooting and debugging
  • Operational best practices

Manual management of Kubernetes clusters has fundamental scalability, reproducibility and reliability limitations. Cluster API provides a declarative and automated approach that solves these problems through infrastructure abstraction and standard Kubernetes controller pattern.

For in-depth information on Cluster API theory and best practices, consult the official documentation and Kubernetes SIG Cluster Lifecycle.

The next part will explore in detail the architecture and components of CAPI, providing the theoretical foundations necessary for practical implementation.