Jonas HietalaModify

Just commands

It won’t take long for me to forget these commands so I’ll add them to Just. These recipes will take care of the secrets we’ve handled so far and the cleartext files (talosconfig, kubeconfig, secrets.auto.tfvars, terraform.tfstate) should allow us to regain cluster control, or to bootstrap the cluster anew from the git repo and the sops key.

[doc("Decrypt required files committed to git")]


decrypt_required:

    just secrets decrypt_cluster_config

    just secrets decrypt_terraform_secrets

    just secrets decrypt_terraform_state




[doc("Encrypt talosconfig and kubeconfig")]


[working-directory('../infrastructure')]


encrypt_cluster_config:

    sops --encrypt kubeconfig.yaml > kubeconfig.encrypted.yaml

    sops --encrypt talosconfig.yaml > talosconfig.encrypted.yaml




[doc("Decrypt talosconfig and kubeconfig")]


[working-directory('../infrastructure')]


decrypt_cluster_config:

    sops --decrypt talosconfig.encrypted.yaml > talosconfig.yaml

    sops --decrypt kubeconfig.encrypted.yaml > kubeconfig.yaml

    chmod 600 talosconfig.yaml kubeconfig.yaml




[doc("Encrypt secrets.auto.tfvars")]


[working-directory('../infrastructure')]


encrypt_terraform_secrets:

    sops --encrypt secrets.auto.tfvars > secrets.auto.encrypted.tfvars




[doc("Decrypt secrets.auto.tfvars")]


[working-directory('../infrastructure')]


decrypt_terraform_secrets:

    sops --decrypt secrets.auto.encrypted.tfvars > secrets.auto.tfvars

    chmod 600 secrets.auto.tfvars




[doc("Encrypt terraform.tfstate")]


[working-directory('../infrastructure')]


encrypt_terraform_state:

    sops --encrypt --input-type json --output-type json terraform.tfstate > terraform.encrypted.tfstate




[doc("Decrypt terraform.tfstate")]


[working-directory('../infrastructure')]


decrypt_terraform_state:

    # If the encrypted state doesn't exist yet (fresh repo), skip silently.

    test -f terraform.encrypted.tfstate || exit 0

    sops --decrypt --input-type json --output-type json terraform.encrypted.tfstate > terraform.tfstate

    chmod 600 terraform.tfstate

I also added encryption to create_cluster_config bootstrap command, to make it harder for me to forget to add them to the repo:

[doc("Create talosconfig and kubeconfig")]


[working-directory('../infrastructure')]


create_cluster_config:

    terraform output -raw talosconfig > talosconfig.yaml

    terraform output -raw kubeconfig > kubeconfig.yaml

    just secrets::encrypt_cluster_config

And for the terraform state too (run with just tf::apply instead of a plain terraform apply):

[doc("terraform apply, then re-encrypt state (encrypts even on failure)")]


[working-directory('../infrastructure')]


apply *args:

    #!/usr/bin/env bash

    set -e

    trap 'just secrets::encrypt_terraform_state' EXIT

    terraform apply {{args}}




[doc("terraform destroy, then re-encrypt state (encrypts even on failure)")]


[working-directory('../infrastructure')]


destroy *args:

    #!/usr/bin/env bash

    set -e

    trap 'just secrets::encrypt_terraform_state' EXIT

    terraform destroy {{args}}

Installation

I’ll install the sealed secrets controller using helm:

helm install sealed-secrets \


    --repo https://bitnami-labs.github.io/sealed-secrets \


    sealed-secrets \


    --version 2.16.2 \


    --namespace sealed-secrets \


    --create-namespace \


    --set fullnameOverride=sealed-secrets-controller


# Wait for it to deploy


kubectl rollout status deployment/sealed-secrets-controller -n sealed-secrets

To create secrets on the client we also need the kubeseal command. It wasn’t available on the Void Linux package manager, so let’s do the hard way:

set -x KUBESEAL_VERSION '0.36.1'


curl -OL "https://github.com/bitnami-labs/sealed-secrets/releases/download/v$KUBESEAL_VERSION/kubeseal-$KUBESEAL_VERSION-linux-amd64.tar.gz"


tar -xvzf kubeseal-$KUBESEAL_VERSION-linux-amd64.tar.gz kubeseal


sudo install -m 755 kubeseal /usr/local/bin/kubeseal


rm kubeseal-$KUBESEAL_VERSION-linux-amd64.tar.gz

Create a secret

To create a secret we can either use the cluster (needs an active connection) or offline via a certificate. I prefer the certificate simply because you need to pass fewer arguments (--cert vs --controller-name and --controller-namespace). Here’s how to fetch the certificate:

kubeseal --fetch-cert \


         --controller-name=sealed-secrets-controller \


         --controller-namespace=sealed-secrets \


    > infrastructure/sealed-secrets-cert.pem

(It’s a public key, safe to commit to git.)

And this is how we can use it to generate a secret my-secret with the two fields username and password:

kubectl create secret generic my-secret \


  --namespace some-namespace \


  --from-literal=username="user" \


  --from-literal=password="password1" \


  --dry-run=client -o yaml \


  | kubeseal --cert infrastructure/sealed-secrets-cert.pem \


             --format yaml \


  > gitops/apps/myapp/my-secret.yaml

(There might be other ways to do this. You can generate json files for example, but this works and I don’t care to do research.)

It will be stored in gitops/apps/myapp/my-secret.yaml that we can apply:

kubectl apply -f gitops/apps/myapp/my-secret.yaml

In the future when we get our GitOps setup up the process is the same except we don’t apply the secret; just create, commit, and push and it’ll get applied automatically. It may feel like a lot of effort, but it’s quite nice to work with in day-to-day operations.

The process to update a secret is exactly the same; update the file with new contents and reapply.

View a secret

Check that the secret has been applied:

kubectl get secret my-secret -n some-namespace -o yaml

The data fields username and password shown above will be base64 encoded. Here’s how to print out the password in cleartext:

kubectl get secret my-secret -n some-namespace -o jsonpath='{.data.password}' | base64 -d

Or you can view the secrets in a dashboard such as Headlamp, which is arguably easier.

Surviving a cluster reset

There’s one gotcha to sealed secrets: when the controller is installed it will generate a new public/private key pair so all existing sealed secrets are invalidated. We’d have to reseal all secrets after we reset the cluster, which is highly annoying.

We can circumvent this by exporting the private key, encrypt it with SOPS + Age, and store it in git. Then during the bootstrap process we can import the private key to the controller, allowing it to reuse all existing sealed secrets.

First export the key and encrypt it so we can keep it in git (gitignore sealed-secrets-key.yaml):

kubectl get secret -n sealed-secrets \


  -l sealedsecrets.bitnami.com/sealed-secrets-key=active -o yaml \


     > sealed-secrets-key.yaml


sops --encrypt sealed-secrets-key.yaml > sealed-secrets-key.encrypted.yaml

This needs a .sops.yaml rule:

creation_rules:

  - path_regex: infrastructure/sealed-secrets-key(\.encrypted)?.yaml$

    age: age1rrkgd5yza053qk9m8lp0ww39apdarz7w0rjyq85493g8l9gufgnq9cehzx

Then to import it we simply apply it:

kubectl apply -f sealed-secrets-key.yaml

Just commands

We’ve added a few steps to the bootstrap process to setup the sealed secrets controller:

[doc("Bootstrap everything from zero")]


full:

    just bootstrap::cluster

    just bootstrap::cilium

    just secrets::restore_sealed_secrets_private_key

    just bootstrap::sealed_secrets

[doc("Restore sealed-secrets-key.yaml")]


[working-directory('../infrastructure')]


restore_sealed_secrets_private_key:

    # If the encrypted private key doesn't exist, skip the whole recipe.

    test -f sealed-secrets-key.encrypted.yaml || exit 0

    just secrets::decrypt_sealed_secrets_private_key

    # Don't exit if namespace already exists.

    kubectl create namespace sealed-secrets || true

    # Restore the private key.

    kubectl apply -f sealed-secrets-key.yaml

    rm sealed-secrets-key.yaml

    # We may need to restart the controller, but it may not exist, which is fine.

    kubectl rollout restart deployment/sealed-secrets-controller -n sealed-secrets || true

I’ve tried to safeguard the recipe to not crash if we haven’t created a key or bootstrapped the controller yet.

[doc("Install sealed secrets controller")]


sealed_secrets:

    helm install sealed-secrets \

        --repo https://bitnami-labs.github.io/sealed-secrets \

        sealed-secrets \

        --version 2.16.2 \

        --namespace sealed-secrets \

        --create-namespace \

        --set fullnameOverride=sealed-secrets-controller

    # Wait for it to deploy

    kubectl rollout status deployment/sealed-secrets-controller -n sealed-secrets

And some extra management recipes:

[doc("Fetch sealed-secrets-cert.pem, necessary to encrypt secrets offline")]


[working-directory('../infrastructure')]


fetch_sealed_secrets_cert:

    kubeseal --fetch-cert \

             --controller-name=sealed-secrets-controller \

             --controller-namespace=sealed-secrets \

        > sealed-secrets-cert.pem




[doc("Fetch sealed-secrets-key.yaml")]


[working-directory('../infrastructure')]


fetch_sealed_secrets_private_key:

    kubectl get secret -n sealed-secrets -l sealedsecrets.bitnami.com/sealed-secrets-key=active -o yaml > sealed-secrets-key.yaml

    just secrets::encrypt_sealed_secrets_private_key

    rm sealed-secrets-key.yaml




[doc("Encrypt sealed-secrets-key.yaml")]


[working-directory('../infrastructure')]


encrypt_sealed_secrets_private_key:

    sops --encrypt sealed-secrets-key.yaml > sealed-secrets-key.encrypted.yaml




[doc("Decrypt sealed-secrets-key.yaml")]


[working-directory('../infrastructure')]


decrypt_sealed_secrets_private_key:

    sops --decrypt sealed-secrets-key.encrypted.yaml > sealed-secrets-key.yaml

With this we’re prepared to setup GitOps with ArgoCD in the next post.

Waiting for nodes

Note the dependency wait_for_api above in the line cilium: wait_for_api. This means wait_for_api will run before cilium, which waits until the Kubernetes nodes are ready (albeit not necessarily healthy according to talosctl health):

[private]


wait_for_api:

    until kubectl get nodes 2>/dev/null | grep -q "NotReady\|Ready"; do sleep 5; done

You can call just bootstrap::wait_for_api inside scripts too, which would give you more control over ordering. Dependencies on the other hand are deduplicated and always run before the recipe.

Modules and directory structure

I wanted to organize the recipes a little and group them (just tf::init instead of just tf_init). For that we use modules:

mod argocd 'just/argocd.just'


mod bootstrap 'just/bootstrap.just'


mod secrets 'just/secrets.just'


# etc...




# With this `just` will give you a list of the modules.


[private]


default:

    just -l

Organized like this in the repository:

home-ops/

├── justfile          # `just` entrypoint, loads `just/*`

├── just/

│   ├── bootstrap.just

│   ├── secrets.just

│   ├── cluster.just

│   └── ...

├── infrastructure/   # Terraform, setup in the previous post

└── gitops/           # GitOps using ArgoCD, setup in the future

Working directories

Another nice feature is to be able to set the working directory:

[doc("terraform init")]


[working-directory('../infrastructure')]


init *args:

    terraform init {{args}}

This means I can run just tf::init from anywhere in the entire repo, and it will just work.

Export kubeconfig / talosconfig

export KUBECONFIG := justfile_directory() / "infrastructure/kubeconfig.yaml"


export TALOSCONFIG := justfile_directory() / "infrastructure/talosconfig.yaml"

Every kubectl/talosctl command in any just file now targets the cluster, no matter what directory they’re run from.

Shebang

By default each line runs in its own shell. A #!/usr/bin/env bash at the top converts a recipe into one script, which allows you to define functions, run for loops, or set traps for cleanup.

For example, this recipe runs terraform apply and then encrypts the terraform state even on failure:

[doc("terraform apply, then re-encrypt state (encrypts even on failure)")]


[working-directory('../infrastructure')]


apply *args:

    #!/usr/bin/env bash

    set -e

    trap 'just secrets::encrypt_terraform_state' EXIT

    terraform apply {{args}}

(I encrypt the state so I can safely commit it to git, meaning I can manage terraform from multiple computers, since you need the up-to-date state.)

Providers

To use Terraform we need to add providers. I used the bpg/proxmox and siderolabs/talos for Proxmox and Talos Linux support:

terraform {

  required_providers {

    proxmox = {

      source  = "bpg/proxmox"

      version = "0.95.0"

    }

    talos = {

      source  = "siderolabs/talos"

      version = "0.10.1"

    }

  }


}

We also need to configure the Proxmox provider:

provider "proxmox" {

  insecure  = true

  endpoint  = "https://10.1.3.1:8006"

  username = "root@pam"

  password = "..."


}

You can (and maybe should) use an API token instead of username/password but I didn’t bother.

Either way it’s not good to commit credentials to git, so let’s move them to secret.auto.tfvars (add the file to .gitignore):

username = "root@pam"


password = "..."

This will get loaded automatically but we need variable blocks and reference them using var:

variable "username" {

  type      = string


}


variable "password" {

  type      = string

  sensitive = true


}




provider "proxmox" {

  username = var.username

  password = var.password


}

I’d like to validate that the Proxmox connection works but Terraform seems to recognize that it’s not used yet, so it won’t try to connect yet. Let’s continue.

Downloading the Talos image

Talos Linux has an image factory where you can find images to download. It’s a helpful tool as there are many parameters you can tweak to get the image you want.

The settings I chose are:

• Cloud server (we’ll install on Proxmox)
• I used version 1.12.6
• Nocloud (again, Proxmox)
• amd64
• The qemu-guest-agent System extension (important for Proxmox) (Note that iscsi-tools and util-linux-tools are also required for Longhorn.)

You’ll receive an image schematic ID, for example: ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515.

You can download this to Proxmox manually but Terraform can automate that.

To make it easier to update I broke it out to variables:

talos_version          = "1.12.6"


talos_image_factory_id = "ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515"

And then reconstruct the download url and tell Proxmox to download it like so:

resource "proxmox_virtual_environment_download_file" "talos_image" {

  content_type            = "iso"

  datastore_id            = "local"

  node_name               = "dorne"

  url                     = "https://factory.talos.dev/image/${var.talos_image_factory_id}/v${var.talos_version}/nocloud-amd64.raw.xz"

  decompression_algorithm = "zst"

  file_name               = "talos-v${var.talos_version}-nocloud-amd64.img"

  overwrite               = false


}

Note that datastore_id should match Proxmox storage that can contain images and node_name should match the name of the Proxmox node (Proxmox can manage multiple machines/nodes, this one is called dorne).

Now we can test that our Proxmox provider is wired correctly:

terraform init


terraform plan

If everything is okay Terraform should tell you that it wants to create a resource. Let’s execute the download:

terraform apply

And it should show up in the Proxmox GUI.

Creating a VM

Creating a VM is straightforward but there are a few settings we need to get right. Here’s a Terraform resource that will create a Talos Linux VM:

resource "proxmox_virtual_environment_vm" "talos" {

  name            = "talos-cp1"

  tags            = ["terraform", "talos"]

  node_name       = "dorne"

  on_boot         = true

  stop_on_destroy = true



  agent {

    enabled = true

  }

  disk {

    datastore_id = "local-lvm"

    file_id      = proxmox_virtual_environment_download_file.talos_image.id

    interface    = "virtio0"

    iothread     = true

    discard      = "on"

    size         = 20

  }

  initialization {

    datastore_id = "local-lvm"

    ip_config {

      ipv4 {

        address = "10.1.4.10/8"

        gateway = "10.0.0.1"

      }

    }

  }

  cpu {

    cores = 4

    type  = "x86-64-v2-AES"

  }

  memory {

    dedicated = 4 * 1024

    floating  = 4 * 1024

  }

  network_device {

    bridge = "vmbr0"

  }

  operating_system {

    type = "l26"

  }


}

• The VM in Proxmox will be called talos-cp1 on the dorne Proxmox node (like before).

• It’s important to enable the QEMU guest agent (which we also enabled in the image factory).

• Using a raw image (instead of an .iso) avoids the installation process as it directly boots from the image.

  The raw image also enables cloud-init configuration (the initialization block), which allows us to set a fixed IP (10.1.4.10) and set the gateway (my router, at 10.0.0.1).

• I create it with 4 CPU cores, 4GB of memory, and an OS disk size of 20 GB.

  This is fine to start with but you may run out of RAM or disk size when you start adding applications (like I did). Bumping up to something like 8 GB RAM and 40 GB storage is probably a good idea.

• There are some extra settings there like the cpu and operating system type that I had to have but I can’t explain why.

Creating multiple VMs

The above will create a single VM but for fun I wanted more.

Let’s go with 3 VMs and let’s break it out in a variable:

nodes = [

  {

    hostname = "talos-cp1"

    ip       = "10.1.4.10"

    cores    = 4

    memory   = 4 * 1024,

  },

  {

    hostname = "talos-cp2"

    ip       = "10.1.4.11"

    cores    = 4

    memory   = 4 * 1024,

  },

  {

    hostname = "talos-cp3"

    ip       = "10.1.4.12"

    cores    = 4

    memory   = 4 * 1024,

  }


]

The declaration looks like this:

variable "nodes" {

  description = "List of nodes and their configurations."

  type = list(object({

    hostname = string

    ip       = string

    cores    = number

    memory   = number

  }))


}

Looping in Terraform feels a bit weird to me as the syntax doesn’t wrap the whole resource, but it’s just a field that creates an each variable you can reference. Something like this (leaving out unchanged fields):

resource "proxmox_virtual_environment_vm" "talos" {

  for_each        = { for node in var.nodes : node.hostname => node }

  name            = each.key

  cpu {

    cores = each.value.cores

    type  = "x86-64-v2-AES"

  }

  memory {

    dedicated = each.value.memory

    floating  = each.value.memory

  }

  initialization {

    datastore_id = "local-lvm"

    ip_config {

      ipv4 {

        address = "${each.value.ip}/8"

        gateway = "10.0.0.1"

      }

    }

  }

  # ...


}

This should now create 3 VMs, with different hostnames and IPs.

Configuring Talos

Now we need to configure Talos Linux. We’ll essentially try to replicate the talosctl apply-config commands from the documentation via Terraform.

First some variables to make life a little easier:

talos_version      = "1.12.6"


kubernetes_version = "1.35.2"


cluster_name       = "talos-cluster"

Then we’ll need three configurations: machine secrets, client config, and the control machine config. The three nodes will be control nodes but if you want to create worker nodes you need a specific config for those, but I’ll skip that in this post.

resource "talos_machine_secrets" "machine_secrets" {

  talos_version = "v${var.talos_version}"


}




data "talos_client_configuration" "client_config" {

  cluster_name         = var.cluster_name

  client_configuration = talos_machine_secrets.machine_secrets.client_configuration

  endpoints            = local.node_ips

  nodes                = local.node_ips


}

The client config references the machine secrets and needs to list the IP addresses for all nodes and its endpoints (the control nodes, for me that’s all the nodes). I use a local to collect those:

locals {

  node_ips = [for node in var.nodes : node.ip]


}

The control machine config follows a similar pattern:

data "talos_machine_configuration" "control_machine_config" {

  cluster_name       = var.cluster_name

  cluster_endpoint   = local.cluster_endpoint

  machine_type       = "controlplane"

  machine_secrets    = talos_machine_secrets.machine_secrets.machine_secrets

  kubernetes_version = "v${var.kubernetes_version}"

  talos_version      = "v${var.talos_version}"



  config_patches = []


}

Of note here is cluster_endpoint which for us will be the first control node IP. We’ll change it later to a Virtual IP (VIP) to avoid a single point of failure, but for now:

locals {

  node_ips                = [for node in var.nodes : node.ip]

  primary_control_node_ip = local.node_ips[0]


}

What about config_patches? They correspond to the patches you apply with talosctl patch and we need to use it for a few things. One thing is to specify the install image so Talos pulls from the image factory during upgrades:

locals {

  install_image = "factory.talos.dev/installer/${var.talos_image_factory_id}:v${var.talos_version}"


}




data "talos_machine_configuration" "control_machine_config" {

  config_patches = [

    yamlencode({

      machine = {

        install = {

          disk  = "/dev/vda" # virtio0 disk

          image = local.install_image

        }

      }

    })

  ]


}

Talos will boot fine without it but if I understand things correctly during updates it’ll then use the official upstream image and will remove any extensions (such as the QEMU agent we need).

Another thing we need to patch is to allow our control nodes to schedule workloads because we don’t have any worker nodes:

data "talos_machine_configuration" "control_machine_config" {

  config_patches = [

    yamlencode({

      cluster = {

        allowSchedulingOnControlPlanes = true

      }

    })

    # Other patches here...

  ]


}

Then we’ll need to apply the configurations to our nodes:

resource "talos_machine_configuration_apply" "control_machine_config_apply" {

  for_each                    = { for node in var.nodes : node.hostname => node }

  depends_on                  = [proxmox_virtual_environment_vm.talos]

  client_configuration        = talos_machine_secrets.machine_secrets.client_configuration

  machine_configuration_input = data.talos_machine_configuration.control_machine_config.machine_configuration

  node                        = each.value.ip


}

Note how we loop through the nodes and target them individually using their IPs, and that we added a dependency to proxmox_virtual_environment_vm.talos to ensure that the VMs are created before we try to apply the configuration.

If you terraform apply this then the VMs will spin up and the nodes will leave Maintenance mode but get stuck in Booting and will print something like:

etcd is waiting to join the cluster, if this node is the first node of the cluster,

please run `talosctl bootstrap` against one of the following IPs:

[10.1.4.10]

(a bunch of other warnings and errors)

With Terraform, bootstrapping is done like this:

resource "talos_machine_bootstrap" "bootstrap" {

  depends_on           = [talos_machine_configuration_apply.control_machine_config_apply]

  client_configuration = talos_machine_secrets.machine_secrets.client_configuration

  node                 = local.primary_control_node_ip

  endpoint             = local.primary_control_node_ip


}

Generating config files

The nodes seem to be running fine and they all signal a Healthy Running state in the Proxmox console. But how do we access them?

We need the Talos and Kubernetes configuration files:

resource "talos_cluster_kubeconfig" "kubeconfig" {

  depends_on           = [talos_machine_bootstrap.bootstrap]

  client_configuration = talos_machine_secrets.machine_secrets.client_configuration

  node                 = local.primary_control_node_ip


}




output "talosconfig" {

  value     = data.talos_client_configuration.client_config.talos_config

  sensitive = true


}




output "kubeconfig" {

  value     = resource.talos_cluster_kubeconfig.kubeconfig.kubeconfig_raw

  sensitive = true


}

And generate them like so:

terraform output -raw talosconfig > talosconfig.yaml


terraform output -raw kubeconfig > kubeconfig.yaml

# Should be ok


talosctl --talosconfig ./talosconfig.yaml health -n 10.1.4.10


# Look at pods


kubectl --kubeconfig ./kubeconfig.yaml get pods -A

You can move them to ~/.talos/config and ~/.kube/config, or set TALOSCONFIG and KUBECONFIG to avoid specifying them all the time.

At this point we have a functional cluster but first I want to change a few things.

Setting up Cilium

I wanted to use Cilium for proxying and as the Container Network Interface (CNI) which means we have to disable them on the Talos nodes. New config_patches:

data "talos_machine_configuration" "control_machine_config" {

  config_patches = [

    # Disables the Flannel, the default CNI for Talos

    yamlencode({

      cluster = {

        network = {

          cni = {

            name = "none"

          }

        }

      }

    }),

    # Disables kube-proxy, the default proxy service

    yamlencode({

      cluster = {

        proxy = {

          disabled = true

        }

      }

    })

    # ...

  ]


}

If we rebuild the nodes we’ll see that talosctl health will stop at not ready:

waiting for all k8s nodes to report ready: some nodes are not ready: [talos-cp1-tmp talos-cp2-tmp talos-cp3-tmp]

This is to be expected as we haven’t installed Cilium yet. First we need to manually install the Gateway CRDs as they need to exist before we install cilium (because we want to use it for Gateway management later as well):

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml

Then we’ll install Cilium using Helm:

helm repo add cilium https://helm.cilium.io/


helm repo update


helm install cilium cilium/cilium \


  --namespace kube-system \


  --version 1.19.2 \


  --set kubeProxyReplacement=true \


  --set k8sServiceHost=10.1.4.10 \


  --set k8sServicePort=6443 \


  --set l2announcements.enabled=true \


  --set externalIPs.enabled=true \


  --set gatewayAPI.enabled=true \


  --set ipam.mode=kubernetes \


  --set operator.replicas=1 \


  --set securityContext.privileged=true

There are a bunch of options here, the most notable:

• kubeProxyReplacement=true use it as a kube-proxy replacement.
• k8sServiceHost=10.1.4.10 target the first control node.
• l2announcements.enabled=true use L2 announcements to give out IP addresses.
• externalIPs.enabled=true allow us to set fixed IPs manually.
• gatewayAPI.enabled=true enable the Gateway API that we’ll use in later posts.
• securityContext.privileged=true needed to work with Talos.

With this installed talosctl health should after a while return all OK again.

Load balancing

Let’s try out a good old classic to see if it works: the nginx test. We’ll use LoadBalancer to get an external IP:

kubectl run nginx --image=nginx --port=80


kubectl expose pod nginx --type=LoadBalancer --port=80

Get the IP:

$ kubectl get svc nginx

NAME    TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE

nginx   LoadBalancer   10.107.132.202   <pending>     80:30952/TCP   2s

Oh right, we haven’t configured load balancing for Cilium yet. Time for our first Kubernetes manifest!

apiVersion: "cilium.io/v2"


kind: CiliumLoadBalancerIPPool


metadata:

  name: first-pool


spec:

  blocks:

    - start: 10.1.4.101

      stop: 10.1.4.255


---


apiVersion: "cilium.io/v2alpha1"


kind: CiliumL2AnnouncementPolicy


metadata:

  name: l2-announcement


spec:

  interfaces:

    - eth0

  loadBalancerIPs: true

This tells Cilium to assign load balancing IPs in the range 10.1.4.101–10.1.4.255.

Apply:

kubectl apply -f cilium_config.yaml

After a while (yes, I hate waiting) kubectl get svc nginx should show an external IP that we can visit in the browser to verify that yes, we have a running app!

Mother always nagged me to clean up after myself:

kubectl delete pod nginx


kubectl delete svc nginx

Virtual IP

Kubernetes is supposed to be a resilient thing but we’ve introduced a central point of failure by using the first control node as the endpoint. If that one node goes down then the entire cluster is now unreachable.

We’ll fix that with a Virtual IP, where all control nodes will share a single IP. If one of them goes down then one of the others will take over. (How? Must be magic!)

Anyway, let’s designate a VIP:

cluster_vip  = "10.1.4.100"

Use it for the cluster endpoint:

locals {

  cluster_endpoint = "https://${var.cluster_vip}:6443"


}

And we’ll also need to patch the nodes to tell them to use the VIP:

data "talos_machine_configuration" "control_machine_config" {

  config_patches = [

    yamlencode({

      machine = {

        network = {

          interfaces = [{

            interface = "eth0"

            vip = {

              ip = var.cluster_vip

            }

          }]

        }

      }

    })

  ]


}

For it to work we also need to specify an interface. eth0 happened to work for me (verify with talosctl get links).

We also need to update the Cilium install parameters to target the VIP:

--set k8sServiceHost=10.1.4.100

To see that it gets assigned we can use talosctl get addresses; one of the nodes should be assigned the VIP. If we regenerate kubeconfig it should also contain the VIP and not the node IPs, so if kubectl can reach the cluster then all is good.

Nameservers

One last thing I’d like to mention is how to add your own nameserver. I’ve got my DNS overrides on my router at 10.0.0.1 that I’d like the nodes to pickup. Here’s how to patch it, with a fallback to 1.1.1.1:

data "talos_machine_configuration" "control_machine_config" {

  config_patches = [

    yamlencode({

      machine = {

        network = {

          nameservers = ["10.0.0.1", "1.1.1.1"]

        }

      }

    })

  ]


}

And with that we have a functional Kubernetes cluster that we can easily tear down and rebuild.

What bags?

I’ve had two backpacks that I used to travel with:

1. Datsusaru Battlepack Core

   It’s a really sturdy backpack that’s great to bring to BJJ training, but it’s very heavy and it’s not ideal as a traveling bag.

2. M-Tac Backpack Urban Line

   I bought this bag recently but I wasn’t too impressed. It was too small for a traveling bag and the zipper broke after just a few trips.

I’ve had a few suitcases that have broken down but I can’t remember the brand of. Most recently I’ve been using the Samsonite Essens 69cm that have held out great so far.

The packing list

• Water bottle

  I later decided not to bring it.

• 5 pair of socks and underwear

• 3 t-shirts, 1 long-sleeve t-shirt

• 1 slightly thicker long-sleeve shirt

• Training gear for Submission Wrestling

  Shorts, rashguard, spats, knee pads, and mouth guard.

• 1 pair of pants

• Mobile phone charger

• A Framework 13 laptop + charger

• Laptop–headphone cable

  I also brought the Sony WH-1000X M3 noise canceling headphones that I wore during the trip.

• Remarkable 2

• A bag for dirty clothes

• Toothbrush & toothpaste

• Power bank

• Vitamins & medicine

• Nail clippers, tape, Whoop body holder

• Earbuds & sleep mask

• A book—the first two books in the Murderbot Diaries series.

Things to wear

I saw the advice that the clothes you travel with is also very important. They have a point.

• Pants with large pockets with zippers.

  I could use the pants the whole week if I wanted to.

• A Houdini hoodie.

  Really warm and cozy, should last the whole week.

• A thin jacket.

  Paired with the hoodie it provides a decent enough protection against the weather. I’m not going on a hiking trip; I’ll be moving between the office, the hotel, and restaurants.

  It’s also thin enough so I can fit it into the fully packed backpack.

• A cap and gloves.

  I’m traveling in Sweden, it’s still fairly cold here.

What I wish I brought

1. It was a bit too cold on some days. A warmer jacket, long-johns, or a nudge would’ve been good.
2. The next Murderbot book. The first was fantastic! I had to visit a local book store where I bought Lock In. Loved that too.

Gearing up

Before I could try out one bag travel I had to get a new backpack. I ended up with the Fyro Levo 30L backpack mostly because the creator had a bunch of cool videos about bags… As shipping was expensive I also got their packing cubes, the retractable key leash, and a 1L sling (that I didn’t use).

This isn’t a review of their stuff; there’s probably better options out there but I’m too inexperienced to say. Maybe I should’ve gotten the 36L backpack, and the sling was an unnecessary purchase, but other than that I’ve been very happy with the Fyro products.

Packing in

Will it fit?

One of the biggest critiques against the bag is that the pockets here are very loose as things might fall out. When the bag is fully packed this has not been my experience—almost the reverse. Fully packed, it’s a bit too tight. But empty they’re too loose.

I did fit everything I wanted to bring. Although I skipped the water bottle it would’ve fit into the water bottle holder on the side. Maybe I’ll bring it next time.

The fully packed bag weighed in at 7.3 kg.

On the road

I do love the large front compartment.

I was a bit worried that the bag wouldn’t fit under the front seat but it worked out well. I didn’t take a picture of it but it fit in the overhead compartment during the flight too.

Although the extra space of the 36L backpack would’ve been greatly appreciated I fear that it might have made the bag too large to fit under the seat. In the end I think I prefer the convenience of having the bag under the seat over the extra 6L of space.

From hotel to office

The bag was also used to transport the laptop and other electronics between the hotel and office. I was worried that a huge bag would be a chore to bring but that’s been a non-issue. The only minor problem is that the bag doesn’t stand by itself without the packing cubes.