Retrieve KOTS Config Values When the Admin Console is Unavailable

[diamon_w]

When the KOTS Admin Console or CLI is unavailable or not functioning properly, we’ve written a tool that can be used to extract and decrypt KOTS configuration values directly from the rqlite database.

When to Use This

• kotsadm pods are down or unhealthy

• kubectl kots get config --decrypt fails or hangs

• Admin Console is unavailable

Prerequisites

• kubectl with access to your KOTS namespace

• Go 1.19+ (for compiling)

Quick Start

Compile the Binary

*# Download source*

wget https://gist.githubusercontent.com/diamonwiggins/d1f4246274039a9a0b1021f69d4b0883/raw/decrypt-kots-config.go



*# Initialize module and get dependencies*

go mod init kots-decrypt

go get sigs.k8s.io/yaml

go mod tidy



*# Compile for Linux (most common)*

GOOS=linux GOARCH=amd64 go build -o kots-decrypt-config kots-decrypt-config.go



*# For macOS Intel*

GOOS=darwin GOARCH=amd64 go build -o kots-decrypt-config kots-decrypt-config.go



*# For macOS Apple Silicon*

GOOS=darwin GOARCH=arm64 go build -o kots-decrypt-config kots-decrypt-config.go

Usage

*# Basic usage*

./kots-decrypt-config -n <namespace>



*# Specific sequence*

./kots-decrypt-config -n default -s 5



*# Custom output directory*

./kots-decrypt-config -n default -o /var/backups



*# Help*

./kots-decrypt-config --help

Output

Creates two files in the output directory (default: /tmp):

• kots-config-raw.yaml - Encrypted config values

• kots-config-decrypted.yaml - Decrypted values in valuePlaintext fields

Example decrypted output:

apiVersion: kots.io/v1beta1
kind: ConfigValues
spec:
  values:
    database_password:
      valuePlaintext: myDecryptedPassword123  *# Decrypted!*
    database_host:
      default: postgres.example.com

How It Works

1. Retrieves encryption key from kotsadm-encryption secret (handles double base64 encoding)

2. Retrieves rqlite password from kotsadm-rqlite secret

3. Port-forwards to rqlite on port 4001

4. Queries rqlite for config values (bypasses kotsadm)

5. Decrypts password fields using AES-192-GCM

6. Outputs both raw and decrypted configs

Security Notes

The decrypted file contains plaintext passwords!

*# Secure the output*

chmod 600 /tmp/kots-config-decrypted.yaml



*# Delete when done*

rm /tmp/kots-config-*.yaml

Source Code

Complete source: kots-decrypt-config.go

[salahalsaleh]

If the rqlite database is unhealthy or the configuration values are missing, you can restore them from MinIO by following these instructions:

1. Exec into the minio pod:

   kubectl exec -it kotsadm-minio-0 -- bash
   

2. Prepare the alias for the minio client:

   mc alias set kotsminio http://$KOTSADM_MINIO_SERVICE_HOST:$KOTSADM_MINIO_SERVICE_PORT_SERVICE "$MINIO_ACCESS_KEY" "$MINIO_SECRET_KEY"
   

3. List apps:

   mc ls kotsminio/kotsadm/
   

4. The command from step 3 will show the app id, then run (replace the app id with the actual value, and make sure there’s a forward slash at the end):

   mc ls kotsminio/kotsadm/<app-id>/
   

5. The command from step 4 will show the sequence numbers of the application versions, if you want the config values for the most recent version created in the admin console, pick the highest number then run (replace app id and sequence number with actual values):

   mc cp kotsminio/kotsadm/<app-id>/<sequence>.tar.gz /tmp/<sequence>.tar.gz
   

6. Extract the retrieved archive:

   mkdir -p /tmp/archive && tar xzf /tmp/<sequence>.tar.gz -C /tmp/archive
   

7. Exit out of the minio pod:

   exit
   

8. Copy the config values out of the kotsadm pod to the host:

   kubectl cp kotsadm-minio-0:/tmp/archive/upstream/userdata/config.yaml ./config.yaml