KOTS: Troubleshoot rqlite

Troubleshooting
,
1

KOTS deploys a rqlite StatefulSet to store the version history, application metadata and other small amounts of data needed to manage the application(s).

These are checks/data retrieval that can be performed on rqlite StatefulSet.

Prerquisites

  • curl, jq, yq installed

Steps

First, port-forward rqlite to access its HTTP API

export NAMESPACE=<namespace>
kubectl port-forward svc/kotsadm-rqlite 4001:4001 -n $NAMESPACE

In another terminal, setup the credential

export NAMESPACE=<namespace>
export URL=kotsadm:$(kubectl get secret kotsadm-rqlite -n $NAMESPACE -o jsonpath='{.data.password}' | base64 -d)@localhost:4001

Now we can perform these checks/data retrieval using curl

Verify that rqlite is up and running

curl $URL/readyz
[+]node ok
[+]leader ok
[+]store ok

Check current database size

curl -s $URL/status\?pretty | jq -r '.store.dir_size_friendly'
3.7 MB

Verify that all rqlite nodes are live and reachable

 curl -s $URL/nodes\?pretty
{
    "kotsadm-rqlite-0": {
        "id": "kotsadm-rqlite-0",
        "api_addr": "http://kotsadm-rqlite-0.kotsadm-rqlite-headless.default.svc.cluster.local:4001",
        "addr": "kotsadm-rqlite-0.kotsadm-rqlite-headless.default.svc.cluster.local:4002",
        "voter": true,
        "reachable": true,
        "leader": true,
        "time": 2.08e-7,
        "time_s": "542ns"
    }
}

Validate that SQL query can be executed

curl -sG $URL/db/query\?pretty\&timings --data-urlencode 'q=SELECT 1'
{
    "results": [
        {
            "columns": [
                "1"
            ],
            "types": [
                "integer"
            ],
            "values": [
                [
                    1
                ]
            ],
            "time": 0.000061166
        }
    ],
    "time": 0.000264582
}

Get latest application config values

curl -sG $URL/db/query\?pretty\&timings --data-urlencode 'q=select av.config_values from app_version av join app a on a.id=av.app_id and a.current_sequence = av.sequence' | jq -r '.results[0].values[0][0]' | yq e 'del(.metadata, .status)' -

apiVersion: kots.io/v1beta1
kind: ConfigValues
spec:
  values:
...

Backup the database

curl $URL/db/backup\?compress -o db.bak.gz

Restore the database

gunzip -d db.bak.gz
curl -v -XPOST $URL/db/load -H "Content-type: application/octet-stream" --data-binary @db.bak