# KOTS: Troubleshoot rqlite

**URL:** <https://community.replicated.com/t/kots-troubleshoot-rqlite/1421>\
**Category:** Troubleshooting\
**Tags:** kots, support\
**Created:** [July 3, 2024, 3:41am UTC](https://community.replicated.com/t/kots-troubleshoot-rqlite/1421 "2024-07-03T03:41:24Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![GerardNguyen](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.replicated.com/gerardnguyen/32/828_2.png) [@GerardNguyen](https://community.replicated.com/u/GerardNguyen)\
**Post date:** [July 3, 2024, 3:41am UTC](https://community.replicated.com/t/kots-troubleshoot-rqlite/1421/1 "2024-07-03T03:41:24Z")

</div>

KOTS deploys a rqlite StatefulSet to store the version history, application metadata and other small amounts of data needed to manage the application(s).

These are checks/data retrieval that can be performed on rqlite StatefulSet.

## Prerquisites

- curl, jq, yq installed

## Steps

First, port-forward rqlite to access its HTTP API

```auto
export NAMESPACE=<namespace>
kubectl port-forward svc/kotsadm-rqlite 4001:4001 -n $NAMESPACE

```

In another terminal, setup the credential

```auto
export NAMESPACE=<namespace>
export URL=kotsadm:$(kubectl get secret kotsadm-rqlite -n $NAMESPACE -o jsonpath='{.data.password}' | base64 -d)@localhost:4001

```

Now we can perform these checks/data retrieval using `curl`

### Verify that rqlite is up and running

```auto
curl $URL/readyz
[+]node ok
[+]leader ok
[+]store ok

```

### Check current database size

```auto
curl -s $URL/status\?pretty | jq -r '.store.dir_size_friendly'
3.7 MB

```

### Verify that all rqlite nodes are live and reachable

```auto
 curl -s $URL/nodes\?pretty
{
    "kotsadm-rqlite-0": {
        "id": "kotsadm-rqlite-0",
        "api_addr": "http://kotsadm-rqlite-0.kotsadm-rqlite-headless.default.svc.cluster.local:4001",
        "addr": "kotsadm-rqlite-0.kotsadm-rqlite-headless.default.svc.cluster.local:4002",
        "voter": true,
        "reachable": true,
        "leader": true,
        "time": 2.08e-7,
        "time_s": "542ns"
    }
}

```

### Validate that SQL query can be executed

```auto
curl -sG $URL/db/query\?pretty\&timings --data-urlencode 'q=SELECT 1'
{
    "results": [
        {
            "columns": [
                "1"
            ],
            "types": [
                "integer"
            ],
            "values": [
                [
                    1
                ]
            ],
            "time": 0.000061166
        }
    ],
    "time": 0.000264582
}

```

### Get latest application config values

```auto
curl -sG $URL/db/query\?pretty\&timings --data-urlencode 'q=select av.config_values from app_version av join app a on a.id=av.app_id and a.current_sequence = av.sequence' | jq -r '.results[0].values[0][0]' | yq e 'del(.metadata, .status)' -

apiVersion: kots.io/v1beta1
kind: ConfigValues
spec:
  values:
...  

```

### Backup the database

```auto
curl $URL/db/backup\?compress -o db.bak.gz

```

### Restore the database

```auto
gunzip -d db.bak.gz
curl -v -XPOST $URL/db/load -H "Content-type: application/octet-stream" --data-binary @db.bak

```