Open Ask AI (⌘/Ctrl+I)
On this page

Database persistent volume is almost full

Symptom

A CustomerDatabasePersistentVolumeAlmostFull alert is firing.

Observations

Here are the details on the alerts:

Alert: CustomerDatabasePersistentVolumeAlmostFull

PropertyValue
Expression(kubelet_volume_stats_available_bytes{persistentvolumeclaim=~"data.*"} / kubelet_volume_stats_capacity_bytes{persistentvolumeclaim=~"data.*"}) * 100 < 1
Delay15 minutes
Labelsteam: customer

Annotations:

  • summary
  • description
  • tenant
  • cluster_id
  • cluster_name
  • pvc
  • namespace

Determine the current size of the database persistent volume

The name of the database persistent volume claim and its namespace are contained in the alert, for example “database-prod-magnolia-helm-public-db-0”. You can find the details on the persistent volume by displaying the persistent volume claim using kubectl:

kubectl -n <namespace from alert> get pvc <persistent volume claim from alert>

Executing the above will list the size of the persistent volume:

$ kubectl -n prod get pvc database-prod-magnolia-helm-public-db-0
NAME                                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
database-prod-magnolia-helm-public-db-0   Bound    pvc-e695274c-5794-4a78-afe0-e86b0efd40ed   10Gi       RWO            local-path     9d

Note the value listed in the “CAPACITY” column.

Solutions

This section provides a solution that should help resolve the issue in most cases.

  1. Increase the capacity of the database persistent volume
  2. Increase capacity via Helm chart
  3. Delete the StatefulSet

Increase the capacity of the database persistent volume

The space allocated to the Database persistent volume can be increased by editing the persistent volume in Rancher and by modifying the values of the Helm chart deploying Magnolia.

You can edit the persistent volume claim and increase the capacity in Rancher or with kubectl.

Edit the persistent volume claim or the persistent volume associated with the claim and increase the size of the storage resource.

  1. Get your desired namespace.
kubectl -n <namespace> get pvc <pvc-name>
  1. Edit the PVC.
kubectl -n <namespace> edit pvc <pvc-name>
  1. In the editor, adjust the storage.
  resources:
    requests:
      storage: 10Gi <1>

Increase the storage property.

Increase capacity via Helm chart

Set the Helm chart value db.persistence.size to an appropriate value in your values.yml file.

Delete the StatefulSet

Because StatefulSet volume definitions cannot be updated in place, you must manually update them.

  1. Run the kubectl command to delete the statefulset.
kubectl delete statefulset <name> --cascade=orphan <1>

Important: The --cascade=orphan flag ensures running pods are not deleted.

  1. Redeploy Magnolia.