Cluster filesystem filling up
Symptom
A CustomerClusterFilesystemFillingUp alert is firing. It indicates a filesystem on a node in one of your cluster(s) is almost full.
Observations
Here are the details on the alerts:
Alert: CustomerClusterFilesystemFillingUp
| Property | Value |
|---|---|
| Expression | predict_linear(node_filesystem_avail_bytes[24h], 8 * 3600) < 0 and instance:node_filesystem_avail:ratio < 0.3 |
| Delay | 30 minutes |
| Labels | team: customer |
Annotations:
summarydescriptiontenantcluster_idcluster_nameinstancedevicemountpoint
Determine the cluster node and filesystem filling up
The alert contains:
- the cluster name/id (the
k8s_cluster_nameandk8s_cluster_idlabels) - the cluster node (the instance label)
- the device (the device label)
- the filesystem (the mountpoint label)
Solutions
This section provides solutions that should help resolve the issue in most cases.
Increase device size for cluster
You can increase the size of the device used by the cluster node in AWS.
- Use the cluster node value (instance label) to find the node and the device (device label) running out of space.
- Increase the device size in the AWS EC2 web interface or AWS command line.
Contact the PaaS Helpdesk
If you have tried the above solutions, and nothing has worked, reach out to us directly via the DX Cloud Helpdesk.