Cluster filesystem is almost full
Symptom
A CustomerClusterFilesystemAlmostFull alert is firing.
Observations
Here are the details on the alerts:
Alert: CustomerClusterFilesystemAlmostFull
| Property | Value |
|---|---|
| Expression | instance:node_filesystem_avail:ratio * 100 < 1 |
| Delay | 15 minutes |
| Labels | team: customer |
Annotations:
summarydescriptiontenantcluster_idcluster_nameinstancedevicemountpoint
Determine cluster node and filesystem filling up
The alert will contain:
- the cluster name/id (the
k8s_cluster_nameandk8s_cluster_idlabels) - the cluster node (the instance label)
- the device (the device label)
- the filesystem (the mountpoint label)
Solutions
This section provides solutions that should help resolve the issue in most cases.
Increase cluster node size
The size of the device used by the cluster node can be increased in AWS. Use the cluster node value (instance label) to find the node and the device (device label) running out of space and increase the device size in the AWS EC2 web interface or AWS command line.