Open Ask AI (⌘/Ctrl+I)

Cluster high memory pressure

Symptom

A CustomerClusterHighMemoryPressure alert is firing. It indicates a node in your cluster is under high memory pressure.

Observations

Here are the details on the alerts:

Alert: CustomerClusterHighMemoryPressure

PropertyValue
Expressioninstance:node_memory_available:ratio * 100 < 5 and rate(node_vmstat_pgmajfault[2m]) > 1000
Delay15 minutes
Labelsteam: customer

Annotations:

  • summary
  • description
  • tenant
  • cluster_id
  • cluster_name
  • instance

Determine the node with high memory pressure

The alert contains:

  • the cluster name/id (the k8s_cluster_name and k8s_cluster_id labels)
  • the cluster node (the instance label)
  • the device (the device label)
  • the filesystem (the mountpoint label)

Solutions

This section provides solutions that should help resolve the issue in most cases.

Increase cluster node instance type

The memory available to a cluster node is determined by the instance type. Cluster nodes are usually t3a-xlarge EC2 instances with 16GB memory. The next larger instance type, t3a.2xlarge, has 32GB memory. There are no instance types larger than t3a.2xlarge.

Add a node to the cluster

We recommend that each Magnolia instance and its database be run on a separate node in the cluster: the author plus two public instances can each be run on separate nodes and each Magnolia instance can have enough memory (10 - 12 GB for author instances and 8 - 10 GB for public instances).

If you want to run more than three Magnolia instances on the cluster, additional nodes can be added to the cluster.