Open Ask AI (⌘/Ctrl+I)
On this page

Magnolia JCR failure with false status

Magnolia fails to connect to its database with an exception.

Symptoms

Exception

When Magnolia fails to connect to the database, an exception is thrown much like the following:

info.magnolia.repository.RepositoryNotInitializedException: org.apache.jackrabbit.core.data.DataStoreException: Can not init data store, driver=org.postgresql.Driver url=null user=null schemaObjectPrefix=ds_ tableSQL=datastore createTableSQL=CREATE TABLE ds_datastore(ID VARCHAR(255) PRIMARY KEY, LENGTH BIGINT, LAST_MODIFIED BIGINT, DATA BYTEA)
...
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (The connection attempt failed.)
...
Caused by: org.postgresql.util.PSQLException: The connection attempt failed.
...
Caused by: java.net.SocketTimeoutException: connect timed out

When Tomcat shuts down Magnolia, you see:

SEVERE org.apache.catalina.core.StandardContext startInternal One or more listeners failed to start. Full details will be found in the appropriate container log file

Tomcat receives 404

Requests sent to Tomcat receive a 404 since Magnolia is no longer running.

Observations

The primary issue is that the pod may appear to be running normally despite Magnolia shutting down due to the exception and hence, no errors are reported in the container logs.

When does this happen?

The failure occurs randomly during Kubernetes upgrades. Because of this, both author and public instances are affected.

Workaround

We are working on fixing this as it is considered a bug. At the time of writing, it has not been fixed, so you should consider the workaround here.

  1. Delete the affected Magnolia instance.

    You can do this via Rancher or using kubectl commands.

    Magnolia StatefulSet recreates the Magnolia pod.

  2. Check the Magnolia logs to ensure the problem is not reoccurring.

Live traffic

If the Magnolia instance is serving live traffic, the 404 response may be cached.

  • If you’re using Fastly, purge the CDN cache in the Content Delivery section of the Cockpit.
  • If you’re using a different CDN, you’ll need to flush the cache as per your CDN instructions.