Troubleshooting CrafterCMS
This section details common problems you might encounter with CrafterCMS and how to solve them.
Topic |
Purpose |
|---|---|
Contains information on troubleshooting search issues |
|
Contains information on troubleshooting publishing issues |
|
Contains information on troubleshooting Deployer issues |
|
Contains information on troubleshooting SSH issues |
|
Contains information on troubleshooting SSL issues |
Troubleshooting Search Issues
How Search Works
CrafterCMS uses OpenSearch as the backing search engine. Crafter Deployer indexes the content and keeps OpenSearch up to date with the latest changes. Crafter Engine uses OpenSearch to perform search queries.
Indexing Process:
Authors create or update content in Crafter Studio
Studio writes and/or publishes the content
The Deployer picks up the changes in a particular target (which is the combination of site/environment). Then:
On preview update, the Deployer inspects the
sandboxand resolves the changes by doing a file diff between the last Git commit it processed and the latest commit in thesandboxOn publish, Studio moves the changes to
publishedand the Deployer in delivery pulls the changes
A processor of the target in the Deployer sends a request to search with the XML or content to index
Querying Process:
The end-user makes a request to Crafter Engine.
Crafter Engine components requiring queries issue a query to search where filters will be added to the query and then executed.
Places Search Indexing Can Get Hung Up
Communication Between Crafter Studio & Crafter Deployer
Crafter Studio will notify the Deployer when content has changed
Ensure that the Deployer is pulling the changes
If the Deployer is not receiving any changes, check network connectivity and ports
Communication Between Crafter Deployer & OpenSearch
The Deployer has a target with a Search Processor
Ensure that the processor is configured to the proper HOST, PORT for OpenSearch
Communication Between Crafter Engine & OpenSearch
Crafter Engine is configured to talk to OpenSearch
Ensure that Crafter Engine is configured to the proper HOST, PORT for OpenSearch
Reviewing Log Files
In order to figure out which of the possible failures described in the previous section happened, you should review the following log files for each one of the components:
Component
|
Log Files
|
|---|---|
Crafter Studio
|
CRAFTER/logs/tomcat/catalina.out |
Crafter Deployer
|
CRAFTER/logs/deployer/deployer.outCRAFTER/logs/deployer/deployer-general.logCRAFTER/logs/deployer/{target}-{env}.log |
Crafter Engine
|
CRAFTER/logs/tomcat/catalina.out |
Configure Deployer Port & Search Port
To configure the Deployer and Search port, open the file CRAFTER/bin/crafter-setenv.sh
export DEPLOYER_PORT=${DEPLOYER_PORT:="9191"}
export SEARCH_HOST=${SEARCH_HOST:="localhost"}
export SEARCH_PORT=${SEARCH_PORT:="9201"}
Troubleshooting Publishing Issues
When publishing fails, here are some things to consider to help track down the cause of publishing issues.
Inspect Publishing Queue
The publishing queue (items that are in the queue for publishing) can be viewed through Studio from the publishing icon at the top bar, or, 
-> Publishing, then scroll down to the Publishing Queue. Here, the user can narrow down the queue for inspection by specifying filters. For more information see Publishing Queue
Publishing Issues When Moving Projects Around in Disk
Publishing may fail when moving projects around in disk. When moving projects around, the reference between the published repository and the sandbox repository may not be valid anymore. To resolve the issue, the reference between the published repository and the sandbox repository needs to be updated.
Typically, the configuration for the published repository can be found in the file path_to_published_repo/published/.git/config and the reference to sandbox may look like this:
[remote "origin"]
url = ../sandbox
fetch = +refs/heads/*:refs/remotes/origin/*
In some cases, the configuration looks like this:
[remote "origin"]
url = /my/absolute/path/to/crafter_install/crafter-auth-env/bin/../data/repos/sites/mysite/sandbox
fetch = +refs/heads/*:refs/remotes/origin/*
To manually fix the configuration problem, either set the url value as a relative path between the published and the sandbox repositories (default ../sandbox) or set it as the absolute path of the sandbox repository.
Troubleshooting Deployer Issues
Unknown Host Key
When configuring a Deployer target to connect to a remote host through SSH you might encounter the following error:
com.jcraft.jsch.JSchException: UnknownHostKey: SERVER_NAME. RSA key fingerprint is 50:db:75:ba:11:2f:43:c9:ab:14:40:6d:7f:a1:ee:e3
This normally means that the SSH host key from the remote server is stored in ECDSA format instead of RSA, under known_hosts.
The Deployer requires the host key to be RSA. To fix this do the following:
Delete the
~/.ssh/known_hostsfile for the user that’s running the Deployer.Connect to the remote server with the following command to add the RSA key:
ssh -o HostKeyAlgorithms=ssh-rsa SERVER_NAME.
Git Errors
If the published folder in the Authoring installation became corrupted or had a Git issue that required manual intervention, and
the Deployer in Delivery is still pulling from that folder, there’s a good probability that the Delivery repository has pull/merge
errors. If you see these errors in the log, after fixing the Authoring published repository, you can do a git reset --hard
in the Delivery repository. This will reset the repository to the latest successful commit, and the Deployer will pull the changes
again from Authoring. You can also reset to a specific commit with git reset --hard <commit-hash>.
You might also see Git diff errors when the Deployer is executing the GitDiffProcessor. This means that the the changes
between the last processed commit and the last pulled commit could not be calculated. To fix this, go to data/deployer/processed-commits
and locate the .commit file for the site. If you know the hash of the last correctly processed commit, update the value and the
Deployer will automatically process from that commit to the latest. If you don’t know, delete the file and the Deployer will reprocess
from the initial commit, but beware because this normally means that all files will be reprocessed/reindexed.
Connection Timed Out
If the Deployer is not able to clone the remote repository and an error like below appears in the logs:
org.eclipse.jgit.errors.TransportException: ssh://jdoe@authoring-server/path/to/repo: Connection timed out (Connection timed out)
at org.eclipse.jgit.transport.JschConfigSessionFactory.getSession(JschConfigSessionFactory.java:159)
at org.eclipse.jgit.transport.SshTransport.getSession(SshTransport.java:137)
at org.eclipse.jgit.transport.TransportGitSsh$SshFetchConnection.<init>(TransportGitSsh.java:274)
at org.eclipse.jgit.transport.TransportGitSsh.openFetch(TransportGitSsh.java:169)
at org.eclipse.jgit.transport.FetchProcess.executeImp(FetchProcess.java:136)
at org.eclipse.jgit.transport.FetchProcess.execute(FetchProcess.java:122)
at org.eclipse.jgit.transport.Transport.fetch(Transport.java:1236)
at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:213)
... 15 common frames omitted
Try a manual clone from the command line: git clone ssh://jdoe@authoring-server/path/to/repo
If the manual clone works, it’s very probable that there’s a proxy between the servers. The Deployer currently does not support connections through proxies but might get support in a future update.
Low Disk
When OpenSearch disk utilization goes above the configured high disk watermark (default is 90%), errors in creating indexes for Deployer targets may appear and the following message will appear in your OpenSearch logs:
2024-10-08 02:13:37 [2024-10-09T06:13:37,787][WARN ][o.o.c.r.a.DiskThresholdMonitor] [30afa082cf18] Putting index create block on cluster as all nodes are breaching high disk watermark. Number of nodes above high watermark: 1.
2024-10-08 02:13:37 [2024-10-09T06:13:37,855][WARN ][o.o.c.r.a.AllocationService] [30afa082cf18] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
2024-10-08 02:13:37 [2024-10-09T06:13:37,980][WARN ][o.o.c.r.a.DiskThresholdMonitor] [30afa082cf18] high disk watermark [90%] exceeded on [Hvfsz707SbiNkljIdIu-2w][30afa082cf18][/usr/share/opensearch/data/nodes/0] free: 39.8gb[8.5%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete
To increase the high disk watermark, set the cluster.routing.allocation.disk.watermark.high property in
CRAFTER_HOME/bin/opensearch/config/opensearch.yml, to desired value. Say we want to increase the high disk watermark
to 95%, add the following to your OpenSearch configuration:
cluster.routing.allocation.disk.watermark.high: 0.95
Troubleshooting SSH Key Issues
In this section, we present a few errors that a user may encounter when using ssh keys with CrafterCMS and how to fix the errors.
SSH Unknown Host
When configuring a Deployer target to connect to a remote host through SSH you might encounter the following error:
Caused by: org.apache.sshd.common.SshException: Server key did not validate
This normally means that the server is not in the known_host file. To fix this run the following:
ssh-keyscan {server url} >> crafter-authoring/data/ssh/known_hosts
To test your SSH config, run the following:
ssh -F crafter-authoring/data/ssh/config -o UserKnownHostsFile=crafter-authoring/data/ssh/known_hosts -T {server url}
Remote Repository Not Found: USERAUTH fail
Here’s another error you may encounter in the logs when creating a site with a link to a remote repository using SSH keys:
org.apache.sshd.common.SshException: USERAUTH fail
This could mean that a passphrase was setup for the SSH keys. Crafter currently doesn’t support using a passphrase with SSH keys. To fix the error, remove the passphrase from the SSH key.
There are a couple of ways to remove the passphrase from the SSH key:
Delete your existing key and generate a new key with NO PASSPHRASE.
Or, another way is to run the following:
$ ssh-keygen -p
It will then prompt you to enter the keyfile location (
CRAFTER_HOME/data/sshby default), the old passphrase, and the new passphrase (leave this blank to have no passphrase).
Troubleshooting SSL Handshake Issues
In this section, we present an error that a user may encounter regarding SSL and how to fix the error.
Starting with JVM version 11.0.11, support for TLS v1.1 and earlier is disabled by default as indicated in the
Consolidated JDK 11 Release Notes under security-libs/javax.net.ssl.
If you encounter the following error in your application:
Caused by: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)
at sun.security.ssl.HandshakeContext.<init>(HandshakeContext.java:171) ~[?:1.8.0_292]
at sun.security.ssl.ClientHandshakeContext.<init>(ClientHandshakeContext.java:98) ~[?:1.8.0_292]
at sun.security.ssl.TransportContext.kickstart(TransportContext.java:220) ~[?:1.8.0_292]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:428) ~[?:1.8.0_292]
at com.mysql.cj.protocol.ExportControlled.performTlsHandshake(ExportControlled.java:317) ~[mysql-connector-java-8.0.23.jar:8.0.23]
at com.mysql.cj.protocol.StandardSocketFactory.performTlsHandshake(StandardSocketFactory.java:188) ~[mysql-connector-java-8.0.23.jar:8.0.23]
at com.mysql.cj.protocol.a.NativeSocketConnection.performTlsHandshake(NativeSocketConnection.java:97) ~[mysql-connector-java-8.0.23.jar:8.0.23]
at com.mysql.cj.protocol.a.NativeProtocol.negotiateSSLConnection(NativeProtocol.java:333) ~[mysql-connector-java-8.0.23.jar:8.0.23]
at com.mysql.cj.protocol.a.NativeAuthenticationProvider.connect(NativeAuthenticationProvider.java:167) ~[mysql-connector-java-8.0.23.jar:8.0.23]
at com.mysql.cj.protocol.a.NativeProtocol.connect(NativeProtocol.java:1350) ~[mysql-connector-java-8.0.23.jar:8.0.23]
at com.mysql.cj.NativeSession.connect(NativeSession.java:157) ~[mysql-connector-java-8.0.23.jar:8.0.23]
at com.mysql.cj.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:953) ~[mysql-connector-java-8.0.23.jar:8.0.23]
at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:823) ~[mysql-connector-java-8.0.23.jar:8.0.23]
... 173 more
This means your application is doing a connection with a version prior to TLSv1.2
Please check the documentation of the library performing the connection to see if there’s a way to enforce TLSv1.2 or higher (for example, when using a MySQL JDBC driver you can enforce it by adding the enabledTLSProtocols=TLSv1.2 parameter to your connection string).
If you must use TLSv1 or TLSv1.1, you can re-enable the disabled versions by removing “TLSv1” and/or “TLSv1.1” from the jdk.tls.disabledAlgorithms security property in the java.security configuration file.
Let’s take a look at an example of re-enabling TLSv1 and TLSv1.1.
Look for the java.security file in JDK_INSTALL_HOME/conf/security, and go to the property jdk.tls.disabledAlgorithms:
jdk.tls.disabledAlgorithms=SSLv3, TLSv1, TLSv1.1, RC4, DES, MD5withRSA, \
DH keySize < 1024, EC keySize < 224, 3DES_EDE_CBC, anon, NULL, \
include jdk.disabled.namedCurves
To re-enable the versions, remove TLSv1 and TLSv1.1 from the property jdk.tls.disabledAlgorithms
jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, \
DH keySize < 1024, EC keySize < 224, 3DES_EDE_CBC, anon, NULL, \
include jdk.disabled.namedCurves