Updated On 4.1.4

Delivery Environment Performance Tuning

This section describes ways on how to enhance a traditional delivery environment setup (non-serverless) performance by tuning delivery environment settings and recommendations for hardware configurations.

Server Requirements

Minimum Installation

8GB of RAM + 8GB Swap Space or Virtual Memory

4GB JVM Memory (-Xms 1G -Xmx 4G)

4 CPU Cores

Medium Installations

16GB+ of RAM + 16GB Swap Space or Virtual Memory

8GB+ JVM Memory (-Xms 2G -Xmx 8G)

8+ CPU Cores

Large Installations

32GB+ of RAM + 16GB Swap Space or Virtual Memory

16GB+ of JVM Memory (-Xms 4G -Xmx 16G)

16+ CPU Cores

Horizontal scaling can be very effective in scaling out delivery of content.

High-level Performance Considerations

The majority of CrafterCMS operations are I/O intensive. Optimizing your installation for better I/O performance will typically pay the biggest dividends in performance gains early on. These general guidelines help address these considerations:

Fast raw storage performance (fast concurrent reads and writes)
Different storage devices are used for different concerns (logging, Git, search index, swap etc.)
Data organization on disk (using different devices for each repos, indexes, etc)
Leave half the RAM for the OS and non-JVM processes

Server Performance Tuning

Server/Hardware Level

Disk/Storage Devices

Crafter Studio’s job is to manage content. A high volume of concurrent reads and writes should be expected. The faster the disk type and connection to the computer, the better the performance you will observe.

Testing Raw Performance

Non-concurrent quick test or the raw device performance can be achieved with sudo hdparm -tT /dev/{device}

Example

Timing cached reads:   24486 MB in  1.99 seconds = 12284.28 MB/sec
Timing buffered disk reads: 3104 MB in  3.00 seconds = 1033.84 MB/sec

Test IOPS using fio https://github.com/axboe/fio

Example

$ fio --randrepeat=1 --ioengine=libaio --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75
    test: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=64
    fio-2.2.10
    Starting 1 process
    Jobs: 1 (f=1): [m(1)] [100.0% done] [495.2MB/164.7MB/0KB /s] [127K/42.2K/0 iops] [eta 00m:00s]
    test: (groupid=0, jobs=1): err= 0: pid=9071: Mon Apr 23 10:49:08 2018
        read : io=3071.7MB, bw=485624KB/s, iops=121406, runt=  6477msec
        write: io=1024.4MB, bw=161945KB/s, iops=40486, runt=  6477msec
        cpu          : usr=12.04%, sys=87.77%, ctx=32, majf=0, minf=8
        IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued    : total=r=786347/w=262229/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency   : target=0, window=0, percentile=100.00%, depth=64

    Run status group 0 (all jobs):
            READ: io=3071.7MB, aggrb=485624KB/s, minb=485624KB/s, maxb=485624KB/s, mint=6477msec, maxt=6477msec
            WRITE: io=1024.4MB, aggrb=161944KB/s, minb=161944KB/s, maxb=161944KB/s, mint=6477msec, maxt=6477msec

Note

Notice the IOPS for READ and WRITE

Test latency with ioping https://github.com/koct9i/ioping

Example

    $ ioping -c 10 .
KiB from . (ext4 /dev/nvme0n1p3): request=1 time=179 us
KiB from . (ext4 /dev/nvme0n1p3): request=2 time=602 us
KiB from . (ext4 /dev/nvme0n1p3): request=3 time=704 us
KiB from . (ext4 /dev/nvme0n1p3): request=4 time=600 us
KiB from . (ext4 /dev/nvme0n1p3): request=5 time=597 us
KiB from . (ext4 /dev/nvme0n1p3): request=6 time=612 us
KiB from . (ext4 /dev/nvme0n1p3): request=7 time=599 us
KiB from . (ext4 /dev/nvme0n1p3): request=8 time=659 us
KiB from . (ext4 /dev/nvme0n1p3): request=9 time=652 us
KiB from . (ext4 /dev/nvme0n1p3): request=10 time=742 us

    --- . (ext4 /dev/nvme0n1p3) ioping statistics ---
requests completed in 9.01 s, 1.68 k iops, 6.57 MiB/s
    min/avg/max/mdev = 179 us / 594 us / 742 us / 146 us

Recommendations

Prefer multiple devices to a single device

Crafter must update content, metadata about the content, search indexes and more on every write. By storing each kind of data on its own storage device, you better enable these activities to occur concurrently and hence vastly improve performance.

Prefer faster disk

Not all storage devices are created equal. The fast the read/write speeds and the more concurrency and lower latency the device supports, the better the performance will be. As a general rule of thumb, use the highest IOPS devices for the most demanding storage concerns, by order of importance:

{CRAFTER_HOME}/data/repos (high-concurrency, important)
{CRAFTER_HOME}/data/db (high-concurrency, important)
{CRAFTER_HOME}/data/indexes
{CRAFTER_HOME}/data/logs
{CRAFTER_HOME}/data/mongodb (if in use)

Avoid high latency connections to disk

High latency connectivity such as Network-Attached Storage (NAS) will typically lead to performance problems. Local disk or Storage Array Network will yield much better performance. NFS or similar protocols will increase latency and cause performance issues.

Use a device for each storage concern when possible

One optimization to raise effective IOPS of a system without buying very expensive storage devices is to distribute the load across many devices. CrafterCMS performs multiple reads/writes to disk from various concerns such as the database, the repository, logs, etc. with very different I/O patterns. For optimal performance, the server should have different storage systems (disks) mounted for different concerns, for example:

/dev/{dev0} -> /
/dev/{dev1} -> /opt/crafter/data/db
/dev/{dev2} -> /opt/crafter/data/repos
/dev/{dev3} -> /opt/crafter/data/indexes
/dev/{dev4} -> /opt/crafter/logs
/dev/{dev5} -> /opt/crafter/data/mongodb
/dev/{dev6} -> /var
/dev/{dev7} -> /home
/dev/{dev8} -> /usr

OS Level

Linux Ulimit

CrafterCMS includes many subsystems that require additional file-handles be available at the operating system level.

Our limits are:

[Service]
# Other directives omitted
# (file size)
LimitFSIZE=infinity
# (cpu time)
LimitCPU=infinity
# (virtual memory size)
LimitAS=infinity
# (locked-in-memory size)
LimitMEMLOCK=infinity
# (open files)
LimitNOFILE=65535
# (processes/threads)
LimitNPROC=65535

The values listed above can be persistently set in the limits.conf file located at /etc/security/

Here’s an example of how the items listed above will look like in a limits.conf file:

/etc/security/limits.conf

#[domain]        [type]  [item]   [value]
...

*                -       fsize    infinity
*                -       cpu      infinity
*                -       as       infinity
*                -       memlock  infinity
*                -       nofile   65535
*                -       nproc    65535

...

where

domain: can be a username, a group name, or a wildcard entry.
type: can be soft, hard or -
item: the resource to set the limit for

For more information on types, other items, etc. that you can configure, see your OS man page for limits.conf (e.g. man limits.conf or visit the online man page for your OS if available:: http://manpages.ubuntu.com/manpages/focal/en/man5/limits.conf.5.html )

Note

On RHEL/CentOS: For the nproc setting, please use /etc/security/limits.d/90-nproc.conf. More information can be found here

On Ubuntu: The limits.conf file is ignored for processes started by init.d . To apply the settings in limits.conf for processes started by init.d, open /etc/pam.d/su and uncomment the following: session required pam_limits.so

Engine Performance Tuning

JVM Level

To configure the heap size, etc for the JVM, open CRAFTER_HOME/bin/crafter-setenv.sh and update the environment variable CATALINA_OPTS to desired value like below:

CRAFTER_HOME/bin/crafter-setenv.sh

export CATALINA_OPTS=${CATALINA_OPTS:="-server -Xss1024K -Xms1G -Xmx2G -Dlog4j2.formatMsgNoLookups=true"}

Tomcat Application Server Level

Connector Thread Count

Update the Tomcat Connector thread count to correlate to the number of CPU cores available on the server. This will ensure that the server is able to handle the maximum number of concurrent requests.

To configure the maximum number of active threads and minimum number of threads (idle and active) alive, open the file CRAFTER_HOME/bin/apache-tomcat/conf/server.xml and set the following in the connector:

maxThreads=”<DESIRED_MAX_THREADS>”
minSpareThreads=”<DESIRED_MIN_SPARETHREADS>”>

In the configuration below, we set maxThreads to 600 and minSpareThreads to 100. For more information on Tomcat thread pools, see https://tomcat.apache.org/tomcat-9.0-doc/config/executor.html

CRAFTER_HOME/bin/apache-tomcat/conf/server.xml

<Connector port="${tomcat.http.port}" protocol="HTTP/1.1" URIEncoding="UTF-8"
           connectionTimeout="20000"
           redirectPort="${tomcat.https.port}"
           maxParameterCount="1000"
           maxThreads="600"
           minSpareThreads="100"
           />

Deployer Performance Tuning

Crafter Deployer is responsible for many operations including publishing content, updating search indexes, updating metadata about content and more. The faster the disk type, network connectivity, and available memory, the better the performance you will observe. For larger installations with a lot to index, the Deployer can run out of resources or be too slow for smooth operation of the system.

To configure the heap size, etc for the JVM, open CRAFTER_HOME/bin/crafter-setenv.sh and update the environment variable DEPLOYER_JAVA_OPTS to desired value like below:

CRAFTER_HOME/bin/crafter-setenv.sh

export DEPLOYER_JAVA_OPTS=${DEPLOYER_JAVA_OPTS:="-server -Xss1024K -Xmx1G -Dlog4j2.formatMsgNoLookups=true"}

Anti Patterns (Things NOT to do)

Here are some things we recommend NOT TO DO when setting up/configuring your authoring environment:

Slow Network Based Storage

Simple network storage such as NAS connected over copper network to compute is known to produce slow performance due to latency across many small operations. Avoid NAS storage.

Use of NFS as a Mounting Protocol

NFS is a particularly slow and unreliable network storage protocol, especially when mounts are configured with default settings.

Putting All Data on the Same Disk

Studio stores content in Git, Metadata about workflow and content in an embedded database and indexes in OpenSearch. All of these stores are updated on each write. Putting them on the same disk can lead to slower access times due to contention in high throughput scenarios.

Using Default Settings for Larger Installations

Installations are pre-configured with settings that assume an average/smaller sized machines. Further OS defaults are not managed by Crafter. To get the best performance you should consider and adjust for your specific environment, hardware, business needs and best practices.