• Document Up to Date
  • Updated On 4.0.0

Introduction to the Developer Workflow with CrafterCMS

Development and content authoring are both a major part of producing digital experiences. In this section, we’ll take a look at the basic mechanics that support content authoring, publishing and developer workflow.

Here’s a typical developer process and set of environments which must be followed in order to get new functionality to production.

Developer Workflow - Typical developer process and set of environments

We’ll first tackle CrafterCMS content management and deployment mechanics, then the DevContentOps workflow and finally how to setup the workflow.

Content Management and Deployment Mechanics

CrafterCMS is implemented with a decoupled architecture, composed of several microservices where content authoring and content delivery services are separated into their own distinct, subsystems.

Authoring is supported via Crafter Studio (a web-based authoring tool) that sits on top of a headless Git-based repository and publishing system. While content authors are performing their work, Crafter is managing all of the Git mechanics, managing locking, creating a time-machine like, Git-based version history and audit trail, all accessible via the Studio UI.

Developer Workflow - Authors work in Sandbox. Delivery nodes pull from Published

Crafter’s publish mechanism deploys content from the Authoring system to the Delivery system. Content logically flows from the authoring environment to the delivery environment. The mechanism for this, given the underlying Git repo, is a “pull” type interaction.

Each delivery node has a Deployer agent that coordinates deployment activities on the node for each project that is being delivered on that node.

  • Delivery nodes can initiate deployment pulls either on a scheduled interval (a “duty cycle”), on-demand via an API call, or both.
  • The Deployer performs a number of activities beyond receiving and updating content on the delivery node. A list of post-commit processors is run. These can be used to execute updates on search indexes, clear caches and perform other such operations.
  • The Delivery node maintains a clone of the Authoring Git-based repository.
  • The Crafter Deployer takes care of managing the synchronization of the delivery node’s clone authoring repository from the authoring environment.
  • Git-mechanics ensure content is in sync
Developer Workflow - Crafter’s Dynamic CMS Publishing via Git

Only approved content is published to the delivery environment. Crafter manages this by using 2 repositories for each project. One called a Sandbox which contains work-in-progress and the other called Published which represents approved, published work and complete content history.

  • Authors use the Crafter Studio UI to review and approve content via workflow.
  • Crafter Studio takes care of moving approved work between Sandbox and Published repositories.
  • Delivery nodes monitor the published repository for updates.
Developer Workflow - Authors work in Sandbox. Delivery nodes pull from Published

The underlying Git repositories and related workflow for Authoring require no setup at all. When you create a project in Crafter Studio it automatically creates the local Sandbox and Published repositories. When you add a new Delivery node a simple command line script is run on that node that configures the node’s deployer to replicate and process content from the Published repository from authoring.

  • Instructions for creating a project can be found here .
  • Instructions for initializing a delivery node can be found here.

DevContentOps Workflow

Content Authors can edit and preview content via Crafter Studio. Edits are saved in the Git-based repository Sandbox. On approval, edits are moved to the repository Published. Crafter Studio handles all of the mechanics for authors. Authors simply click edit and make changes, perform reviews and publish work via the UI. In the background, Crafter Studio will lock content, perform commits and take any other actions necessary to perform the low-level repository tasks necessary.

The authoring environment is akin to a work and test area for the authors. Relative to the live project or app the content authors are deploying content to, the Crafter Studio (and the Sandbox repository) where they edit, preview and approve content is what we would call a lower environment. Work is promoted from the lower environment (authoring) to the live environment (delivery.)

Let’s take a look at a typical developer workflow that’s integrated with the production authoring environment. In this diagram, you will find a mechanism for both a “code forward” activity in which code can be moved through a development process through to production as well as a “content back” activity in which production content, including in-process content, can be brought back to lower development environments, including local development machines.

Developer Workflow - Typical DevContentOps Workflow

DevContentOps workflow that describes in general how content flows back to the development process and code flows forward through the certification process to production

Let’s take a closer look at the diagram above.

Step 1: Create an Upstream Repository

In order to facilitate a developer workflow, you want to establish an upstream repository for your Sandbox in your production authoring environment. Any Git repository will work. It’s typical to use a Git repository that has a UI/Workflow atop of it such as GitHub, Bitbucket, GitLab and other similar systems. The upstream repository is the root of the developer workflow.

Developer Workflow - Create upstream repository

A Git repository with support for “developer/team workflow” such as GitHub, GitLab or Bitbucket is configured as a remote upstream repository of the Production Authoring Sandbox repository. This is the primary point of contact between the authoring process and the development and code release process come together.

Step 2: Use Automation to Sync Sandbox with the Upstream

Note that the repository is what the Git community refers to as an “upstream” remote repository. That means in a sense the Production Sandbox becomes subordinate to it from a Git log perspective. This upstream repository is where content and development work will be merged before making its way to the production Sandbox and ultimately to the Published repository and the delivery nodes themselves. Also, note that nothing changes about the publishing and deployment configuration and topology of the Production environment.

You will want to keep the upstream repository up to date with authoring. The best way to accomplish this is to use a DevContentOps automation platform such as Jenkins or Bamboo to orchestrate a push to the upstream on a schedule. Keeping the upstream repository up to date with the authoring Sandbox repository provides downstream development and testing repositories with the latest content and helps to avoid conflicts when it’s time to promote from the upstream repository to the authoring Sandbox repository.

Developer Workflow - Use Automation to Sync Sandbox with the Upstream

Content flows from Production Authoring Sandbox to the Developer repository via automated Git Push

Step 3: Use the Upstream Repository to Support Development and Testing Environments and Workflow

From this upstream repository, you will feed all lower environments and branch for each of your development efforts. From the upstream repository, you can support any development process and supply content to any development or testing environment. Lower environments may consist of Authoring and Delivery, or just Authoring, or just Delivery. It depends on the needs. For example, Development often contains both Authoring and Delivery, while QA tends to focus only on Delivery. Simple “Git pull” mechanics are used to move code and content from the upstream repository, typically from a branch.

Developer Workflow - Use the Upstream Repository to Support Development and Testing Environments and Workflow

Development branches and forks can be created to meet your specific workflow needs. Lower environments use these repositories and branches as their upstream.

Step 4: Merge Code Updates Into the Upstream Master and Pull them to the Production Authoring Sandbox Repository

When you are ready to move code from a development branch to the authoring Sandbox you will first merge the work into the master of the upstream repository. You can do merge all of your commits or you can merge them into a single commit by using the Git rebase command. By Squashing all of the commits into a single commit you make it easier to move the workaround as a single unit. Merges are typically done via pull requests on repositories that support development workflow. Once the merge operation is complete in the upstream repository your automation will carry that work to the production authoring Sandbox.

Developer Workflow - Merge Code Updates Into the Upstream Master and Pull them to the Production Authoring Sandbox Repository

Use “Pull Requests” and other Git workflow mechanics to promote code through the process. When it’s ready to go live merge it to the “Master” of the Developer Repository. At this point, it will flow via automation to the Production Authoring Sandbox.

Step 5: Use Crafter Studio’s Publish Commits API to Publish Code to Delivery

Once your development work is in the Production Sandbox you will want to publish it. To do this Crafter Studio provides a REST API that will trigger the publishing on one or more commits. Simply call Crafter Studio’s Publish Commits REST API (/api/1/services/api/1/publish/commits.json) via the DevContentOps automation platform passing the required parameters including the commit IDs to be published and Crafter Studio will move the work to the Published repository from which it will be replicated to your delivery nodes.

Developer Workflow - Use Crafter Studio’s Publish Commits API to Publish Code to Delivery

Move code from Sandbox to Published with a simple API call.

Setting Up the Workflow

If you don’t have Crafter Studio up and running, begin with Step 0, otherwise, go to Step 1

Step 0: Install Crafter Studio

Before we begin setting up the workflow for DevContentops, we need to have Crafter Studio up and running.

First, please review the following for requirements and supported platforms: Requirements and Supported Platforms.

We’ll then install the Crafter Studio (authoring instance). You can follow the How do I install CrafterCMS section here to install and start authoring.

Alternatively, you can also install CrafterCMS and start authoring by cloning the craftercms repo and building and deploying it using gradle.

git clone https://github.com/craftercms/craftercms.git ./gradlew clone build deploy -Penv=authoring ./gradlew start -Penv=authoring

Step 1: Create your upstream

Create an empty repository in your upstream (GitHub, Bitbucket etc.) The specific steps depend on the repository you are using. The key here is that you want to create an empty or what’s sometimes called a “bare” repository.

Developer Workflow - Create a bare repo in Git

On GitHub, once created you will see the following screen. Here you will note the repository URL (https://github.com/myuser/devworkflowexample.git) which you will need for the next step. Also, if you’re trying to create an upstream for an existing project, you’ll find the instructions below in the “push an existing repository from the command line” section.

Developer Workflow - Bare repo created in Git

Step 2: Create Your Project In Crafter Studio

Next, you want to log in to Crafter Studio as the admin user. The admin user has the rights to create new projects (called projectss.) Click Create Project.

Developer Workflow - Create Project via Crafter Studio

Clicking Create Project will present you with the Create Project dialog. This dialog changes depending on what you choose. Below is an example of the dialog filled out in a way that creates your project locally. We will then add the Github repository as its upstream remote and push the initial project contents to the upstream repository.

Let’s walk through each part of the dialog:

Developer How Tos - Create Project Dialog Walk Through step 1 :width: 50 %
  1. The first thing you need to do is to choose your blueprint. There are several out of the box blueprints provided by default. Choose one of these or one of your own. For our example, we’ll choose the Website Editorial blueprint.

    Developer How Tos - Create Project Dialog Walk Through step 2

  2. The next thing to do is give your project an ID. The ID itself doesn’t matter in a sense. It doesn’t need to match anything per se, technically speaking the only requirement is that it’s unique. That said, it’s a best practice to provide an ID that is meaningful/recognizable to the team. If your website is called FreshFlowers.com a good ID might be “freshflowerscom”.

    Click on the Review button, then finally click on the Create Project button to create your project. CrafterCMS will create the local repositories, search index and internal data structures required to support the project and install the blueprint. Your project should be created in a short while.

  3. Once your project is created, the next step is to add a remote repository to your newly created project. Open the Sidebar then click on Project Tools -> Remote Repositories, then click on the New Repository on the top right.

    This will open up a dialog where we can fill in all the information for our remote repository.

    Create Repository dialog to fill in information of remote repository being added to the project

  4. In the Remote Git Repository Name field you want to provide a repository name that makes sense. It’s common to use “origin” or “upstream.”

  5. In the Remote Git Repository URL field you must provide the link to the Git repository discussed in Step #1: https://github.com/myuser/mysweetdotcom.git

  6. Provide your credentials based on the authentication method selected. For example, if we chose Username & Password as the authentication method, we’ll need to fill in Username and Password

    Note

    When using ssh keys for authentication, the keys can be generated using one of the following: RSA, ECDSA, ED25519 or DSA as the algorithm and with no passphrase (Crafter currently doesn’t support using a passphrase with SSH keys.)

    To generate your Secure Shell (SSH) keys for authentication, run the following command:

    Generate an RSA SSH keypair with a 4096 bit private key

    ssh-keygen -t rsa -b 4096

    Generate an DSA SSH keypair with a 2048 bit private key

    ssh-keygen -t dsa -b 1024

    Generate an ECDSA SSH keypair with a 521 bit private key

    ssh-keygen -t ecdsa -b 521

    Generate an ed25519 SSH keypair- this is a new algorithm added in OpenSSH.

    ssh-keygen -t ed25519

    Your output should look something like this:

    ✗ ssh-keygen -m PEM -b 4096 -t rsa
    Generating public/private rsa key pair.
    Enter file in which to save the key (/Users/myuser/.ssh/id_rsa):
    Enter passphrase (empty for no passphrase):
    Enter same passphrase again:
    Your identification has been saved in /Users/myuser/.ssh/id_rsa.
    Your public key has been saved in /Users/myuser/.ssh/id_rsa.pub.
    .
    .
    

    After generating your private and public keys, you will need to add your new public key to where your remote git repository is located. If you are using GitHub, you will need to add your public key (e.g., id_rsa.pub) into your GitHub account. If your remote Git repository is hosted on a server, you will need to copy your public key (e.g., id_rsa.pub) to the host server.

  7. Once you’re satisfied with your entries for your remote repository, click on the Create button. Once complete we can now connect to the upstream and push the contents of the Sandbox repository to the remote by clicking on the Push button (button with the up arrow).

    Remotes screen displaying newly added remote repository to project

    Figure:: Project is created and a remote repository added

Step 3: Set up Your Delivery Nodes

Now that your project is created you can set up the rest of your production environment by initializing your delivery nodes to receive deployments from authoring. Remember these delivery nodes will pull from Crafter Studio’s repositories, not the upstream remote repository.

When you add a new delivery node a simple command line script is run on that node that configures it to replicate and process content from the “Published” repository from authoring.

Instructions for creating a project can be found here .

Instructions for initializing a delivery node can be found here.

Step 4: Set up your Developer Workflow and Lower Environments

Now that your upstream repository is initialized in GitHub you can set up any developer workflow you want. It’s typical to consider Master to be in-sync with the Production Authoring Sandbox. Given that, you don’t want to work in Master. Create branches to isolate development work from work that’s ready to move to Production Authoring. Below is an example topology that shows multiple environments and developer workflow that include feature branches, developer forms, and local developer clones.

Developer Workflow - Set up Developer Workflow and Lower Environments

Full DevContentOps “Code Forward, Content Back” workflow for CI/CD with CrafterCMS leveraging Git mechanics and DevContentOps automation

Setting Up to Work Locally Against the Upstream

After setting up the topology above, if you just want to get a local environment up and going (say a new developer joining the team), follow these instructions:

  1. Install Crafter Studio locally (Binaries download or Source build)
  2. Login as Admin
  3. Click Create Project
Developer Workflow - Setting up to work locally against the upstream

  1. Fill out the Create Project Form as in a similar fashion described in Step 2 above, except this time, instead of selecting a blueprint, select Remote Git Repository to create your project based on an existing upstream repository. This can be your team’s branch or your own fork. The exact workflow is up to you.
Developer How Tos - Setting up to work locally against the upstream

Developer How Tos - Setting up to work locally against the upstream review entries

Set up a project based on existing remote Git repository via Crafter Studio