Sitecore SXA CI/CD Improvements

It is a copy of my article initially placed here to keep all things in one place.

Saving compiled files in the source code is a bad practice. No one will argue that having either exe or dll or minified css or minified js is not good for your source control. But I continue to see serialized Sitecore SXA theme files added to source control. Many projects have pre-optimized-min.yml (or separate main.yml, styles.yml, scripts.yml, etc.) files saved in source control. And the common explanation, why it is there, is that it is hard to do on CI because it requires Sitecore to get these files. Let's figure out is it really complex and how we can avoid it.

The Problem

Firstly, let's talk about why having exe, dll, minified css, minified js, and pre-optimized-min.yml saved in your repository is not good for your project. All companies use version control systems for codebase management. Majorly it is Git. It is an awesome tool that allows the simultaneous work of many developers on the same project. But it works properly only when you don't save compiled (dll, exe) or transpiled (minified css, minified js) files. Once you start to save these files in the source control, you start to get conflicts when developers need to merge their changes. It happens because developers can work on different .cs, .js, .css, etc. files, but these files will be compiled or transpiled into the single file and you will get a conflict there. And pre-optimized-min.yml is the only serialization of transpiled minified js or css. It means that it will cause the same issue. Your developers will struggle with constant source conflicts and will spend time on resolving them. It means that it is better don't have them in source code. And there are 2 solutions, how we can get rid of them.

Solution 1

The easiest solution is to move execution of SXA gulp tasks(buildSass, buildCss, buildJs, cssOptimization, jsOptimization, uploadCss, uploadJs) to your CI\CD pipeline. You will need:

1. Remove all files produced by SXA gulp tasks from source control, including serialized theme files.
2. Add execution of SXA gulp tasks to your deployment step

Solution 2

Having pre-optimized-min.yml means that you already have either Unicorn or TDS in place. And you already delivering content to your environments using serialization. Let's look at content of this file:

---
ID: "bb97e335-fb50-4c14-b0e8-270f0f246f0a"
Parent: "e9eacb5b-ba97-46fb-af75-184cba337e1a"
Template: "962b53c4-f93b-4df9-9821-415c867b8903"
Path: "/sitecore/media library/Themes/Experiment/styles/pre-optimized-min"
DB: master
SharedFields:
- ID: "06d5295c-ed2f-4a54-9bf2-26228d113318"
  Hint: __Icon
  Value: "-/media/9be3803bcbfd44afaf16a8d2c94f15a2.ashx?h=16&thn=1&w=16"
- ID: "40e50ed9-ba07-4702-992e-a912738d32dc"
  Hint: Blob
  BlobID: "2afa94b5-0b9a-4319-82a1-8b17a411e275"
  Value: 6bGVmdDtjbGVhcjpi..................=
- ID: "6954b7c7-2487-423f-8600-436cb3b6dc0e"
  Hint: Size
  Value: 72312410
- ID: "6f47a0a5-9c94-4b48-abeb-42d38def6054"
  Hint: Mime Type
  Value: text/css
- ID: "c06867fe-9a43-4c7d-b739-48780492d06f"
  Hint: Extension
  Value: css
Languages:
- Language: en
  Versions:
  - Version: 1
    Fields:
    - ID: "25bed78c-4957-4165-998a-ca1b52f67497"
      Hint: __Created
      Value: 20210526T095146Z
    - ID: "5dd74568-4d4b-44c1-b513-0af5f4cda34f"
      Hint: __Created by
      Value: |
        sitecore\anton

Almost everything that is present in this file is text. There are only 2 fields that cause merge conflicts: Blob and Size. Are we able to fill these 2 fields without Sitecore? Yes, we are! Blob is base64 encoded content of the file: pre-optimized-min.css(or pre-optimized-min.js). And Size is the size of this file. We can use this PowerShell script to get both these values from pre-optimized-min.css:

<#
Get source file, convert to base64 string, put into the template, and put into destination
#>
param (
    [string]$source,
    [string]$destination,
    [string]$template
)

$content = [IO.File]::ReadAllText($source)
$bytes = [System.Text.Encoding]::UTF8.GetBytes($content)
$encodedText =[Convert]::ToBase64String($bytes)

$size = (Get-Item $source).length

$templateContent = [IO.File]::ReadAllText($template)

$destinationContent = $templateContent -replace '\${{size}}', $size
$destinationContent = $destinationContent -replace '\${{blob}}', $encodedText

[IO.File]::WriteAllText($destination, $destinationContent)

All additional steps that you need is to create a template, with Blob and Size values replaced with ${{blob}} and ${{size}} and run this script before Unicorn sync in your CI/CD pipeline.

Conclusion

Adding binary files to the source control is a bad practice. And your serialized SXA theme files are binary files (because of the presence of base64 string). Removing these files from source control and improvement of your CI/CD pipeline will save a lot of time and the mental health of your developers. And they will be grateful for this improvement.

Using WebP Format in Sitecore

It is a copy of this blog post to have all my articles in one place.
WebP is promising format that allows good optimization comparing to JPEG and PNG. According to tests it provides lossless images that are 26% smaller compared to PNGs and lossy images that are 25-34% smaller than comparable JPEG images at equivalent SSIM quality index.
This format becomes more and more popular over the web nowadays. Google promotes WebP format, it released WebP as open source to allow anyone works with it and suggest improvements. But, is it supported by major browsers? Going to CanIUse gives answer that not:

(support details are on 2018/01/21, it could be changed over time)

It is supported by Chrome, Opera, Chrome for Android. Safari and Firefox are experimenting(not supported yet) with supporting WebP images. IE and Edge doesn’t support it.
But how it can be used now, when not all browsers support it? Each browser(web client) provides Accept header when getting resources from server. If this header contains image/webp, web server know that it can returns WebP format. As example Akamai CDN use this behavior. It can return optimized WebP images for web clients who support it and JPEG and PNG for those who doesn’t support.
Let’s consider how it could be used in Sitecore. It has no sense to add support of WebP format to media library for now, we can’t return this file format to all web clients. But it makes sense to return WebP format to web clients that can use it. Saving about 25% of time on loading images can make big difference for user experience, especially on mobile. I decided don’t write new image optimizer from the scratch and add support of WebP to well known tool for images optimization for the Sitecore: Dianoga.
To make it works, we should get understanding how Sitecore Media Library and Sitecore Media Library cache work. Sitecore media library creates files on disk after each request, or uses previously created files. By default these files are located under \Website\App_Data\MediaCache\website\{some folders}. There are few files in each folder. First type of files is cache for image. It is image itself, Sitecore don’t need to preprocess width, height and other parameters for each media request. It does processing only once. Second type of files is .ini file, it is metadata for cached object. Metadata contains next values:
Key. e.g. ?as=False&bc=0&h=0&iar=False&mh=0&mw=0&sc=0&thn=False&w=0. Key contains all media query parameters.
Extension - extension of file.
Headers - cached headers that should be returned to web client
DataFile - file name of object that should be returned to web client.

We need to extend forming of key, to add one more parameter that will indicate support of WebP format. It will require extending of MediaRequestHandler:

public class MediaRequestHandler : Sitecore.Resources.Media.MediaRequestHandler
{
 protected override bool DoProcessRequest(HttpContext context, MediaRequest request, Media media)
 {
  if (context?.Request.AcceptTypes != null && (context.Request.AcceptTypes).Contains("image/webp"))
  {
   request.Options.CustomOptions["extension"] = "webp";
  }

  return base.DoProcessRequest(context, request, media);
 }

 private static bool AcceptWebP(HttpContext context)
 {
  return context?.Request.AcceptTypes != null && (context.Request.AcceptTypes).Contains("image/webp");
 }
}

Our key will contain one more additional parameter: extension. E.g.: ?as=False&bc=0&h=0&iar=False&mh=0&mw=0&sc=0&thn=False&w=0&extension=webp
Let’s add handler that will process WebP compatible requests based on
CommandLineToolOptimizer:

public class WebPOptimizer : CommandLineToolOptimizer
{
 public override void Process(OptimizerArgs args)
 {
  //If WebP optimization was executed then abort running other optimizers
  //because they don't accept webp input file format
  if (args.AcceptWebP)
  {
   base.Process(args);
   args.AbortPipeline();
  }
 }

 protected override string CreateToolArguments(string tempFilePath, string tempOutputPath)
 {
  return $"\"{tempFilePath}\" -o \"{tempOutputPath}\" ";
 }
}

It is very easy, it runs cwebp.exe tool that converts JPEG or PNG to WebP. It doesn’t utilize all available command line options, and could be tuned depending on requirements. All others code changes are more about Dianoga configuration and unit tests, I will not stop on them in this article. If you want more details, you can review all changes in GitHub repository.

How to enable Dianoga WebP support for your project:

1. Clone GitHub repository and build project
2. Enable Dianoga.WebP.config.disabled config
3. Open web.config and change line <add verb="*" path="sitecore_media.ashx" type="Sitecore.Resources.Media.MediaRequestHandler, Sitecore.Kernel" name="Sitecore.MediaRequestHandler" /> to <add verb="*" path="sitecore_media.ashx" type="Dianoga.MediaRequestHandler, Dianoga" name="Sitecore.MediaRequestHandler" />
4. If you have custom MediaRequestHandler (e.g. Habitat is used) then skip step 3 and override DoProcessRequest method with detection of support of WebP format. See MediaRequestHandler code listening above.

P.S. It is experimental feature, use it on your own risk. :-)