Saturday, November 27, 2021

Sitecore SXA 10.2 Improvements: CSS and JS Source Maps

It is a copy of my article initially placed here to keep all things in one place.

Remember my article about enabling of Source Map for Sitecore SXA?

November 2021 Sitecore SXA 10.2 was released. But if you check the list of new features/improvements in release notes, you will not find one important improvement for developers: Now enabling CSS Source Maps becomes much easier!

Steps that you need:

  1. Open gulp\config.js file
  2. Set sassSourceMap value to true
  3. Set js>JsSourceMap value to true
  4. Set css>cssSourceMap value to true
  5. Restart gulp

If you will look inside SXA Source Maps implementation, you will see that implementation is very similar to what I did for earlier Sitecore versions. Probably SXA Developers Team was inspired by my previous post.

Enjoy your work on the SXA theme with enabled source maps! And remember, if you need to get the source maps for Sitecore SXA earlier 10.2 version, you can use this article.

Saturday, November 13, 2021

Sitecore Solr Proxy Console: Update to Sitecore 10 Version

It is a copy of my article, initially placed here to keep all things in one place.

 2 years ago I had problems with troubleshooting issues with search based on Sitecore Solr. I solved that problem by writing a small tool that makes the content management server acts as a reverse proxy. It receives a request and depending on the requested URL, it either handles the request by Sitecore or redirects it to the Solr server. It made things simple, I didn't need any additional access. All that I needed was the ability to log in on Sitecore CM.

Time passed, and the problem with access to the Solr admin console didn't disappear. It became even more actual with Azure, Docker, and Kubernetes hosting. Of course, when you are an admin on your Azure subscription, you can easily give yourself permission. But sometimes, especially when you work with big companies, giving permission is not a quick process. But you need to solve the issue with search right now. And one of my colleagues recently faced the same problem. It means that issue is still actual.

The version that was created for Sitecore 8, didn't work and required a few updates. The first was the way how Sitecore Solr connection is configured. The second was a change in arguments for HttpRequestProcessor implementation. The third was changes in the authentication process. After these small changes, now module is ready for Sitecore 8-10 versions.

How to use module

  1. Download update package from GitHub releases page. You need to use 1.+ version for Sitecore 8.+ and 2.+ version for Sitecore 9.+ and 10.+
  2. Install update package using Sitecore update installation wizard
  3. You will get a link on Sitecore desktop to run Solr proxy
  4. You will be able to access Sitecore Solr Proxy using https://yourwebsite/solr for Sitecore 8.+ and https://yourwebsite/sitecore/shell/solr/ for Sitecore 9.+ and Sitecore 10.+

I hope, this module will save your time during troubleshooting search issues in different environments.

Saturday, October 2, 2021

Be Careful with BlobID Value During Merge of YML Files

It is a copy of my article initially posted here to keep all things in one place.

Let's imagine that you have a Sitecore project with configured Unicorn serialization. And you are working on this project with the team. One day you get a conflict in one of your files in Media Library:

winmerge

It is important to remember that BlobID is important as well as Value property. If during solving of conflict you update only Value and update your item then nothing will be changed. Media item will still have old media attached to the item.

We have faced with this issue during the automation of our processes. We wrote a CI script that creates YML serialized files from the SXA theme. We updated the Value but didn't touch BlobID, it caused items to remain the same.

Thursday, July 1, 2021

Sitecore SXA: Turning on CSS Source Map Files

It is a copy of my article, initially posted here to keep all things in one place.

Update from 26 November 2021: If you use Sitecore 10.2 version, please use out of the box configuration. Or you can consider use @sca/celt 10.2 for your SXA theme for lower Sitecore versions as well.

CSS source map files are the cornerstone thing for modern web development. Sitecore SXA has the ability to turn on the usage of CSS map files. But there are use cases when you could not turn on SXA source maps out of the box. Let's examine why do you need to have source map files and how to configure using them for different cases in the Sitecore experience accelerator.

Theory: CSS Source Map Files

If you are familiar with CSS source map files, you can skip this chapter and scroll to the next one. It is present here because there will be parts that rely on source map specification in further explanation.

Source maps were introduced a long time ago. It was always easier for developers to save files with proper formatting. But proper formatting takes precious bytes. Each space, tab, line end character, clear variable name make your file bigger. And each additional byte makes page speed slower and users' experience worse. And developers found a way how to solve this problem. You can post-process your source and get rid of everything that doesn't affect styles and JS code execution. That is how minification was introduced. But minification created another problem. You made your page fast, but you also made troubleshooting the frontend much harder. At this point source files mapping appeared. The idea was pretty simple, we can provide a map from minified file to its real sources. And when you open sources in your developer's toolbar you will see a reference to your source file and it will allow you to save pricey seconds during troubleshooting of issues.

Web development evolved and new meta-languages and approaches appeared. SASS, SCSS, Babel, CoffeeScript, TypeScript opened new abilities for developers and made development faster and more comfortable. But browsers didn't evolve fast. And you still can use only plain CSS and JS for your pages. (Fortunately new languages specifications, but still only this approaches). And source map files became even more valuable as you write code in one meta-language, but browsers use a different one. Because browser can pretty print page sources, but it has no idea how file was written initially.

We used that source map has somename.css.map file name. But that is actually not a standard. It is a common location, where the source map is placed. But it could be configured! We can specify sourceMappingURL parameter at the end of .css file.

We can use different sourceMappingURL parameter values:

  • Absolute link to source map file
  • Relative link to source map file
  • Data URL: base64 encoded source map file. sourceMappingURL=data:application/json;charset=utf8;base64,eyJ2ZXJzaW.....

Another option, how to specify the source map file is setting X-SourceMap (or SourceMap in the newer specification) header. It will require changes in server-side code or configuring rewrite rules, which will make our solution more complex.

Adding CSS Source Map when Minification is Disabled

SXA Creative Live Exchange allows working in different modes. One of the options is to upload all produced .css files to Sitecore Media Library. (When you have enableMinification: true and disableSourceUploading: true settings in your config.js file.

In this case, everything is easy. You need to open your config.js file and change cssSourceMap and sassSourceMap values to true. In this case, SXA gulp scripts will add source maps to the end of each .css file of your theme.

Challenges of Adding CSS Source Map when Minification is Enabled

When minification is enabled, SXA gulp scripts prepare pre-optimized-min.css file, which is uploaded to Sitecore Media Library and then served as a styles file for your SXA theme.

With enabled cssSourceMap and sassSourceMap values, SXA gulp scripts will start to create pre-optimized-min.css.map source map file. But there is no watcher for this type of file. Also, we will need to have item pre-optimized-min.css with .map extension in Sitecore Media Library. Or pre-optimized-min item with .css.map extension. But Sitecore doesn't accept dots in item names by default. And we will not be able to use .css.map extension because in this case we will need to have 2 Sitecore items in the same level, which is also not allowed. But from the theory part, we know that we can add source map directly inside .css or configure different source map file name. Let's do it!

Placing Source Map inside pre-optimized-min.css

If we will look at how minification is done in SXA gulp scripts we will notice that there it uses gulp-sourcemaps npm package. And this package allows the flexible configuring location of source maps.

All that we need is to open \node_modules\@sxa\celt\util\cssMinificator.js file, which is responsible for concatenation and minification of .css files, and patch it. (Of course, it is better to copy this file to your source control and modify it separately. I am writing about changing in this file only for simplification). We need to replace gulpSourcemaps.write('.\') on gulpSourcemaps.write() . Changing of argument will write source map inside pre-optimized-min.css. It is a quick and easy change. But it could be used only for development environments, but not for production. Because it will double the size of css file and we will lose all improvements added by minification.

Proper Way of Adding CSS Source Map File to Your SXA Theme

We have already reviewed few options, how we can add source map files. But that options don't work for production. It means that you will need to have different approaches for different environments. It is not good. Let's figure out how to do it in a long way, but properly.

First of all, similar to other approaches, we need to set cssSourceMap and sassSourceMap values to true in config.js file.

To avoid problems with dots in item names and the same filenames on the same level, let's name our source maps filename pre-optimize-min-css.map instead of pre-optimized-min.css.map . This small change will make our work easier. We will not need to have any changes in the Sitecore backend or configuration. To achieve it, let's copy cssMinificator.js file and make small change:

// tasks\override\cssMinificator.js
// File is based on \node_modules\@sxa\celt\util\cssMinificator.js
// It allows to use different name for map file
// For our case we need [file]-css.map instead [file].css.map

const gulp = require('gulp');
const cleanCSS = require('gulp-clean-css');
const path = require('path');
const gulpSourcemaps = require('gulp-sourcemaps');
const gulpif = require('gulp-if');
const gulpConcat = require('gulp-concat');
const config = require(path.resolve(process.cwd(), './gulp/config'));

module.exports = function (cb) {
    let conf = config.css;
    if (!conf.enableMinification) {
        if (process.env.debug == 'true') {
            console.log('CSS minification is disabled by enableMinification flag'.red)
        }
        return cb();
    }
    let streamSource = conf.minificationPath.concat(['!' + conf.cssOptimiserFilePath + conf.cssOptimiserFileName])
    let stream = gulp.src(streamSource)
        .pipe(gulpif(function () {
            return conf.cssSourceMap;
        }, gulpSourcemaps.init()))
        .pipe(cleanCSS(config.minifyOptions.css))
        .pipe(gulpConcat(conf.cssOptimiserFileName))
        //Changed part: we use different filename suffix for Sitecore compatibility
        .pipe(gulpif(function () {
            return conf.cssSourceMap;
        }, gulpSourcemaps.write('./', {
            mapFile: function (mapFilePath) {
                // source map files are named *-css.map instead of *.css.map
                return mapFilePath.replace('.css.map', '-css.map');
            }
        })))
        //Disable previous pipe and enable this one if you want to add CSS source maps to the same file
        //.pipe(gulpif(function () {
        //  return conf.cssSourceMap;
        //}, gulpSourcemaps.write()))
        .pipe(gulp.dest('styles'));
    stream.on('end', function () {
        console.log('Css minification done'.grey)
    });
    return stream

}

watchCSS built-in task has a dependency on cssMinificator.js. That is why we need also to override it and use our new minification script.

// tasks\override\watchCss.js
// File is based on \node_modules\@sxa\celt\util\watchCss.js
// We need to call different CSS optimizer with changed map file name

const gulp = require('gulp');
const colors = require('colors');
const vinyl = require('vinyl-file');
const config = require(global.rootPath + '/gulp/config');
//Changed relative path to absolute
const { fileActionResolver } = require('@sxa/celt/util/fileActionResolver');
//Changed path to overridden module
const cssMinificator = require('./cssMinificator');

module.exports = function watchCss() {
    setTimeout(function () {
        console.log('Watching CSS files started...'.green);
    }, 0);
    let conf = config.css,
        indervalId,
        watch = gulp.watch(conf.path, { queue: true });
    watch.on('all', function (event, path) {
        var file = {
            path: path
        };
        if (event !== 'unlink') {
            file = vinyl.readSync(path);
        }

        file.event = event;
        if (!conf.disableSourceUploading || file.path.indexOf(conf.cssOptimiserFileName) > -1) {
            fileActionResolver(file);
        } else {
            if (process.env.debug == 'true') {
                console.log(`Uploading ${file.basename} prevented because value disableSourceUploading:true`.yellow);
            }
        }
        if (conf.enableMinification && file.path.indexOf(conf.cssOptimiserFileName) == -1) {
            indervalId && clearTimeout(indervalId);
            indervalId = setTimeout(function () {
                indervalId = clearTimeout(indervalId);
                cssMinificator();
            }, 400)
        }
    })
}

The next step is adding watch and upload tasks for .map files:

// tasks\watchMap.js
const gulp = require('gulp');
const vinyl = require('vinyl-file');
const config = require(global.rootPath + '/gulp/config');
//Changed relative path to absolute
const { fileActionResolver } = require('@sxa/celt/util/fileActionResolver');

module.exports = function watchMap() {
    setTimeout(function () {
        console.log('Watching MAP files started...'.green);
    }, 0);
    let conf = config.map
        watch = gulp.watch(conf.path, { queue: true });
    watch.on('all', function (event, path) {
        var file = {
            path: path
        };
        if (event !== 'unlink') {
            file = vinyl.readSync(path);
        }

        file.event = event;
        fileActionResolver(file);
    })
}
// tasks\uploadMap.js
const gulp = require('gulp');
const tap = require('gulp-tap');
const config = require(global.rootPath + '/gulp/config');
//Changed relative path to absolute
const { fileActionResolver } = require('@sxa/celt/util/fileActionResolver');

module.exports = function uploadMap() {
    var conf = config.map;
    const promises = [];

    return gulp.src(conf.path)
        .pipe(tap(
            function (_file) {
                let file = _file;
                file.event = 'change';
                promises.push(() => fileActionResolver(file));
            })
        )
        .on('end', async () => {
            for (const prom of promises) {
                await prom();
            }
        })
}

These files need configuration, where to look for .map files. We configure it in the additional setting in config.js file:

// config.js

...
map: {
        path: ['styles/*.map']
    },
...

Now, let's bring it all together in gulpfile.js

// gulpfile.js

...
const watchCssTasks = require('./gulp/tasks/override/watchCss');
// Instead of:
// const watchCssTasks = getTask('watchCss');

...

const cssOptimizationTasks = require('./gulp/tasks/override/cssMinificator');
// Instead of:
// const cssOptimizationTasks = getTask('cssOptimization');

...

// New tasks for .map files
const watchMapTasks = require('./gulp/tasks/watchMap');
const uploadMapTasks = require('./gulp/tasks/uploadMap');
module.exports.watchMap = gulp.series(login, watchMapTasks);
module.exports.uploadMap = gulp.series(login, uploadMapTasks);

...

// Changed default task with added watchMapTasks
module.exports.default = module.exports.watchAll = gulp.series(login,
    gulp.parallel(
        watchHtmlTasks,
        watchCssTasks,
        watchJSTasks,
        watchESTasks,
        watchImgTasks,
        watchScribanTasks,
        watchSassTasks.watchStyles,
        watchSassTasks.watchBase,
        watchSassTasks.watchComponents,
        watchSassTasks.watchDependency,
        watchSassSourceTasks,
        watchMapTasks
    )
);

...

// Extended Build + upload tasks
module.exports.rebuildAll = gulp.series(
    login,
    jsOptimizationTasks, sassComponentsTasks, cssOptimizationTasks,
    uploadJsTasks, uploadCssTasks, uploadImgTasks, uploadMapTasks
)
module.exports.rebuildMain = gulp.series(
    login,
    jsOptimizationTasks, sassComponentsTasks, cssOptimizationTasks,
    uploadJsTasks, uploadCssTasks, uploadMapTasks
)

...

Voila! Now after running gulp buildAll or triggering watch task we get a proper link at the end of pre-optimize-min.css and pre-optimize-min-css.map file itself. And both these files are uploaded to the media library. Now when you troubleshoot any CSS issues using the developer tools, you see, where it is done in sources.

Conclusion

I hope that someone from the Sitecore SXA team will read my article and include the ability to upload CSS source map files for pre-optimize-min.css out of the box with SXA gulp scripts. It is easy to change, but it can save a lot of developer hours.

But even if not, you still have control over your sources and your project. And if you need something to be improved then everything is in your hands! No needs to wait when it will be implemented by someone else.

Thursday, June 10, 2021

Sitecore SXA CI/CD Improvements

It is a copy of my article initially placed here to keep all things in one place.

Saving compiled files in the source code is a bad practice. No one will argue that having either exe or dll or minified css or minified js is not good for your source control. But I continue to see serialized Sitecore SXA theme files added to source control. Many projects have pre-optimized-min.yml (or separate main.yml, styles.yml, scripts.yml, etc.) files saved in source control. And the common explanation, why it is there, is that it is hard to do on CI because it requires Sitecore to get these files. Let's figure out is it really complex and how we can avoid it.

The Problem

Firstly, let's talk about why having exe, dll, minified css, minified js, and pre-optimized-min.yml saved in your repository is not good for your project. All companies use version control systems for codebase management. Majorly it is Git. It is an awesome tool that allows the simultaneous work of many developers on the same project. But it works properly only when you don't save compiled (dll, exe) or transpiled (minified css, minified js) files. Once you start to save these files in the source control, you start to get conflicts when developers need to merge their changes. It happens because developers can work on different .cs, .js, .css, etc. files, but these files will be compiled or transpiled into the single file and you will get a conflict there. And pre-optimized-min.yml is the only serialization of transpiled minified js or css. It means that it will cause the same issue. Your developers will struggle with constant source conflicts and will spend time on resolving them. It means that it is better don't have them in source code. And there are 2 solutions, how we can get rid of them.

Solution 1

The easiest solution is to move execution of SXA gulp tasks(buildSass, buildCss, buildJs, cssOptimization, jsOptimization, uploadCss, uploadJs) to your CI\CD pipeline. You will need:

  1. Remove all files produced by SXA gulp tasks from source control, including serialized theme files.
  2. Add execution of SXA gulp tasks to your deployment step

Solution 2

Having pre-optimized-min.yml means that you already have either Unicorn or TDS in place. And you already delivering content to your environments using serialization. Let's look at content of this file:

---
ID: "bb97e335-fb50-4c14-b0e8-270f0f246f0a"
Parent: "e9eacb5b-ba97-46fb-af75-184cba337e1a"
Template: "962b53c4-f93b-4df9-9821-415c867b8903"
Path: "/sitecore/media library/Themes/Experiment/styles/pre-optimized-min"
DB: master
SharedFields:
- ID: "06d5295c-ed2f-4a54-9bf2-26228d113318"
  Hint: __Icon
  Value: "-/media/9be3803bcbfd44afaf16a8d2c94f15a2.ashx?h=16&thn=1&w=16"
- ID: "40e50ed9-ba07-4702-992e-a912738d32dc"
  Hint: Blob
  BlobID: "2afa94b5-0b9a-4319-82a1-8b17a411e275"
  Value: 6bGVmdDtjbGVhcjpi..................=
- ID: "6954b7c7-2487-423f-8600-436cb3b6dc0e"
  Hint: Size
  Value: 72312410
- ID: "6f47a0a5-9c94-4b48-abeb-42d38def6054"
  Hint: Mime Type
  Value: text/css
- ID: "c06867fe-9a43-4c7d-b739-48780492d06f"
  Hint: Extension
  Value: css
Languages:
- Language: en
  Versions:
  - Version: 1
    Fields:
    - ID: "25bed78c-4957-4165-998a-ca1b52f67497"
      Hint: __Created
      Value: 20210526T095146Z
    - ID: "5dd74568-4d4b-44c1-b513-0af5f4cda34f"
      Hint: __Created by
      Value: |
        sitecore\anton

Almost everything that is present in this file is text. There are only 2 fields that cause merge conflicts: Blob and Size. Are we able to fill these 2 fields without Sitecore? Yes, we are! Blob is base64 encoded content of the file: pre-optimized-min.css(or pre-optimized-min.js). And Size is the size of this file. We can use this PowerShell script to get both these values from pre-optimized-min.css:

<#
Get source file, convert to base64 string, put into the template, and put into destination
#>
param (
    [string]$source,
    [string]$destination,
    [string]$template
)

$content = [IO.File]::ReadAllText($source)
$bytes = [System.Text.Encoding]::UTF8.GetBytes($content)
$encodedText =[Convert]::ToBase64String($bytes)

$size = (Get-Item $source).length

$templateContent = [IO.File]::ReadAllText($template)

$destinationContent = $templateContent -replace '\${{size}}', $size
$destinationContent = $destinationContent -replace '\${{blob}}', $encodedText

[IO.File]::WriteAllText($destination, $destinationContent)

All additional steps that you need is to create a template, with Blob and Size values replaced with ${{blob}} and ${{size}} and run this script before Unicorn sync in your CI/CD pipeline.

Conclusion

Adding binary files to the source control is a bad practice. And your serialized SXA theme files are binary files (because of the presence of base64 string). Removing these files from source control and improvement of your CI/CD pipeline will save a lot of time and the mental health of your developers. And they will be grateful for this improvement.

Monday, May 10, 2021

Sitecore TinyMCE: New Life

It is a copy of my article initially posted here to keep all things in one place.

Sitecore uses Telerik Rich Text Editor for a long time. It is quite a powerful and reliable editor. But sometimes it is really hard to extend it if you want to do something complex. And also, there are a lot of free open-source alternatives that could be better. And it is not a new idea, people from time to time ask, how to replace Sitecore RTE with something else.

One of the options is TinyMCE. This advanced rich text editor has different pricing plans. But even the free plan has too many features. It can cover 95% of editing cases.

And the idea to integrate Sitecore with TinyMCE also is not new. After a quick search, we are able to find:

Both modules are open-source and have Github repositories(1, 2). We are more interested in the extension of Content Editor that is why I took the Emanuele module as a basis. Then I started to work on improvements:

  1. There was added Unicorn serialization. It allows developers to work easier with Sitecore content items that are required for this module.
  2. The way, how TinyMCE is included in the project was changed. Instead of adding static frontend files, now Tiny MCE is installed as npm module.
  3. Version of TinyMCE was updated to 5.6.1
  4. Continuous integration and continuous delivery were configured using AppVeyor. Now, each commit to GitHub triggers AppVeyor build and provides Sitecore .update package as an artifact. This package could be installed on any Sitecore CM instance. If you are interested in how it was done, I described it in detail in my blog for another Sitecore module.
  5. SonarCloud was configured to ensure code quality. And, yeah, there are still many things that could be improved in code. Contributions are welcome!
  6. Added one more Sitecore field type. Now you should not choose between classic Sitecore RTE and TinyMCE. Now you have both at once. Rich Text editor has buttons to open Sitecore RTE as well as TinyMCE RTE.
  7. Added Sitecore 10 support.
  8. Changed the way how references are used. Locally copied assemblies were replaced with Nuget packages.
  9. Added style formats support.
  10. Improved local development experience: added publishing profiles and proper development setup description.
  11. Many other small improvements.

I hope all these changes will bring this module to a new life. But there is still room for other improvements. Contributions to the Github repository are welcomed. If you find any bugs - report them. If you fixed bugs or added new functionality - create pull requests.