Continuous integration and deployment of Sitecore Habitat/Helix as an Azure Web App

Hello happy people, don’t we all just love Habitat/Helix. If you don’t know what I’m talking about(shame on you), please read about it here:
Sitecore Helix Documentation
Sitecore Habitat – good demo with code and all
Do a Helix Technical Workshop at Sitecore

Sitecore is now available as an Azure web app. This is nothing new but people are finally seeing it as the default option, which is great 😄

Rob Habraken and Bas Lijten have some really good posts regarding Sitecore Deployments on Azure:
– Blue Green Sitecore Deployments on Azure (Rob Habraken)
– Sitecore 8.2-1 on Azure Web Apps (Rob Habraken)
– Zero downtime deployments with Sitecore on Azure (Bas Lijten)
– Use the Sitecore Azure toolkit to deploy your on premises environment (Bas Lijten)

Ok lets move on… In this post I would like to share with you guys how to setup a Habitat/Helix solution as an Azure web app, where unicorn will be our deliverer/savior 😉
Let me explain:
We will setup a clean Sitecore website in Azure.
Next will be to change the ci(continuous integration) gulp file, for copying and transforming files to a deploy folder.
Instead of using (item) packages we will use Unicorn as our “item deliverer”.
It will all be deployed to the azure web app instance with good old ftp(this is not the best approach but I just want to show you how easy it could be)
Everything will run in TeamCity or Visual Studio Team Services.

As always let’s divide our work into following steps:
1. Setup a clean Sitecore instance in Azure – Azure web app.
2. Setup folders and config files to transform on.
3. Prepare and add transformations for the continuous integration script.
4. Update the continuous integration script file – gulpfile-ci.js.
5. TeamCity – Add build steps.
6. Visual Studio Team Services – Add build tasks.

1. Setup a clean Sitecore instance in Azure – Azure web app

There are a lot of good posts on how you to setup Sitecore as an Azure web app.
Sitecore Azure Toolkit
Deploy a new Sitecore environment to Azure App Service
And really good posts from the Sitecore community:
An introduction to Sitecore on Azure Web Apps
Sitecore 8.2 update 1: Azure deployments, ARM, Web Deploy and the Sitecore Azure Toolkit
Sitecore on Azure: Create custom web deploy packages using the Sitecore Azure Toolkit

Let the posts guide you 🙂

2. Setup folders and config files to transform on.

Instead of publishing and transforming files to the website folder, we will do that in a Deploy folder(Which will be placed in the root of the solution).
We will also have a folder for configs to copy from – ConfigsToBuildFrom(It will be placed in the root of the solution), the folder will contain the following files:

• Website
  • Web.config
  • App_Config
    • Include
      • DataFolderUnicornMaster.config
      • UnicornData.config
  • Security
    • Domains.config
• WebsiteCD
  • Web.config

Here we place the web.config’s (from the Azure CM instance and the Azure CD instance), some unicorn configs and the Domain.config.
UnicornData.config sets the sourceFolder:

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <sc.variable name="sourceFolder" value="$(dataFolder)\unicorn" />
  </sitecore>
</configuration>

DataFolderUnicornMaster.config holds the path to the source of the serlialization files

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
  <sitecore>
    <unicorn>
      <defaults>
        <targetDataStore physicalRootPath="$(dataFolder)\Unicorn\$(configurationName)" useDataCache="false" type="Rainbow.Storage.SerializationFileSystemDataStore, Rainbow" singleInstance="true" xdt:Transform="Replace" xdt:Locator="Match(type)"/>
      </defaults>
    </unicorn>
  </sitecore>
</configuration>

3. Prepare and add transformations for the continuous integration gulp script.

Next will be to change(and add) transforming files. Instead of one transforming “type” we will have one per role. Here we have updated/changed the domain.config.transform file(from the habitat project) and the web.config.transform file(from the common project):
domain.config.local.transform
domain.config.cm.transform
domain.config.cd.transform

web.config.local.transform
web.config.cm.transform
web.config.cd.transform

Then in the gulpfile-ci.js(continuous integration script) we will have one transforming task(method) per build “role”.

We also need to transform some of the config patch files. In Foundation.Indexing we need to change the Foundation.Indexing.config file(right now its using the lucene index configuration). We will create two new transforms:
– Foundation.Indexing.config.cm.transform
– Foundation.Indexing.config.cd.transform.
They will replace the lucene index configuration with the cloud index configuration:

<?xml version="1.0" encoding="utf-8"?>

<configuration  xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
  <sitecore>
    <contentSearch>
      <indexConfigurations>
        <defaultLuceneIndexConfiguration>
          <add xdt:Transform="Remove" />
        </defaultLuceneIndexConfiguration>
        <defaultCloudIndexConfiguration type="Sitecore.ContentSearch.Azure.CloudIndexConfiguration, Sitecore.ContentSearch.Azure" xdt:Transform="InsertIfMissing">
          <fieldMap type="Sitecore.ContentSearch.Azure.FieldMaps.CloudFieldMap, Sitecore.ContentSearch.Azure" >
            <fieldNames hint="raw:AddFieldByFieldName">
              <field fieldName="all_templates" storageType="YES" indexType="TOKENIZED" vectorType="NO" boost="1f" type="System.Collections.Generic.List`1[[System.String, mscorlib]]" settingType="Sitecore.ContentSearch.Azure.CloudSearchFieldConfiguration, Sitecore.ContentSearch.Azure" >
                <Analyzer type="Sitecore.ContentSearch.LuceneProvider.Analyzers.LowerCaseKeywordAnalyzer, Sitecore.ContentSearch.LuceneProvider" />
              </field>
              <field fieldName="has_presentation" storageType="YES" indexType="TOKENIZED" vectorType="NO" boost="1f" type="System.Boolean" settingType="Sitecore.ContentSearch.Azure.CloudSearchFieldConfiguration, Sitecore.ContentSearch.Azure"  />
              <field fieldName="has_search_result_formatter" storageType="YES" indexType="TOKENIZED" vectorType="NO" boost="1f" type="System.Boolean" settingType="Sitecore.ContentSearch.Azure.CloudSearchFieldConfiguration, Sitecore.ContentSearch.Azure"  />
              <field fieldName="search_result_formatter" storageType="YES" indexType="UNTOKENIZED" vectorType="NO" type="System.String" settingType="Sitecore.ContentSearch.Azure.CloudSearchFieldConfiguration, Sitecore.ContentSearch.Azure" />
            </fieldNames>
          </fieldMap>
          <virtualFields type="Sitecore.ContentSearch.VirtualFieldProcessorMap, Sitecore.ContentSearch">
            <processors hint="raw:AddFromConfiguration">
              <add fieldName="content_type" type="Alite.Foundation.Indexing.Infrastructure.Fields.SearchResultFormatterComputedField, Alite.Foundation.Indexing"/>
            </processors>
          </virtualFields>
          <documentOptions type="Sitecore.ContentSearch.Azure.CloudSearchDocumentBuilderOptions,Sitecore.ContentSearch.Azure" >
            <fields hint="raw:AddComputedIndexField">
              <field fieldName="has_presentation" storageType="no" indexType="untokenized">Sitecore.Foundation.Indexing.Infrastructure.Fields.HasPresentationComputedField, Sitecore.Foundation.Indexing</field>
              <field fieldName="all_templates" storageType="no" indexType="untokenized">Sitecore.Foundation.Indexing.Infrastructure.Fields.AllTemplatesComputedField, Sitecore.Foundation.Indexing</field>
              <field fieldName="has_search_result_formatter" storageType="no" indexType="untokenized">Sitecore.Foundation.Indexing.Infrastructure.Fields.HasSearchResultFormatterComputedField, Sitecore.Foundation.Indexing</field>
              <field fieldName="search_result_formatter" storageType="no" indexType="untokenized">Sitecore.Foundation.Indexing.Infrastructure.Fields.SearchResultFormatterComputedField, Sitecore.Foundation.Indexing</field>
            </fields>
          </documentOptions>
        </defaultCloudIndexConfiguration>
      </indexConfigurations>
    </contentSearch>
  </sitecore>
</configuration>

In Foundation.LocalDatasource we need to change the Foundation.LocalDatasource.config file(right now its using the lucene index configuration).
We will create two new transforms:
– Foundation.LocalDatasource.config.cm.transform
– Foundation.LocalDatasource.config.cd.transform.
They will replace the lucene index configuration with the cloud index configuration:

<?xml version="1.0" encoding="utf-8"?>

<configuration  xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
  <sitecore>
    <contentSearch>
      <indexConfigurations>
        <defaultLuceneIndexConfiguration xdt:Transform="Remove" />
        <defaultCloudIndexConfiguration type="Sitecore.ContentSearch.Azure.CloudIndexConfiguration, Sitecore.ContentSearch.Azure" xdt:Transform="InsertIfMissing">
          <documentOptions>
            <fields hint="raw:AddComputedIndexField">
              <field fieldName="local_datasource_content" storageType="NO" indexType="TOKENIZED">Sitecore.Foundation.LocalDatasource.Infrastructure.Indexing.LocalDatasourceContentField, Sitecore.Foundation.LocalDatasource</field>
            </fields>
          </documentOptions>
        </defaultCloudIndexConfiguration>
      </indexConfigurations>
    </contentSearch>
  </sitecore>
</configuration>

In Common.Website we need to change the Common.Website.config file.
We will create two new transforms:
– Common.Website.config.cm.transform
– Common.Website.config.cd.transform
They will replace the rootHostName, Analytics.CookieDomain and (if you want) the Analytics.ClusterName:

<?xml version="1.0" encoding="utf-8"?>

<configuration  xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
    <sitecore>
      <sc.variable name="rootHostName" value="yourdomain" xdt:Transform="SetAttributes(value)" xdt:Locator="Match(name)"/>
      <settings>
        <setting name="Analytics.CookieDomain" value="yourdomain" xdt:Transform="SetAttributes(value)" xdt:Locator="Match(name)"/>
        <setting name="Analytics.ClusterName" value="yourdomain" xdt:Transform="SetAttributes(value)" xdt:Locator="Match(name)"/>
      </settings>
    </sitecore>
</configuration>

Finally in Habitat.Website we need to change the Habitat.Website.config file.
We will create two new transforms:
– Habitat.Website.config.cm.transform
– Habitat.Website.config.cd.transform

In Habitat.Website.config.cm.transform we will just change the target host name:

<?xml version="1.0" encoding="utf-8"?>

<configuration  xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
  <sitecore>
    <sites>
      <site name="habitat" targetHostName="yourSiteName.$(rootHostName)" xdt:Transform="SetAttributes(targetHostName)" xdt:Locator="Match(name)"/>
    </sites>
  </sitecore>
</configuration>

And in Habitat.Website.config.cd.transform we will follow jammykam’s advice from his great post – Disable Edit and Preview Modes on your CD servers:

<?xml version="1.0" encoding="utf-8"?>

<configuration  xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
  <sitecore>
    <sites>
      <site name="habitat" targetHostName="yourSiteName.$(rootHostName)" enablePreview="false" enableWebEdit="false" enableDebugger="false" xdt:Transform="SetAttributes(targetHostName,enablePreview,enableWebEdit,enableDebugger)" xdt:Locator="Match(name)" />
    </sites>
  </sitecore>
</configuration>

Instead of transforming you can go for Patch Override, there is a great post from jammykam – Helix: Patching your Config Patches for Specific Environments

4. Update the continuous integration script file – gulpfile-ci.js

So now we have setup the config and transformation files. Next will be to update our gulpfile-ci.js file, which will be used in TeamCity or Visual Studio Team Services.

We will add some new variables to the script, by pointing out paths to the new folders. (By default the “data folder” is the App_Data folder in a Sitecore Azure web app)

var webTransformationsToBuildFrom = path.resolve("./ConfigsToBuildFrom/Website");
var webCDTransformationsToBuildFrom = path.resolve("./ConfigsToBuildFrom/WebsiteCD");
var dataFolder = path.resolve("./Deploy/Data/App_Data");
var tempWebsite = path.resolve("./Deploy/Website");
var tempWebsiteCD = path.resolve("./Deploy/WebsiteCD");
config.websiteRoot = tempWebsite;

Here is the main task – CI-And-Prepare-Files-CM-CD:

gulp.task("CI-And-Prepare-Files-CM-CD", function (callback) {
    runSequence(
        "CI-Clean",
        "CI-Copy-Configs-CM",
        "CI-Publish",
        "CI-Copy-Website-CD",
        "CI-Copy-Configs-CD",
        "CI-Prepare-Files-CM",
        "CI-Prepare-Files-CD",
        "CI-Apply-Xml-Transform-CM",
        "CI-Apply-Xml-Transform-CD",
        "CI-Copy-Items-For-Unicorn",
        callback);
});

Let’s go trough the different tasks…
In CI-Clean we just clean the deploy folder.

gulp.task("CI-Clean", function (callback) {
    rimrafDir.sync(path.resolve("./Deploy"));
    callback();
});

CI-Copy-Configs-CM, will copy files from the ConfigsToBuildFrom folder to the Deploy folder.

gulp.task("CI-Copy-Configs-CM", function () {
    return gulp.src(webTransformationsToBuildFrom + "/**")
        .pipe(gulp.dest(tempWebsite));
});

CI-Publish, will publish all your projects to the Deploy/Website folder.

gulp.task("CI-Publish", function (callback) {
    config.buildConfiguration = "Release";
    runSequence(
        "Nuget-Restore",
        "Build-Solution",
        "Publish-Foundation-Projects",
        "Publish-Feature-Projects",
        "Publish-Project-Projects", callback);
});

CI-Copy-Website-CD, will copy all the files from Deploy/Website to Deploy/WebsiteCD.

gulp.task("CI-Copy-Website-CD", function () {
    return gulp.src(tempWebsite + "/**")
        .pipe(gulp.dest(tempWebsiteCD));
});

CI-Prepare-Files-CM, will go through and remove unwanted files and dll’s for our CM instance. I got the idea from this great post – Sitecore Habitat Deployment.

gulp.task("CI-Prepare-Files-CM", function (callback) {
    var excludeList = [
        tempWebsite + "\\bin\\{Sitecore,Lucene,Newtonsoft,System,Microsoft.Web.Infrastructure}*dll",
        tempWebsite + "\\bin\\*.pdb",
        tempWebsite + "\\compilerconfig.json.defaults",
        tempWebsite + "\\packages.config",
        tempWebsite + "\\App_Config\\Include\\{Feature,Foundation,Project}\\z.*DevSettings.config",
        tempWebsite + "\\App_Data\\*",
        "!" + tempWebsite + "\\bin\\Sitecore.Support*dll",
        "!" + tempWebsite + "\\bin\\Sitecore.{Feature,Foundation,Project}*dll",
        tempWebsite + "\\bin\\{Sitecore.Foundation.Installer}*",
        tempWebsite + "\\App_Config\\Include\\Foundation\\Foundation.Installer.config",
        tempWebsite + "\\README.md",
        tempWebsite + "\\bin\\HtmlAgilityPack*dll",
        tempWebsite + "\\bin\\ICSharpCode.SharpZipLib*dll",
        tempWebsite + "\\bin\\Microsoft.Extensions.DependencyInjection*dll",
        tempWebsite + "\\bin\\MongoDB.Driver*dll",
        tempWebsite + "\\bin\\Microsoft.Web.XmlTransform*dll"
    ];
    console.log(excludeList);

    return gulp.src(excludeList, { read: false }).pipe(rimraf({ force: true }));
});

Notice that we are not removing the serialization.config’s from the CM instance, we will use them for some Unicorn magic later 🙂

CI-Prepare-Files-CD, will go through and remove unwanted files and dll’s for our CD instance. I got the idea from this great post – Sitecore Habitat Deployment.

gulp.task("CI-Prepare-Files-CD", function (callback) {
   var excludeList = [
        tempWebsiteCD + "\\bin\\{Sitecore,Lucene,Newtonsoft,System,Microsoft.Web.Infrastructure}*dll",
        tempWebsiteCD + "\\bin\\*.pdb",
        tempWebsiteCD + "\\compilerconfig.json.defaults",
        tempWebsiteCD + "\\packages.config",
        tempWebsiteCD + "\\App_Config\\Include\\{Feature,Foundation,Project}\\z.*DevSettings.config",
        tempWebsiteCD + "\\App_Data\\*",
        "!" + tempWebsiteCD + "\\bin\\Sitecore.Support*dll",
        "!" + tempWebsiteCD + "\\bin\\Sitecore.{Feature,Foundation,Project}*dll",
        tempWebsiteCD + "\\bin\\{Sitecore.Foundation.Installer}*",
        tempWebsiteCD + "\\App_Config\\Include\\Rainbow",
        tempWebsiteCD + "\\App_Config\\Include\\Unicorn",
        tempWebsiteCD + "\\App_Config\\Include\\Rainbow*.config",
        tempWebsiteCD + "\\App_Config\\Include\\Unicorn*.config",
        tempWebsiteCD + "\\App_Config\\Include\\Foundation\\*.Serialization.config",
        tempWebsiteCD + "\\App_Config\\Include\\Feature\\*.Serialization.config",
        tempWebsiteCD + "\\App_Config\\Include\\Project\\*.Serialization.config",
        tempWebsiteCD + "\\App_Config\\Include\\DataFolderUnicornMaster.config",
        tempWebsiteCD + "\\App_Config\\Include\\Foundation\\Foundation.Installer.config",
        tempWebsiteCD + "\\README.md",
        tempWebsiteCD + "\\bin\\HtmlAgilityPack*dll",
        tempWebsiteCD + "\\bin\\ICSharpCode.SharpZipLib*dll",
        tempWebsiteCD + "\\bin\\Microsoft.Extensions.DependencyInjection*dll",
        tempWebsiteCD + "\\bin\\MongoDB.Driver*dll",
        tempWebsiteCD + "\\bin\\Microsoft.Web.XmlTransform*dll",
        tempWebsiteCD + "\\bin\\Rainbow*dll",
        tempWebsiteCD + "\\bin\\Unicorn*dll",
        tempWebsiteCD + "\\bin\\Kamsar.WebConsole*dll"
    ];
    console.log(excludeList);

    return gulp.src(excludeList, { read: false }).pipe(rimraf({ force: true }));
});

Here we will remove everything that has to do with Unicorn.

CI-Apply-Xml-Transform-CM, will run all the CM transformations.

gulp.task("CI-Apply-Xml-Transform-CM", function () {
     var layerPathFilters = ["./src/Foundation/**/*.CM.transform", "./src/Feature/**/*.CM.transform", "./src/Project/**/*.CM.transform", "!./src/**/obj/**/*.CM.transform", "!./src/**/bin/**/*.CM.transform"];
    return gulp.src(layerPathFilters)
        .pipe(foreach(function (stream, file) {

            var fileToTransform = file.path.replace(/.+code\\(.+)\.CM.transform/, "$1");
            util.log("Applying configuration transform: " + file.path);
            return gulp.src("./scripts/applytransform.targets")
                .pipe(msbuild({
                    targets: ["ApplyTransform"],
                    configuration: "Release",
                    logCommand: false,
                    verbosity: "minimal",
                    stdout: true,
                    errorOnFail: true,
                    maxcpucount: 0,
                    toolsVersion: config.buildToolsVersion,
                    properties: {
                        Platform: config.buildPlatform,
                        WebConfigToTransform: tempWebsite,
                        TransformFile: file.path,
                        FileToTransform: fileToTransform
                    }
                }));
        }));
});

CI-Apply-Xml-Transform-CD, will run all the CD transformations.

gulp.task("CI-Apply-Xml-Transform-CD", function () {
     var layerPathFilters = ["./src/Foundation/**/*.CD.transform", "./src/Feature/**/*.CD.transform", "./src/Project/**/*.CD.transform", "!./src/**/obj/**/*.CD.transform", "!./src/**/bin/**/*.CD.transform"];
    return gulp.src(layerPathFilters)
        .pipe(foreach(function (stream, file) {

            var fileToTransform = file.path.replace(/.+code\\(.+)\.CD.transform/, "$1");
            util.log("Applying configuration transform: " + file.path);
            return gulp.src("./scripts/applytransform.targets")
                .pipe(msbuild({
                    targets: ["ApplyTransform"],
                    configuration: "Release",
                    logCommand: false,
                    verbosity: "minimal",
                    stdout: true,
                    errorOnFail: true,
                    maxcpucount: 0,
                    toolsVersion: config.buildToolsVersion,
                    properties: {
                        Platform: config.buildPlatform,
                        WebConfigToTransform: tempWebsiteCD,
                        TransformFile: file.path,
                        FileToTransform: fileToTransform
                    }
                }));
        }));
});

CI-Copy-Items-For-Unicorn, will grab all the serialized items(yml files) and put them in following path – Deploy/Data/App_Data/Unicorn(for our CM instance).

gulp.task("CI-Copy-Items-For-Unicorn", function () {
    return gulp.src("./src/**/serialization/**/*.yml")
        .pipe(gulp.dest(dataFolder + "/Unicorn/"));
});

CI-Sync-Unicorn-CM. will be called from TeamCity or Visual Studio Team Services to sync the Sitecore items.

gulp.task("CI-Sync-Unicorn-CM", function (callback) {
    var options = {};
    options.siteHostName = habitat.getStageUrl();
    options.authenticationConfigFile = config.websiteRoot + "/App_config/Include/Unicorn.SharedSecret.config";

    unicorn(function () { return callback() }, options);
});

When task CI-And-Prepare-Files-CM-CD is done, we should have a Deploy folder ready (for deployment):

To make it even better, next step could be to make a custom web deploy package (using the Sitecore Azure Toolkit).
Check out Sitecore on Azure: Create custom web deploy packages using the Sitecore Azure Toolkit from Bas Lijten

5. TeamCity – Add build steps

The next move is to automate our deployment, I was thinking of using TeamCity and Visual Studio Team Services.

Let’s start with TeamCity. I will not explain on how to setup TeamCity, but it’s quite straight forward. You can read all about it here – Installing and Configuring the TeamCity Server

I added some plugins in order to run the xunit tests and execute the gulp tasks:
– xUnit-TeamCity
– TeamCity.Node (gulp)

Next we will create the build steps:
1. Nuget Restore
NuGet Installer
Solution: MySolution.sln

2. Build solution
Visual Studio (sln)
Build file path: MySolution.sln
Targets: Rebuild
Configuration: Release

3. Run all tests
xUnit
Version 2.1.0 (AnyCPU/MSIL/.NET 4.5)
Included assemblies: **\bin\debug*Test*.dll

4. npm install
Command Line
Custom script: npm install

5. Generate deploy files for Azure
Gulp
Run targets: CI-And-Prepare-Files-For-CM-CD

6. Upload files To Azure CM
FTP Upload
Target FTP server: ftp://xxxx.ftp.azurewebsites.windows.net/site/wwwroot/
Source folder: %teamcity.build.workingDir%/Deploy/Website/**

7. Upload files To Azure CD
FTP Upload
Target FTP server: ftp://xxxx.ftp.azurewebsites.windows.net/site/wwwroot/
Source folder: %teamcity.build.workingDir%/Deploy/WebsiteCD/**

8. Copy Unicorn files
FTP Upload
Target FTP server: ftp://xxxx.ftp.azurewebsites.windows.net/site/wwwroot/
Source folder: %teamcity.build.workingDir%/Deploy/Data/**

9. Sync items with Unicorn
Gulp
Run targets: CI-Sync-Unicorn-CM

In step 8 and 9 that’s where all the magic happens 🙂
We will upload all the serialized Sitecore items and thanks to the serialization.config’s(which we didn’t remove) we can now call task CI-Sync-Unicorn-CM and sync all the serialized sitecore items to our Azure CM instance – love it !

6. Visual Studio Team Services – Add tasks

Visual Studio Team Services is a great service and it’s easy to get started with. The best of it all – it’s all in the cloud. Read all about it here – I want to to do it now!!

Here are our created build tasks:

Finally we need to setup a Trigger, we don’t want to start the build manually. In this case we will listen to the master branch from our Bitbucket repository:

That’s it guys, I must say that I fell in love with Visual Studio Team Services – Put it all in the cloud !!

That’s all for now folks 🙂