Sitecore 8 Content Testing

Anything with the codename Skynet immediately gets my attention so I dug into Sitecore’s Content Testing. Essentially Content Testing can learn from AB and Multivariate tests to suggest changes and create tests which can improve conversion rates, engagement values, site usage. So lets create some robotic Overlords!

What is Content Testing?

The principles of Sitecore’s Content Testing is to test everything, remove assumptions, learn what produces benefits, have fun while doing it then get Sitecore to do all the work for you. Tests can completed at three levels;

1. Page Level Testing – Tests two or more pages, or versions of pages, against one another for the best outcome
2. Component Testing – Measures value provided by differing components
3. Content Testing – Measures value provided by differences in text, imagery, video etc.

Testing one of these ‘variables’ against another one is called an AB Test. For example, Promo Panel 1 and Promo Panel 2 will be added to different versions of the same page and Sitecore will measure the value generated by each over the duration of the test.

A Multivariate test will consist of n number of variables; layout, content, components, personalization. For each combination of the variables Sitecore produces an Experience. Sitecore then tests each Experience and evaluate the value it generates to recommend which change the User should implement.

In this post I’ll cover what makes up a Content Test, the tools to manage them, where the data is stored and of course how to create a Content Test and use its outcome.

Firstly, a Sitecore Test consists of the following attributes

• Variables – the Items, Components, Content, Personalization, Component tests which are being varied
• Experience – a combination of the variables, ultimately what is being tested
• Expected Effect – the test creator’s opinion if the test will produce a negative, positive or no change to the engagement value produced by the experience
• Traffic Allocation – a percentage of the visitors that will be exposed to the test
• Confidence Level – the statistical confidence percentage required before a result has been obtained
• Test Objective – a measure in which the test is evaluated
• End Test Options – the conditions when a test can end; automated or manual
• Duration – the minimum time a test can run before a relevant result is obtained and the maximum time a test can run

There are two methods to create a Test.

Sitecore Experience Optimization

The Experience Optimization app has been created to manage all tests and their outcomes in one tool fully integrated into Sitecore, written in SPEAK and can be found on the Launch Pad.

From the Experience Optimization Tool you can view Tests in all states; Draft, Active and Completed and once a number of tests have been completed Sitecore will have learned enough to begin displaying Suggested Tests. These Suggested Tests are what Sitecore has determined would be good changes to better achieve the goals, campaigns etc.

The stats of the gamification feature can also be accessed to see which User suggests changes that produce the highest effect, who accurately predicts results of tests and so on.

Most importantly, the Experience Optimization tools is where you create Page Tests. Page Tests in Sitecore is a method of creating AB tests for entire pages however you are not limited to only two pages. You can include n number existing pages, previous versions of pages or create new page versions to run tests against. You can edit the Pages from the the Experience Optimization tool before you begin the test.

Sitecore Workflow Content Testing

The other method to create Tests in Sitecore is via Workflow. These differ from Tests created in the Experience Optimization Tool; opposed to testing pages and against each other you test variables on the Item you are pushing through workflow. These variables are components, content, personalization, layouts and even AB Tests on components which make up a Multivariate Test.

To create a Test via Workflow requires unique Workflow Commands to present the Test interface to the User and submit the test. Once a test has been created it will be present in the Experience Optimization Tool to track.

I’ve recently posted how to add Content Testing to existing workflow.

Location of Tests in Sitecore

When a Test is created an Item called a Test Definition Item is created to contain the attributes and is stored under the Test Lab in the Marketing Control Panel.

This Test Definition Item is then linked to the Item in the test via the ‘Page level test set’ field within the Layout node of that Item.

A key consideration that Developers need to take note of. If the test includes more than one version of an Item, multiple versions of an Item will be present in the Web database. Until now only one version of an Item was ever present in the Web database.

Creating an Content Test in Sitecore

I’ll use an example to help explain how to create a Multivariate Test through Workflow. I chose Multivariate over Page Test as Page Tests follows the same process but simplified.

The first step is to create the variables. Add a new version of an Item by locking and then make your changes to the content, layout, components, personalization, component tests etc via the Experience Editor. After that it can be pushed through Workflow and begin creating the Multivariate Test. The interface will list out all variables that will be varied in the test based on the changes made to the new version of the Item.

When creating AB Tests via the Experience Optimization tool you can include more pages to test against at this step, including previous item versions, new item versions and other existing items. Once chosen you can further edit the pages to Test against by clicking the Pencil Icon on each Page bringing up the Experience Editor. If prefer you can make the changes to the version in the Content Editor before accessing the Experience Optimization tool.

The next step of configuring is the gameification where you can state if you believe the outcome of your changes will have a positive or negative effect on Engagement Value. Not only is this used for the gameification Leaderboards but Sitecore will learn which Users are suggest changes which have a negative effect on Engagement Value and react, for example showing the test to less visitors.

Multivariate Tests will display all Experiences which will be tested and the variables that created them. At this point you can choose to Enable or Disable a variable from the test if you so wish. From my example eight Experiences will be tested.

Sitecore will use historical data to inform how long it should take to determine which Experience is the winner.

The Traffic Allocation slider sets the percentage of visitors that will experience the Test when they access the applicable pages. Higher the allocation the bigger the sample size and in theory, the faster you will achieve a result. You may opt for a lower allocation if the change is risky, therefore only a small subset of the visitors will see it.

The Confidence level list defines at what statistical confidence percentage needs to be achieved before deciding an Experience has come out as a winner. The higher the percentage the more certain result needs to be which could take significant time to achieve if at all.

The Test Objective is used to measure how successful each Experience is. Therefore the Objective needs to be something that can be affected by users going through the Test Experience. Perhaps the Test affects users journey through the sales funnel ultimately leading to an order or maybe the test changes the content and layout of the account creation page. The former’s Test Objective would be the default Trailing Value per Visit and the latter’s would be Registrations.

This section also lets you define how a winner is a chosen; manually selected by a User or automatically by Sitecore.

The final setting of the Page Test is the duration measured in days. It is best practice to choose a reasonable minimum amount of time the test needs to run before a winner can be selected to ensure the sample size is large enough to produce representative results. A definitive maximum duration is required to prevent the test from running indefinitely, this is especially important when using the automated options for selecting a winning experience.

And that’s it.. for now!

Before this post becomes any longer I’ll split off into another future post concerning how to analyse your Test results and how to use Suggested Tests to do all the work for you.