All websites are built to achieve a goal and A/B testing is an extremely important tool to make sure your website maximizes its goal. Setting up an A/B test, especially in the Jamstack, has traditionally required a developer who would have to manually set up and wire multiple tools for each test. Stackbit’s Studio empowers anyone to set up, run and analyze multiple A/B tests easily in a few minutes.
A note before we dig in, the Stackbit Studio itself does not provide a direct way to view the results of your A/B test. While you can set up and make changes to the variants from the Studio, you will need to determine how you will measure the results. Within this doc, we will show how to leverage Google Analytics' Custom Definitions to measure results. If you have other measurement tools you would like to integrate please let us know.
You Will Need:
- A site using the Stackbit Studio
- Access to the A/B testing feature
- A Google Analytics account (optional)
Accessing A/B Testing
This feature is only available to users at the Business and Enterprise plan levels. To gain access to try A/B testing, you can start a free 7-day trial by selecting the Business plan from your account dashboard.
TODO: Screenshot of the Plans menu from the dashboard.
Identifying Your Hypothesis
Once you have access to A/B testing, the next step is determining what you want to test. This means creating a clear hypothesis. There are plenty of resources available online that dive deeper into this process. A good hypothesis will directly tie the Variant modifications to a measurable change in user behavior. Here are a couple examples of good ones:
Example Hypothesis 1: “Changing the Call to Action button text that appears above the fold to ‘Buy Now’ instead of ‘See All Items’ will result in more clicks of the CTA button over the same 7 days time period”
Example Hypothesis 2: “Given that most users navigate the site with the top nav links per heat mapping data, moving the Newsletter signup link to the top menu will result in higher newsletter signups over the same 5 day time period”
In our examples, we are testing one hypothesis for our site at a time. Testing more than one change at a time on the same page can add complexity, making it more difficult to validate an hypothesis.
For this example, we will use Example Hypothesis 1, as stated above and modify the Call to Action
Naming Your Variants And Split %
Now that you have your hypothesis, you can go ahead and set up your Variants. Under the hood Stackbit sets up the needed Github branches, leverages the deployment platform's routing, creates a separate CMS environment and generates a preview for each of these sets.
In the upper right hand side of the screen, click on the A/B Testing menu and you will see Variants and Split %. By default this is populated with “Baseline: 50%” and “Copy Baseline: 50%”
Change the name of these Variants to reflect the hypothesis. In this example we are using the long variable names: “Baseline-CTA-see-all-items” and “Variant-CTA-buy-now” for clarity.
You will also set what percentage of traffic will be shown each Variant. When deployed the traffic will be balanced per the set “Split %”. By default we pre-set these to be a 50/50 split. For our example, we want to show our Variant equally, so we will leave it as the default 50/50.
Once set, hit ‘Save Changes’.
Saving changes can take a couple minutes as it provisions all the new environments in the various tools of your setup (repository, deployment, CMS).
Editing Your Site Variants
Now that Variants are defined, Stackbit will provision a Variant to make changes against and deploy. The new Variants will appear in the upper left hand corner beside your site name. While you can make changes to both, it is best practice to leave the Baseline as is.
The control to select Variants will appear next to the project name on the top left, allowing easy switching between them.
Switching to the new Variant, you will see an identical copy of the site, which you can modify like any other page using the Stackbit Studio. On this Variant, for this example, we have modified the CTA to the Variant text “Buy Now”
Modifying Variants To Show Different Behavior
In this example we only modified the call to action text for simplicity sake. You can modify any element of the page to track different behavior.
Here are a few examples:
- Manually tagging the conversion action by using a UTM parameter of "baseline" vs. "variant".
- Modifying the Variant links with different
ref=parameters and see which was followed more.
- Changing the Variant link to go to a different or new landing page.
- Modifying which document a user downloads from the Variant.
There are no limits to what you can modify in your site Variants. Just keep your hypothesis in mind and your plan on how to measure the test results.
Deploying The A/B Test
If you are planning to integrate with Google Analytics, you need to supply the Google Tracking ID and Dimension Index before proceeding. Please see the directions for setting up Google Analytics further down this page before starting your A/B test.
Once everything looks good, you can deploy the test by hitting the “Start A/B Test” button under the “A/B Testing” menu. This will publish the Variants and begin splitting the traffic.
TODO REPLACE SCREENSHOT WITH GA FILLED IN
Concluding The Test
Once you have run the test for the predetermined period of time and measured the results, you should conclude the test by selecting “Conclude Test” from the A/B Testing menu.
Concluding the test means declaring which Variant you wish to publish as your main site and use it on an ongoing basis.
Once you hit Publish, the selected Variant will be published as your main site and the other Variant will be permanently deleted.
Making Changes While Test Is In Progress
We recommend not changing the site while the test is in progress. Sometimes though, changes need to be made.
If it's not going to affect the test, like fixing a typo or a change on a page outside the test, then the change should be made in both Variants. If the change is specifically affecting the test, then one should only look at the results after concluding the test.
Determining Which Version of Google Analytics you are using
In October 2020, Google Analytics rolled out a major change to their Google Analytics platform. As a result, how you should integrate your A/B testing with Stackbit varies quite a bit between versions. We have included both sets of directions below.
One of the simplest ways to determine which version of Google Analytics you are using is to look at your property's left hand menu.
If your menus look like this, you are using Google Analytics 4:
If your menus look like this, you are using Google Analytics 3:
This should be the default for new web properties set up inside Google Analytics. If you are not familiar with this upgrade, we recommend this overview of the rather significant changes in Google Analytics.
Integrating Google Analytics 4 Using Custom Dimensions
Google Analytics is one of the most popular ways to measure website performance and how users are interacting with the site. Stackbit Studio A/B Testing is pre-configured to leverage Google Analytics. This part of the docs will guide you through setup and how to find results in Google Analytics.
If you have not done so yet, you will need to create a new property for the site you want to report on. Here is a handy guide for creating new properties in Google Analytics 4 (GA4)
Now that we have our property created, there are two items needed to connect Stackbit's A/B testing with GA4, one you get from Google Analytics and one you will set up in Google Analytics.
- Measurement ID
- Dimension Index
Getting Your Tracking ID
This is what Google Analytics 4 calls a ‘Measurement ID”. The fastest way to get this ID is to simply ask Google for it from within the Google Analytics dashboard. It will start with a ‘G-’.
Setting Your Dimension Index
What Is A Dimension Index?
In order to determine which Variant of a site’s users are interacting with, you need a unique identifier for each. In Google Tags Manager terms this is called a Custom Dimension. Stackbit will insert a dimension denoted as ‘dimension(index number). For example, if you set the dimension index to be ‘1’, then the inserted dimension for Google Analytics to track will be ‘dimension1’.
Since multiple tools leverage this and you might already be using ‘dimension1’ for something else, the tooling allows you to use any number here to append to the end of ‘dimension’.
Set Up A Custom Dimension In Google Analytics
First, pick a dimension index number in the Stackbit Studio A/B Testing Menu, under “Integrate with Google Analytics”. This can be any number you like.
Next, under your project in Google Analytics, under “Events” click on “All Events”
In the upper right corner click ‘Manage Custom Definitions’. This will open the menu to allow you to create “Custom Definitions”. Next click “Create Custom Definitions and fill in the “Event Parameter Name” as ‘dimension[index number]’. For example, setting index Dimension in Stackbit to 1 would mean I need to use the parameter name ‘dimension1’, as I have done in this example.
Before you hit ‘Save’, notice that the ‘Custom Dimension name’ field automatically populated with the parameter name. You can leave this as is or you can name it to something more meaningful. Here it is changed to be ‘StackbitAB_Test’ to make it easier to find and use in Google Analytics. Now, you can hit Save.
Start the test from the Stackbit Studio.
As is normal for Google Analytics, you will need to wait up to 24 hours to see any traffic utilizing this custom dimension.
Viewing Your Results In Google Analytics
Back in our Google Analytics dashboard, on the left hand menu, select “Engagement” and click to open “Pages and Screens”
Click on the ‘+’ link beside ‘Page title and screen class’ and select the Custom(Event-scoped) category. From here click on your custom dimension, which we created up earlier. Since our example did not change the ‘Custom Dimension name’ field it shows up as ‘dimension1’.
Selecting our custom dimension adds a secondary field to the view and if we have had traffic it will show what Variant was used.
Integrating Google Analytics 3 - Custom Dimensions
Google Analytics is currently on version 4, but many websites are still utilizing the earlier version. This guide is intended for folks who are adding A/B testing to an existing property or creating new properties that will behave as Google Analytics 3.
If you do not have an existing property for your site in Google Analytics, you will need to create a new property for the site you want to report on. By default Google Analytics will make all new properties using Google Analytics 4, unless you follow these directions.
Now that we have our property created, there are two items needed to connect Stackbit's A/B testing with GA4.
- Tracking ID
- Dimension Index
Getting Your Tracking ID
You can find your Google Analytics property’s Tracking ID from the Admin view of the property you are working on. Here we are using "example" to cover the actual Tracking ID, but the format will look something like
In your Stackbit account, click on the A/B Testing dropdown and click on "Integrate Google Analytics". In the text box, input the Tracking ID you received from Google.
Creating A Custom Dimension And Using The Dimension ID In Google Analytics, navigate to Admin -> Create Property, and scroll down and click "Custom Definitions".
Next, click the "Add Custom Dimension" button. A form will appear. Enter the name of your Custom Dimension and the scope. Finally, to submit it, hit "Create".
You will see a few different code Variants for tracking purposes. Click "Done" and you should see your new customization created.
Under “Index” you will find a number that signifies your Custom Dimension Index. Insert that number into the A/B Testing module on Stackbit, and hit "Save changes".
Voila! Now you have your Google Analytics set up to track and analyze the results of your test.
Finding your results on Google Analytics 3
At this point, your test is live and gathering results. To see these results in Google Analytics, go to the "Reports" section of your sidebar, and click "Audience".
Under the drop down select "Overview" then click "Add Segment".
Click on "+NEW SEGMENT". This will bring you to the segment creation area. Since your dashboard segment will be reliant on your custom dimension, you’ll need to head to the "Advanced" segment section and select "Conditions".
Here you will add the options you had set up for your A/B test. Navigate to "Custom Dimensions".
Then enter the conditions you are testing and click Save.
Once you get traffic to your site, you will see the statistics you had set out to measure.
Further Understanding Results In Google Analytics
Now that you know how to find your results, it is time to contemplate what they mean and what they are showing. Now is the time to look at your hypothesis. Did the change result in the expected results? If so, then you can confidently declare which of your Variants is the winner. If not, think about how you might change elements to get the results you want and try again.
There are a lot of resources that dive into A/B testing further than we will here. To read further we would recommend starting with these links: