Types of A/B testing
OpenWrap supports three types of A/B tests:
|Client-side vs Server-side |
Measures the effect of bidder timeouts on monetization. This test allows you to set a different auction timeout on a specified percentage of traffic to see the effect on revenue and latency.
- Short timeouts result in fast page load and slightly improves viewability.
- Longer timeout is better for monetization.
Measures the effect of adding or removing bidders on monetization. This test allows you to quantify the overall effect of adding or removing bidding partners.
- Adding a bidder will usually show some revenue on that bidder, but how much of it is true incremental revenue and how much is just a shift from other bidders?
- The same is true for removing a bidder.
Measures the effect of adding or removing Identity Providers on monetization. This test allows you to quantify:
- How much incremental revenue is obtained from adding a particular ID provider.
- How much revenue is lost when removing an id provider.
When looking at results, keep in mind that gains are much larger for cookieless traffic (Safari and Firefox). Chrome traffic shows a smaller effect, so the percentage gain is larger on the uncookiable share of traffic. Reporting doesn’t currently break this out.
Set up an A/B test
Only one A/B test can be performed at at time. If you want to perform more than one test, create a new profile version for each one.
- Create an OpenWrap profile as you normally would.
- Enable A/B Testing. You can edit the profile version to disable the test at anytime.
- Select the Test Group Size and Test Type.
- Enter the test group criteria. The control and test configurations must be different otherwise the system will not let you save the profile.
- Access the results in the Profile Details page.
Sample results page:
Custom A/B tests
You can use code on the page to run custom tests.
- Set PWT.testGroupId to a number between 0 and 15
- 0 is the control group
- 1-15 are test groups
The results won’t show up in the A/B test enabled in the version list, so you'll need to use Report Builder to get the results.
Frequently asked questions
How long should an A/B Test for?
One week. This will avoid biased results that might occur from day of the week seasonality.
What sampling percentage should I choose?
Best practice is to set the percentage high enough to get at least 100,000 paid impression in the test group. Fewer than 100,000 paid impressions can result in a higher sampling error
Can I run more than one test at time?
Not at this time. This feature currently supports one test per profile per profile version.
Can I test multiple combinations?
No. This feature doesn't support multi-variate testing.