A/B test

Optimize ad monetization by conducting A/B tests on waterfall configurations and ad strategies, adjusting traffic splits, and analyzing performance metrics to determine the most effective setup.

Read time 7 minutes

The A/B test tool enables you to test different ad monetization strategies on the ad unit level. You can split user traffic, add ad units to the test at any time, and remove an individual ad unit without closing the entire test.

You can start an A/B test from the following locations:

  • Mediation management page: test different waterfall settings, including:
    • Waterfall setup
    • Mediation groups
    • New networks
    • Instance pricing - To test your waterfall configuration 
  • Ad units page: create any type of ad strategy test for all of the supported features on the ad unit and placement level: 
    • Ad unit capping and pacing
    • Banner refresh rate 

Start an A/B test
Start an A/B test

The following process details how to start an A/B test from the Mediation management page.

  1. Under Management, select Mediation.
  2. Select the app, and then select the ad unit to run an A/B test on.
  3. In the upper right corner, select Start test.
  4. Enter a name and description for your test.
  5. Decide how to split the traffic between the test group and the control group. By default, the split is set at 50/50. You can decrease or increase it by increments of 5%.
  1. Under Reward, select the edit icon to adjust the type of reward or the number of rewards a user gets for watching an ad.

Modify your waterfall setup
Modify your waterfall setup

On the Mediation Management page, use the toggle to switch between group A (control) and B (test). Adjust the waterfall configuration for group B (test group) according to the variables you want to test. Each change you make will apply only to group B.

Supported waterfall components include:

  • Addition (or subtraction) of instances from the mediation group
  • Group configurations 
  • Delete / add groups
  • Instance rates
  • Instance level capping 

Add an ad unit to a test
Add an ad unit to a test

To add an ad unit to an A/B test, complete the following steps.

  1. Selecting the ad unit name (Rewarded, Interstitial, or Banner) from the dropdown menu in the upper left corner of the page.
  2. Select Add to test.
  3. Adjust the waterfall configuration for your test group (B) according to the variables you want to test.
  4. Select Save, and the ad unit is immediately be added to the test.  
An example configuration for an ad unit in an AB test

Test monitoring and reports
Test monitoring and reports

From the left navigation menu, select A/B to open the A/B Management page, on which you'll see a list of the apps that currently are running an A/B test and apps that have ended an A/B test.

An A/B dashboard is automatically created whenever you start an A/B test in your app. The dashboard displays key metrics for each ad unit test to help you decide which test group you should continue with. You can also choose to end an individual ad unit test or the entire test.

To access the A/B dashboard, complete the following steps:

  1. Select the action button at the end of an app's row
  2. Select View A/B dashboard.

On the A/B dashboard page, the Overview table displays an overall comparison of the performance of groups A and B by using the following metrics:

  • Revenue
  • ARPDAU
  • D0 ARPU
  • D3 ARPU
  • D7 ARPU
  • D1 Retention
  • D3 Retention
  • D7 Retention

You'll also see a direct comparison for each ad unit test that you started in the app through the following metrics:

  • Revenue
  • Impressions 
  • eCPM
  • ARPDAU
  • ARPDEU
  • Impressions per DAU
  • Engagement rate
  • Fill rate 

To further examine the data, select filters, such as date range, countries, and ad sources, from the top of the page. To view the ad units that were included in the test, select View test logs.

An overview of performance metrics for test groups A and B

View additional metrics by selecting View report or Real Time Pivot at the top of the page. 

  • Select View report to see granular monetization performance data per ad unit in your Performance Report.
  • Select User Activity or Cohorts in the left navigation menu to view additional insights. 
  • Select Real Time Pivot to view your A/B test results in real time.
An example performance report for an AB test

Adjust your traffic split during a test
Adjust your traffic split during a test

To adjust your traffic split at any time, complete the following steps:

  1. Go to to the A/B management page.
  2. Select the action button on a test.
  3. Select Adjust traffic split.
  4. Use the slider to adjust your traffic split to your new configuration.
A slider for splitting traffic during an AB test

End a test
End a test

Individual ad units
Individual ad units

You can remove an ad unit from your test at any time without ending the entire AB test.

On the Mediation management page, remove an ad unit from a test by completing the following steps:

  1. Select the ad unit to remove from the test.
  2. Select the dropdown menu at the top of the page and select either group A or B.
  3. Select End test.

On the A/B dashboard page, remove an ad unit from a test by completing the following steps:

  1. Select either group A or B.
  2. Select End test.

End all ad unit tests
End all ad unit tests

You can end an A/B test across all ad units.

From the A/B Management page, complete the following steps:

  1. Select the app with the active test.
  2. Select the action button.
  3. Select either group A or B.
  4. Select End test.

From the A/B dashboard page, complete the following steps:

  1. On the Overview table, find the test to modify, and then select the button for either group A or B in the corresponding row.
  2. Select End test.