Matchmaker A/B testing#

Finding the right matchmaking rule that provides the best game experience is important for player engagement. A/B testing is a feature that allows you to test, evaluate and optimize matchmaking rules without tampering with game operations or revenue. You can experiment with different matchmaking rule configurations and compare match outcomes.

The A/B testing dashboard displays real-time metrics about how the different matchmaking rules are performing. This data helps you evaluate and decide which ones align with your goals. The dashboard displays retention, engagement and monetization data to give a full picture of each of the matchmaking rule variants. You can also use the dashboard real-time data to tweak the matchmaking rules to best fit your needs.

Requirements#

To use A/B testing, install the latest SDK package for the following services:

1. In the Unity Editor's Package Manager, select Unity Registry.
2. Search for the following packages, then Install them:
   • Matchmaker:
     • For Unity 2022 LTS and later: com.unity.services.multiplayer
     • For earlier versions of Unity: com.unity.services.matchmaker
   • Analytics: com.unity.services.analytics

To set up A/B testing:#

1. In the Unity Cloud Dashboard, go to Matchmaker > Queues > A/B Testing. Select a pool and then A/B testing.
2. Navigate to the pool variant you want to do the A/B test on and select A/B Test with Game Overrides.
3. Select Create Override and then enter an A/B test name.
4. Set the target audience that you want to aim your A/B test to:
   • Audiences: Build custom stateful audiences using Unity Analytics tracking.
   • JEXL is not supported at the moment.
5. Choose your content type and create different matchmaking rule variations.
6. Set when the A/B test will run and how long it’ll run for. Adjust the start and end dates for your A/B test.

Troubleshooting#

Why aren't my tickets showing up in the variant pool?#

First, check the A/B test result using the following code:

var ticketResponse = await MatchmakerService.Instance.CreateTicketAsync(players, options);

object abTestingSerialize = (object)ticketResponse.AbTestingResult;
string abTestingJsonOutput = JsonConvert.SerializeObject(abTestingSerialize, Formatting.Indented);
Debug.Log(abTestingJsonOutput);

The A/B test result should return an object similar to the following example:

{
  "isAbTesting": true,
  "variantId": "aaaaaaaa-cdf2-4752-bae1-52ecc90d38c6:Variant 1",
  "poolId": "bbbbbbbb-722f-40f2-87f4-28ec307b7d0c",
  "basePoolId": "cccccccc-39a0-42ca-a15e-77544d7ffe41"
}

The A/B test result might return null, if the:

• Analytics SDK is not installed.
• Pool variant has been deleted.
• Game Override is disabled, expired, or otherwise misconfigured.

In any of these cases, tickets will be sent to your pool's primary variant.

To fix, review the initial steps on this page to validate your configuration.

Limitations#

The current version of Matchmaker A/B Testing does not currently support the following Game Overrides features:

• Contextual data attributes (JEXL) for player targeting are not supported at the moment.
• On the Unity Cloud Dashboard, in Game Overrides, the Reporting tab does not show results from Matchmaker A/B tests. However, users can view results from the graphs in the Reporting tab.
• In the Reporting tab, Total participants refers to the total number of participants involved in the A/B tests, but it is not equal to the total number of confirmed players who have been served by the Matchmaker configurations.