Creative Testing campaign best practices
Before setting up your test campaign, take some time to consider what you want to discover about your creatives. The following information about test parameters and best practices can help you set up the best test campaign for your needs.
Creative testing parameters
This section details the parameters to consider before starting your test campaign. For step-by-step instructions on how to set up your campaign, refer to the Start a Creative Testing campaign documentation.
Creative pack quantity
There is no limit to how many creative packs you can include in a test campaign. Remember that each creative pack you add will slightly decrease the scale for all creative packs in the test campaign though. This reduction happens because campaign delivers the packs on an equal-opportunity basis, and the creative packs are competing for impressions as a result.
Creatives combinations within packs
To test how different combinations of end cards, videos, and playables perform for a particular market, you can create packs with different combinations of the same creatives. A Creative Testing campaign will deliver all of these packs evenly and identify which combinations are statistically most successful. For more information about creative pack configurations, refer to the documentation on Creative packs.
Country targeting
When conducting a test campaign, it's recommended best practice to test creative packs in the same market you intend to use the creative packs in real implementation. Within this scope, you can select multiple countries for one test campaign, or you can create multiple test campaigns that each target a different country.
Recommendations for best results
The following are the recommended best practices for conducting a Creative Testing campaign:
Test with a control creative pack
To get the most exact results, Unity recommends using a control creative pack. A control pack is a creative pack you have already used within Unity’s ad networks. Using a creative pack that already has a performance history will yield results much faster than using only new, untested creative packs in your experiment.
Note: Although a control pack isn't a prerequisite for creative testing, running a test campaign with only new creative packs will result in a much slower testing process.
Calculate sample size
Before acting on the results of your Creative Testing campaign, it’s recommended best practice to calculate the number of impressions needed to be confident in the results. A reputable test calculator such as the Sample Size Calculator from Evan Miller will calculate how big your impression sample size needs to be to deliver statistically significant results.
Sample size calculators will help you identify whether the differences in your creative packs’ performance are significant enough to make changes to your user acquisition strategy.
Test with a hypothesis
As with any experiment, Creative Testing campaigns are most effective when used to test a theory or hypothesis. This means it's best practice to have an idea of something specific you expect to learn from the campaign.
For example: If you want to test how a video creative performs when paired with differing end cards, you can build a Creative Testing campaign that delivers creative packs made up of different combinations of the same creatives.
If you want to test which playable creatives get the most clicks, you can build a creative testing campaign and use the simplified overview to compare clicks across creatives.
Keep creative packs consistent
If your creatives have any distinct differences among them, your creative testing campaign might distribute your creative packs unevenly. Uneven distribution will artificially alter the results of your experiment. Differences in creative packs are often caused by sensitive attributes or inconsistent start dates. Sensitive attributes include things like gambling, violence, or any content that might receive a higher age rating when the packs undergo human moderation. For more information on Unity’s moderation process, refer to the Creative pack moderation documentation.
Differences in start dates often occur when you add or change the creative packs used in a creative testing campaign after the campaign has already gone live. For more information about how to adjust the start date of a creative pack in you test campaign, refer to the Change or add creative packs section of this documentation.
When these content differences result in unequal impression distribution, you will receive an Uneven Distribution warning. For more information on this warning, refer to the Campaign management section of this documentation.
Maintain consistent testing parameters
The recommended best practice for Creative Testing campaigns is to avoid adding or removing creative packs during an active test campaign. Although you can pause a Creative Testing campaign to add additional creative packs, doing so can lead to inaccurate and misleading results. To learn more about the effects of adding creative packs to a live Creative Testing campaign, continue to the Campaign management section of this documentation.
Next steps: Analyze your Creative Testing campaign results.