Optimization can coincide with external growth
You upgraded your landing page on Friday, and your conversion went up over the weekend. Does that mean the new screen is better? Not necessarily. At the same time, traffic patterns, promotional activity, or overall demand may have changed.
Set the context of change
In the experiment journal next to the page version, note notable external events: the start of the campaign, a change in the advertising budget, a new GEO, a major influencer exit, a technical update.
Before and After Comparison Weaker Than Control Group
If possible, leave some of the traffic on the old version. Both groups live in the same calendar period. This is much stronger than comparing this week to last.
What can distort the outcome
| Factor. | What it looks like in the data |
|---|---|
| promo | growth in multiple sources |
| Weekend/holiday | time-shift |
| New placement | volume-leap |
| Changing GEO mix | Average CR moves unchanged within countries |
| Technical problem | flop from the moment |
Don’t just use common CR.
If KZ’s share of traffic has increased from 20% to 50%, and KZ has historically converted differently, the overall figure will change even with stable performance within each GEO.
What to do with the promo
The promo can be analyzed as a separate period or a separate campaign tag. Then you don't carry the "promo-CR" into normal weeks and build an inflated forecast.
Conclusion
Any strong jump has to be tested: has only the variable being tested changed? If not, the result is useful as an observation, but weaker as a proof of causality.
Use an Event Calendar
Have a simple calendar that highlights promos, holidays, major releases, budget changes, and platform events. When the metric changes dramatically, first look at that context, and then declare the victory of the new hypothesis.
Compare similar days
Monday and Saturday may have different audience profiles. For a short analysis, it is more useful to compare the same days of the week or use a control group within the same period.
Post-promo effect
Sometimes the action doesn’t just affect the time of the action: users come back later. Therefore, for retention metrics, it is better to mark the promo cohort separately and not mix with the usual baseline.
Don't optimise by peak
The peak week creates a temptation to increase the permanent budget. Before doing this, check whether the economy persists after the end of the external factor. The basic plan is better to build on the usual period, and the promo is considered as a separate scenario.
Separate the supply effect from the traffic effect
During the promo, the behavior of users and the volume of purchases can change. If the budget has increased at the same time, you can not attribute all the growth to the promo itself. If possible, keep the control segment or at least normalize the result by traffic volume.
Post-peak plan
Even before the end of the campaign, determine which budget will return to the usual level and what creatives will be used after it. This reduces the risk that the campaign will continue to work on a message that has already lost relevance.
Seasonal base
A year from now, your own story is more important than general advice. Store weekly metrics and context to compare similar periods and build a forecast on your data.
