Analytics
What is Statistical Significance?
A judgement that a measured difference is unlikely to be chance alone. It answers whether the result is real, not whether it is large enough to be worth acting on, and those are separate questions.
Why it matters in practice
The everyday failure is calling a test early. Results bounce around a lot in the first days, so if you check every morning and stop the moment one version is ahead, you will declare winners that are pure noise. Decide the sample size and the run length before you start, then look at the end. The second failure is the opposite: a result can be significant and still be trivial, a fraction of a percent that cost weeks of traffic to prove. For most stores the honest position is that they do not have the volume to detect small changes, which is an argument for testing bigger swings rather than for testing nothing.
Want this working in your favor instead of against you? The Strategy Audit names the gap on your own account, the optimizations, a phased plan, and a potential outcome, delivered free.
Back to the full glossary, every term on one page.