Control & Disentangle
Controlling Experimental Variables for Making Causal Inferences and Disentangling Effects
To find out what happens when you change something, it is necessary to change it.¹ This axiom describes the relationship between inference and experimentation, but there’s often more to discovering what’s really driving outcomes. Experimentation has advantages in its ability to discern causal relationships between events or circumstances. These factors can be changed and measured to compare the effect of different variants. However, as is often the case in experimentation, information that is lacking invites competing interpretations.²
Humans are adept cognitive misers, unmatched in our ability to take in diverse sources of information and draw conclusions. The human brain is built to associate events with definite causes and therefore can have a hard time considering the influence of unrelated or random factors. Evolutionarily, this quick and confident (albeit sometimes wrong) decision-making has served us well, helping us avoid uncertain dangers for generations to come. However, in the domain of experimentation, these instincts have contributed to challenges including a replication crisis in academic research.
Opportunities to experiment online continue to grow as the Internet industrializes. Still, one of the greatest challenges facing digital marketers, product managers, and market researchers is measurement. Where who is doing what — it’s a tricky problem space. By reframing hard problems into easier problems, there are simple solutions anyone can use to get ahead of the curve in most situations.
Controlled A/B tests have 100% attribution.
In high school biology, our teacher walked us through writing lists of controlled variables when designing experiments — it was great for papers double-spacing a list of conditions ranging from humidity to atmospheric pressure. Luckily, when page count isn’t a primary concern, it’s not as arduous controlling digital experiments.³
Just change one thing at a time.
Simple enough, but changing one thing at a time can be contradictory to the ambitions of many experimenters. In business experimentation, the overarching goal may be to broadly improve specific outcome metrics, through which the mechanisms are of secondary importance. Therefore, experimentation focused on improving results may rely on changing multiple factors, kind of a spray and pray scattershot approach to optimization. This makes sense — if you can stack improvements, you’ll get better results as small wins cumulate into big changes.
However, if the goal is improving results, a reliable stop on the way to results is achieving understanding. But trying to understand everything can lead to testing everything, unfortunately at the expense of measuring anything.
Here’s an example:
Imagine you’re a home bartender. Your friends have come over for drinks, and you’ve received praise all night for the cocktail you invented earlier in the evening.
The next morning, you awake to remember very little at all.
The texts on your phone are clear — you made the cocktail of your life last night. But what was in it?
Your fridge is stocked full with lemons and limes. Probably a good place to start. And there was something on the rims of the glasses. Salt? Maybe sugar?
Okay. For a starting place, you’re not doing badly.
Salt and lime work, as do sugar and lemon. So you make your cocktails, one with salt and lime and one with sugar and lemon, and you invite all your friends back over for night two.
What’s the mistake?
Well, drinking habits aside, the primary mistake is that we’ve failed to control variables.
Your Drink of the Gods may rely on lemon or lime, salt or sugar, but with the current setup, you would be unable to determine if lemon or lime were a step in the right direction, or if salt or sugar were the missing ingredient. The reason is that by changing more than one variable at once, the effects of either variable cannot be measured, obscured by the effects of other variables.
The solution: changing one variable at a time, you’ll need to make four different drinks.
Only after examining the effects and interactions of each of the factors at each level can researchers, bartenders, and business experimenters alike determine the relative merits of different potential combinations of options.
Another common best practice is setting aside a control group of participants to be present but not exposed to the experimental variables of interest in order to gather a baseline measurement against which any effects can be compared. Further, to successfully disentangle effects of each of the experimental variables, similarly to high school biology, potentially confounding factors such as weather should be as controlled as possible. Practically, an example would be inviting your friends all on the same night or similar times of week, maybe Fridays. The key is maintaining consistency and control of experiment conditions that may affect the variables being examined.
Now, whether or not to invite all your friends back for four drinks each — that’s a different question.
[1]: Box, G. E. P., Hunter, W. G., Hunter, J. S. (1978). Statistics for experimenters: an introduction to design, data analysis, and model building.
[2]: Mlodinow, L. (2009). The drunkard’s walk: how randomness rules our lives.
[3]: Kind of. Generally, third-party platforms offering participant recruitment services will solve a lot of these problems for online experimenters. The rest of the hard technical problems are generally solved by the platform hosting your experiment or survey, such as Google Forms, Survey Monkey, or Guided Track. Remaining problems are best solved by counterbalancing often through randomization.
