Home > Affiliate Marketing Forum >

---

PART 2 - Everything You Need To Know To Kill It In AM Was Written 2500 Years Ago (5)

---

Welcome back to part 2 of STM's introduction to the Scientific Method - the basis behind pretty much everything we do in AM.

You can find Part 1 here, in which we went over defining your question and formulating your hypothesis.

Now, we dive into Part 2 - the during and after phase.

Experimentation and Measurement

TL: DR Summary

• Make sure your tracker is set up to record all the data it possibly can.
• Keep a journal. Keep it organised.
• Record and check your assumptions about the experiment you're running.
• Choose a sensible end point in advance and stick to it.
• Make sure your experiments end as soon as they can - but no sooner.
• "I will stop when I have run x rounds of testing to statistical significance" is a good general rule.
• DO NOT change other variables half-way through an experiment.
  

The Details

When most people think of science, they think of labs. Test tubes. Particle accelerators.

All of those things are used for experimentation.

As affiliates, we're almost certainly already good at the experimentation phase, because that's the process of running a campaign. Your hypothesis is usually your angle, and you spend money on data to test that hypothesis. (You may have other sub-hypotheses too - see "Nesting", below).

The key thing to realise about science for affiliates is that experimentation is worthless without measurement. If you run an experiment and don't accurately measure the results, you might as well have not run it - it doesn't help or prove anything.

So, at the most basic level, make sure you're running a good tracker. Beyond that, record anything else that could be of use.

• Times of day.
• Exactly what you're testing.
• What you are assuming to be true already - checking those assumptions is incredibly useful.
• Any confounding factors - like public holidays or other major events in your geo.
• Any mistakes you make which invalidate some of your results, or otherwise are worth noting.
  

Vortex wrote a post a few days ago about keeping a campaign journal. Journaling is a vital component of science, and that's highly recommended. Make sure you're clear in your journal as to what experiment you're running, and which data you're associating with it - it's very easy for journals to end up very disjointed and messy, and that substantially reduces their usefulness.

The other important feature of an experiment is that it has a definite end point. Run the experiment, check your hypothesis, and then conclude the experiment.

Example: Hiran and Anton are both trying to solve the problem of getting a specific offer to convert profitably at more than 200 leads a day. Anton decides that he's going to conclude that any experiment he's taken through three optimisation rounds is finished, and move on. Hiran, meanwhile, decides he'll go with his gut, and ends up spending nearly fifteen optimisation rounds on a single hypothesis that keeps looking like it's almost working. Anton ends up crushing the offer before Hiran gets through his third test.

That's so important because the scientific method relies on repeating the process. Try, fail, learn, repeat until you don't fail. One of the subtlest ways to get that wrong is to be slow to brand an experiment a success or a failure.

Make sure, though, that your experiment doesn't end too soon. Scientists developed statistical mathematics for just this, and later on developed Bayesian statistics too. I've gone on about this in plenty of other places so won't talk about it again at length here. Just make sure your math is solid and your approach - whether you're using a Frequentist approach like Chi-squared testing or confidence intervals, or a Bayesian approach - is sound.

"I will stop when I have run x rounds of testing to statistical significance" is a good general rule.

Finally - make sure to keep all other variables constant whilst you're testing. If you suddenly go "hey, you know what, this could work on Facebook!" half way through your experiment, that's great. But wait until the end of the experiment to test that hypothesis. Switching hypotheses half-way through an experiment is an extremely common way to invalidate your results.



Nesting

TL: DR Summary

• If you hit problems whilst running your main experiment, use the scientific method to solve these sub-problems too.
• Don't skip steps: do the sub-problem science as rigorously as the main problem's science.
  

The Details

One thing that's worth remembering - just because you're running one experiment doesn't mean you can't run others at the same time, or even as a sub-element of the first experiment.

So, if you're testing a hot new offer which you hypothesise could help you break $x,xxx a day, you can then use the scientific method to hypothesise and test angles for that offer - sub-experiments within an experiment. And then, you can use the scientific method again to improve CVR on a landing page, or CPC on a banner, or even ROI on bids.

Often you'll end up with a nested structure where you've run an overarching experiment which then had a dozen or more sub-problems within it, each of which you've then run a bunch of experiments on to find working solutions to the problem, which has led you to conclude the result of the main experiment.

Example: Clare has an overall angle hypothesis for a new traffic source which she thinks she can scale massively. However, as it's a new traffic source there are lots of sub-problems to solve. She goes through each problem - what bid type to use, what creative format, how to structure creatives, ideas for landers, bidding strategies - and for each of them formulates a question, tests hypotheses, and comes up with solutions. That means she bulldozes through problems one after another and never ends up spinning her wheels unable to tell quite what she should test next.

As with the main process, it's important to go through all the steps here for each of the sub-problems. Do remember to define your problem appropriately, formally brainstorm hypotheses and rank them, and so on. You'll save a lot of time and energy that way.



Conclusions

TL: DR Summary

• Check whether you answered your original question, not whether you made a profit.
• Remember: a single result doesn't necessarily mean the result will be repeatable.
• Don't assume results will be true forever. Keep an eye out for signs that your method no longer works.
  

The Details

Once you've run the experiment, you need to answer the universal question: was your hypothesis right or wrong?

(Technically a hypothesis can usually only be proved wrong - that's why we call scientific theories "theories" rather than facts, even if they're well-supported ones like the theory of gravity. But for AM, you only need to be not-wrong for a few months, rather than 200 years, and so we'll skip that particular element of longer-lasting science.)

It's important not to confuse this question with "did I make a profit?". Go back to your original question, and see whether your hypothesis solved it. If you defined your problem as "How do I make $X,XXX a day with mobile?" and your hypothesis got you to $535 a day, your hypothesis was not proven. You need to keep working.

If you did solve your question, the next part of the method is to check whether your results were repeatable. Most of the time if your initial question is financially related, you'll want to check this anyway, because the way you check it is by keeping running your campaign! However, if you're solving a nested question, it's important to bear this in mind. Just because your initial test showed that Lander Design B outperformed lander design A, that doesn't mean it's always the case. Keep an eye on results, see if Lander B's ongoing performance tracks with your initial experiment, and occasionally re-test to make sure you're not leading yourself up a blind alley.

Finally, remember, we're in the soft sciences here. We deal with people, not universal laws. As a result, your results won't stay true forever. Keep an eye on your stats to check whether your methods are starting to lose effectiveness - if CVRs are dropping, costs are increasing, or similar, you might need to start coming up with new hypotheses and start testing again, even to solve a problem you solved before.



Learning

TL: DR Summary

• You spent money on this data: get the most out of it you can.
• Stop and take time to learn and study after a failure OR a success.
• Failures can point the way to success if you take time to study them in depth.
• Don't just look at the overall result: drill down.
• And once again, take notes.
  

The Details

So, that's it, right? Either keep running the successful thing or go back to "hypothesis" and choose another one?

Absolutely not.

The next stage, regardless of whether you proved or disproved your hypothesis, is to take the new data you have and study it to extract as much information and learning as possible.

If it didn't work:

• Did it work better or worse than previous experiments? Is there a trend line here?
• Was there a point where it looked like it might work? Can you learn anything from that?
• If it failed really disastrously - why? The first explanation that springs to mind might not be the best. Look at the data.
• Were there any pockets of success? You'll need to deep-dive into your stats to see if any individual component worked better than expected - banner, lander, placement, offer, whatever.
• Can you use a data-mining tool like WEKA to extract any more information from your data? I highly recommend that if possible.
• Are there any anti-proofs in the information? Is there anything you can definitively say is very unlikely to work, beyond your initial hypothesis, based on the data you've gathered so far?
• Does any of your result set remind you of something you've read about elsewhere? Another Follow-Along, a case study, anything else? Go find that thing and see if there are parallels.
  

If you want to learn more about learning from experiments, I heartily recommend the classic Silicon Valley book Lean Startup. Lean Startup methodology is strongly based on the scientific method, and has powered companies like Dropbox, Wealthfront and arguably Facebook to massive success.

When you're doing this learning, it is once again vitally important you take notes. Don't assume you'll remember your conclusions. You won't remember all of them.

Once you've extracted all the learning you can from the experiment, then you should move on. Either keep running and build on your success with a new question, or go back to the hypothesis stage and come up with another hypothesis.



Communicating Your Results

TL: DR Summary

• Consider sharing what learning you can with the community.
• It helps other people and it helps you!
  

The Details

And here's a final step that most people don't think of: an absolutely crucial element of the Scientific Method, and one of the main reasons it has conquered the world, is that it's built in that you should communicate your results.

Obviously, in affiliate marketing, you don't always want to do that - just like you don't want to do that in pharma research or military research. Don't share all the details of the campaign you've just spent months testing!

But if you can, it's a great idea to share knowledge that won't disadvantage you. It helps the community advance, it pays your success forward, and it also helps develop relationships and respect from other affiliates that can be very useful in the future!

So - communicate what results you can. Write a case study after you've wrung the campaign dry. Write a guide to an element of what you did. Share some hypotheses that didn't work!

It'll help you and help others - awesome!

And that's it! If you have any questions, comments, or would otherwise like to engage in peer review of this article, post 'em below!

---

Very valuable article. I read it carefully every word.
I'm new to affiliate marketing, so I'm confused here:
Don't change variable during a experiment, and test main hypothesis and sub hypothesis at the same time. how is that possible?
For example, I want test the bidding type, and at the same time, I want test the angle. Now I should change two variables at the same time: bidding tipe and angle. How can I solve this problem?

---

Don't test both of those at the same time. Either launch a new campaign with a new angle but the bidding system you're familiar with, or launch a new campaign with a different bidding system but an angle you've already gotten results from.

Otherwise you can't tell whether the angle or the bidding was responsible for your results.

(You can fuzz this rule a bit in practice if you're familiar with the traffic source, because you can look at expected results from your experience. But in an ideal world - and probably for best results - you should only test one variable at once.)

---

Hello @caurmen
I just read part 1 and 2.
Good work man!

I wonder where can I see examples about TESTING sub-problems at the same time testing the main problem. Thanks so much!

---

@angelok:

Thanks!

Sub-problems and main problem: here's a simple example.

You're attempting to get to $x,xxx a day on a traffic source using your existing offer suite. That's your main problem.

To do that, you are currently testing a hypothesis that a specific angle with a specific offer will work.

You're optimising that offer, and you find that your landers just aren't getting a high enough click-through rate to the offer. So you start working on solving that problem using the same process: define problem, hypothesis, experiment, etc.

That's an example of solving a sub-problem. And it has all the usual features of solving a problem using science: for example, you have to be accurate about your problem definition. ("How do I get a higher CTR?" is NOT a good question. "How do I get a higher CTR whilst retaining my existing CVR?" is better. "How do I achieve a balance of CVR and CTR from this lander/offer combination that gets me to the profit levels I want?" is better still. )

Hope that helps!

---

Home > Affiliate Marketing Forum >