Home > Questions and Answers > General Questions

Statistical Signifiance Formula - Any Maths Guru Here ? (10)

06-06-2014 12:59 PM #1 givizator (AMC Alumnus)
Statistical Signifiance Formula - Any Maths Guru Here ?

Hi guys,

I'm looking for a good formula to calculate statistical signifiance.

I've campagns with Ads, LPs and Offers.

For each item, I have an eCPM.

I cannot use the conversion rate because I globally use too many differents kind of payout in my Adserver (mixing CPM, revshare and lead) and I want a global formula I can apply everywhere in my adserver.

So I'd like to base my calculation on the eCPM only.

Let see some results :

Click image for larger version.  Name: Capture d’écran 2014-06-06 à 14.53.45.jpg  Views: 58  Size: 15.8 KB  ID: 2682

What's in your opinion the best way to calculate statistical signifiance in this datas and have the script said "ok, it's time to disable ad #880".

Thanks for any advice.

06-06-2014 03:09 PM #2 joshogle (Member)

IANAMathematician, fyi:

It's not nearly as simple as just a "formula," since you're trying to do this in a way that's unique to your own situation.

Typically, you'll have an LP or an ad or whatever, and have a finite amount of possibilities as the end-result (e.g.: converted; did-not-convert), after which point you can determine the degree of confidence you're looking for (95% as an example; this is known as your "alpha"), and run through any number of tests to get significance.

Statistical significance is not here to tell you whether you should or shouldn't stop running an ad -- it's going to tell you the likelihood of your ad performing at a certain level, which is your base hypothesis, vs other ads -- then you yourself determine whether to run it or not (or set things up to do so automatically based on a certain eCPM or something).

In your case, I'm not sure I even understand what you're attempting to test as your hypothesis, so I can't imagine an off-the-shelf formula that would help you here, unfortunately.

06-06-2014 03:30 PM #3 givizator (AMC Alumnus)

Quote Originally Posted by joshogle View Post
IANAMathematician, fyi:

It's not nearly as simple as just a "formula," since you're trying to do this in a way that's unique to your own situation.

Typically, you'll have an LP or an ad or whatever, and have a finite amount of possibilities as the end-result (e.g.: converted; did-not-convert), after which point you can determine the degree of confidence you're looking for (95% as an example; this is known as your "alpha"), and run through any number of tests to get significance.

Statistical significance is not here to tell you whether you should or shouldn't stop running an ad -- it's going to tell you the likelihood of your ad performing at a certain level, which is your base hypothesis, vs other ads -- then you yourself determine whether to run it or not (or set things up to do so automatically based on a certain eCPM or something).

In your case, I'm not sure I even understand what you're attempting to test as your hypothesis, so I can't imagine an off-the-shelf formula that would help you here, unfortunately.
Thank you for your answer, I agree with you, more than a formula I need something for my unique situation.

On the screenshot I provide in the post, I'm testing 5 ads, each one get nearly 5.3 millions prints, the CTR is in a range of 0.2 to 0.46% and the eCPM in a range of 0.013 to 0.033€.

The Best Ad for now is #861 with an eCPM of 0.033€ after 5.3 millions prints
The Worst Ad for now is #880 with an eCPM of 0.013€ after 5.1 millions prints

Based on that, I can said that #880 make me lost 101.8€ for the 5.1 millions times it was used, I will have that 101.8€ if I haved replace it by #861 all that time.

My problem is : how can I calculate that this is a safe assumption and there is enough datas to be sure that #861 is, and will probably remain, better than #880, so I can disabled #880 and test a new Ad ?

Hope it's more clear.

Thank you.

06-06-2014 03:37 PM #4 t0mmy (Member)

I am not a mathematician either, though I did end up with a math minor somewhat by accident.

Anyway josh is correct, statistical significance mostly just tells you whether or not a given performance level is the result of chance or not. If I recall correctly (which I may not), it is mostly concerned with sample size and ensuring you have enough data.

06-06-2014 06:12 PM #5 caurmen (Administrator)

I'd recommend having a read of Section 10 of the Getting Started Guide - I discuss exactly the problem you're facing there, and show how to figure out the predicted ROI of ads using statistical significance tools.

Whilst statistical significance calculations can't make choices for you, they can give you accurate predictions of future performance - assuming all variables stay the same - which I think will help a lot with your decision-making process here.

06-06-2014 07:45 PM #6 givizator (AMC Alumnus)

Quote Originally Posted by caurmen View Post
I'd recommend having a read of Section 10 of the Getting Started Guide - I discuss exactly the problem you're facing there, and show how to figure out the predicted ROI of ads using statistical significance tools.

Whilst statistical significance calculations can't make choices for you, they can give you accurate predictions of future performance - assuming all variables stay the same - which I think will help a lot with your decision-making process here.
Thank for your answer.

I've already read that post, but it doesn't fit my needs.
I cannot use the conversion rate variable in my process, I have to base it on the eCPM.

I understand now that with eCPM, statistical signifiance is out of topic.

I assume I will have to compare things based on eCPM and an amount of money allocated to the running test.

Best Ad : ecpm X
Test Ad : ecpm Y < X

When the test Ad will be Z euros behind the best Ad, I will cut it.

06-07-2014 05:53 AM #7 zeno (Administrator)

I can see what you're after but hell if I know how to approach it! Maybe generating some sort of distribution curve based on impressions and eCPM then comparing their overlap.

Sort of like this: http://www.peakconversion.com/2012/0...al-calculator/ but with impressions and eCPM as the factors. Unfortunately that tool won't deal with such large numbers.

06-07-2014 10:54 AM #8 caurmen (Administrator)

If you have the eCPM and you have the payout of the offer, you can derive the conversion rate, then use the post I mention.

eCPM = 1000* Payout * CVR (as percent)

So CVR = eCPM / Payout *1000.

For example, if the eCPM is $5, and the payout is also $5, that means that you're making one conversion every thousand impressions: CVR = 5/(5*1000), so CVR = 0.1% .

06-09-2014 11:56 AM #9 givizator (AMC Alumnus)

Quote Originally Posted by caurmen View Post
If you have the eCPM and you have the payout of the offer, you can derive the conversion rate, then use the post I mention.

eCPM = 1000* Payout * CVR (as percent)

So CVR = eCPM / Payout *1000.

For example, if the eCPM is $5, and the payout is also $5, that means that you're making one conversion every thousand impressions: CVR = 5/(5*1000), so CVR = 0.1% .
That can be a way to go.
I haven't the payout as it is not fix (can be $100 one time and $20 the other with the same offer).

But I can said that I consider the payout to be $5 per conversion
For a $100 payout I will count 100 / 5 = 20 conversions
For a $20 payout I will count 20 / 5 = 4 conversions

06-10-2014 05:26 PM #10 caurmen (Administrator)

That sounds like a reasonably solid way to get a decent estimate without getting into hideously complex math.

Good luck!

Home > Questions and Answers > General Questions