For a new campaign I am focusing on several smaller placements, about 15000 impressions/day.
- What would you think is the optimal number of banners to test at the same time?
- How do you get over small differences, sometimes 1 conversion can be the difference between -20% and +20%
Basically I am looking to optimize my process. I've made the mistake uploading 30 banners for 1 placement, now I should wait at least 10 days before cutting any of them statistically. Any tips for smaller placements are welcome 
There is not a set rule, what I do is inverse thinking.
Example for a product A:
1) I know or guess that I need 50 clicks to get a conversion
2) I'm aiming for an average CTR of 0.5%
1 + 2 means I need 10,000 impressions to get 1 conversion.
3) For product A, with a low CPA, I need 5 conversions to lessen the chances of 1 lucky/unlucky conversion to a 20%, that for a low CPA product, means an small $ difference.
1+2+3 means I need 50,000 impressions to get enough data to make proper judgement about a banner that falls in my premise 2), if the CTR for all banners is over it, i'll need less impressions, if the CTR is under it for all banners, then i´ll need more impressions. In both cases I would adjust premise 2).
I wouldn't want to wait more than 10 days to judge a campaign, in your case 10 days = 150000 impressions, that divided by my 50,000 needed to judge a banner, means 3 different banners.
The good news is that if you start with totally different banners, you will be able to eliminate some really soon, as the CTR just after 1 day will be different between them (conversion probably not due to not enough statistical data), and you can eliminate fast those that fall below your premise 2, no point on leaving them there for 50,000 impressions with a low ctr as in this case, with little amount of impressions, you need a high CTR to have some traffic sent to your landings.
So in resume, in those 15,000 impressions spots, I would start with 3 banners and eliminate the one with the lowest ctr every day, adding a new one, but only if the difference in ctr between the 3 banners is enough to detect a bad banner.
Hope it helped!
Awesome, awesome post, thnx Ivancy.
Going to cut my losses and set it up again.