Home > General > Affiliate Marketing Forum

---

Best high volume multivariate testing platform (23)

---

We used to use GWO and serve around 30-40 million clicks a month. So curious what peoples opinions are. We've looked at VWO and Optimizely and both seemed cost prohibitive for large volume. Also, I found this which is pretty interesting.

http://www.whichmvt.com/

Let me know your thoughts.

---

Why not build something inhouse if you guys are pushing that kind of volume?

In terms of performance to cost feasibility, I don't see how you can beat GWO.

---

GWO doesn't exist anymore.

And of course we could build something in house but it's all about allocation of resources.

If I can deploy those tech resources to automate bid management or whatever on google it makes a lot more sense if there's solution that are reasonable that exist.

I honestly building stuff in house is a time sink and pulls you away from core competencies.

Core competencies is what makes the money.

---

Vertster is pretty powerful and easy to use. Taguchi method, decent reporting, cookies users to make sure they seem same version when returning, etc

---

I definitely concur with your reasoning as certain development is a huge time sink.

But I am sure you can hire/use a dedicated developer to make something within a week. Not to mention make future internal updates and even sell it in the future.

At the end, I agree with the companies core competence. Which is to make "money".

---

Hey! No need to build this in house - we can build you something badass. App development is our thing.

Our Skype: custompsds

EDIT
To add some value to this post as @ari_ pointed out:

So I guess the main question is: How many variations are you testing at a time?

Optimizely is a really cool platform and a really handy tool in the toolbox of variation testing and I highly recommend it, but as you pointed out the problem is you need to host those variations on their servers. On small scale, it's ok - and there was actually an affiliate I was having a conversation with the other day states it had a hard time firing pixel properly - not sure.

But when you are dealing with volume on the scale you are, an array of problems pop up - cost is a big one. Probably the best solution that we have actually looked at, is taking something like Optimizely's awesome editor and version creator ( it is extremely awesome as long as your markup is written properly ) following the normal Optimizely steps, but exporting the code to a .ZIP variation on the fly so you have the code and you can test on your servers. That works if your testing a few versions with high traffic, as your tracker can tell you the winners. But the problem...if you have 50-100 variations your testing can become quite difficult.

As of right now, I personally don't know of anything really powerful and that I would recommend to use. It would have to be a hosted solution on your end however - as SaaS stuff would be really costly.

There is definitely a hole in the marketplace that needs to fill this gap. We have discussed a "perfect" tool like this here at the office many times.

---

Jason,
In theory this is true. The problem is when the software you're using affects your core competencies - i.e. creates lag time or loses clicks or doesn't measure properly. At some point you start spending so much time with the other company's API that it would have made sense to roll your own.

The big problem I have right now with Optimizely, VWO and Google Content Experiments is that they don't handle dynamic pages in a really elegant way. Certainly the first two are very "Visual Editor" oriented, and I did not like VWO's jquery API. I also found its conversion tracking was less then adequate. We also ran into similar problems with Optimizely. Also at your volume they are cost prohibitive, and I am not 100% certain that they could reliably handle that volume of traffic without affecting performance in a noticeable way.
(VWO anyway, Optimizely seems to have higher volume clients)

Right now on a e-commerce site that we have tied into analytics, we're forced to use all sorts of cludges an hacks to make content experiments work properly. And even then the conversion and goal tracking is suboptimal. It's still good enough that we're seeing improvements, but frustrating b/c we're flying half blind.

I don't have experience with the larger more enterprise focused ones on that list, but they might be a better solution. I think either way you might be better off working with the Squirrel people anyway (the people who own whichmvt) and see along the way if they can give you the right solution
Dude, I appreciate that you're hustling - that's the nature of this forum. And I've actually talked to you on Skype and will probably hire you in the coming weeks/months. But if everyone started promoting themselves in every thread, there would be no value in being a member. If you think you can help Jason then send him a PM or post in the offers section. I mean this in a nice way, not trying to flame etc, just trying to keep some integrity within non commercial threads.

---

Apologize. But we never promoted ourselves in anyones thread before - just seemed like a really cool project and looked super interesting. Won't happen again. Apologies for offending, and I agree with you.

P.S. - Couldn't send a PM as we had less than 10 posts. That was our 10th post I believe.

---

Jason,

Can you tell us any more about exactly what you would need the software to do. Does it have to be MV or would A/B suffice?

Is this for landing pages or for in-site pages as a part of a wider funnel/site?

Thanks,

Dave

---

check out Landing Page Genius - a plugin for CPVLabs. It does Multivariate testing just like VWO... but you only pay for it once ($150 or so) versus paying monthly based on impressions. And you can track conversions on 3rd party sites - which is something VWO can't do

---

I use VWO daily but it's seriously lacking in many aspects. For example the basic technical implementation is such that if you are loading elements with JS AFTER the page has loaded, you'll have a bad time - if you want to obfuscate your LP and deobfuscate (is that even a word?) on-the-fly with JS, you are going to have flashing problems with VWO.

I used to advocate for VWO and their JS in the head approach but now I've seen too many cases where it just simply sucks ass. I don't even use VWO's Visual Editor anymore, it just doesn't work for anything else than simple image or text switches.

I hope CPVLab guys can soon deliver new version with MVT testing like they say on their page. Landing Page Genius looks good but I'm still a bit vary of buying stuff that can be rendered useless after the next update. But until that, I'm in the process of testing my own solution for testing which is way more comprehensive than your average MVT.

---

Sorry for playing devil's advocate but usually investing in a testing platform is a very lucrative investment. If you can't justify the cost of it maybe it's because what you're testing doesn't matter as much as it should. Could you justify spending money with a testing platform if you'd double the conversion rate or ecpm of your pages?

You also don't need to send 100% of your traffic to it, but just enough to get statistically significant results and roll out the changes to all your pages.

---

Clicky? They're very cheap - about $240/month for 1 million daily page views, or $1900 a year. I personally like their platform though it's probably not in the greatest setup right now for multivariate testing. However, they are quite transparent with their tracking API and how their internal processing works - quite easy to use server2server calls to trigger goals etc. rather than relying solely on javascript. I use them for dynamic split testing where passed subids are used to on-the-fly determine the naming of split tests/versions etc.

---

optimizely rocks.

---

optimizely sucks

---

- ^ interesting

---

Does LPG do all the things that the big software packages like VWO do?

Does it have any weakneses compared to the pay-per-traffic solutions?

---

polar.... do tell...

---

Yea I'm interested in this too ...

I'm getting ready to setup a google content experiment -- looking forward to the low down on optimizely and others.

---

What are you looking for, exactly?

There's two dimensions for consideration.

Dimension 1 - How you deliver the tests, e.g., do you just want to collect random permutations of page elements?, or, do you want a structured approach (e.g., Taguchi, D-Optimal, etc)?

Dimension 2 - Will you analyze the data by hand? Or, do you need the system to auto-optimize?

Personally, when I do high-volume MVT, I just data on (semi-) random permutations, and then bulk analyze the data by hand. (The less data you have, the more important it is to have a well thought out design of experiment.) The reason I analyze by hand is because it allows for much more flexibility in analyzing the data, and can provide insights you don't get with auto-optimization. Of course, unless you know what you're doing (and have the right tools), extracting value from large, multi-variate data sets in very difficult. Most end up just summarizing by one variable at a time, which loses the point of multi-variate. Might as well just a/b test.

---

Not sure if that was directed at me.

I'm currently only doing (and have ever done) A/B split tests.

I was under the impression unless you have a lot of traffic (and consequently "actions" or "conversions") MVT wasn't very useful/accurate?

So basically I'm just looking at a simple way to do split-tests on my current landing pages.

Like I've used CPVLab for split tests -- but then you have to change the actual link you're using to your website -- but I already have a lot of links/ads whatever sending traffic to a particular landing page so I just want to change/test things on that page like:

-- split test the VSL (video sales letter)
-- split test the order button/call to action
-- split test the headline

etc.

So loking for a good replacement for GWO which I used to use to do that.

I also used Visual Website Optimizer for a while but found it:

1. expensive for high traffic and
2. I don't "trust" the just clicking and dragging shit on your page you know?

I don't know maybe I don't understand the new tech and that's why I don't "Trust" something like VWO ...

---

The question was for OP.

But, you ask a good question . . . how much traffic you need to make MVT useful.

Here's the tradeoff - If you do full MVT (testing all the elements against each other) it takes a long time to get meaningful signals, but, you can sometimes find interactions that have a big effect. If you do a designed MVT (e.g., Taguchi), you can get signals a lot faster. The downside is you have to know how to design the test, and you're less likely to spot interactions between elements.

The best thing, in my opinion, is to test the major elements using Taguchi to get "close enough" as quickly as possible, and then use automatic MVT optimization to fine tune over time.

With that said, I'm not currently doing that. With what I'm doing in mobile, much of the optimization is getting the click to the right offer, so I do a lot less creative testing (and just A/B).

---

This stuff makes me think about the use of PCA and SVD etc. for analysis in the science I do. Something I avoid like the plague.

---

Home > General > Affiliate Marketing Forum