Ideas are powerful – you don’t need to watch Inception to know that, either. Results released by Microsoft, Amazon and App Sumo indicate most of their ideas to improve their websites have failed to improve the metrics they were designed to lift.

Here are the results:

• Amazon - 50% of experiments fail to show improvement
• Microsoft - 1/3 ideas had a significant positive impact
• AppSumo - 1/8 tests produce results
• QualPro - 75% of important business decisions have no impact on, or actually hurt performance (these guys have done MVT for nearly 30 yrs)

That’s an average success rate of roughly 30%. That sounds pretty grim to me. In fact, it sounds as though it’s dangerous to implement ideas without testing them.

Test ideas from any old place might sound like a good idea, but sometimes it pays off to work for your ideas (through research).

Surely those results are on the low side?

I’d say Microsoft and Amazon are doing some pretty ground breaking stuff. Ultimately, that’s the point of testing. By pushing the envelope on counter-intuitive ideas, these guys are producing test results where they learn something and push the boundaries of what’s expected from website changes. They don’t have a lot issues on their sites because they already have specialist teams developing awesome sites.

Through my work at Hitwise, our success rate is a bit higher than that but it could mean we’re not pushing the envelope far enough. On the other hand, the websites we test on are not exactly as advanced as MSFT or AMZN, so typically it’s easier to setup tests that will work. We also tend to research our test ideas and prescribe what has worked for past clients.

How hard should you push the envelope?

If you’re seeing a high rate of success from your tests, perhaps you need to start looking outside the box and broadening your tests’ horizons. On the other hand if you push it too hard, you risk slowing down the results (or screwing things up), whilst potentially finding massive, game-changing ideas.

Alternatively, you could also do a bit of research and start to understand your visitors before you prescribe “solutions”.

Sources: Ron Kohavi, Visual Website Optimizer.