nethy canning   Oct 09 2012   Flag

Nice article. I like blogs that present their experience as their experience (basically anecdotes) as opposed to trying to build some theory of everything based on it. A few notes: - $252k on consultants sounds like a lot of money. That doesn't mean it's not worth it, but it would be interesting to hear what he think of the decision to spend this all/mostly on consultants. Why not employees. My feeling on this is that employees (a) know the business itself more, which can bring insights & (b) bring knowledge in. Having highly ranked employees responsible for testing can help spread the idea of testing throughout an company. - Multi Variate doesn't work. This is my experience & I've heard it from other as well. Strangely, the reason it doesn't work doesn't seem to be different every time. Technical problems. - Validate results with another test, preferably using different tools. Especially for important decisions (eg price). Through human or machine error, tests can be wrong. - A "testing tool" is a tool. Sometimes you use it exactly as it says on the box. Sometimes you don't. Some important questions will require multiple tools (including your brain) to get answers to. Databases, excel, crm, google analytics, coding. You may have to bang together a toolset yourself for important tests.