The only way to be fantastic at creating products that customers love and keep on loving (which is a core tenant of marketing) is to have a culture of testing in your company.
When a product is young it’s easy to have ideas about how to improve them. It’s easy. There is low hanging fruit everywhere and everyone has an opinion – from your mother through to investors.
But which ideas do you pursue? That’s where leadership comes in, to prioritise and (dare I say it) focus. However, having a priority for implementation is not even half the challenge. How do you know the idea made a difference? Did your mother or the investor now low it 100 times more? Does that matter? The only way you know is by testing.
The ability for a company to be able to implement something quickly is pointless without the ability to see if it made a difference. To begin with it is easier, since things may have been broken and now they work at all, but just because something wasn’t there before and now it is doesn’t mean the overall product is better. In fact most the time more is a lot, lot, lot less.
Testing Culture
Having a culture of testing starts with the mindset and the commitment. You have to want it, because it’s not fun. It takes the inspiration down a peg or two and makes it work for it’s money. It means you spend more time looking at numbers, graphs and spreadsheets and less time working on what’s next. An big failure of many young businesses is that they move onto ‘next’ before ‘just added’ is working. Be careful about the term ‘working’ too. Functioning doesn’t mean that it actually makes the product better and that people care about it. Take Dave’s advice and try removing a feature and see if people kick up a stink. (see the slideshare below). This takes guts and that means strong leadership which forms strong culture.
Testing Structure
You also have to want it enough to make some changes. A culture of testing without the right tools is pointless. Everyone at least uses Google Analytics, but it’s not enough. You need to be able to quickly see if people are talking about the change and loving it. See if it’s changing other behaviour. If more people are coming back. More people are converting. More paying. Paying more. Telling their friends more. Measure everything that is actually really important. Again, see the slideshare below for more on what to measure.
Testing (Your) Patience
One of the hardest things about testing is that it takes time which can frustrate the 1,000 kms per hour entrepreneur. The anal retentive team member needs to step in and pull on the hand brake. Testing takes time. You have to think about it, implement it, let it run live, analyse the results and then you learn whether it worked or not. If you’re agile and making changes every week or two, this is still 3-4 weeks.
Testing Rhythm
The great news is that once you’re team is into testing, the tools are in place and you’ve done a couple of laps/iterations of it, you’ll start to get in the groove. You’ll start to to talk like testers. It goes something like this:
Co-Founder A “Hey, I think if we made the ‘sign up’ button bigger we’d get more sales?”
Co-Founder B “Good idea, let’s test it! Create 3 versions, AB test it, and check conversions.”
Instead of how it used to go:
Co-Founder A “Hey, I think we should make the ‘sign up’ button bigger?”
Co-Founder B “Good idea. Team, stop what you’re doing and make the button bigger.” A day goes by…. “Wow, that new button looks good. OK, what’s the next feature we should add?”
Absolutely no idea whether the idea was any good. Just lots of untested guesses. In fact one thing that a testing culture does is adds a little more discipline to ideas. When people know they will be tested, they really think ideas through instead of throwing them out there.
What to Test
I’d like to write the definitive piece on what to test, but it’s been written. Dave McClure, the dude of startup dudes, has put this together which is worth studying. Don’t just flick through it. Print it out and make notes. Crosses and ticks for what you don’t and do measure.
Tell me how you test
If you’re doing some testing, I’d love to hear what you test, how you test it and how it changes the way you think / run your business. Add a comment below.
Nice post. We’re nailing down a baseline for features in our web/iphone app at the moment. Focusing on less is more which helps time to market and will iterate design/features with feedback/stats once launched.
This speaks to feature “validation” as opposed to just functional testing, e.g. “don’t stop at acceptance testing” if you’re delivering software product. In terms of product the question “is it being used (amp; loved!?) ?” is much more interesting than “does it work?”.
On the subject of setting up a feedback loop on product features — here’s the old $1M question around feature creep in general: If the average user of Microsoft Word uses less than 10% of it’s functionality (true), would Microsoft still have grabbed the same market share if they had been prescient enough to only deliver and maintain that lt; 10% of the product ?
Interesting thoughts. I’ve been somewhat pathologically obsessed by fine-grained unit testing (call me crazy, many do) for a long time – in fact, way before it became cool because I was lucky enough to work with some really amazing people at Object Oriented Pty Ltd in Sydney in the early nineties.
It was lucky in the sense that I was exposed to fine-grained unit testing as part of the normal software process, not something tacked on the end. As my first real job, I just thought that was how you wrote software.
Anecdotes aside, all this talk of testing reminded me of a book that I read when I was at OOPL by Brad Cox called “Superdistribution”. Check out a review here: http://bit.ly/cL0CM6
The point of all this is that one of the main theses of the book is that it was testing that kick-started the industrial revolution (in the US at least) when the Government put out a contract for the manufacture of guns. The contract was open to anyone, as long as they could make the guns according to a public specification, and the specification was tested by the use of a couple of simple tools that were very easy to make, used to check the length and width of the various components (eg barrel, lock, etc).
The parallels to software engineering are obvious.
Cox goes on to make the point that such a testing methodology, and the development of appropriate tools is a necessary first step in taking software out of the cottage industry stage into an era of “superdistribution”.