“Measuring in particular … is often based on ethics and values rather than effectiveness. Due to heavy investment in projects through personal relationships it can be hard to be ‘objective’ and use SMART goals as you know you should.”
Helen, Madly in Love Project Manager, YouthNet
Third sector decision making is often driven more by ethics and values than hard evidence. Given the importance of our values base this is in some ways understandable. However in tech projects values driven decisions tend to be weaker and lead to poorer end products. Data driven decisions are far stronger.
However, there are so many things you could be measuring that knowing what to measure and when throughout your product’s life cycle can be difficult to work out. Nominet Trust have produced an excellent guide to the different metrics to consider at each stage. Highly recommended.
Measuring During Product Development
All of the advice we’ve provided in this guide is geared towards setting your project up for crunchier and leaner decision making based on evidence. Examples of evidence include customer discovery findings, test results, prototype feedback, unsolicited problem descriptions, device usage statistics. Verbal reports of wants and solutions make for weak evidence while verbal reports of problems are strong. We strongly recommend reading the Lean Startup or attending a Lean Startup Weekend workshop. Lean’s principles are heavily focused on finding the right thing to measure and the appropriate metric to use when designing and iterating your product.
Lean methods aren’t a panacea for every app or service design problem but they are a solid foundation that will quantum leap you beyond the world of focus groups and vanity metrics and into the world of measurable impact, validated learning and smarter decision making.
Measuring Live Products
If you ran a restaurant you wouldn’t check with the maître to see how many diners there were each month, you’d be checking by the hour and getting feedback on what was consumed. While you may only be changing the menu and prices occasionally those decisions would be made from a deep understanding of regular user habits, gleaned on a regular basis.
Measuring Impact
One of the many things digital products and services are good at measuring is use. However, what they are often far less good at is in telling you what effect that use is having. If people are using a service you would expect them to be gaining some underlying utility – in this case an improvement in their wellbeing – but proving this is incredibly difficult without resorting to traditional research methods – questionnaires etc, which are only a very approximate tool.
Measuring Use
Many technology projects will cite visitors, sessions and ‘conversion rates’ as KPIs. These three variables provide a sense of engagement and popularity. However, these can easily become ‘vanity metrics’ as Dan Barker states in his excellent post conversion rates are often used uncritically and need to be considered with other variables as not all visitors will convert equally, higher visitor numbers and lower conversion levels may still be more beneficial and a more engaging site may convert less well (as people visit more often but don’t action what you want them to – they may just be coming to read your blog).
Careful consideration of what metrics suit the project are needed and it’s worthwhile making sure each project really understands how to measure and analyse data – and if they don’t then to find someone who does.
Tools
Analytics tools for web and mobile applications ought to be built into the way your team works so they are always there to help you make ongoing development decisions. iOS and Android markets provide data on mobile applications and services like Flurry can generate more detailed data. For web based projects Google Analytics is an incredibly powerful tool for monitoring a range of metrics as well as testing different ideas and experiences during early design phases. Google has its own Analytics Academy which takes you through step-by-step videos of how to use it.
Read more…
- Nominet Trust Guide to Lean Social Impact Measurement
- Measuring the Growth of your Social Tech Business
- The Lean Startup Book
- Lean Startup Machine Weekends UK
This article is taken directly from ‘Measure Lean and Smart’ a chapter of Learning from the Labs: How to fund and deliver social tech for charities and social enterprises.
