Skip to main content

4 ways you are probably doing Product Analytics wrong

An article that I read recently stated that 41% of businesses struggle to turn data into decisions. This got me thinking about my own experience with product analytics.
Over the last two decades, the Web has transitioned from being a space for the hobbyists to one where billion dollar businesses are made.
This transition of Web made way to another kind of transition. What used to be Web Analytics, evolved into what we now know as Product Analytics.
In their glossary, Optimizely defines Web Analytics as:
Web analytics is the measurement and analysis of data to inform an understanding of user behavior across web pages.
Product Analytics, on the other hand, is a combination of quantitative and qualitative insights that inform product decisions.
For these purposes, you may find yourself looking at different kinds of data, ex:
- user interactions with the website and apps,
- behaviour patterns such as retention and churn,
- purchases and cancellations
- u
ser feedback and Net Promoter Score (NPS)
In my product journey as a PM for Vistara (Tata Singapore Airlines JV) and FlixBus (one of Europe’s largest online travel portal), these are the 4 common mistakes ( with solutions) that I have observed when it comes to Product Analytics —

1. Blind trust in the tool is dangerous

First things first — having the best tools at your disposal means nothing, if:
- You have no idea why, where and how the data is being collected;
- You don’t use the tool to actually deep dive into the data to gather insights;
- You never question the correctness of the data you see;
- You are sure of the correctness of the data but not what it says about your product.

In the end, any analytics tool is only as good as the the people using it.
To move into the direction of product analytics, once you have set-up the tool(s), ensure you have done data sanity checks, and actually start using the tool but with a curious and a critical mind-set.

2. Not defining success early on

One of the first lessons I learnt very early on is to clearly define what success looks like for your product.
It is possible that your ideal success metric is not the same as the organization’s success metric. As long as your success metric feeds into the overall organization’s success metric, you are doing fine.
Say you are a product manager responsible for the Search Listing page in an eCommerce organization in fashion domain. A true success metric for your organization could be conversion rate (total orders/ total sessions) and average order value (Total revenue/Total orders).
However, your success metrics could be Progression rate to the next step in the ideal customer path to conversion and avg. basket size.

While selecting a success metric, I follow these 3 steps -
1. Identify an actionable core metric for the product that feeds into the organization’s goals.
2. Choose a supplementary metric, ex: Avg basket size along with progression rate
3. Apply the “So What Test”

3. Missing Issue Trees/ hypothesis driven analytics

We all know that gathering insights about product usage is important for product development. Then why do most of us give up on digging them out?
After having spoken to many of my peers, I have narrowed it down to 1 thing: Data can be overwhelming.
However, the problem often is not the amount of data or shortage of time at hand. It usually is the lack of a hypothesis driven approach, or a preference for, what I call, diving-in-head-first approach.
Let’s say your daily email dashboard shows a 5% dip in Day over day conversion rate (CVR). What do you do?
If your first instinct is to open your favourite analytics tool — Stop! Take a breath. Grab a pen and paper and try to answer — 

What is the impact on Y if X changes.

Y being CVR in this case, and X could be any number of factors that you think could impact CVR.
Example Issue Tree/ Hypothesis driven analytics
Once the list of factors is in place, start plotting an issue tree.
This might seem like a lot of work the first time, but trust me, you already know the factors by heart.
By following this approach, you will be able to narrow down the exact problem that could be causing the dip in CVR and you won’t have to dig into 500 different reports to find clues to this not so big mystery.

4. Not combining qualitative with quantitative

My biggest learning in the realm of Product analytics came from House of Cards. In season 1, episode 12, Raymond Tusk says - 
Decisions based on emotions aren’t decisions at all. They’re instincts, … which can be of value. The rational and the irrational complement each other. Individually they’re far less powerful.

By bringing quantitative and qualitative together, you can triangulate the solution and have greater confidence and richer insights for product development.
One way to combine the two approaches is to use Qualitative methods like user research to formulate hypothesis that you can validate by using quantitative approaches like A/B testing.
By combining the forces of Quantitative and Qualitative methodologies, you can answer the WHY with an evidence-based decision making approach rather than only answering the WHAT by sticking with data-based decision making.

We all appreciate good product instincts in our fellow Product managers, but I can’t agree with Alistair Croll more when says in his book Lean Analytics,
Instincts are experiments. Data is proof.

And in the end, as my dear friend put it once, if you’ve never enjoyed a detective story, if you’ve never spent hours in front of a data anomaly and felt a rush of “guilty excitement” trying to solve a problem, consider doing something else, something that gives you pleasure.
Further Reading -


Popular posts from this blog

Probably one of the most simple and extremely effective tools for managers

One of the topics that I discussed with quite some mentees via the Mentoring Club is how to manage managers and what tools and frameworks I would suggest in order to build strong relationships with everyone in my team.
One tool that I established in my teams last year is a "User Guide" for working with me. Yes right: I wrote a document that describes several aspects of how I would like to work, about my expectations but also my flaws. I gave this document to all my team leads and senior team members as a reference. 
With this article I publish the contents of my 'user guide' together with some explanations. You should take your time to reflect about what is important for you. List everything that comes to your mind, try to get to the root of everything. In a second step put yourself into the shoes of whoever works with you. What might be important to them? Why should they know? What do you want them to be 100% aligned with? What would they need from you in order to m…

9 perspectives on why the corona pandemic will have a positive impact

The Mentoring Club is an initiative that was founded when the corona pandemic hit the german and european business world. The two founders (Jessica & Bastian) actually started this initiative because their employer needed to reduce the amount of work for almost everyone to massively reduce costs. In just a few days there were many other people who asked to join and in not even two month time The Mentoring Club counts almost 70 mentors from around the world, who already donated about 200 hours of free mentoring, support and counseling to others. 

One positive side effect of corona is indeed that the humanity as a whole and the business world showed a very positive and humble face. We asked our mentors what they think how the pandemic might have a positive effect on businesses and organisations. Read here what nine of them think. 
Do you agree? Do you see other aspects? Please add as a comment below!

"In CORONA times, even the most conservative managers were pushed to try remote w…

Let’s talk about women in leadership

By Monique Zytnik
I’ve been inspired by the amazing female leaders on the global stage who have coolly and effectively handled the recent crisis and are still negotiating, supporting and inspiring others as their countries head on the road to recovery. I see these leaders as role models.However, looking at numbers of women in leadership roles last year we still have a way to go to achieve fair representation. This year the outlook is grimmer. Many social commentators fear that women have just taken a giant leap backwards and under the increased family and social strain are less likely to put their hands up for leadership roles. What do you think?