Skip to main content

4 ways you are probably doing Product Analytics wrong

An article that I read recently stated that 41% of businesses struggle to turn data into decisions. This got me thinking about my own experience with product analytics.
Over the last two decades, the Web has transitioned from being a space for the hobbyists to one where billion dollar businesses are made.
This transition of Web made way to another kind of transition. What used to be Web Analytics, evolved into what we now know as Product Analytics.
In their glossary, Optimizely defines Web Analytics as:
Web analytics is the measurement and analysis of data to inform an understanding of user behavior across web pages.
Product Analytics, on the other hand, is a combination of quantitative and qualitative insights that inform product decisions.
For these purposes, you may find yourself looking at different kinds of data, ex:
- user interactions with the website and apps,
- behaviour patterns such as retention and churn,
- purchases and cancellations
- u
ser feedback and Net Promoter Score (NPS)
In my product journey as a PM for Vistara (Tata Singapore Airlines JV) and FlixBus (one of Europe’s largest online travel portal), these are the 4 common mistakes ( with solutions) that I have observed when it comes to Product Analytics —

1. Blind trust in the tool is dangerous

First things first — having the best tools at your disposal means nothing, if:
- You have no idea why, where and how the data is being collected;
- You don’t use the tool to actually deep dive into the data to gather insights;
- You never question the correctness of the data you see;
- You are sure of the correctness of the data but not what it says about your product.

In the end, any analytics tool is only as good as the the people using it.
To move into the direction of product analytics, once you have set-up the tool(s), ensure you have done data sanity checks, and actually start using the tool but with a curious and a critical mind-set.

2. Not defining success early on

One of the first lessons I learnt very early on is to clearly define what success looks like for your product.
It is possible that your ideal success metric is not the same as the organization’s success metric. As long as your success metric feeds into the overall organization’s success metric, you are doing fine.
Say you are a product manager responsible for the Search Listing page in an eCommerce organization in fashion domain. A true success metric for your organization could be conversion rate (total orders/ total sessions) and average order value (Total revenue/Total orders).
However, your success metrics could be Progression rate to the next step in the ideal customer path to conversion and avg. basket size.

While selecting a success metric, I follow these 3 steps -
1. Identify an actionable core metric for the product that feeds into the organization’s goals.
2. Choose a supplementary metric, ex: Avg basket size along with progression rate
3. Apply the “So What Test”

3. Missing Issue Trees/ hypothesis driven analytics

We all know that gathering insights about product usage is important for product development. Then why do most of us give up on digging them out?
After having spoken to many of my peers, I have narrowed it down to 1 thing: Data can be overwhelming.
However, the problem often is not the amount of data or shortage of time at hand. It usually is the lack of a hypothesis driven approach, or a preference for, what I call, diving-in-head-first approach.
Let’s say your daily email dashboard shows a 5% dip in Day over day conversion rate (CVR). What do you do?
If your first instinct is to open your favourite analytics tool — Stop! Take a breath. Grab a pen and paper and try to answer — 

What is the impact on Y if X changes.

Y being CVR in this case, and X could be any number of factors that you think could impact CVR.
Example Issue Tree/ Hypothesis driven analytics
Once the list of factors is in place, start plotting an issue tree.
This might seem like a lot of work the first time, but trust me, you already know the factors by heart.
By following this approach, you will be able to narrow down the exact problem that could be causing the dip in CVR and you won’t have to dig into 500 different reports to find clues to this not so big mystery.

4. Not combining qualitative with quantitative

My biggest learning in the realm of Product analytics came from House of Cards. In season 1, episode 12, Raymond Tusk says - 
Decisions based on emotions aren’t decisions at all. They’re instincts, … which can be of value. The rational and the irrational complement each other. Individually they’re far less powerful.

By bringing quantitative and qualitative together, you can triangulate the solution and have greater confidence and richer insights for product development.
One way to combine the two approaches is to use Qualitative methods like user research to formulate hypothesis that you can validate by using quantitative approaches like A/B testing.
By combining the forces of Quantitative and Qualitative methodologies, you can answer the WHY with an evidence-based decision making approach rather than only answering the WHAT by sticking with data-based decision making.


We all appreciate good product instincts in our fellow Product managers, but I can’t agree with Alistair Croll more when says in his book Lean Analytics,
Instincts are experiments. Data is proof.

And in the end, as my dear friend put it once, if you’ve never enjoyed a detective story, if you’ve never spent hours in front of a data anomaly and felt a rush of “guilty excitement” trying to solve a problem, consider doing something else, something that gives you pleasure.
 
Further Reading -
  1. https://hbr.org/2012/09/metrics-are-easy-insights-are-hard
  2. https://www.computerworld.com/article/2984988/all-your-big-data-will-mean-nothing-without-systems-of-insight.html
  3. https://spotify.design/articles/2018-12-05/cross-disciplinary-insights-teams-integrating-data-scientists-and-user-researchers-at-spotify/

Comments

Popular posts from this blog

What it takes to move to Product Management from another function successfully.

When I reflect on all the mentoring sessions that I have had so far, I can say that the different sessions couldn’t have been more colourful. I tested several early stage versions of apps, I brainstormed on new ideas, I have shared career advice and I spoken to many product managers who are in a similar state as I am and we exchanged experiences and learned from each other. Nevertheless, I do see some patterns of topics, which occur regularly and I want to take one of those today and write down my thoughts for more people to read. The topic that I chose for this blog post is around how to transition to product management from other disciplines. Spoiler alert: it is possible, always. But it requires endurance, passion and commitment. Tl;dr  For me, far and foremost it is the passion to solve customer problems, which means to be 100% customer centric and have the ability to put yourself in your customers shoes at any given point in time. The more empathy a product manager has towards the

12 surprising facts about what happened since The Mentoring Club started

Taming Advice Monsters While Mentoring

Photo by Yaopey Yong on Unsplash Let me introduce you to the advice monster that I got to know in Michael Bungay Stanier’s TED talk “ How to tame your advice monster ”. Imagine someone asking you for advice. Before they even finish explaining the problem, your advice monster awakens already waiting for its turn to burst out all the brilliant advice. That advice monster is convinced that you know the situation well even if you may not have the full context at all.  When we think of a mentor, we often expect someone who is capable of giving great advice. We may also become mentors because we feel we have enough experience, passion, and ability to advise people. However, mentoring is much more complex than simply providing advice. And, mentoring environments usually are extremely satiating spots for advice monsters. For a better mentoring experience we need to tame our advice monsters. Mentoring requests often start with a small description of a person’s challenges and often it ends with