sd

If you are starting out to build a product, you will hear these words every day — AARRR, Cohort, CAC, CLV, Conversion rate, etc. Most of you are probably curious about using the proven ideas such as ‘set up to measure your KPIs,’ ‘if you can’t measure then you can’t improve,’ ‘make data-driven decisions.’

But, I have good news, you don’t need to implement everything at once. With this playbook, you’ll know when and what metrics to apply at the various stages of the product and the company.

If you are building the product for the first time, this blog gives you the framework and tools to implement product analytics into your product. And if you are building a product for the Nth time, then this blog refreshes the framework and tools to integrate into your product.

A product can be in one of the following stages:

Idea
Version 1
Version 2 to Product-Market Fit
Growth (Post Product-Market Fit)

Product Analytics at the Idea Stage

Product analytics at the idea stage

Source: unsplash.com

When you are at this stage words such as AARRR, Cohort, CLV, etc. are not useful for you. The one thing you should care about is whether your idea has merit. There are two ways to figure that out.

Qualitative feedback: Run interviews

The obvious thing you will do is talk to friends, family, and colleagues and listen to what they have to say right?

Their feedback is not the real feedback you want. You have to go beyond your immediate circle. You have to get your idea out to the people you don’t know. Reach out to people in the problem domain and start collecting feedback. Even better, start cold calling, or cold emailing the target users and see if you can get them to open your email, take your call, and then listen what their problems are.

You may have a question. Don’t I need the product to sell right now? You are not alone if you have that thought. Many people have thought so.

A smarter way to begin is to start selling before building the product. Gather positive market signals as early as possible. You will be surprised how far you’ll get without building anything. Remember, you are at the idea stage. Sharpen the idea before you start using all your funds to develop the product.

Another question you may have is how many people should you talk to?

I suggest you talk to 8-10 people that you don’t know who is your target audience. After you talk to 8-10 people, you’ll start noticing that people are repeating things. You may want to stop there. You will have learned 80% from 8-10 people. That much is enough; you don’t want 100% accuracy. Rather than spending another 4x effort to learn additional 20%, 80% is good enough.

Here is the summary of steps you should follow:

  1. Define the target audience
  2. Define a problem and a solution
  3. Set your hypothesis and success criteria
  4. Talk to 8-10 people that you don’t know and that are your target users
  5. Synthesize your result, by discerning if it meets your success criteria
  6. If yes move on to next step, if not iterate

Quantitative feedback: Run a smoke test/landing page

The best part of the qualitative feedback step is that you learn to define the problem and solution in the language that resonates with your target audience. The purpose of this step is to validate whether the conclusion you reached from a small sample size accurately represents the bigger market. The best way to do that is to run a smoke test/landing page and measure the conversion rate.

The question you may have is how to get people to the landing page.

  1. Run google or facebook ads
  2. Write a blog post and distribute on channels such as Quora, Medium, Reddit, or where your target audience hangs around
  3. Email Campaigns

There is no one way to do this. You will have to experiment with the channels and figure out what works best for you.

For the sake of moving forward, let us assume that you have traffic to your website. You have written the value propositions, and there is a clear call to action button. People click on the button. They enter their email address and name and submit the form.

Measure the following metrics:

  1. Measure the conversion rate
  2. Measure the sources of the traffic

The question you may have is what the conversion rate you should target? It depends on your market. If you have a small sample size visiting the website, then around 20% is good and higher than that is better. But if you have a huge sample visiting the website, then 5%-10% is good. Sample size means that you are targeting to the people you need this product. Huge sample size means that your marketing is so good that even those who don’t need your product end up visiting the website.

Get in touch with people who have entered their email address and name,  and listen why they are interested in the product. Don’t worry if many people don’t respond. People have their lives to live. Only for you is your idea your life.

When you talk to people, focus more on their problems and less on the idea you have in your head. I have mentioned in the customer interview section what types of questions to ask.

The plan sounds good for a consumer product. Will it work for B2B products?

I’d say two things. First, it works. Second, if it doesn’t work, you will still need a landing page anyway for the optics of looking legitimate when your sales guy navigates the complex system of the enterprise sales. The exercise will help you crystallize your value proposition.

Another question you may have is what if I don’t get the conversion rate that I have intended?

In that case, find out if you are above or below your market average. If you are below, change your value prop and repeat or change your target audience and repeat.

Remember that you are looking for signals that the market for your product exists. Selling before building is excellent advice. Also, you are at the idea stage. The sooner you start investing in the distribution channels, the better it is. Don’t wait until you build the product. You will need inputs from this step to shape your product. If you do these, then you have set up the foundation and mindset for validated data-driven decision making.

Finally, useful tools for you:

  1. Google Analytics
  2. Unbounce, Launchrock, WordPress, Wix, etc. for website building without coding
  3. MailChimp for email campaigns

Let us move on to the next step.

Product analytics for the first version of the product

Product analytics for the first version of the product
Source: unsplash.com

You validated the idea has merit, talked to a few users, perhaps even sold the idea. It’s time to build the product. Does that mean coding? Not yet. Now, we are moving to the details of the product.

By now, you’ll have the list of features that you want to build. You prioritize those based on the riskiest assumptions, the most value, and the highest return on investment. I won’t talk much about that. It will require a separate topic. For the sake of this blog, I will assume that you have the list of features for the minimum viable product. You will focus on building the minimum value product based on your prioritized features that delight the user.

Here is how you can use the analytics at this stage.

Qualitative Feedback: Customer Interviews and Usability Testing

Customer Interviews

At this stage, you will have a mockup of the minimum viable product. I suggest starting the building the mockup in the following fidelity order:

  • Paper Based Mockup

     Tools: Of course Pen and Paper, you dummy. ☺

  • Low-Fidelity Clickable Wireframe

    Tools: Powerpoint or Keynote, or Balsamiq. I use Balsamiq

  • High-fidelity Clickable Prototype

Tools: Figma, Sketch, Invision

Run a customer interview in each fidelity and move on to the next step only after you have results from the previous level. I say that because ideas develop and evolve as you talk to the users and lower the fidelity the faster it is to iterate.

What type of questions should I ask?

When you run the customer interview, you want to understand whether the product solves the user problems. That’s why you want to focus on the behavior of the user than the attitude of the user. You can ask questions such as “when was the last time you had that problem?”, “what were you doing?”, “how did you do solve it?”,”describe your process and your experience solving that problem.

Remember again; the product is about the user it is not about you. Avoid asking questions such as “Would you use it?” , “Would you see someone using it?”. You would rarely hear ‘no’ to these questions. If you are asking these questions, then you are just flaming your vanity. That won’t help you.

You may ask, should I interview in-person or remote? The best is to do in-person. If in-person is too expensive, then remote is the option. If you run remote interview use tools such as zoom.us. If possible, take video calls.

‘Wait! Isn’t this blog about analytics? Until now I feel like I haven’t collected data. And what I am doing is unscalable. I want to have data to take my decisions?” You are not alone if you have this thought. But what you need to remember is your stage. You are in the first version of the product. Therefore, you want to invest your time in why people use your product before you build the product.  And if you run the interviews well, I promise you will learn a lot.

One tip: run at least five interviews with the same script, i.e., ask the same set of questions to 5 users before changing them because you don’t want to change the direction because one person said so. You want to check whether it is just one person’s experience or it is universal.

Second tip: record the interviews taking notes. You want to record for two reasons. First, you want to evaluate how did by replaying the interviews. Second, you want to listen to the users again so that you don’t miss what they said.

Iterate at this stage for a couple of rounds. You will surprise yourself how fast your idea evolves and how quick you sharpen your MVP.

Usability Testing

In the customer interview stage, you focus on learning why users use your product. In this stage, you focus on how they will use it. Another way to describe it is that the customer interview is to identify how valuable it is to the users, and usability testing is to determine how usable the product is. Your product has to be both— valuable and usable.

You create the clickable prototype using tools such as Invision. You don’t tell the users that its a prototype. You fake it. Then you request them to speak out loud as they use it. Again, focus on what they do and what they say. Sometimes they may be different. You measure whether they can read your value props clearly, whether they click on the click on the action button, how easy they navigate from one step to the next, and get back to the previous. Identify the friction points.

As you watch them using it, your heart will cringe, and you will be impatient. You will want to put the words in their mouth and hold their hands to click on the places you want them to click. The more that happens, the more you know that your product has usability issues. Act a bit. Avoid showing your emotions. Otherwise, you will inject your biases into the user’s mind.

One tip: Test with the real users. That’s why I asked you to start selling first or running email campaigns so that you will have real users to do customer interviews and usability testing. Otherwise, you will have to settle for paid users. I still don’t know what to make out of paid testers, they are paid to give the test, and their job is to test it simulated environment and share their opinion rather than provide real answers whether product solves their problem or not.

Quantitative

You can create a demo video out of the product and put it on the website and measure its impact on the conversion rate. At this stage, you will have a polished video. You will have language your users speak to define the problems and the value propositions. Of course, you will change the copy of the website.

You test with the more number of target audience what you build works or not.

Now repeat what you did in the idea stage. Measure the before and after the conversion rate. Your conversion rate should go up. If it doesn’t improve, then you rinse and repeat the previous stage.

You can install tools such as www.fullstory.com to watch how people use the website.

You can install Mixpanel to measure conversion rate and measure the funnel that leads to the conversion rate.

Second tip: Run 8-10 usability tests. It should take less time than customer interview for each usability test.

In the customer interview stage, you focus on learning why users use your product. In this stage, you focus on how they will use it. Another way to describe it is that the customer interview is to identify how valuable it is to the users, and usability testing is to determine how usable the product is. Your product has to be both— valuable and usable.

You create the clickable prototype using tools such as Invision. You don’t tell the users that its a prototype. You fake it. Then you request them to speak out loud as they use it. Again, focus on what they do and what they say. Sometimes they may be different. You measure whether they can read your value props clearly, whether they click on the click on the action button, how easy they navigate from one step to the next, and get back to the previous. Identify the friction points.

As you watch them using it, your heart will cringe, and you will be impatient. You will want to put the words in their mouth and hold their hands to click on the places you want them to click. The more that happens, the more you know that your product has usability issues. Act a bit. Avoid showing your emotions. Otherwise, you will inject your biases into the user’s mind.

One tip: Test with the real users. That’s why I asked you to start selling first or running email campaigns so that you will have real users to do customer interviews and usability testing. Otherwise, you will have to settle for paid users. I still don’t know what to make out of paid testers, they are paid to give the test, and their job is to test it simulated environment and share their opinion rather than provide real answers whether product solves their problem or not.

Second tip: Run 8-10 usability tests. It should take less time than customer interview for each usability test.

Survey

The survey is another method to validate whether your MVP features are valuable or not. But I prefer landing page to surveys. However, I don’t dismiss them as useless.

You can use the survey to get quantitative numbers before you build the website. You want to focus on the behavior rather than attitude. You can ask questions such as: When was the last time you faced this problem? How frequently you have the problem? How do you solve the problem? How much time does it take to solve the problem? How much does it cost to solve the problem? Rate how valuable is the current solution, rate how satisfied are you with the current solution, etc.

I use google forms to do the survey. It is quick and easy. It’s cheap. There are other tools as well. For me, it works fine.

Unscalable Analytics

When we hear about analytics, we assume that we must implement scalable things from the early stage. I recommend doing the opposite. At an early stage, you want to focus on learning why users use the product, learn their languages that they define the problems and solutions, and understand their behavior. When I say unscalable, I mean the data points you enter using your hand rather than getting from the software.

Product analytics from the version 2 to the product market fit

Product Analytics from the version 2 to the product market fit
Source: unsplash.com

Launch version 1. You have data. You have moved from qualitative data to quantitative data. You have the real usage data. Before this step, your data from customer interview, usability data, smoke test, and survey looked somehow unreal. But those data points pave the way to gather real usage data. You want to measure the following metrics and optimize them in the following order.

  1. Acquisition
  2. Activation
  3. Retention
  4. Revenue
  5. Referral

Measure your acquisition from several distribution channels

The most likely post v1 launch is that you won’t see users signing up in huge numbers. The hockey stick growth after the first release is a glamorized media storytelling. It almost never happens. You will need to grind and hustle to get even a small activity on your website. So don’t be disheartened if you see very less activity. That’s why you need to measure your distribution channel and double down on the distribution channels that work for you. Remember to do the unscalable things.

Some of the things you will notice are if your marketing is excellent you will have a low activation rate. If your marketing is weak, then you will have high activation rate. Let me explain. If your marketing is excellent then more people— even people who don’t need your product— will visit the landing page. If your marketing is weak, then only people who search organically and need your product will visit the landing page. So you also have to look at the absolute number of people who visit your landing page. If you have ten people who visit the website and eight people activate then the activate rate is 80%. That doesn’t tell the real story.

So, you want to increase the acquisition to statistically significant sample size. Say 1000 or more. If the sample size is small, then statistical analysis will produce a skewed result. That sample size depends on the deal size. Your deal size is an ant, rabbit or an elephant? The sample size inversely proportional to the deal size. Consumer products need high absolute number visiting the landing page whereas B2B need low because of how sales engine operate.

Define your activation

Is it when someone signs up or is it when someone does some activity in the product?

After you achieve a statistically significant acquisition at a reasonable rate, then you optimize the activation rate. I use the word reasonable because it depends on the type of market. B2B will have high activation rate in comparison to the consumer product.

Define the retention that makes sense for your product

It may be daily, weekly or monthly. Our reading list is full of stories from Facebook, Google, Amazon, Apple. So we have a bias that all products should have high daily activity. Most products don’t require daily usage. It depends upon the frequency of the users and when they need their problem solved. When you define retention, please keep in mind the frequency of the user problems. For example, if your user does resource planning every Monday of the week, and your product is resource planning software, then you need measure whether user returns every Monday. If you measure daily activity, then you have the wrong metric. For this reason, I suggest iterating on the customer interviews to understand the triggers, frequencies, and environment in which your users need to solve the problem.

Measure revenue

The most important metric to measure value. If people use every day, but they don’t want to pay for the product, then your product is a dispensable utility. You haven’t achieved the product market fit until your users don’t want to pay. I am not going to talk about consumer products where users are the product. ( For Facebook, Google, Twitter, Instagram, WhatsApp users are the product). For products like those, referral matter more than revenue.

Measure referrals

Your product can be for a single user or multiple users. Even if your product is single-use, if your users refer the product to their friends, then you know that you have achieved the product market fit. If you have users, who pay and refer then that’s the ultimate place you want to be.

Of course, that’s not going to happen as soon as you launch. You will have to iterate based on the metrics defined above.

Tip: Think in the flywheel. Don’t focus on improving one metric. For example, don’t focus on improving acquisition. After you have the significant sample size, focus on improving the activation rate, and move on the next metric after you reach the reasonable rate.  You can iterate on the product only when you have this flywheel running. You will get close to the product market fit.

As you iterate more, you want to do further analysis on the cohort, customer acquisition cost, and so on. I will talk about those in the next stage.

Product analytics for growth (Post Product-Market Fit)

When you have the flywheel smooth and fast, then you know that you have achieved product-market fit. There are not many foundational metrics to talk about here. I have already mentioned above. You continue doing everything you started with the step 1 to step 3. That is how you will know what feature to test and build next. You will identify usability issues, growth channels, and custom features for your cohorts.

At this stage, you will do more quantitative analysis than qualitative analysis. If you have a feature in mind, then you can test quickly by building the facade of the feature. You focus on optimizing each step of the AARRR metrics.

You want to make sure it decreases with time. Think about incentivizing users to refer to their friends. Word of mouth is the best user acquisition channel. You want to instrument that in the product. To achieve that, you want to play around with the product a lot. You will test the user experience of the product. You will add new features to increase the value of the product.

It makes sense to talk about these assuming that you have a sufficient number of users to do a statistical analysis that produces statistically valid results. Many people try to do this at an early stage but waste their effort. There is a time to do the following, and that time is when you have the statistically relevant sample size.

A/B testing

You can use A/B to test the usability of the product and test new features. Put two versions and check which one will yield a better result.

Tip: Be cautious that an increase in one metric doesn’t decrease another metric. So think about pairing the metrics that negative correlation. If there is metric A and metric B, then you want to set a target to improve metric A and set the acceptable decrease in the metric B.

You want to test new features then you can add a button or link to the product and check whether users will click on it or not. After they click on the link or button, you can show that this feature is a work in progress. You can meanwhile do a customer interview for those who clicked on the link or button and listen to them why they clicked on it and what did they expect after they clicked on it. Don’t tell them what you are building and let them define their expectation.

Cohort analysis

At the growth stage, you want to be specific while targeting your users and positioning your product. To do so, you want to segment the users. That’s where the cohort will come. You can segment based on the engagement and find how to improve engagement for those who are not engaged. For those who refer more people, find out why they do that and see whether you can get other users do the same. It is an essential tool not just to build new features but also to channel product positioning and storytelling.

Conclusion

Therefore, you want to know the stage you are in and invest in the product analytics that is relevant to your phase, i.e., qualitative and unscalable at the idea and v1, and quantitative and scalable at the pre-product-market fit and post-product-market fit.

There are more metrics to measure. I won’t discuss that in this post because those metrics depend on the product type such as consumer, SaaS, Marketplace, Social networks, and so on. However, analytics playbooks, processes, and tools are fundamentals to building every type of product.

If you are looking to build the product and you need a full stack product engineering consulting and development then contact us. Or if you have to talk in depth about incorporating the product analytics in your product then reach out to us. We will be happy to chat.

Read our previous blog:

Playbook to build a defensible product


Also published on Medium.


Share this blog

More awesome Augmentation awaits...