Let me start by getting something out of the way right here and now:
*EVERY metric looked in isolation is a vanity metric*
Tweet ThisThink about that… 300% increase in signups, 50% increase in traffic, 10% increase in MRR….
All these mean nothing unless you give them context and compare them to other data.
An increase in signups doesn’t necessarily mean an increase in paying customers and so on and so forth.
Nowadays, every startup wants to be data driven, yet with all this data available to us, what metrics and KPIs should we pay attention to and what is noise?
It’s a tough question to answer and it depends a lot on who wants to to view the data (different data for sales, marketing, the CEO etc), as well as the stage the startup is at.
However, I will go ahead and take a stab at it mapping out some guidelines for distinguishing between vanity and actionable metrics for SaaS growth and marketing.
Let’s start with defining what a vanity metric is:
What is a vanity metric: a definition
Vanity metrics are the metrics that are easy to track, look big on the surface and of course, make us feel good, but don’t give any real insights into the business and its performance.
These metrics tend to look good no matter what you do and have neither direct impact nor serve as a close proxy for growth, profit or MRR.
An example would be total customers ever (not necessarily active) since the startup’s inception.
This is a vanity metric because no matter what you do, the number will keep increasing as you acquire paying users, regardless of your churn and it provides no insights in your business.
Ok that was pretty obvious, but another one, less obvious vanity metric is CTR.
Before we move on to see some common vanity metrics that you must avoid, I have to point out something *very important*…
The metrics tracked in a startup, flow from the heads of the company or the department.
This means that if the CEO or other people in managerial positions evaluate the performance of the growth team based on vanity metrics, then employees will work to improve those vanity metrics and not necessarily growth. It really makes sense…
It is the job of the founders and VPs to instill a culture where true growth and not vanity, matters.
Also, you need to pay attention to both percentages/ratio changes and to actual numbers.
For example, you need to know your MRR growth both in % and actual $$ numbers (i.e. 10% growth in MRR, which is $10K extra this month).
Anyway, let’s see some common vanity metrics…
Examples of vanity metrics in B2B SaaS
These are some popular metrics that I have found and cross-checked with other experienced marketers to be vanity metrics in B2B SaaS startups:
1) Traffic
I know, this sounds crazy at first sight, but traffic is the ultimate, most common vanity metric. Why?
Well if you are a media site (blog, online newspaper, etc) of course traffic and pageviews are your lifeblood.
But, if you are a SaaS you absolutely must care much more about conversions, MRR and churn and much less about traffic!
For example you might have over 100,000 monthly visitors, but if you haven’t optimized the flow from visitor to becoming a user of your software, it’s useless bringing in more traffic.
Also, having substantial visitors means nothing if they aren’t your target customers or if they don’t try your product.
Traffic just for the sake of it or thinking “If a lot of people visit our site some will convert”, is useless, dangerous and a distraction.
Of course, if you lack traffic go get it, but it’s better to focus on the conversion funnel and users.
2) Active Users
Active users is the number of users (or accounts) that are paying you. In essence, they are your customers.
This is a vanity metric, because unless your churn is higher than your growth, this number will keep rising no matter what you do!
However, it’s good to have a rough number of your active users in mind, since certain growth hacks and marketing activities need a minimum number of paying users to have a possibility of being successful.
3) Email Subscribers
Yes I said it. The number of your email subscribers is the most overvalued vanity metric in existence.
The amount and intensity of some people suggesting us to get huge amount of subscribers is off the charts!
Justin Brooke has said it before me and I will echo his voice:
You can have hundreds of thousands of subscribers and make peanuts.
You can have hundreds of subscribers and make 6 or 7 figures.
It’s all about the quality of the subscribers, trust & value, their engagement and if they want to hear what you have to say.
Take for example WeeklyGrowth which is my personal blog.
I have seen what kind of people share my posts, follow me on Twitter and subscribe to my biweekly newsletter. They are high quality people (and some pretty influential ones) from serious startups and companies, who open my emails and if interested click to read the whole article.
That’s why I don’t write insanely simple. I don’t need to increase my readership and email subscribers in lieu of their quality.
The same goes for your SaaS. Gear your marketing activities towards getting subscribers that match the profiles of your ideal customers.
Do you target busy C-level executives?
Don’t write huge blog posts just because they (might) get ranked higher in search engines and get more shares! Write meaty posts that your busy audience will read, appreciate and subscribe to learn more!
4) Ad Impressions
Of course ad impressions is among the standard vanity metrics reported mainly in bigger corporations. They are nothing to get excited about.
However, when looked together with other metrics, you can evaluate the performance of the ad and the angle you are using.
Every metric can be a vanity metric
As I said at the opening of this post, conversions, paying users, ARPU (average revenue per user), traffic, MRR and virtually everything can easily become a vanity metric if you don’t combine it with more metrics.
Let me show you 2 examples in some detail:
– “MRR grew by 50%”
Scenario: MRR might have been $1,000 and we got 50% by spending $10,000. Not a good move…
– “Conversions increased by 30% this month”
Scenario: we might have brought less traffic, more targeted, but with lower budgets, so we got more conversions percentage wise, but a lower actual number and even lower average transactions per user.
You see where this is going…
One of the things an experienced growth hacker has to do, is constantly be looking at all the metrics, at the big picture of the business and evaluate any new marketing activities that might have taken place according to how they influenced the metrics.
Actionable Metrics
We saw the essence of vanity metrics and how even good metrics can become vanity metrics just by being viewed in isolation or by being manipulated.
Let’s see now some actionable metrics and how to use them:
1) Cohort analysis
Although it’s more of a tool than a metric, it’s very handy at seeing patterns usually in Churn or MRR.
I have used it only for Churn, so I can’t talk about MRR.
Here is an image from ChartMogul on how a typical churn cohort analysis looks like:
I am not going to go into much depth in this post, but what you want to look at in the chart above, is not each and every cell. You want to look at the big picture and see mostly the red boxes and identify patterns.
For example, you can see that in months February and April 2014, after 4 months churn is very high. Why is that? Did some product features get implemented or stripped that increased churn? Something else?
You try to identify the cause and see if you can influence it in some way.
As I said, I won’t go into much depth in this post, but feel free to check this very useful pdf from ChartMogul to learn more about cohort analyses and this post from RJMetrics about revenue cohorts.
2) Comparisons
What I find very useful looking at on a daily basis is a chart of the past 12 months that has the following columns:
– Month
– Free trials (or free signups if you have a free version)
– New customers
– Signup to Conversion (percentage of trials/signups to becoming paying customers)
– Lost customers
– Active customers (how many paying customers you have)
– Revenue
– Churn
– ARPU (Average Revenue Per User – 1 User = 1 Paying account)
– LTV (LifeTime Value)
If you want the above template, just subscribe to my blog by using this form (valid email required).
Also, I can identify major changes that happened, what influenced them and keep a pulse on the most important growth marketing metrics and KPIs.
For example if I see that traffic increased by 50% over the course of 5 months, but signups increased by a 5%, then I know that there is an issue there. It might mean that we need to change the signup flow, or that we attract the wrong audience/traffic, or something else.
So, we will have to address this issue before we focus on bringing in more traffic.
3) Well Documented Experiments
Running experiments and documenting them is a great way for growth. I started this process by modeling after Referral SaaSquatch’s process. They have some pretty good templates and a process that works for me.
Of course I occassionally adapt/edit it according to the circumstances, but overall it’s a pretty good starting point.
Here is an example of how I had formulated a growth experiment:
Hypothesis: Here, you make a hypothesis of what you believe is happening.
For example: “I believe having a signup form with 5 required fields discourages the user from signing up and thus recudes signups”.
Description: Here you describe the action that you want to take to tackle and test the above hypothesis/problem.
For example: “We will reduce the required fields to 2 and track the change in signups over the course of at least 2000 signups and 4 weeks.
We will also track statistical significance and it should be 99% or above.
Lastly we will track the effect this change will have on other metrics such as the conversions from signups to paying customers.”
Goal: Here you write what is the best case scenario and what you want to achieve with numbers/percentages.
For example: “Increase signups by 10%, without reducing the conversions from signups to paying customers.”
Minimum success criteria (KPIs): Here you describe what is the minimum change in specific KPIs that you could see and consider the experiment a success.
For example: 5% increase in signups with 99% statistical significance and with a 0.3% or less reduction in conversions from signups to paying customers.
Conclusion
To sum it up, every metric is a vanity metric when looked in isolation and without context. Most common culprits are traffic and email subscribers. Actionable metrics/tools include cohort analysis, comparisons and well documented experiments.
Remember that being data driven is great, but you also need intuition and experience. Otherwise growth would be replaced by software and businesses would be run by robots.
Last, although most startups nowadays gather data, they don’t necessarily act on them. Make a habit of acting on them and tracking the results.
If you found this article useful, please share it with your audience on social. It would mean a lot to me.
As always, leave a comment below if you have something to add or ask.
Thanks for reading!
