Archive for February, 2011
Quora: What are the best metrics for measuring user engagement?

I posted this answer to Quora and figured I would share it here as well. You can find me on Quora here.
What are the best metrics for measuring user engagement?
Metrics are merely a reflection of the product strategy that you have in place.
What you are trying to do should lead what you want to measure, not the other way around. It’s for this reason that the blanket questions and answers around “best” metrics are meaningless- the question is, what are you trying to do.
For example, if you are an ecard that is driven based on holiday traffic, your strategy might be:
- people should come to my site at every major holiday
- people should send as many ecards as possible
In this case, your week-to-week retention isn’t important. The only question is whether or not you are sending out ecards, and whether or not you’re a new user or if you came back last holiday.
On the other hand, if you are trying to be more of a communications product, then you might want something like:
- people should come back for short durations every day to check their messages
- people should write messages occasionally, but mostly read a bunch of messages
In this case, you care a lot about DAUs and +1 day and +1 week retention. You might also put in a qualifier to that to make sure that people are actually reading/writing messages and not just showing up to an empty messages area.
So ultimately, the important part is to figure out what you are trying to do and what the expected behavior is around it. Only once you have that should you then ask yourself how you’d validate and test it using metrics.
Quora: What is considered a significant number of users for a free consumer internet product?
I posted this answer to Quora and figured I would share it here as well. You can find me on Quora here.
What is considered a significant number of users for a free consumer internet product?
If it’s a mass market product and you are looking to build a venture-scale startup, you need 10s of millions of users, maybe more.
Looking at the end state
To pick an arbitrary end state, let’s say you want to end up at $100M revenue runrate. If this seems to high or low to you in defining a “significant” number of users, then just pick your own number and apply the reasoning below.
So starting with the $100M number, this is why you need 10s of millions of users, typically:
Ad-based business models
If you go with advertising-based models, CPMs are traditionally quite low for mass market products- usually <$1 per 1000 ad impressions[0]. For social sites that number is more like $0.25 CPM.
So if you want to make, let’s say $100M a year, then it’ll take you 100B impressions per year, or 8.3B per month, to build that kind of business. You need a LOT of users- certainly in the 10s of millions of uniques per month who are quite engaged, in order to make that work. You would have a top 50 website to make this happen. We’ll read about you in the news.
You can cut this number down if you manage to create, say, a viable search engine. Then you might have CPMs more like $50-100, which cuts down your ad impression requirements significantly, but then you’re competing with Google. Similarly, you’d need millions of users on your email list to compete with Groupon, but you don’t need 100M email subs to get to a good revenue number.
Facebook has 570B+ pageviews/month[4], which is 5X more than Google, but their revenues are still 1/10 or 1/20 that of Google’s[5][6].
Social gaming business models
The same is true for mass market consumer internet models based on transactions. You end up with about 3% of users converting, and their ARPPU is in the single digit $ figures. So you still end up needing 10s of millions to hit a big revenue number like that.
To generate $100M runrate, you need $8.3M revenue per month. At 3% conversion and $5 ARPPU, that’s still 55M uniques per month.
The way you build a Zynga is you build a company with 266M MAU[3].
Transactional and vertical markets
If you are building a more transactional product, then the numbers above can be significantly increased. For example, if you’re a free consumer internet site for job hunters, then you’re getting a % of a transaction that’s $50k-$100k, so that’s much better.
The downside is that vertical applications tend to have a tough time acquiring and holding onto their users, whereas horizontal sites focused on communication or content publishing usually are viral and hold on to their users. If you have a tough time acquiring or holding onto users, then you eventually pay your margin out to Google, Facebook, etc. and your profits go to zero. It’s a tradeoff.
(Thus the focus of so many companies to take a transactional thing and make it social, to try to capture the social benefits- like social shopping, social job hunting products, etc.)
Just starting out?
So after reading all of the above, you might want to know how likely you are to get into a trajectory to a substantial user number. If you’re just starting out, I might look at the following:
- size of market (do I think 50M uniques/month want to do this?)
- how fast is it growing (could you approach 5k-50k new users per day?)
- have you proven your product out with a sizeable base? (50k-500k active users per month?)
- Are you part of a large existing category of products that has 100M+ uniques per month?
I think the above could all be clues to evaluating your particular product. But you never know
[0] Here’s a more detailed CPM breakdown: http://andrewchenblog.com/2008/0…
[1] Some ARPU numbers: http://giffconstable.com/2009/07…
[2] Some ARPPU numbers: http://andrewchenblog.com/2009/1…
[3] Zynga stats: http://www.appdata.com/devs/10-z…
[4] http://www.businessinsider.com/h…
[5] http://investor.google.com/finan…
[6] http://mashable.com/2011/01/17/f…
Stanford CS major seeks sales/marketing monkey
Silicon Valley is mean to MBAs
This tumblr, Whartonite Seeks Code Monkey, made me laugh.
It’s full of emails from clueless Wharton MBAs which read like this:
LOL right?
This also reminds me of the famous quote on valuing startups:
Add $1,000,000 in value for every engineer.
Subtract $500,000 in value for every MBA.
Here’s why it’s hard: The nerd perspective is, they don’t need you
Much of the reason why it’s insanely hard to find a really good technical cofounder is that the best ones really don’t need you. Or at least they don’t think they need you.
Because there’s an illustrious track record of engineering-founded companies succeeding, spanning from HP to Facebook, there’s a lot of datapoints that say that a 20-yo Stanford computer science major can do it himself, or at least with his other CS roommates. Similarly, the very best alums out of places like Facebook and Google have lots of access to capital, advice, and people- these are all recipes for making you (the biz founder) completely irrelevant.
So I think the right point of view is just to accept that the amount of leverage strong technical folks in the Valley have is just the facts, and you’ll have to work around that.
Remember this:
They are not the code monkey. You are the biz monkey.
That’s just how it is.
Picking the right idea
One key way to mitigate this is to pick the right idea that doesn’t require ridiculous amounts of technical expertise upfront. You can build a great company that’s extremely sales driven rather than product driven in categories like:
- Enterprise sales
- Groupon for X
- Blog/media sites aka Content farms
- Marketplaces
- Ad network
I’m sure I’m leaving many other categories out.
For anything above, a lot of the work is in sales, and the actual technical infrastructure doesn’t require a strong engineer to pull together, at least initially. You’ll need them to scale it, but at that point hopefully you’ll have more money and more momentum.
For the kinds of ideas above, they might be easy enough to build in the short-run that you can get a different kind of coder at first. You can get someone who can code up a site and potentially have some visual design background, rather than an “engineer” who has theoretical understanding of computer science, understands performance tradeoffs, etc. There are more of the former than that latter in the world.
At the same time, note that many of the ideas above may not be particularly exciting to an engineer that wants to play with technologies. So perhaps something that combines the two can help – for example, MySQL is a great example of a cool technology (at the time) but clearly couldn’t have been turned into a company without a lot of business types running around.
Understanding and communicating what you really bring to the table
If you read through the Wharonite Seeks Code Monkey blog, you can see that obviously they are mostly noobs and don’t know what exactly is the valuable part of what a biz cofounder can do versus not. This is true of many startups, both biz and geek-led, but there is huge overvaluation of the initial idea.
What do geeks really need help with? It’s very simple- there’s a class of purely business-related stuff that adds value:
- selling stuff and making money
- getting partnerships and marketing/distribution of the product
- funding the company
- scalable marketing/monetization strategy (ad arb / viral / freemium / etc.)
- team recruiting, particularly of other engineers and disciplines (not other MBAs please)
If you are an expert at any of the above and can show it, then there’s a lot more value. Very few business folks, particularly newly-minted MBAs (with the exception of Stanford folks) or industry-switchers can really deliver on these though, which is why they’re not bringing much to the table.
Then there’s a class of things that are much more product-oriented, and while it overlaps with the skillset of some engineers, if you have great skills in any of the following, they are clearly valuable too:
- design, especially visual design
- UI/frontend skills – HTML/CSS/JS – even if mediocre!
- copywriting within the product for help text, marketing, etc
- user research and customer development
- usability testing
Again, it all depends on what you’re really good at and what the particular product needs – enterprise might require less of the above, but a more solid initial product might help.
Worst comes to worst, write it yourself
And finally, there’s a nice track record of technical-enough people writing the first version of something and then having great engineers build it up later. Foursquare was like that, for example. More recently, David Binetti of Votizen wrote the first version of his product. I have immense respect for folks who do this, because it means they’re making “good-enough” progress without waiting for exactly the right technical partner to show up.
Any other thoughts or tips to share?
If you guys have other thoughts on ideas or thoughts on this topic, especially from those who are on the technical side, of how to attract and partner with engineers, write me a note in the comments! I’ll update this post as we go.


