Is your website a leaky bucket? 4 scenarios for user retention
Do you have happy, smiling users?
I’ve previously written a lot about metrics and user acquisition – just look at the left bar of this blog – but have not written much about metrics and user retention. By retention, I mean the process in which you convert new users who don’t care about your site into recurring users that are loyal and continually drive pageviews.
In general, I would say that more people care about this than pure user acquisition, which is great, but they are often using aggregate numbers to measure this retention. By aggregate data, I mean looking at an overall Google Analytics number, or looking at an Alexa rank, or some other rolled-up metric which doesn’t differentiate between new users that are discovering your site for the first time versus loyal users that are returning to your site.
In fact, in general I think of websites as "leaky buckets" where users are constantly getting poured into the top, and the site is constantly leaking users. In fact, you can imagine that if you pour 1,000 users into any website and then stop additional new users from joining, that 1,000 can only decrease. Over time, some users become loyal and throw off pageviews, but over time, they disappear. The rate at which this happens can be a turned into a metric just like any other number.
Pop quiz: Is Twitter retaining users?
First off, take a look at this graph and tell me if you think Twitter is retaining its userbase month over month. What do you think?
Think you have any answer?
The growth disambiguation problem
And of course, it was a trick question. In fact, it’s basically impossible from purely outside data to disambiguate the following scenarios:
- Pageviews are coming ONLY from new users
- Pageviews are coming ONLY from one generation of users (like early adopters)
- Pageviews are coming ONLY from retained users
- Pageviews are coming from new users and retained users
This should be totally obvious to people, but instead I see people pointing at Alexa graphs and saying that site A or site B is doing well, when in fact they could have a deep systemic problem.
In fact, let me argue the following in this post:
From aggregate data (like Alexa), you can figure out what sites are doing poorly at retention, but not what sites are doing well
Let’s start with the first scenario:
1. Pageviews are coming ONLY from new users
In this first scenario, the retention on your site totally sucks meaning that you lose all your people after the first session. That means that the drop off from a 1,000 users flowing in is 1,000 dropping to 0. Your retention rate is 0% from week 1 to week 2 
That said, how could you still get pageviews? First off, you obviously get any pageviews a user might create in the first session, even if they never come back. I think the most common scenarios are the following:
- Users create text content which is SEO’d and placed in the Google index
- Users send invites via e-mail which are then accepted
In either case, they are some form of "viral loop" that attracts new users even if the original user is never retained. In fact, I bet you that a lot of sites out there are buoyed by their search engine traffic, even when they have really terrible retention rates. All that matters is that they do enough work to generate a couple pageviews, and then bring in the next generation.
Using the bucket analogy, this is a bucket that has a firehose filling it, but all the water leaks out almost immediately. With a big enough firehose, the aggregate stats could look good when they are in fact rather shitty.
2. Pageviews are coming ONLY from one generation of users (like early adopters)
3. Pageviews are coming ONLY from retained users
Similar to the first scenario, you might have a situation where the numbers look great, but it’s because the bucket was able to fill well in the first group of users, but after then, the site sucks at retention. Or the inverse, where there’s no growth at all, but the retention is great.
In either case, this might hint at a bad systematic condition within the site, but ultimately the aggregate numbers hide the problem. In either case, not being able to acquire and retain brand new users is a problem, and without measuring the groups separately, it seems impossible to assess the true situation.
Back to Twitter for a second
So in fact, looking at the Twitter chart, the right answer is "we don’t know." A plateau’d chart like that could mean that Twitter is doing fine at retaining some set of users, and it’s stalled on new users, or that it’s acquiring news users like crazy but not retaining them, or anything in the middle.
That said, given the fact that Twitter pages show up in Google, which will provide them with a steady stream of new users, and that the average time on site looks closer to a heavily SEO’d site like Yelp than a social site like MySpace (5min instead of 30min, according to Compete.com), I’d guess that they are actually bleeding users pretty rapidly. Again, it’s hard to do an analysis like this without a lot more data to back it up, but that’d be my high-level analysis.
How do you figure out the health of the site then? Measuring "cohorts"
In general, the solution to the retention measurement problem lies in separating out NEW users and RETURNING users within the analytics. So at the minimum, you’d have to be able to talk about the following:
- 1 million uniques to the site
- 100,000 new uniques
- 900,000 returning uniques from the month before
That’d give you a sense that the site was actually retaining users well. But to take this further, what you really care about is to carve up your userbase into "cohorts," and measure drop-off rates from time period to time period. Here’s the definition of a time-based cohort:
A cohort is all the users that joined through a particular time period
Only then can you track the retention rate of a SPECIFIC set of users, and then measure other users experiencing an independent scenario. In the "cohort model" you’d end up with a group like:
Users that joined in Week 1
week 1 uniques: 100,000
week 2 uniques: 50,000
week 3 uniques: 25,000
In this model, you’d see that 100k users joined in week 1, and if you follow that "cohort" through, you end up with a 50% drop-off rate from week to week.
But then, in week 2, new users joined as well, which creates a week 2 cohort. Of course, in your aggregate metrics, the site would have 100k uniques in week 1, then 125k+50k uniques in week 2.
Users that joined in Week 2
week 2 uniques: 125,000
week 3 uniques: 50,000
Note that this cohort only goes through 2 weeks because it starts at week 2 and ends at week 3, whereas the week 1 cohort is able to run 3 weeks.
When you compare to the week 1 to week 2 cohort, you can tell that 1) there was a 25% increase in new users (100k to 125k), and that the retention rate DECREASED to 40% (50k/100k versus 50k/125k). This would be a red flag that your site was sucking, even if your aggregate stats looked good:
Total site stats
week 1 uniques: 100,000
week 2 uniques: 175,000
week 3 uniques: N/A*
(*since week3 cohort is not defined, 25k+50k+week3 cohort stats)
It’s not clear what your time period should be – perhaps weeks, perhaps days, perhaps months. Probably it depends on the average time between your users logging in, or something similar.
Is there a retention coefficient?
In fact, one might argue that in analyzing these cohorts that in addition to a "viral coefficient" which is measured in viral marketing, there’s in fact a "retention coefficient" that measures how well you are able to keep ahold of users.
This would be true if the cohorts you chose typically lose a constant % from week to week. That would mean that every cohort decays exponentially, which would give you a coefficient. (i.e., f(x) = e^-ax, where a is the retention coefficient)
Please measure and e-mail me your findings 
Like this post?
If you liked this post, please subscribe or follow me on Twitter. You can also find more essays here.
-
http://fantasycongress.com Andrew Lee
-
http://www.optimizeandprophesize.com/ Jonathan Mendez
-
http://christianbusch.blogspot.com Christian Busch
-
http://500hats.typepad.com dave mcclure
-
http://kickstand.typepad.com Jordan Mitchell
-
http://www.cheapertravel.com Craig
-
http://www.webdesignexperts4u.com Web Design Company USA
-
http://www.onlinepharmacy.vg/catalog/-c-32_469.html Creig Buccanan
-
http://www.onlinepharmacy.vg/catalog/-c-32_469.html Joseph Lee
-
jonesalbert
-
Ricky Martin
-
http://andrewchenblog.com/2009/08/19/to-my-first-10000-blog-subscribers-thank-you/ To my first 10,000 blog subscribers: Thank you! | Andrew Chen (@andrew_chen)
-
http://www.WallStreetSurvivor.com/ Mark
-
mfarney
-
http://www.antonybrydon.com abrydon
-
http://www.antonybrydon.com abrydon
