As a Ph D Statistician and search quality engineer, I know a lot about how to properly measure things. In the past few months I’ve become an active Twitter user and very interested in measuring the influence of individuals. Klout provides a way to measure influence on Twitter using a score also called Klout. The range is 0 to 100. Light users score below 20, regular users around 30, and celebrities start around 75. Naturally, I was intrigued by the Klout measurement, but a careful analysis led to some serious issues with the score.
Everything in life can be measured. Some quantities live on natural measurement scales: height, weight, temperature, etc. Some quantities are derived measurements: happiness, deliciousness, hunger, etc. Though all useful measurements, research has repeatedly shown derived measurements to be inconsistent and not trustworthy individually. Specifically, if two individuals tell you their happiness levels are an 8 and a 9 on a scale of 10, we have no way to know:
- what this means for each individual without significant amounts of context
- which individual is “happier” even if 8 is less than 9
I argue that Klout is far more similar to a derived measurement and has several suboptimal properties. Specifically, there are 3 basic, desirable properties the Klout score should satisfy:
- Ordering by Klout should make sense in the real world – the score should roughly represent the degree to which one is influential or has clout
- The score should not be easy to game – people should not be able to hack their klout in a few days by getting bots to RT, squatting on hashtags, or simply connecting a Facebook account
- The score should be monotonic – if another member has higher stats than me in ALL categories, then he/she should have a higher score
To demonstrate the issues Klout has with these principles, we provide 4 groups of Klout score comparisons:
- a set of individuals with Klout in the 40-49 range
- a set of individuals with Klout in the 55-64 range
- a set of individuals with Klout in the 70-79 range
- a set of individuals with Klout >= 80
The four groups were chosen to span the Klout range and contain bloggers, executives, tech pundits, and celebrities of varying levels of activity in social media, notoriety, influence, importance, etc.
Group 1 (Klout 40-49)
Alex Braunstein (me), @alexbraunstein – Statistician, Research Scientist at Chomp, X-Googler
Ben Keighran, @benkeighran – the CEO of Chomp
Binh Tran, @binhtran – the co-founder and CTO of Klout,
Chomp, @chomp – app search engine
Vic Gundotra, @vicgundotra – SVP and head of social at Google
Carla Borsoi, @u_m – VP of Consumer Insights at AOL
Let’s consider a few pairwise comparisons. First, Ben’s stats dominate mine excepting likes per post and comments per post, however, his Klout score is 7 points lower than mine. Next, Binh’s stats completely dominate my own in EVERY category, often by very large factors, yet we have identical Klout scores. Carla’s scores also completely dominate mine, but her score is lower. Finally, consider Chomp and Vic Gundotra. Vic’s stats blow Chomp out of the water, yet his Klout score is lower. In the “real world” sense of the word clout, Vic should dominate this group. The group 1 comparisons demonstrate the Klout score violating rules 1 and 3 from above.
Group 2 (Klout 55-64)
Paul Graham, @paulg – the fearless leader of Y Combinator.
Y Combinator, @ycombinator – startup incubator
500 Startups, @500startups – startup incubator
Adria Richards, @adriarichards – a tech consultant and popular blogger (also my roommate)
Stefanie Michaels, @adventuregirl – go to person for everything travel
In group 2, my roommate has a higher Klout score than Paul Graham? Really? By 5 points? Paul has 6x more followers, 2x total RTs, and 4x as many unique RTs, but he hasn’t linked his FB account. Adria has incredibly low FB stats (she uses it sparingly), but apparently that still gives her a tremendous boost. Adding a FB account is far too easy a way to game your score higher. I understand that Klout wants to incentivize the attachment of FB accounts and keep growing virally, but this aspect of the Klout score seems broken. Additionally, the pairwise comparison of Y Combinator and Paul is confusing. Paul’s stats are much higher, but they are assigned the same score. One could argue different, perhaps more Klout-tastic people, are following Y Combinator, however, I find that unlikely given that Paul is in charge of it. Finally, its wrong that Adventure Girl’s Klout is so low. She has been named one of the top 100 people on Twitter, has been featured in Time magazine, etc., but her Klout is only two points higher than Adria’s.
Group 3 (Klout 70-79)
Tim Ferriss, @tferriss – author of the 4 Hour Workweek and 4 Hour Body
Jack Dorsey, @jack – Executive Chairman of Twitter and CEO of Square
Matt Cutts, @mattcutts – head of web spam team at Google
MG Siegler, @parislemon – my favorite writer for Techcrunch
Klout, @klout – the service I’m trashing in this post
David Pogue, @pogue – tech guy from the NYT
Jeffrey Zeldman, @zeldman – designer, writer, and publisher
Things get very confusing in this group. Jack Dorsey’s stats dominate those of David Pogue, but his score is 4 points lower. Matt Cutts has 4000 more total RTs but 1.5M fewer followers relative to Jack Dorsey, so his Klout score is 1 point higher? I’ll go out on a limb and state that 4000 incrementral RTs seem FAR less valuable than 1.5M incremental followers. Klout, the company, has fewer followers, total RTs and unique RTers by a factor of at least 6, but 7K more unique mentioners, so Klout’s Klout score is 4 points higher than Jacks? But if unique mentions are so valuable, how can Jack Dorsey have a lower score than Matt Cutts when he has 16K additional unique mentioners? This is just the start of the inconsistencies.
Without FB, MG Siegler’s score would likely be 10 points lower. Jeffrey Zeldman’s blog is super high quality, but does he deserve to have more Klout than David Pogue? Again, Facebook puts him over the top. I think that Klout’s score is far too high, though perhaps its not surprising Klout does well on its own metric. Finally, I included Tim Ferriss not just because I’m a huge fan, but his stats provide an interesting counterpoint for even more interesting pairwise comparisons. It will lead you to several more contradictions concerning the relative value of followers, RTs, unique RTers, and unique mentioners.
Group 4 (Klout >= 80)
Robert Scoble, @scobleizer – blogger, tech evangelist, and author
Perez Hilton, @perezhilton – master of celebrity gossip
Charlie Sheen, @charliesheen – #winning
Guy Kawasaki, @guykawasaki – entrepreneur and former Chief Evangelist at Apple
Justin Bieber, @justinbieber – never saying never
The pairings of Scoble/Hilton and Sheen/Kawasaki again demonstrate the severe miscalibration regarding Facebook scores. Also, I’m not sure I trust any system which has Justin Bieber as most influential.
In conclusion, there are some serious inconsistencies with Klout that render it nearly meaningless in some circumstances. It often does not correctly order individuals in terms of how influential they are, is easy to game higher simply by adding a Facebook account, and does not respect some very basic monotonicity rules. Put simply, it acts like a derived measurement. From this analysis, I have gleaned the following rough rules of thumb for understanding your Klout score:
- Connecting an additional account (ie Facebook) will ALWAYS increase your Klout.
- The degree to which your followers are influential seems to be irrelevant or matter very little
- The differential between number of people you follow seems to be irrelevant or matter very little.
- In terms of value to your Klout score: follow < RT < unique RT < unique mention but this can be inconsistent
- In terms of value to your Klout score: like < comment but this can be inconsistent
To be fair, Klout does not want their score to be completely transparent. Then it would be easy to rip off and even easier to game. That being said it should be possible respect the three conditions I enumerated and still keep a lid on their secret sauce. As I have time, I’m going to mess around with the Klout API a bit and gather more comprehensive data to further demonstrate the points made in this post, including a similar study concerning the Klout of companies/brands. Additionally, I will submit several questions regarding my analysis to Joe Fernandez’s (the CEO of Klout) Klout chat, and hope the company follows up. I’ll post any details/answers I receive here.
As any good statistician should, I need to qualify my analysis. There is of course selection bias in the examples enumerated above. Although not as egregious, these head scratching scores are the rule, not the exception. All data was pulled on 5/29/11, and may not reflect current scores. Finally, please remember that this is my personal blog and reflects my opinion alone. In particular, it does not reflect the opinions of any employer past, present, or future.