Hey Klout, Adding More Decimal Places Does Not Make Your Score More Accurate

Posted: October 26th, 2011 | Author: | Filed under: Klout, PeerIndex, Statistics | No Comments »

Klout has been hyping up their score changes for a week now. The CEO Joe Fernandez has claimed that this makes the score more accurate, more transparent, and may cure some forms of cancer (well maybe not the last claim). Let’s just say I haven’t been this disappointed since the 2000 election. Let’s start with their first claim: accuracy. See figure 1, my new score

It’s exactly the same graphic as before, but with two decimal places. While my 8th grade Chemistry teacher may be glad that they are using more significant digits, I honestly don’t care. They were there before, just not displayed. Lame.

In their blog post, they claim: “This project represents the biggest step forward in accuracy, transparency and our technology in Klout’s history.” They support this vague claim with the histogram below, showing the differences in Klout scores, before and after the change:

This histogram leaves tons of open questions. Is this different than your normal daily shift in scores? The histogram reminds me of a t-distribution with a fatter positive tail. If more people are signing up for Klout than are leaving, thats probably what it should look like anyways as users hookup more networks and gradually become more active online. The graphic doesn’t show that your score is any better, just that it changed. That’s not impressive at all.

My beef with Klout remains simply that the service provides us with no real validation or explanation of our scores. They don’t show us how many times we have been RT’ed, mentioned, etc. On Google, you can look up your page rank, on app stores you can see your average rating and number of ratings, on Klout, you are told that your true reach has increased, but not told what that implies or how you can verify it.

Klout is still the social influence measurement leader, but with Peerindex rapidly improving (and better in many ways in my opinion), and new competitors such as Proskore and Kred popping up, Klout should be worried. I’ll have a review of both Proskore and Kred up shortly as well so you can easily compare them for yourself.

Klout improves score by making it less transparent and even harder to explain

Posted: August 29th, 2011 | Author: | Filed under: Klout, PeerIndex, Statistics | No Comments »

After taking a few weeks off from reaming Klout, their newest “improvements” have left me with no choice but to write a sardonic and snarky response. Klout has added 5 new services (Instagram, Flickr, tumblr, Last.fm, and Blogger) and removed ANY secondary statistics from our profile pages. I’m still not sure which is worse, just that both are stupid. I’ll start by criticizing the addition of new services with a simulated conversation between Klout and myself.

Part 1: A conversation with Klout about their new signals

Alex: This brings the total services to 10. Really Klout, you need 10 services?
Klout: Of course this will help make your Klout score even better!
Alex: But you didn’t do a good job with just Twitter and Facebook, how can I expect you to do a good job with 10?
Klout: More data always improves the performance of complicated, black box, machine learning algorithms like our own.
Alex: That’s actually false.
Klout: Ummmm, look dude, I’m just a data whore and want to sell your data to the man.
Alex: So you just want all of my data to sell it to the man and give me nothing in return?
Klout: We actually have a terrific Klout perks program. I see you’ve received two Klout perks.
Alex: Yup, you sent me a Rizzoli and Isles gift pack, a TV show on a network I don’t have and literally hadn’t heard of before receiving the gift pack. Did I mention that the gift pack came with handcuff earrings?
Klout: But what about your other Klout perk, a party at Rolo, a store in SF that sells jeans. Careful analysis of your Twitter, Facebook, Foursquare and LinkedIn data led us to believe that you like or wear jeans.
Alex: Everyone wears jeans. That’s similar to predicting that I like to go on vacation or eat tasty food. These jeans happened to be $175, which doesn’t sound like much of a perk to me.

On top of this, android users actually can’t even connect their Klout accounts to Instagram because the app is iPhone only. Ironically, the Klout blog just posted about the average Klout of iPhone and Android users, finding the former beat out the latter 42.0 to 40.6. Perhaps the comparison would be more equal if Android users were allowed to connect 10 services rather than 9? Does MG Siegler actually need more Klout?

Part 2: Klout removes any accountability from website

Finally, let’s discuss, the complete lack of transparency imposed by their recent banishment of the majority of profile stats. Here is a screen shot of my “Network Influence” before:

and after:

You will notice that the supporting stats are gone. Though this absence makes it much harder for me to criticize the inconsistencies in their score, it also takes away most of the utility I received from Klout. Unless you have your own Twitter analytics, most people don’t have access to this info, thats one of the reason Klout was cool. It indulges my latent nerd narcissism. How many RTs do I get? How many @ mentions? How many unique ones? Now I just get a number with little explanation. Luckily, Klout competitor, Peerindex, still has much of that info:

From Klout’s point of view, I completely understand why they would want to add more services: greater reach, more data, more partners, etc. I suppose they could justify the removal of more specific stats by saying that things could get too crowded on the main page, but then put the data on another page, don’t take it away. Twitter and Facebook still drive the large majority of usage. Do you really think Blogger cares if their stats aren’t on the main page? Seems nefarious to me.

Comparing Klout competitors and alternatives: PeerIndex and Twittergrader

Posted: June 8th, 2011 | Author: | Filed under: Klout, PeerIndex | 2 Comments »

After spending two blog posts on the shortcomings of Klout, it only seems fair that I look into the quality of its competitors PeerIndex and Twitter Grader.

Like Klout, both consume your Twitter stream and provide a summary score of your “influence” between 0 and 100. PeerIndex also allows you to connect your Facebook, LinkedIn and Quora accounts, while Twitter Grader is limited to Twitter. In my previous post I claimed that any good measure of influence should have the following properties:

  1. Ordering should make sense in the real world – the score should roughly represent the degree to which one is influential or has clout
  2. The score should not be easy to game – people should not be able to hack their score in a few days by getting bots to RT, squatting on hashtags, or simply connecting a Facebook account
  3. The score should be monotonic – if another member has higher stats than me in ALL categories, then he/she should have a higher score

The primary competitor is PeerIndex, so we will start by seeing if their score satisfies the above conditions more than Klout. Here is my PeerIndex summary:

Though overall PeerIndex better satisfies the three rules better, the first thing I noticed is a bug: my PeerIndex is reported as 50 at the top of this screenshot and 44 at the bottom. PeerIndex focuses on three areas: Activity, Authority, and Audience (as compared to Network Influence, Amplification Probability, and True Reach on Klout). These individual numbers are also different at the top and bottom of the screenshot. Though I don’t have a great idea of how these scores are calculated, Activity and Audience (followers) sound much more easily gamed to me. Authority is in theory a great measure, but in practice it falls short. I encountered several other bugs. Mouseover elements in the graph don’t always disappear, and some accounts just aren’t able to be added to the comparision. @Williamsonwines (my favorite winery) and @uberjon a product marketing executive at Facebook and X-Googler are two examples. Several other issues and bugs are highlighted in the comparison screenshot below:

PeerIndex claims that @feltron should have an audience score of 0, however, the well known data analyst and designer had 11,219 Followers and was listed 907 times at the time of this post. @coachella, the epic music festival in Indio, CA, has an activity score of 0, though the account has tweeted 436 times and several immediately preceding this blog post. Finally, @MummNapaWinery has 0’s across the board even though the account has 312 tweets, 3,122 followers, and is one of my favorite wineries in Napa. I also find it ironic and mildly awkward that PeerIndex gives itself a low authority score and Klout a slightly higher one.

For due diligence purposes, I performed the same group comparisons as in my original Klout post, but I didn’t find the same monotonicity issues. As a result, they are harder to poke fun at, so I placed them at the bottom of this post. You will notice a few individuals are left out relative to the original Klout comparisons. The UI requires 6 or less people, so I took a few out randomly. If you stare at the authority scores in these examples long enough, I think you’ll agree that they are kind of wonky (check out Vic Gundotra and Carla Borsoi, VPs at Google and AOL, respectively). Put more succinctly:

PeerIndex Advantages:

  • No or fewer monotonicity issues
  • Facebook, LinkedIn, and Quora already integrated (Klout is just adding LinkedIn)
  • Twitter Elite Lists are VERY interesting: Top Users, Top Women, Top Brands, Top Cities

PeerIndex Disadvantages:

  • Can’t see or track your score over time
  • Fewer details in summary
  • VERY slow to update (~7 days initially and updates only every couple of days)
  • Authority score needs more explanation

In summary, PeerIndex is a legitimate competitor, but they need to fix the user facing bugs I highlighted and really speed up their scoring cycle. If a user shows up to the site and can’t immediately access their score they will have HUGE retention issues. One of the most under appreciated features of Klout, regardless of your feelings concerning the legitimacy of its score, is its infrastructure. The fact that they can ingest the Twitter and Facebook streams, process the data, and update every day is an incredible engineering feat, especially for a startup of its size. Perhaps there is an unseen tradeoff between speed and quality of score within Klout and PeerIndex.

Next we consider Twitter Grader.

Their approach is totally opaque. The summary simply lists stats that I can get from my own twitter account, a number from 0 to 100, and a relative rank. I have no idea where this rank comes from, especially because there are way more than 9 million people on Twitter. Perhaps this is the number of accounts ever scored on Twitter Grader? I certainly hope not, because that would be incredibly biased. They do provide an article in their help center: How does Twitter Grader Calculate Twitter Rankings. Honestly their score may be great, but I just don’t have enough information and their site is lacking many of the features found in PeerIndex and Klout.

Twitter Grader Advantages:

  • Pulls your data and calculates score instantly
  • Associated with Hubspot (which has a great reputation)
  • It only uses twitter (see also disadvantages)

Twitter Grader Disadvantages:

  • It only uses twitter
  • Can’t see or track your score over time
  • Lack features

In my next post on the subject, I’ll have more followup from the ranking folks at Klout and the CEO of PeerIndex

As previously mentioned, the PeerIndex score comparisons mirroring those from the original Klout post can be found below:

Comparison 1

Alex Braunstein (me), @alexbraunstein – Statistician, Research Scientist at Chomp, X-Googler
Binh Tran, @binhtran – the co-founder and CTO of Klout,
Chomp, @chomp – app search engine
Vic Gundotra, @vicgundotra – SVP and head of social at Google
Carla Borsoi, @u_m – VP of Consumer Insights at AOL

Comparison 2

Paul Graham, @paulg – the fearless leader of Y Combinator.
Y Combinator, @ycombinator – startup incubator
500 Startups, @500startups – startup incubator
Adria Richards, @adriarichards – a tech consultant and popular blogger (also my roommate)

Comparison 3

Tim Ferriss, @tferriss – author of the 4 Hour Workweek and 4 Hour Body
Matt Cutts, @mattcutts – head of web spam team at Google
MG Siegler, @parislemon – my favorite writer for Techcrunch
Klout, @klout – the service I’m trashing in this post
Jeffrey Zeldman, @zeldman – designer, writer, and publisher

Comparison 4

Robert Scoble, @scobleizer – blogger, tech evangelist, and author
Perez Hilton, @perezhilton – master of celebrity gossip
Charlie Sheen, @charliesheen – #winning
Guy Kawasaki, @guykawasaki – entrepreneur and former Chief Evangelist at Apple
Justin Bieber, @justinbieber – never saying never