Last week, tech writer Alexis Madrigal noticed that over half the the traffic coming into The Atlantic was direct traffic to its articles. Since it's unlikely that people would manually type out the URL "http://www.theatlantic.com/technology/archive/2012/10/dark-social-we-have-the-whole-history-of-the-web-wrong/263523/" they were obviously coming from somewhere — it's just that no one knows where, precisely. Madrigal proposed that this traffic mostly comes from email and IM, which generally lack referral data, representing "a vast trove of social traffic [that's] essentially invisible to most analytics programs." He grouped all of that direct traffic to article pages together and dubbed it "dark social," in reference to dark matter.
But there's an important distinction: All "dark social" traffic is direct traffic, but not all direct traffic is dark social. In fact the question here is how much of direct traffic really is dark social. Direct traffic, it turns out, is a slippery category: while you might think it's simply traffic from bookmarks and people punching out URLs manually, it's actually any traffic to a page that doesn't tell you where it comes from. There's no referral data, in other words.
It might be more accurate to call the universe of direct traffic the noumenal web — a big, messy bowl of stuff that we know is there but whose composition we can't actually probe with any of our traditional senses (in this case, web analytics, or our nose). So the question is how much this noumenal web is secretly social, and how much of this dark social is just, well, dark. BuzzFeed's data scientists have gathered data from the BuzzFeed Network — a set of sites like TMZ and The Daily Mail that collectively have over 300 million users — that sheds more light on dark social and direct traffic.
And a deeper dive into the extensive data BuzzFeed collects on the topic suggests that Madrigal may have overstated the case. "Dark social" is a catch-all term for traffic that includes sharing on email and IM — but also, according to BuzzFeed's data, "regular" social traffic that's being miscategorizedlike sharing on Facebook's mobile app and other apps. (Update: It turns out that Facebook mobile app traffic without referral data is in fact from Facebook chat and messaging. More on that below.) Madrigal's piece, though, is an important reminder that for some users who haven't moved to the newer social platforms — data suggests it skews toward older people — email remains the preferred way to share.
But first let's look at a study commissioned by the New York Times on how Americans get their news, reported on by Poynter. For Boomers and Generation X, the two most popular ways to share stories are by word of mouth and email, with email being more important to Gen X than either Boomers or Millennials. It can also be assumed that they're more heavily reliant on email blasts and subscriptions. (And it's debatable if those things are that's truly social.) But for Millennials, posting a link — in other words social media like Facebook or Twitter — has overtaken email as a more popular way to share stories. This is very much an age thing: Gen Xers, who are between Millennials and Boomers agewise, fall exactly in the middle on social media usage to share stories. And other research, by Pew, shows that among users ages 12 to 17, only 6 percent use email daily and only 37 percent use it even once a week. Pew's data for younger users also shows, notably, that only half of them use IM once a week. Most of their messaging appears to happen on social sites, with 66 percent of them using Facebook or MySpace messaging once a week.
In other words, there's a clear demographic shift happening in the way that people share content: Millennials and younger users are using social media like Facebook and Twitter — instead of email — to share more and more and to get news. This is in line with data from the BuzzFeed Network released a couple of weeks ago, showing a longer-term downward trend for email sharing compared to other forms of social sharing. (It's not simply a question of non-referrals either — all of the most popular webmail clients provide referral data, for instance, so the data is indicative of a larger trend, at least for the audiences that come to sites in the BuzzFeed Network.) Poynter also notes that analytics data company AddThis has found that email is driving 5 percent less traffic to websites than it was six months ago. As Millennials and younger users make up and more of the web's users, it seems that the proportion of direct traffic that's truly "dark social" — made up of email — will potentially drop over time, with more of what's left of dark social made up of IM and private messages exchanged over regular social sites. There'll always be the stories people only want to talk about over email and IM, but the data seems to indicate that more and more of what people share is going to happen out in the light.
This is borne out by the clearest finding from the BuzzFeed network data: The more social a story is, the less direct traffic it receives, proportionally speaking. Ranking stories by viral views, those in the 40th percentile get as much as 60 percent of their traffic from direct, while the top 1 percent of viral stories (the 99th percentile) get only 20 percent of their traffic from direct.
There are a couple of possible explanations for this: for one, regular social media like Facebook and Twitter are far more virulent than dark social media like IM and email. A story that explodes on Facebook really explodes, and propagates itself in a way that it doesn't, and can't, via email. So once Facebook or Twitter sharing take off, email and IM and other sources of direct traffic never catch up. Consider the feedback loop between the two: you email something you saw on Facebook, and one person sees it. You post something on Facebook that was emailed, and lot of people see it.
The other is in the kinds of stories that get the most social traffic vs. the most direct traffic. Across the BuzzFeed network, the top 20 most social stories only have a 45 percent overlap with the top 20 stories with the most direct traffic — and the majority of stories on the top direct traffic list are either celebrity focused (lots of Kate Middleton), among the most searched or long-tail traffic (traffic to older stories). They're less likely to be the kinds of stories that people want to share on social networks with lots of other people, in other words — you might IM your friend a link to Katle Middle n00dz, but you're probably not going to post it on your Facebook wall.
All of that said, there is a huge correlation — 95 percent, to be precise — between regular social and dark social traffic. If a story or site gets one, it will get the other. To a certain extent, duds are duds, and hits are hits, no matter the medium.
The other major finding is that in the BuzzFeed network, nearly 50 percent of direct traffic comes from mobile devices. That's a hugely disproportionate share, since mobile only makes up 23.1 percent of all traffic to the network. The question is why. To break out the mobile a bit more, 50 percent of iOS direct traffic comes from the Safari browser — which, notably, is how Mail.app opens links — but nearly 20 percent of it comes from the Facebook app, which doesn't always set a referrer but reveals itself through other information. Update: It turns out that instances of Facebook traffic without referral information come from Facebook chat and messages, which jibes with the Pew study above about younger users and social network messaging services. I suppose one could debate whether Facebook messages are true "dark social." I would say yes, BUT either way that's a lot of direct traffic from Facebook messaging/chat, which says interesting things about what dark social activity looks like for younger users — namely that a lot of it's happening on social sites themselves.
If Android usage patterns matched iOS's (and the data is not definitive on this point), the overall mix of dark social would look something like 25 percent mobile email and messaging, 10 percent mobile Facebook chat and messaging, 15 percent other mobile apps (like third-party Twitter clients since Twitter wraps all links in t.co, it's unlikely much is coming from Twitter) and 50 percent non-mobile. So a lot of the truly "dark social" traffic from email and younger users is in fact happening on mobile devices — which makes a lot of sense, because according to another Pew study, email is still the most popular thing people do with their smartphones.
The non-mobile is far harder to break down than mobile, unfortunately.
What we can say is that for non-mobile devices, default browsers, particularly on the Mac, make up a disproportionate share of direct traffic. In other words, more direct traffic is coming from the kind of users who are less inclined to switch their default browser. Which becomes important when we also see who is behind the direct traffic. According to the BuzzFeed Network's data, direct traffic is less likely to come from urban communities than from non-urban areas, disproportionately originating from the middle of the country instead of the coast. In other words, direct traffic skews toward less tech savvy (so, potentially older) and less urban users.
Another point: It indicates how non-homogenous "dark social" likely is. There's the part of it that's older and non-urban, coming from non-mobile devices, and then there's the part coming from younger users, which is more highly mobile. To oversimplify: old people on desktops reading emails, young people on phones.
It's hard to see the whole picture of this noumenal web. But it's clear that direct traffic is a mix of email and IM (dark social), a hint of mischaracterized regular social traffic, long-tail traffic and other bits — and a whole heck of a lot of it is mobile. And given how much people use mobile devices for email, that indicates that's where a lot of the truly dark social is happening. The more social a piece of content is, the less direct traffic it gets. And that direct traffic tends to skew toward coming from older, non-urban, less tech savvy people. The demographic and behavioral shifts for Millennials and younger people paint a picture where, as people share more and more on regular social networks, traffic from regular social will continue to grow, but the proportion of direct traffic that's truly dark social declines and changes in composition — where much of that dark social activity in fact takes place on regular social networks using chat, and less on email. The huge Facebook messaging numbers for mobile seem to show that. In other words, more and more social is going to happen out in the light, even if those spaces are sometimes only dimly lit.