“The movement of news to the Internet makes it possible to quantify something that was otherwise very hard to measure — the temporal dynamics of the news,” said Kleinberg. “We want to understand the full news ecosystem, and online news is now an accurate enough reflection of the full ecosystem to make this possible. This is one [very early] step toward creating tools that would help people understand the news, where it’s coming from and how it’s arising from the confluence of many sources.”
The researchers also say their work suggests an answer to a longstanding question: Is the “news cycle” just a way to describe our perception of what’s going on in the media, or is it a real phenomenon that can be measured? They opt for the latter, and offer a mathematical explanation of how it works.
The research was presented at the Association for Computing Machinery Special Interest Group on Conference on Knowledge Discovery and Data Mining Conference June 28-July 1 in Paris.
The ideal, Kleinberg said, would be to track “memes,” or ideas, through cyberspace, but deciding what an article is about is still a major challenge for computing. The researchers sidestepped that obstacle by tracking quotations that appear in news stories, since quotes remain fairly consistent even though the overall story may be presented in very different ways by different writers.
Even quotes may change slightly or “mutate” as they pass from one article to another, so the researchers developed an algorithm that could identify and group similar but slightly different phrases. In simple terms, the computer identified short phrases that were part of longer phrases, using those connections to create “phrase clusters.” Then they tracked the volume of posts in each phrase cluster over time. In the August and September data they found threads rising and falling on a more or less weekly basis, with major peaks corresponding to the Democratic and Republican conventions, the “lipstick on a pig” discussion, rising concern over the financial crisis and discussions of a bailout plan.