Web Data Commons - Hyperlink Graphs

The graphs have been extracted from the 2012 and 2014 versions of the Common Crawl web corpera. The 2012 graph covers 3.5 billion web pages and 128 billion hyperlinks between these pages. To the best of our knowledge, the graph is the largest hyperlink graph that is available to the public outside companies such as Google, Yahoo, and Microsoft. The2014 graph covers 1.7 billion web pages connected by 64 billion hyperlinks. Below we provide instructions on how to download the graphs as well as basic statistics about their topology.

We hope that the graphs will be useful for researchers who develop

• search algorithms that rank results based on the hyperlinks between pages.
• SPAM detection methods which identity networks of web pages that are published in order to trick search engines.
• graph analysis algorithms and can use the hyperlink graphs for testing the scalability and performance of their tools.
• Web Science researchers who want to analyze the linking patterns within specific topical domains in order to identify the social mechanisms that govern these domains.