Common Crawl

News

A freely accessible index of 5 billion web pages, their page rank, their link graphs and other metadata, hosted on Amazon EC2, was announced today by the Common Crawl Foundation. “It is crucial [in] our information-based society that Web crawl data be open and accessible to anyone who desires to utilize it,” writes Foundation director Lisa Green on the organization’s blog.