success fail Oct JUL Oct 18 2008 2011 2012 15 captures 13 Oct 2008 - 21 Oct 2017 About this capture COLLECTED BY Organization: Internet Archive The Internet Archive discovers and captures web pages through many different web crawls. At any given time several distinct crawls are running, some for months, and some every day or longer. View the web archive through the Wayback Machine. Collection: Wide Crawl started March 2011
Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi.
Whatâs in the data set:
Crawl start date: 09 March, 2011
Crawl end date: 23 December, 2011
Number of captures: 2,713,676,341
Number of unique URLs: 2,273,840,159
Number of hosts: 29,032,069
The seed list for this crawl was a list of Alexaâs top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT) crawler software and respected robots.txt directives. The scope of the crawl was not limited except for a few manually excluded sites.
However this was a somewhat experimental crawl for us, as we were using newly minted software to feed URLs to the crawlers, and we know there were some operational issues with it. For example, in many cases we may not have crawled all of the embedded and linked objects in a page since the URLs for these resources were added into queues that quickly grew bigger than the intended size of the crawl (and therefore we never got to them). We also included repeated crawls of some Argentinian government sites, so looking at results by country will be somewhat skewed.
We have made many changes to how we do these wide crawls since this particular example, but we wanted to make the data available âwarts and allâ for people to experiment with. We have also done some further analysis of the content.
If you would like access to this set of crawl data, please contact us at info at archive dot org and let us know who you are and what youâre hoping to do with it. We may not be able to say âyesâ to all requests, since weâre just figuring out whether this is a good idea, but everyone will be considered.TIMESTAMPS
( 73 )
"Sounds Of A Playground Fading"
( 94 )
"In Grisly Rapture"
( 8 )
THE BLACK DAHLIA MURDER
( 65 )
"Darkness in the Light"
( 26 )
Last Updated: July 17, 2011 9:09 PM
Drummer JOSH FREESE To Leave NINE INCH NAILS - Oct. 9, 2008 NINE INCH NAILS has just completed a three-week tour of South America and Mexico and is preparing to resume its North American trek on October 23 in San Antonio. Mainman Trent Reznor writes on the band's web site, "Josh Freese [NIN's current drummer] fans, be sure to check out a show on the upcoming North American leg of the tour, as sadly they will be the last ones he's playing with us. He informed me he's leaving the tour at the end of this year. Yes, we are still dear friends. No, I'm not sure what we are going to do."
NINE INCH NAILS recently announced plans to release its own branded version of the iPhone music game TapTap Revenge. The special edition of the popular game includes 16 songs from the band's most recent albums, "The Slip" and the all-instrumental "Ghosts I - IV". Some of the tracks have been re-edited or remixed specifically for the game. A statement from developer Tapulous added that this edition of the game will have a "special NIN look and feel."
Trent Reznor recently asked fans to take an online survey, saying, "As we've moved from the familiar world of record labels and BS into the unknown world of doing everything yourself, we've realized it would benefit us and our ability to interact with you if we knew more about what you want, what you like, what you look like naked, etc."
Reznor offered a video download of a live performance to everyone who completed the questionnaire, hinting at other prizes as well. Tweet To report any abusive, obscene, defamatory, racist, homophobic or threatening comments, or anything that may violate any applicable laws, please send an e-mail to [email protected] with pertinent details. Anyone posting such material will be immediately and permanently banned. IP addresses are recorded to aid us in enforcing these conditions.