success fail Jun AUG Oct 20 2010 2011 2012 70 captures 01 Jun 2008 - 23 Aug 2019 About this capture COLLECTED BY Organization: Internet Archive The Internet Archive discovers and captures web pages through many different web crawls. At any given time several distinct crawls are running, some for months, and some every day or longer. View the web archive through the Wayback Machine. Collection: Wide Crawl started March 2011
Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi.
What’s in the data set:
Crawl start date: 09 March, 2011
Crawl end date: 23 December, 2011
Number of captures: 2,713,676,341
Number of unique URLs: 2,273,840,159
Number of hosts: 29,032,069
The seed list for this crawl was a list of Alexa’s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT) crawler software and respected robots.txt directives. The scope of the crawl was not limited except for a few manually excluded sites.
However this was a somewhat experimental crawl for us, as we were using newly minted software to feed URLs to the crawlers, and we know there were some operational issues with it. For example, in many cases we may not have crawled all of the embedded and linked objects in a page since the URLs for these resources were added into queues that quickly grew bigger than the intended size of the crawl (and therefore we never got to them). We also included repeated crawls of some Argentinian government sites, so looking at results by country will be somewhat skewed.
We have made many changes to how we do these wide crawls since this particular example, but we wanted to make the data available “warts and all” for people to experiment with. We have also done some further analysis of the content.
If you would like access to this set of crawl data, please contact us at info at archive dot org and let us know who you are and what you’re hoping to do with it. We may not be able to say “yes” to all requests, since we’re just figuring out whether this is a good idea, but everyone will be considered.TIMESTAMPS Classifieds: | | | Home News Travel Money Sports Life Tech Weather Wash/Politics Washington home Washington briefs Government guide Health&Behavior H&B home Medical resources Health information Opinion Opinion home Columnists Cartoons More News Top news briefs Nation briefs World briefs States Lotteries By the numbers Special reports Day in pictures Snapshots Offbeat Video Talk Today Marketplace Real estate Arcade Newspaper
Classifieds Posted NaN/NaN/NaN NaN:NaN PM
Post-9/11 report recommends police, fire response changesNEW YORK (AP) Police and fire officials pledged Monday to improve command procedures and communications as they released two reports examining emergency response in the wake of the World Trade Center attack.
The two reports were prepared by high-ranking department officials and management consultant McKinsey & Co., who together conducted dozens of interviews and reviewed hundreds of pages of computer records and hours of radio transmissions.
"There is no doubt in my mind that we are doing today what the heroes of 9-11 would have wanted us to do," Mayor Michael Bloomberg said. "It is in that spirit that we present these reports."
Both Police Commissioner Ray Kelly and Fire Commissioner Nicholas Scoppetta promised to improve they way they deploy officers and firefighters in disasters.
"We've revised our mobilization procedures, controlling the number of personnel who respond at any one time to an event," said Kelly.
Dozens of firefighters directed to go to staging points on streets surrounding the twin towers on Sept. 11 instead went straight into the buildings, officials have said. Kelly said too many police officers also went directly to the scene.
A total of 343 firefighters and 23 NYPD officers died that day.
City officials also discussed improving the fire department's radio system and linking it to police communications.
The roughly 100-page fire department document also recommends that the department bolster its single hazardous materials unit with new staff and equipment, allowing the FDNY to better respond to potential chemical, biological or radiological attacks.
It says the FDNY and neighboring fire departments should develop mutual-aid procedures for assisting each other during massive emergencies.
Similarly, procedures should be developed under which the fire department, police department and agencies as diverse as the CIA and Coast Guard could better coordinate the dissemination of information.
The FDNY document praises what it calls firefighters' historic evacuation of an estimated 25,000 people from the twin towers. It emphasizes that it would have been nearly impossible for any fire department to prepare for an event of such unprecedented scale.
"The goal was to learn from the events of 9-11 so that the city can learn from the experiences of that day — from what worked and what didn't so that we can be better prepared for any future large-scale emergency," Bloomberg said.
Beyond the sheer loss of life, the fire department was devastated by the deaths of some of its most senior commanders, including Chief of Department Peter Ganci. A number of commanders were in the lobbies of the towers and others were stationed elsewhere in the trade center complex.
One of the FDNY's most senior officers should oversee large-scale emergencies from the department's Brooklyn operations center instead of the scene itself, the report said.
Copyright 2005 The Associated Press. All rights reserved. This material may not be published, broadcast, rewritten or redistributed. Newspaper Home Delivery - Subscribe Today USATODAY.com partners: USA WEEKEND Sports Weekly Education Space.com
Home Travel News Money Sports Life Tech Weather
Resources: Mobile news Site map FAQ Contact us E-mail news
Advertise Press Room Media Lounge Electronic print edition Reprints and Permissions
Add USATODAY.com RSS feeds
The Nation's Homepage
Copyright 2008 USA TODAY, a division of Gannett Co. Inc.