This website does readability filtering of other pages. All styles, scripts, forms and ads are stripped. If you want your website excluded or have other feedback, use this form.

Internet Archive Web Crawls : Free Web : Free Download, Borrow and Streaming : Internet Archive

Skip to main content Search the history of over 339 billion web pages on the Internet.

Featured
texts All Texts latest This Just In Smithsonian Libraries FEDLINK (US) Genealogy Lincoln Collection Additional Collections Books to Borrow
Top
American Libraries Canadian Libraries Universal Library Community Texts Project Gutenberg Biodiversity Heritage Library Children's Library Open Library Lantern Library: Cousins of Anarchism What's Left Boston Lantern Library: Marx
Featured
movies All Video latest This Just In Prelinger Archives Democracy Now! Occupy Wall Street TV NSA Clip Library TV News
Top
Animation & Cartoons Arts & Music Community Video Computers & Technology Cultural & Academic Films Ephemeral Films Movies Understanding 9/11 News & Public Affairs Spirituality & Religion Sports Videos Television Videogame Videos Vlogs Youth Media
Featured
audio All Audio latest This Just In Grateful Dead Netlabels Old Time Radio 78 RPMs and Cylinder Recordings Live Music Archive
Top
Audio Books & Poetry Community Audio Computers & Technology Music, Arts & Culture News & Public Affairs Non-English Audio Radio Programs Librivox Free Audiobook Spirituality & Religion Podcasts
Featured
software All Software latest This Just In Old School Emulation MS-DOS Games Historical Software Classic PC Games Software Library Internet Arcade
Top
MS-DOS APK Community Software CD-ROM Software Kodi Archive and Support File Software Sites Tucows Software Library Console Living Room CD-ROM Software Library IPA Software Shareware CD-ROMs CD-ROM Images Apple Computer ZX Spectrum ZX Spectrum Library: Games
Featured
image All Image latest This Just In Flickr Commons Occupy Wall Street Flickr Cover Art USGS Maps Metropolitan Museum
Top
NASA Images Solar System Collection Ames Research Center Brooklyn Museum

Internet Archive Web Crawls

The Internet Archive discovers and captures web pages through many different web crawls. At any given time several distinct crawls are running, some for months, and some every day or longer. View the web archive through the Wayback Machine. MORE share Share
No_Favorite Favorite
edit Edit
time History
ABOUT COLLECTION

Share This Collection


1,170,600 RESULTS rss


Media Type
111 collections 1,168,957 web 1,532 data
Year
98,113 2018 204,614 2017 167,853 2016 139,882 2015 100,828 2014 138,507 2013 More right-solid
Topics & Subjects
1,022,584 crawldata 2,263 no404 1,452 wikipedia 811 wordpress 252 amazonbooks 7 end of term More right-solid
Collection
1,170,600 Internet Archive Web Crawls 1,150,742 Web Crawls 568,947 Worldwide Web Crawls 376,841 Youtube Videos 87,943 Survey Crawls 83,780 Wide Crawl Number 16: Started June 3rd, 2017 - Still running More right-solid
Creator
1,015,217 internet archive 73,846 internetarchive 32,949 alexa internet 1,275 university of north texas 31 google, inc. 3 [email protected] More right-solid
Language
3,915 English SHOW DETAILS up-solid down-solid SORT BY VIEWS TITLE DATE ARCHIVED DATE PUBLISHED DATE REVIEWED CREATOR eye Title Date Archived Creator 4.7B 4.7B Worldwide Web Crawls collection 568,747 ITEMS 4.7B VIEWS Oct 5, 2010 10/10 collection
eye 4.7B
Wide crawls of the Internet conducted by Internet Archive. Please visit the Wayback Machine to explore archived web sites. Since September 10th, 2010, the Internet Archive has been running Worldwide Web Crawls of the global web, capturing web elements, pages, sites and parts of sites. Each Worldwide Web Crawl was initiated from one or more lists of URLs that are known as "Seed Lists". Descriptions of the Seed Lists associated with each crawl may be provided as part of the metadata for...
2.4B 2.4B Survey Crawls collection 87,942 ITEMS 2.4B VIEWS Nov 17, 2012 11/12 collection
eye 2.4B
Survey crawls are run about twice a year, on average, and attempt to capture the content of the front page of every web host ever seen by the Internet Archive since 1996.
Topic: survey crawls
1.9B 1.9B Live Web Proxy Crawls collection 37,507 ITEMS 1.9B VIEWS Apr 26, 2011 04/11 collection
eye 1.9B
Content crawled via the Wayback Machine Live Proxy mostly by the Save Page Now feature on web.archive.org. Liveweb proxy is a component of Internet Archive’s wayback machine project. The liveweb proxy captures the content of a web page in real time, archives it into a ARC or WARC file and returns the ARC/WARC record back to the wayback machine to process. The recorded ARC/WARC file becomes part of the wayback machine in due course of time.
739.9M 740M Survey Crawl Number 0 - Started May 18th, 2013 - Ended May 15, 2014 collection 16,282 ITEMS 739.9M VIEWS Nov 17, 2012 11/12 collection
eye 739.9M
The seed for this crawl was a list of every host in the Wayback Machine This crawl was run at a level 1 (URLs including their embeds, plus the URLs of all outbound links including their embeds) The WARC files associated with this crawl are not currently available to the general public.
556.2M 556M Wide Crawl started April 2013 collection 25,005 ITEMS 556.2M VIEWS Apr 18, 2013 04/13 collection
eye 556.2M
Web wide crawl with initial seedlist and crawler configuration from April 2013.
502M 502M Wide Crawl Number 14 - Started Mar 4th, 2016 - Ended Sep 15th, 2016 collection 71,730 ITEMS 502M VIEWS Mar 4, 2016 03/16 collection
eye 502M
The seed for Wide00014 was: - Slash pages from every domain on the web: -- a list of domains using Survey crawl seeds -- a list of domains using Wide00012 web graph -- a list of domains using Wide00013 web graph - Top ranked pages (up to a max of 100) from every linked-to domain using the Wide00012 inter-domain navigational link graph -- a ranking of all URLs that have more than one incoming inter-domain link (rank was determined by number of incoming links using Wide00012 inter domain links)...
470.1M 470M Survey Crawl Number 2 - Started Dec 17th, 2014 - Ended Jul 31st, 2015 collection 13,890 ITEMS 470.1M VIEWS Dec 17, 2014 12/14 collection
eye 470.1M
The seed for this crawl was a list of every host in the Wayback Machine This crawl was run at a level 1 (URLs including their embeds, plus the URLs of all outbound links including their embeds) The WARC files associated with this crawl are not currently available to the general public.
431.3M 431M Wide Crawl started June 2014 collection 45,313 ITEMS 431.3M VIEWS Jun 6, 2014 06/14 collection
eye 431.3M
Web wide crawl with initial seedlist and crawler configuration from June 2014.
422.2M 422M Wayback Indexes collection 554 ITEMS 422.2M VIEWS Apr 4, 2012 04/12 collection
eye 422.2M
Wayback indexes. This data is currently not publicly accessible.
407M 407M Wide Crawl Number 12 - started March, 14th 2015 collection 49,621 ITEMS 407M VIEWS Jan 9, 2015 01/15 collection
eye 407M
Web wide crawl with initial seedlist and crawler configuration from January 2015.
357.4M 357M Wide Crawl started August 2013 collection 21,911 ITEMS 357.4M VIEWS Jul 30, 2013 07/13 collection
eye 357.4M
Web wide crawl with initial seedlist and crawler configuration from August 2013.
332.4M 332M Wide Crawl started January 2012 collection 30,362 ITEMS 332.4M VIEWS Dec 30, 2011 12/11 collection
eye 332.4M
Web wide crawl with initial seedlist and crawler configuration from January 2012 using HQ software.
322.4M 322M Survey Crawl Number 4 - Started Jan 9th, 2016 - Ended Feb 25th, 2016 collection 12,622 ITEMS 322.4M VIEWS Jan 9, 2015 01/15 collection
eye 322.4M
The seed for this crawl was a list of every host in the Wayback Machine This crawl was run at a level 1 (URLs including their embeds, plus the URLs of all outbound links including their embeds) The WARC files associated with this crawl are not currently available to the general public.
300.3M 300M Wide Crawl started April 2012 collection 39,252 ITEMS 300.3M VIEWS Mar 31, 2012 03/12 collection
eye 300.3M
Web wide crawl with initial seedlist and crawler configuration from April 2012.
279.8M 280M Survey Crawl Number 3 - Started Aug 1st, 2015 - Ended Feb 11th, 2016 collection 10,137 ITEMS 279.8M VIEWS Jan 9, 2015 01/15 collection
eye 279.8M
The seed for this crawl was a list of every host in the Wayback Machine This crawl was run at a level 1 (URLs including their embeds, plus the URLs of all outbound links including their embeds) The WARC files associated with this crawl are not currently available to the general public.
269M 269M Wide Crawl Number 13 collection 46,049 ITEMS 269M VIEWS Jan 9, 2015 01/15 collection
eye 269M
Web Wide Crawl Number 13
239.3M 239M Survey Crawl Number 1 - Started May 17th, 2014 - Ended Dec 10th, 2014 collection 6,909 ITEMS 239.3M VIEWS Apr 25, 2014 04/14 collection
eye 239.3M
The seed for this crawl was a list of every host in the Wayback Machine This crawl was run at a level 1 (URLs including their embeds, plus the URLs of all outbound links including their embeds) The WARC files associated with this crawl are not currently available to the general public.
222.5M 223M Wide Crawl started October 2010 collection 15,839 ITEMS 222.5M VIEWS Oct 5, 2010 10/10 collection
eye 222.5M
Web wide crawl with initial seedlist and crawler configuration from October 2010
218.6M 219M Wide Crawl Started January 2013 collection 15,138 ITEMS 218.6M VIEWS Jan 1, 2013 01/13 collection
eye 218.6M
Wide crawls of the Internet conducted by Internet Archive. Access to content is restricted. Please visit the Wayback Machine to explore archived web sites.
216.5M 217M Wide Crawl started September 2012 collection 22,402 ITEMS 216.5M VIEWS Aug 24, 2012 08/12 collection
eye 216.5M
Web wide crawl with initial seedlist and crawler configuration from September 2012.
215M 215M Wikipedia Outlinks March 2016 collection 27,070 ITEMS 215M VIEWS Mar 3, 2016 03/16 collection
eye 215M
Crawl of outlinks from wikipedia.org started March, 2016. These files are currently not publicly accessible. Properties of this collection. It has been several years since the last time we did this. For this collection, several things were done: 1. Turned off duplicate detection. This collection will be complete, as there is a good chance we will share the data, and sharing data with pointers to random other collections, is a complex problem. 2. For the first time, did all the different wikis....
198.9M 199M Wide Crawl Number 15: Started Oct 1st, 2016 - Ended May 8th, 2017 collection 38,782 ITEMS 198.9M VIEWS Jan 9, 2015 01/15 collection
eye 198.9M
Web wide crawl.
193.8M 194M Wide Crawl started October 2011 collection 12,648 ITEMS 193.8M VIEWS Sep 30, 2011 09/11 collection
eye 193.8M
Web wide crawl with initial seedlist and crawler configuration from March 2011 using HQ software.
177.8M 178M Wide Crawl started February 2014 collection 9,789 ITEMS 177.8M VIEWS Nov 15, 2013 11/13 collection
eye 177.8M
Web wide crawl with initial seedlist and crawler configuration from February 2014.
173M 173M Around The World Crawl collection 2,150 ITEMS 173M VIEWS Jul 16, 2012 07/12 collection
eye 173M
Data crawled by Sloan Foundation on behalf of Internet Archive
170.4M 170M Wide Crawl started March 2011 collection 8,528 ITEMS 170.4M VIEWS Oct 5, 2010 10/10 collection
eye 170.4M
Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi. What’s in the data set: Crawl start date: 09 March, 2011 Crawl end date: 23 December, 2011 Number of captures: 2,713,676,341 Number of unique URLs: 2,273,840,159 Number of hosts: 29,032,069 The seed list for this crawl was a list of Alexa’s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT)...
165.6M 166M GDELT collection 43,763 ITEMS 165.6M VIEWS Aug 27, 2014 08/14 collection
eye 165.6M
A daily crawl of more than 200,000 home pages of news sites, including the pages linked from those home pages. Site list provided by The GDELT Project
Topics: GDELT, News
162.6M 163M .com survey started January 2011 collection 2,535 ITEMS 162.6M VIEWS Jan 20, 2011 01/11 collection
eye 162.6M
Survey crawl of .com domains started January 2011.
Topic: webcrawl
133.7M 134M Wikipedia Outlinks February 2012 collection 2,951 ITEMS 133.7M VIEWS Feb 3, 2012 02/12 collection
eye 133.7M
Crawl of outlinks from wikipedia.org started February, 2012. These files are currently not publicly accessible.
94.5M 95M Wide Crawl Number 16: Started June 3rd, 2017 - Still running collection 83,780 ITEMS 94.5M VIEWS Jan 9, 2015 01/15 collection
eye 94.5M
Web wide crawl number 16 The seed list for Wide00016 was made from the join of the top 1 million domains from CISCO and the top 1 million domains from Alexa.
74.2M 74M International News Crawls collection 5,367 ITEMS 74.2M VIEWS Oct 4, 2010 10/10 collection
eye 74.2M
Crawls of International News Sites
66.8M 67M Shallow Crawls collection 1,042 ITEMS 66.8M VIEWS Nov 17, 2012 11/12 collection
eye 66.8M
Shallow crawls that collect content 1 level deep including embeds. This data is currently not publicly accessible.
62.2M 62M Wikipedia Outlinks May 2011 collection 1,638 ITEMS 62.2M VIEWS Jul 11, 2011 07/11 collection
eye 62.2M
Crawl of outlinks from wikipedia.org started May, 2011. These files are currently not publicly accessible.
57.8M 58M Survey Crawl Number 6: Sep 11th, 2017 - running now collection 13,711 ITEMS 57.8M VIEWS Jan 9, 2015 01/15 collection
eye 57.8M
The seeds for this crawl came from: 251 million Domains that had at least one link from a different domain in the Wayback Machine, across all time ~ 300 million Domains that we had in the Wayback, across all time 55,945,067 Domains from https://archive.org/details/wide00016 This crawl was run with a Heritrix setting of "maxHops=0" (URLs including their embeds) The WARC files associated with this crawl are not currently available to the general public.
53.4M 53M Wayback CDX Shards collection 1,214 ITEMS 53.4M VIEWS Aug 21, 2013 08/13 collection
eye 53.4M
CDX Index shards for the Wayback Machine. The Wayback Machine works by looking for historic URL's based on a query. This is done by searching an index of all the web objects (pages, images, etc) that have been archived over the years. This collection holds the index used for this purpose, which is broken up into 300 pieces so they fit into items more naturally and distribute the lookup load. Each of these 300 pieces is stored in at least 2 items, and then those are also stored on the backup...
50.7M 51M Hackernews crawl number 0 collection 6,154 ITEMS 50.7M VIEWS Mar 3, 2016 03/16 collection
eye 50.7M
Hacker News Crawl of their links.
43.1M 43M Wikipedia Outlinks July 2011 collection 1,011 ITEMS 43.1M VIEWS Jul 11, 2011 07/11 collection
eye 43.1M
Crawl of outlinks from wikipedia.org started July, 2011. These files are currently not publicly accessible.
40.4M 40M End of Term Web Crawls collection 22,178 ITEMS 40.4M VIEWS Sep 13, 2012 09/12 by IA collection
eye 40.4M
This collection includes web crawls of the Federal Executive, Legislative, and Judicial branches of government performed at the end of US presidential terms of office.
Topics: web, end of term, US, federal government
37.9M 38M Survey Crawl Number 5: Oct 21st, 2016 to Sep 10th, 2017 collection 3,887 ITEMS 37.9M VIEWS Jan 9, 2015 01/15 collection
eye 37.9M
The seed for this crawl was a list of every host in the Wayback Machine This crawl was run at a level 1 (URLs including their embeds, plus the URLs of all outbound links including their embeds) The WARC files associated with this crawl are not currently available to the general public.
29.5M 29M COM Survey Crawl 2009-2010 collection 729 ITEMS 29.5M VIEWS Apr 6, 2012 04/12 collection
eye 29.5M
COM survey crawl data collected by Internet Archive in 2009-2010. This data is currently not publicly accessible.
26.6M 27M Shallow Crawl Started 2013 collection 544 ITEMS 26.6M VIEWS Feb 19, 2013 02/13 collection
eye 26.6M
Shallow crawl started 2013 that collects content 1 level deep, including embeds. Access to content is restricted. Please visit the Wayback Machine to explore archived web sites.
26.1M 26M Geocities Closing Crawl collection 149 ITEMS 26.1M VIEWS Apr 4, 2012 04/12 collection
eye 26.1M
Geocities crawl performed by Internet Archive. This data is currently not publicly accessible. from Wikipedia : Yahoo! GeoCities is a Web hosting service. GeoCities was originally founded by David Bohnett and John Rezner in late 1994 as Beverly Hills Internet (BHI), and by 1999 GeoCities was the third-most visited Web site on the World Wide Web. In its original form, site users selected a "city" in which to place their Web pages. The "cities" were metonymously named after...
24.5M 25M Shallow Crawl Started 2013 collection 252 ITEMS 24.5M VIEWS Mar 5, 2013 03/13 collection
eye 24.5M
Shallow crawl started 2013 that collects content 1 level deep, including embeds. Access to content is restricted. Please visit the Wayback Machine to explore archived web sites.
24M 24M Collections News Crawls v3 collection 3,227 ITEMS 24M VIEWS Nov 12, 2013 11/13 by [email protected] collection
eye 24M
Miscellaneous high-value news sites 
Topics: World news, US news, news
20.3M 20M Youtube Videos collection 376,827 ITEMS 20.3M VIEWS Jan 26, 2011 01/11 collection
eye 20.3M
Captures of pages from YouTube. Currently these are discovered by searching for YouTube links on Twitter.
Topics: YouTube, Twitter, Video
20.2M 20M nsdlweb collection 91 ITEMS 20.2M VIEWS Jul 12, 2012 07/12 collection
eye 20.2M
this data is currently not publicly accessible.
19.7M 20M survey_net00000 collection 300 ITEMS 19.7M VIEWS Nov 24, 2010 11/10 collection
eye 19.7M
Survey crawl of .net domains started December 2010.
Topic: webcrawl
18.6M 19M Collections news crawls v2 collection 820 ITEMS 18.6M VIEWS Dec 13, 2012 12/12 by [email protected] collection
eye 18.6M
17.4M 17M End of Term 2012 Web Crawls collection 2,382 ITEMS 17.4M VIEWS Sep 13, 2012 09/12 collection
eye 17.4M
This collection contains web crawls performed on the US Federal Executive, Legislative & Judicial branches of government in 2012-2013.
Topics: end of term, US, Federal government, 2012, Obama
15.2M 15M International News Crawl started in September 2010 collection 1,083 ITEMS 15.2M VIEWS Oct 4, 2010 10/10 collection
eye 15.2M
Crawl of International News Sites with initial seedlist and crawler configuration from Sep 1, 2010.
12M 12M ORG Survey Crawls collection 191 ITEMS 12M VIEWS Jun 20, 2011 06/11 collection
eye 12M
Survey of .org domains. This data is currently not publicly accessible.
11.6M 12M Survey Crawl Number 7 collection 6,541 ITEMS 11.6M VIEWS Feb 24, 2018 02/18 collection
eye 11.6M
This "Survey" crawl was started on Feb. 24, 2018. This crawl was run with a Heritrix setting of "maxHops=0" (URLs including their embeds) Survey 7 is based on a seed list of 339,249,218 URLs which is all the URLs in the Wayback Machine that we saw a 200 response code from in 2017 based on a query we ran on Feb. 1st, 2018.   The WARC files associated with this crawl are not currently available to the general public.
9.8M 9.8M Shallow Crawl Started February 2013 collection 86 ITEMS 9.8M VIEWS Feb 19, 2013 02/13 collection
eye 9.8M
Shallow crawl started February 2013 that collects content 1 level deep, including embeds. Access to content is restricted. Please visit the Wayback Machine to explore archived web sites.
Live Web Proxy Crawls 9.6M 9.6M Liveweb Capture 2011-03-27T22:10:09PDT to 2011-03-28T05:27:05PDT May 3, 2011 05/11 by Internet Archive web
eye 9.6M
favorite 1
comment 0
Internet Archive Liveweb Capture from WaybackMachine, captured by wwwb-proxy0.us.archive.org:wbm from Sun Mar 27 22:10:09 PDT 2011 to Mon Mar 28 05:27:05 PDT 2011.
Topic: crawldata
8.6M 8.6M Edu & Gov Crawl, June 2010 collection 704 ITEMS 8.6M VIEWS Jun 21, 2010 06/10 collection
eye 8.6M
TEST COLLECTION: Crawl of .edu and .gov sites started in June 2010.
Topic: crawldata
8.3M 8.3M survey_net00001 collection 170 ITEMS 8.3M VIEWS Dec 29, 2011 12/11 collection
eye 8.3M
Survey crawl of .net domains started October 2011.
Topics: webwidecrawl, net
8.3M 8.3M End of Term 2016 Web Crawls collection 19,084 ITEMS 8.3M VIEWS Nov 21, 2016 11/16 collection
eye 8.3M
This collection contains web crawls performed as part of the End of Term Web Archive, a collaborative project that aims to preserve the U.S. federal government web presence at each change of administration. Content includes publicly-accessible government websites hosted on .gov, .mil, and relevant non-.gov domains, as well as government social media materials. The web archiving was performed in the Fall and Winter of 2016 and Spring of 2017. For more information, see...
Topics: end of term, federal government, 2016, president, congress, government data
7.5M 7.5M End of Term 2008 California Digital Library Crawl collection 340 ITEMS 7.5M VIEWS Apr 3, 2012 04/12 collection
eye 7.5M
End of term 2008 crawl data gathered by Internet Archive on behalf of the California Digital Library. This data is currently not publicly accessible.
6.7M 6.7M Crawl Data collection 32,956 ITEMS 6.7M VIEWS Jun 20, 2011 06/11 collection
eye 6.7M
Crawl Data. This data is currently not publicly accessible.
6.7M 6.7M crawl_UNK collection 32,949 ITEMS 6.7M VIEWS Jan 26, 2011 01/11 collection
eye 6.7M
Crawl data. This data is currently not publicly accessible.
6.2M 6.2M Wide Crawl started September 2010 collection 332 ITEMS 6.2M VIEWS Oct 5, 2010 10/10 collection
eye 6.2M
Web wide crawl with initial seedlist and crawler configuration from September 2010
5.9M 5.9M 2004 Election collection 178 ITEMS 5.9M VIEWS Apr 4, 2012 04/12 collection
eye 5.9M
2004 Election crawl performed by Internet Archive. This data is currently not publicly accessible.
5.3M 5.3M YouTube 2007 Crawl collection 73,846 ITEMS 5.3M VIEWS Jun 12, 2012 06/12 collection
eye 5.3M
Data crawled from YouTube.com in 2007 by Internet Archive. These files are not currently accessible.
Live Web Proxy Crawls 5.2M 5.2M liveweb-20151110214543 Nov 10, 2015 11/15 web
eye 5.2M
favorite 0
comment 0
5.1M 5.1M World Wars Crawl collection 13 ITEMS 5.1M VIEWS Apr 5, 2012 04/12 collection
eye 5.1M
Web data related to World Wars I and II collected by Internet Archive in an experimental crawl sponsored by National Endowment for the Humanities and JISC. This data is currently not publicly accessible.
4.6M 4.6M Shallow Crawl Started November 2012 collection 137 ITEMS 4.6M VIEWS Nov 17, 2012 11/12 collection
eye 4.6M
Shallow crawl started November 2012 that collects content 1 level deep, including embeds. This data is currently not publicly accessible.
4.4M 4.4M Hurricane Katrina collection 111 ITEMS 4.4M VIEWS Apr 6, 2012 04/12 collection
eye 4.4M
Data related to Hurricane Katrina collected in 2005 by Internet Archive. This data is currently not publicly accessible. from Wikipedia : Hurricane Katrina was the deadliest and most destructive Atlantic hurricane of the 2005 Atlantic hurricane season. It was the costliest natural disaster, as well as one of the five deadliest hurricanes, in the history of the United States. Among recorded Atlantic hurricanes, it was the sixth strongest overall. At least 1,833 people died in the hurricane and...
4.4M 4.4M End of Term 2008 UNT Dot Gov Crawl collection 281 ITEMS 4.4M VIEWS Apr 3, 2012 04/12 collection
eye 4.4M
End of term 2008 crawl of .gov domains gathered by University of North Texas . This data is currently not publicly accessible. UNT is a student-focused, public, research university located in Denton, Texas.
Wide Crawl started January 2012 4.2M 4.2M Webwide Crawldata 2012-01-17T08:02:53PST to 2012-01-17T01:16:20PST Jan 17, 2012 01/12 by Internet Archive web
eye 4.2M
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl423.us.archive.org:wide from Tue Jan 17 08:02:53 PST 2012 to Tue Jan 17 01:16:20 PST 2012.
Topic: crawldata
4M 4.0M End Of Term 2016 Pre-Inauguration Crawls collection 4,693 ITEMS 4M VIEWS Dec 15, 2016 12/16 collection
eye 4M
This collection contains web crawls performed as the pre-inauguration crawl for part of the End of Term Web Archive, a collaborative project that aims to preserve the U.S. federal government web presence at each change of administration. Content includes publicly-accessible government websites hosted on .gov, .mil, and relevant non-.gov domains, as well as government social media materials. The web archiving was performed in the Fall and Winter of 2016 to capture websites prior to the January...
Topics: end of term, federal government, 2016, president, congress
3.8M 3.8M Wide Crawl Number 17: Started August 3rd, 2018 - Still running collection 15,176 ITEMS 3.8M VIEWS Aug 2, 2018 08/18 collection
eye 3.8M
Wide17 was seeded with the "Total Domains" list of 256,796,456 URLs provided by  Domains Index   on June 26th, and crawled with max-hops set to "3" and de-duplication set "on".   
Live Web Proxy Crawls 3.8M 3.8M Liveweb Capture 2011-05-08T07:07:52PDT to 2011-05-08T08:00:29PDT May 8, 2011 05/11 by Internet Archive web
eye 3.8M
favorite 0
comment 0
Internet Archive Liveweb Capture from WaybackMachine, captured by wwwb-proxy0.us.archive.org:wbm from Sun May 8 07:07:52 PDT 2011 to Sun May 8 08:00:29 PDT 2011.
Topic: crawldata
Wayback Indexes 3.7M 3.7M Crawldata from Internet Archive from 2002-11-01T06:23:33PDT to 2002-11-19T23:24:07PDT Jul 18, 2012 07/12 web
eye 3.7M
favorite 0
comment 0
Data crawled by Internet Archive on behalf of Internet Archive from Fri Nov 01 06:23:33 PDT 2002 to Tue Nov 19 23:24:07 PDT 2002
Topic: crawldata
World Wars Crawl 3.6M 3.6M Crawldata from Internet Archive from 2008-08-08T00:17:40PDT to 2008-06-26T05:29:33PDT Jul 18, 2012 07/12 web
eye 3.6M
favorite 0
comment 0
Data crawled by National Endowment for the Humanities and JISC on behalf of Internet Archive from Fri Aug 08 00:17:40 PDT 2008 to Thu Jun 26 05:29:33 PDT 2008
Topic: crawldata
Live Web Proxy Crawls 3.4M 3.4M Liveweb Capture 2011-03-28T12:43:47PDT to 2011-03-28T16:56:17PDT May 3, 2011 05/11 by Internet Archive web
eye 3.4M
favorite 0
comment 0
Internet Archive Liveweb Capture from WaybackMachine, captured by wwwb-proxy0.us.archive.org:wbm from Mon Mar 28 12:43:47 PDT 2011 to Mon Mar 28 16:56:17 PDT 2011.
Topic: crawldata
MORE RESULTS
Fetching more results DESCRIPTION The Internet Archive discovers and captures web pages through many different web crawls. At any given time several distinct crawls are running, some for months, and some every day or longer. View the web archive through the Wayback Machine.
ACTIVITY

comment


Created on June 11
2010 ARossi
Archivist ADDITIONAL CONTRIBUTORS Wayback Machine Web Crawling
Archivist kngenie
Archivist VIEWS

10,287,909,434

ITEMS

1,170,269

TOP REGIONS (LAST 30 DAYS) – BETA

(data not available)