{"id":79574,"date":"2026-02-06T00:39:30","date_gmt":"2026-02-06T05:39:30","guid":{"rendered":"https:\/\/world.edu\/?p=79574"},"modified":"2026-02-06T00:39:30","modified_gmt":"2026-02-06T05:39:30","slug":"news-sites-are-locking-out-the-internet-archive-to-stop-ai-crawling-is-the-open-web-closing","status":"publish","type":"post","link":"https:\/\/world.edu\/news-sites-are-locking-out-the-internet-archive-to-stop-ai-crawling-is-the-open-web-closing\/","title":{"rendered":"News sites are locking out the Internet Archive to stop AI crawling. Is the \u2018open web\u2019 closing?"},"content":{"rendered":"<p><img fetchpriority=\"high\" decoding=\"async\" class=\"aligncenter size-full wp-image-79575\" src=\"https:\/\/world.edu\/wp-content\/uploads\/2026\/02\/internet-archive.jpg\" alt=\"\" width=\"680\" height=\"383\" srcset=\"https:\/\/world.edu\/wp-content\/uploads\/2026\/02\/internet-archive.jpg 680w, https:\/\/world.edu\/wp-content\/uploads\/2026\/02\/internet-archive-250x141.jpg 250w\" sizes=\"(max-width: 680px) 100vw, 680px\" \/><\/p>\n<p>When the World Wide Web went live in the early 1990s, its founders\u00a0<a href=\"https:\/\/www.theguardian.com\/technology\/2017\/mar\/11\/tim-berners-lee-web-inventor-save-internet\">hoped<\/a>\u00a0it would be a space for anyone to share information and collaborate. But today, the free and open web is shrinking.<\/p>\n<p>The\u00a0<a href=\"https:\/\/archive.org\/\">Internet Archive<\/a>\u00a0has been recording the history of the internet and making it available to the public through its\u00a0<a href=\"https:\/\/web.archive.org\/\">Wayback Machine<\/a>\u00a0since\u00a0<a href=\"https:\/\/theconversation.com\/the-internet-archive-has-been-fighting-for-25-years-to-keep-whats-on-the-web-from-disappearing-and-you-can-help-163867\">1996<\/a>. Now, some of the world\u2019s biggest news outlets are\u00a0<a href=\"https:\/\/www.engadget.com\/ai\/publishers-are-blocking-the-internet-archive-for-fear-ai-scrapers-can-use-it-as-a-workaround-204001754.html\">blocking<\/a>\u00a0the archive\u2019s access to their pages.<\/p>\n<p>Major publishers \u2013 including The Guardian, The New York Times, the Financial Times, and USA Today \u2013 have confirmed they\u2019re ending the Internet Archive\u2019s access to their content.<\/p>\n<p>While\u00a0<a href=\"https:\/\/www.niemanlab.org\/2026\/01\/news-publishers-limit-internet-archive-access-due-to-ai-scraping-concerns\/\">publishers say<\/a>\u00a0they support the archive\u2019s preservation mission, they argue unrestricted access creates unintended consequences, exposing journalism to AI crawlers and members of the public trying to skirt their paywalls.<\/p>\n<p>Yet, publishers don\u2019t simply want to lock out AI crawlers. Rather, they want to\u00a0<a href=\"https:\/\/digiday.com\/media\/media-briefing-associated-press-deal-cements-microsofts-quiet-rise-in-ai-licensing\/\">sell their content<\/a>\u00a0to data-hungry tech companies. Their back catalogues of news, books and other media have become a\u00a0<a href=\"https:\/\/digiday.com\/media\/ap-makes-its-archive-ai-ready-to-tap-the-enterprise-rag-boom\/\">hot commodity<\/a>\u00a0as data to train AI systems.<\/p>\n<h4><strong>Robot readers<\/strong><\/h4>\n<p>Generative AI systems such as ChatGPT, Copilot and Gemini require access to large archives of content (such as media content, books, art and academic research) for\u00a0<a href=\"https:\/\/dl.acm.org\/doi\/10.1145\/3442188.3445922\">training<\/a>\u00a0and to\u00a0<a href=\"https:\/\/doi.org\/10.1007\/s12599-025-00945-3\">answer user prompts<\/a>.<\/p>\n<p>Publishers claim technology companies have accessed a lot of this content for free and\u00a0<a href=\"https:\/\/theconversation.com\/meta-allegedly-used-pirated-books-to-train-ai-australian-authors-have-objected-but-us-courts-may-decide-if-this-is-fair-use-253105\">without the consent of copyright owners<\/a>. Some began taking tech companies to court, claiming they had stolen their intellectual property. High-profile examples include\u00a0<a href=\"https:\/\/www.nytimes.com\/2023\/12\/27\/business\/media\/new-york-times-open-ai-microsoft-lawsuit.html\">The New York Times<\/a>\u2019 case against ChatGPT\u2019s parent company OpenAI and\u00a0<a href=\"https:\/\/www.theguardian.com\/technology\/2024\/oct\/21\/rupert-murdoch-ai-lawsuit-new-york-post-dow-jones\">News Corp\u2019s lawsuit against Perplexity AI<\/a>.<\/p>\n<h4><strong>Old news, new money<\/strong><\/h4>\n<p>In response, some tech companies have\u00a0<a href=\"https:\/\/digiday.com\/media\/news-corp-in-talks-with-google-for-ai-licensing-deal\/\">struck<\/a>\u00a0<a href=\"https:\/\/www.theguardian.com\/gnm-press-office\/2025\/feb\/14\/guardian-media-group-announces-strategic-partnership-with-openai\">deals<\/a>\u00a0to pay for access to publishers\u2019 content. NewsCorp\u2019s contract with OpenAI is reportedly\u00a0<a href=\"https:\/\/www.wsj.com\/business\/media\/openai-news-corp-strike-deal-23f186ba\">worth more than US$250 million<\/a>\u00a0over five years.<\/p>\n<p>Similar deals have been struck between academic publishers and tech companies. Publishing houses such as\u00a0<a href=\"https:\/\/theconversation.com\/academic-publishing-is-a-multibillion-dollar-industry-its-not-always-good-for-science-250056\">Taylor &amp; Francis and Elsevier<\/a>\u00a0have come under scrutiny in the past for locking publicly funded research behind commercial paywalls.<\/p>\n<p>Now,\u00a0<a href=\"https:\/\/www.insidehighered.com\/news\/faculty-issues\/research\/2024\/07\/29\/taylor-francis-ai-deal-sets-worrying-precedent\">Taylor &amp; Francis<\/a>\u00a0has signed a US$10 million nonexclusive deal with Microsoft granting the company access to over 3,000 journals.<\/p>\n<p>Publishers are also using\u00a0<a href=\"https:\/\/www.editorandpublisher.com\/stories\/revenue-roundup-can-ai-marketplaces-save-publishings-future,258345\">technology to stop unwanted AI bots<\/a>\u00a0accessing their content, including the crawlers used by the Internet Archive to record internet history. News publishers have referred to the Internet Archive as a \u201c<a href=\"https:\/\/www.niemanlab.org\/2026\/01\/news-publishers-limit-internet-archive-access-due-to-ai-scraping-concerns\/\">back door<\/a>\u201d to their catalogues, allowing unscrupulous tech companies to continue scraping their content.<\/p>\n<h4><strong>The cost of making news free<\/strong><\/h4>\n<p>The Wayback Machine has also been used by members of the public to\u00a0<a href=\"https:\/\/www.niemanlab.org\/2026\/01\/news-publishers-limit-internet-archive-access-due-to-ai-scraping-concerns\/\">avoid newspaper paywalls<\/a>. Understandably, media outlets want readers to pay for news.<\/p>\n<p>News is a business, and its\u00a0<a href=\"https:\/\/www.cjr.org\/tow_center_reports\/platform-press-how-silicon-valley-reengineered-journalism.php\">advertising revenue model<\/a>\u00a0has come under increasing pressure from the same tech companies using news content for AI training and retrieval. But this comes at the expense of public access to credible information.<\/p>\n<p>When newspapers first started moving their content online and making it free to the public in the late 1990s, they contributed to the ethos of sharing and collaboration on the early web.<\/p>\n<p>In hindsight, however, one commentator called free access the \u201c<a href=\"https:\/\/www.theatlantic.com\/technology\/archive\/2014\/08\/advertising-is-the-internets-original-sin\/376041\/\">original sin<\/a>\u201d of online news. The public became accustomed to getting their digital editions for free, and as online business models shifted, many mid- and small-sized news companies struggled to fund their operations.<\/p>\n<p>The opposite approach \u2013 placing all commercial news behind paywalls \u2013 has its own problems. As news publishers move to\u00a0<a href=\"https:\/\/doi.org\/10.26493\/1854-6935.18.323-348\">subscription-only models<\/a>, people have to juggle multiple expensive subscriptions or limit their\u00a0<a href=\"https:\/\/reutersinstitute.politics.ox.ac.uk\/sites\/default\/files\/2025-06\/Digital_News-Report_2025.pdf\">news appetite<\/a>. Otherwise, they\u2019re left with whatever news remains online for free or is served up by social media\u00a0<a href=\"https:\/\/link.springer.com\/book\/10.1007\/978-3-030-87086-7\">algorithms<\/a>. The result is a more closed, commercial internet.<\/p>\n<p>This isn\u2019t the first time that the Internet Archive has been in the\u00a0<a href=\"https:\/\/theconversation.com\/internet-archives-digital-library-has-been-found-in-breach-of-copyright-the-decision-has-some-important-implications-212091\">crosshairs of publishers<\/a>, as the organisation was previously sued and found to be in breach of copyright through its Open Library project.<\/p>\n<h4><strong>The past and future of the internet<\/strong><\/h4>\n<p>The Wayback Machine has served as a public record of the web for\u00a0<a href=\"https:\/\/theconversation.com\/the-internet-archive-has-been-fighting-for-25-years-to-keep-whats-on-the-web-from-disappearing-and-you-can-help-163867\">more than three decades<\/a>, used by researchers, educators, journalists and amateur internet historians.<\/p>\n<p>Blocking its access to international newspapers of note will leave significant holes in the public record of the internet.<\/p>\n<p>Today, you can use the\u00a0<a href=\"https:\/\/web.archive.org\/web\/19970601173955\/http:\/\/www.thenewyorktimes.com\/\">Wayback Machine<\/a>\u00a0to see The New York Times\u2019 front page from June 1997: the first time the Internet Archive crawled the newspaper\u2019s website. In another 30 years, internet researchers and curious members of the public won\u2019t have access to today\u2019s front page, even if the Internet Archive is still around.<\/p>\n<p>Today\u2019s websites become tomorrow\u2019s historical records. Without the preservation efforts of not-for-profit organisations like The Internet Archive,\u00a0<a href=\"https:\/\/theconversation.com\/academic-publishing-is-a-multibillion-dollar-industry-its-not-always-good-for-science-250056\">we risk losing vital records<\/a>.<\/p>\n<p>Despite the actions of commercial publishers and\u00a0<a href=\"https:\/\/theconversation.com\/wikipedia-at-25-can-its-original-ideals-survive-in-the-age-of-ai-273473\">emerging challenges of AI<\/a>, not-for-profit organisations such as the Internet Archive and\u00a0<a href=\"https:\/\/theconversation.com\/wikipedia-at-25-can-its-original-ideals-survive-in-the-age-of-ai-273473\">Wikipedia<\/a>\u00a0aim to keep the dream of an open, collaborative and transparent internet alive.<\/p>\n<p><em><strong>Author Bio:<\/strong> Tai Neilson is a Senior Lecturer in Media at Macquarie University<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<p>When the World Wide Web went live in the early 1990s, its founders\u00a0hoped\u00a0it would be a space for anyone to share information and collaborate. But today, the free and open web is shrinking. The\u00a0Internet Archive\u00a0has been recording the history of the internet and making it available to the public through its\u00a0Wayback Machine\u00a0since\u00a01996. Now, some of [&hellip;]\n&hellip; <a href=\"https:\/\/world.edu\/news-sites-are-locking-out-the-internet-archive-to-stop-ai-crawling-is-the-open-web-closing\/\" class=\"more\">learn more&rarr;<\/a><\/p>","protected":false},"author":33,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[434724],"tags":[435631],"coauthors":[11733],"class_list":["post-79574","post","type-post","status-publish","format-standard","hentry","category-internet","tag-archives"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.2 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>News sites are locking out the Internet Archive to stop AI crawling. Is the \u2018open web\u2019 closing? - World leading higher education information and services<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/world.edu\/news-sites-are-locking-out-the-internet-archive-to-stop-ai-crawling-is-the-open-web-closing\/\" class=\"yoast-seo-meta-tag\" \/>\n<meta property=\"og:locale\" content=\"en_US\" class=\"yoast-seo-meta-tag\" \/>\n<meta property=\"og:type\" content=\"article\" class=\"yoast-seo-meta-tag\" \/>\n<meta property=\"og:title\" content=\"News sites are locking out the Internet Archive to stop AI crawling. Is the \u2018open web\u2019 closing? - World leading higher education information and services\" class=\"yoast-seo-meta-tag\" \/>\n<meta property=\"og:description\" content=\"When the World Wide Web went live in the early 1990s, its founders\u00a0hoped\u00a0it would be a space for anyone to share information and collaborate. But today, the free and open web is shrinking. The\u00a0Internet Archive\u00a0has been recording the history of the internet and making it available to the public through its\u00a0Wayback Machine\u00a0since\u00a01996. Now, some of [&hellip;]\" class=\"yoast-seo-meta-tag\" \/>\n<meta property=\"og:url\" content=\"https:\/\/world.edu\/news-sites-are-locking-out-the-internet-archive-to-stop-ai-crawling-is-the-open-web-closing\/\" class=\"yoast-seo-meta-tag\" \/>\n<meta property=\"og:site_name\" content=\"World leading higher education information and services\" class=\"yoast-seo-meta-tag\" \/>\n<meta property=\"article:published_time\" content=\"2026-02-06T05:39:30+00:00\" class=\"yoast-seo-meta-tag\" \/>\n<meta property=\"og:image\" content=\"https:\/\/world.edu\/wp-content\/uploads\/2026\/02\/internet-archive.jpg\" class=\"yoast-seo-meta-tag\" \/>\n\t<meta property=\"og:image:width\" content=\"680\" class=\"yoast-seo-meta-tag\" \/>\n\t<meta property=\"og:image:height\" content=\"383\" class=\"yoast-seo-meta-tag\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" class=\"yoast-seo-meta-tag\" \/>\n<meta name=\"author\" content=\"Kevin\" class=\"yoast-seo-meta-tag\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" class=\"yoast-seo-meta-tag\" \/>\n\t<meta name=\"twitter:data1\" content=\"Kevin\" class=\"yoast-seo-meta-tag\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" class=\"yoast-seo-meta-tag\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" class=\"yoast-seo-meta-tag\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/world.edu\\\/news-sites-are-locking-out-the-internet-archive-to-stop-ai-crawling-is-the-open-web-closing\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/world.edu\\\/news-sites-are-locking-out-the-internet-archive-to-stop-ai-crawling-is-the-open-web-closing\\\/\"},\"author\":{\"name\":\"Kevin\",\"@id\":\"https:\\\/\\\/world.edu\\\/#\\\/schema\\\/person\\\/42c1c1d5ff7ffbdf0bd50ed94c6ae6ee\"},\"headline\":\"News sites are locking out the Internet Archive to stop AI crawling. Is the \u2018open web\u2019 closing?\",\"datePublished\":\"2026-02-06T05:39:30+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/world.edu\\\/news-sites-are-locking-out-the-internet-archive-to-stop-ai-crawling-is-the-open-web-closing\\\/\"},\"wordCount\":840,\"image\":{\"@id\":\"https:\\\/\\\/world.edu\\\/news-sites-are-locking-out-the-internet-archive-to-stop-ai-crawling-is-the-open-web-closing\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/world.edu\\\/wp-content\\\/uploads\\\/2026\\\/02\\\/internet-archive.jpg\",\"keywords\":[\"archives\"],\"articleSection\":[\"Internet\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/world.edu\\\/news-sites-are-locking-out-the-internet-archive-to-stop-ai-crawling-is-the-open-web-closing\\\/\",\"url\":\"https:\\\/\\\/world.edu\\\/news-sites-are-locking-out-the-internet-archive-to-stop-ai-crawling-is-the-open-web-closing\\\/\",\"name\":\"News sites are locking out the Internet Archive to stop AI crawling. Is the \u2018open web\u2019 closing? - World leading higher education information and services\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/world.edu\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/world.edu\\\/news-sites-are-locking-out-the-internet-archive-to-stop-ai-crawling-is-the-open-web-closing\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/world.edu\\\/news-sites-are-locking-out-the-internet-archive-to-stop-ai-crawling-is-the-open-web-closing\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/world.edu\\\/wp-content\\\/uploads\\\/2026\\\/02\\\/internet-archive.jpg\",\"datePublished\":\"2026-02-06T05:39:30+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/world.edu\\\/#\\\/schema\\\/person\\\/42c1c1d5ff7ffbdf0bd50ed94c6ae6ee\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/world.edu\\\/news-sites-are-locking-out-the-internet-archive-to-stop-ai-crawling-is-the-open-web-closing\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/world.edu\\\/news-sites-are-locking-out-the-internet-archive-to-stop-ai-crawling-is-the-open-web-closing\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/world.edu\\\/news-sites-are-locking-out-the-internet-archive-to-stop-ai-crawling-is-the-open-web-closing\\\/#primaryimage\",\"url\":\"https:\\\/\\\/world.edu\\\/wp-content\\\/uploads\\\/2026\\\/02\\\/internet-archive.jpg\",\"contentUrl\":\"https:\\\/\\\/world.edu\\\/wp-content\\\/uploads\\\/2026\\\/02\\\/internet-archive.jpg\",\"width\":680,\"height\":383},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/world.edu\\\/news-sites-are-locking-out-the-internet-archive-to-stop-ai-crawling-is-the-open-web-closing\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/world.edu\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"News sites are locking out the Internet Archive to stop AI crawling. Is the \u2018open web\u2019 closing?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/world.edu\\\/#website\",\"url\":\"https:\\\/\\\/world.edu\\\/\",\"name\":\"World leading higher education information and services\",\"description\":\"Global Higher Education Jobs, University jobs, Lecturer &amp; tenure jobs.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/world.edu\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/world.edu\\\/#\\\/schema\\\/person\\\/42c1c1d5ff7ffbdf0bd50ed94c6ae6ee\",\"name\":\"Kevin\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/1e8d491a5f9b9bf3db67c9e46abbba7bb06b26b458bd5025c55e0a4b9b60ea8a?s=96&d=mm&r=g8ba4c352a279c450aa9435d770c1ebf4\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/1e8d491a5f9b9bf3db67c9e46abbba7bb06b26b458bd5025c55e0a4b9b60ea8a?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/1e8d491a5f9b9bf3db67c9e46abbba7bb06b26b458bd5025c55e0a4b9b60ea8a?s=96&d=mm&r=g\",\"caption\":\"Kevin\"},\"description\":\"Kevin is founder of the world.edu project. The past 28 years have been involved in publishing to the education sector in print and the internet. Kevin has a degree in Education and has a many years experience in developing companies and projects.\",\"sameAs\":[\"https:\\\/\\\/world.edu\"],\"url\":\"https:\\\/\\\/world.edu\\\/author\\\/kevin\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"News sites are locking out the Internet Archive to stop AI crawling. Is the \u2018open web\u2019 closing? - World leading higher education information and services","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/world.edu\/news-sites-are-locking-out-the-internet-archive-to-stop-ai-crawling-is-the-open-web-closing\/","og_locale":"en_US","og_type":"article","og_title":"News sites are locking out the Internet Archive to stop AI crawling. Is the \u2018open web\u2019 closing? - World leading higher education information and services","og_description":"When the World Wide Web went live in the early 1990s, its founders\u00a0hoped\u00a0it would be a space for anyone to share information and collaborate. But today, the free and open web is shrinking. The\u00a0Internet Archive\u00a0has been recording the history of the internet and making it available to the public through its\u00a0Wayback Machine\u00a0since\u00a01996. Now, some of [&hellip;]","og_url":"https:\/\/world.edu\/news-sites-are-locking-out-the-internet-archive-to-stop-ai-crawling-is-the-open-web-closing\/","og_site_name":"World leading higher education information and services","article_published_time":"2026-02-06T05:39:30+00:00","og_image":[{"width":680,"height":383,"url":"https:\/\/world.edu\/wp-content\/uploads\/2026\/02\/internet-archive.jpg","type":"image\/jpeg"}],"author":"Kevin","twitter_misc":{"Written by":"Kevin","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/world.edu\/news-sites-are-locking-out-the-internet-archive-to-stop-ai-crawling-is-the-open-web-closing\/#article","isPartOf":{"@id":"https:\/\/world.edu\/news-sites-are-locking-out-the-internet-archive-to-stop-ai-crawling-is-the-open-web-closing\/"},"author":{"name":"Kevin","@id":"https:\/\/world.edu\/#\/schema\/person\/42c1c1d5ff7ffbdf0bd50ed94c6ae6ee"},"headline":"News sites are locking out the Internet Archive to stop AI crawling. Is the \u2018open web\u2019 closing?","datePublished":"2026-02-06T05:39:30+00:00","mainEntityOfPage":{"@id":"https:\/\/world.edu\/news-sites-are-locking-out-the-internet-archive-to-stop-ai-crawling-is-the-open-web-closing\/"},"wordCount":840,"image":{"@id":"https:\/\/world.edu\/news-sites-are-locking-out-the-internet-archive-to-stop-ai-crawling-is-the-open-web-closing\/#primaryimage"},"thumbnailUrl":"https:\/\/world.edu\/wp-content\/uploads\/2026\/02\/internet-archive.jpg","keywords":["archives"],"articleSection":["Internet"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/world.edu\/news-sites-are-locking-out-the-internet-archive-to-stop-ai-crawling-is-the-open-web-closing\/","url":"https:\/\/world.edu\/news-sites-are-locking-out-the-internet-archive-to-stop-ai-crawling-is-the-open-web-closing\/","name":"News sites are locking out the Internet Archive to stop AI crawling. Is the \u2018open web\u2019 closing? - World leading higher education information and services","isPartOf":{"@id":"https:\/\/world.edu\/#website"},"primaryImageOfPage":{"@id":"https:\/\/world.edu\/news-sites-are-locking-out-the-internet-archive-to-stop-ai-crawling-is-the-open-web-closing\/#primaryimage"},"image":{"@id":"https:\/\/world.edu\/news-sites-are-locking-out-the-internet-archive-to-stop-ai-crawling-is-the-open-web-closing\/#primaryimage"},"thumbnailUrl":"https:\/\/world.edu\/wp-content\/uploads\/2026\/02\/internet-archive.jpg","datePublished":"2026-02-06T05:39:30+00:00","author":{"@id":"https:\/\/world.edu\/#\/schema\/person\/42c1c1d5ff7ffbdf0bd50ed94c6ae6ee"},"breadcrumb":{"@id":"https:\/\/world.edu\/news-sites-are-locking-out-the-internet-archive-to-stop-ai-crawling-is-the-open-web-closing\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/world.edu\/news-sites-are-locking-out-the-internet-archive-to-stop-ai-crawling-is-the-open-web-closing\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/world.edu\/news-sites-are-locking-out-the-internet-archive-to-stop-ai-crawling-is-the-open-web-closing\/#primaryimage","url":"https:\/\/world.edu\/wp-content\/uploads\/2026\/02\/internet-archive.jpg","contentUrl":"https:\/\/world.edu\/wp-content\/uploads\/2026\/02\/internet-archive.jpg","width":680,"height":383},{"@type":"BreadcrumbList","@id":"https:\/\/world.edu\/news-sites-are-locking-out-the-internet-archive-to-stop-ai-crawling-is-the-open-web-closing\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/world.edu\/"},{"@type":"ListItem","position":2,"name":"News sites are locking out the Internet Archive to stop AI crawling. Is the \u2018open web\u2019 closing?"}]},{"@type":"WebSite","@id":"https:\/\/world.edu\/#website","url":"https:\/\/world.edu\/","name":"World leading higher education information and services","description":"Global Higher Education Jobs, University jobs, Lecturer &amp; tenure jobs.","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/world.edu\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/world.edu\/#\/schema\/person\/42c1c1d5ff7ffbdf0bd50ed94c6ae6ee","name":"Kevin","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/1e8d491a5f9b9bf3db67c9e46abbba7bb06b26b458bd5025c55e0a4b9b60ea8a?s=96&d=mm&r=g8ba4c352a279c450aa9435d770c1ebf4","url":"https:\/\/secure.gravatar.com\/avatar\/1e8d491a5f9b9bf3db67c9e46abbba7bb06b26b458bd5025c55e0a4b9b60ea8a?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/1e8d491a5f9b9bf3db67c9e46abbba7bb06b26b458bd5025c55e0a4b9b60ea8a?s=96&d=mm&r=g","caption":"Kevin"},"description":"Kevin is founder of the world.edu project. The past 28 years have been involved in publishing to the education sector in print and the internet. Kevin has a degree in Education and has a many years experience in developing companies and projects.","sameAs":["https:\/\/world.edu"],"url":"https:\/\/world.edu\/author\/kevin\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/posts\/79574","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/users\/33"}],"replies":[{"embeddable":true,"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/comments?post=79574"}],"version-history":[{"count":2,"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/posts\/79574\/revisions"}],"predecessor-version":[{"id":79583,"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/posts\/79574\/revisions\/79583"}],"wp:attachment":[{"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/media?parent=79574"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/categories?post=79574"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/tags?post=79574"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/world.edu\/wp-json\/wp\/v2\/coauthors?post=79574"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}