Lend a Helping Hand with the End of Term Crawl

Over the past several weeks, we have seen a sharp increase in questions and suggestions across social media about how members of the public can support long term preservation and access for born-digital government information, including data, websites, reports, and other digital tools. Some of the information we have seen is verifiably accurate and helpful, while other information is incomplete or misleading, even about effective activities. Fortunately, there are efforts out there that are legitimate and could use your help! We’re posting this information as a public service announcement and would appreciate it if you share this update across your networks. 

One of the easiest and most effective ways to get started with federal government information preservation is by contributing to the 2024 End of Term (EoT) web crawl. The resulting archive is hosted by academic and not-for-profit organizations for public access and analysis, meaning that the collected government information will be readily available for public access and reuse.

Who is responsible for the EoT crawl? 

EoT is a partnership among the Internet Archive, not-for-profit organizations, and libraries. The PEGI Project is not an official partner of EoT but we celebrate their efforts to create a comprehensive snapshot of federal web content at the end of each presidential administration. 

Can I contribute to the EoT crawl? 

Yes! Individuals and groups are invited and encouraged to contribute. The 2024 EoT Crawl is accepting nominations until the crawl ends, which is estimated to be late spring 2025. You can contribute individual URLs directly through their nomination tool, or you can submit bulk lists of URLs to the project. Keep in mind, building bulk lists is a great project for groups of people interested in spinning up “data rescue”-style events! 

What kinds of websites can I look for to help with the EoT crawl? 

In Scope: Federal government websites (.gov) in the Legislative, Executive, or Judicial branches of the U.S. government, and their related social media accounts. 

  • Federal government websites on other domains, such as .mil, .edu, and .com, including websites for federally-funded programs and projects, are also in scope. For example, the SLAC National Accelerator Laboratory is a public-private partnership between the Department of Education and Stanford University, with a website hosted on stanford.edu. 

  • The greatest priority this year is to identify and submit websites that are likely to change dramatically or disappear during or soon after the presidential administration transition in early 2025.

Out of Scope: Local or state government websites, and any other non-government sites including those documenting the U.S. elections.

 

What else can I do?

Stay tuned as we prepare to share information about additional projects and efforts that would benefit from your time and effort. Please contact us if you’re working on a government information or data preservation project that needs public participation. And, if you see something out there and you’re not sure if it’s legitimate, feel free to reach out to us too. We’ll do our best to help you untangle the truth.