I’ve written before about the Bush administration’s use of the robots exclusion standard to tell search engines and web archives to stay away from chunks of the Whitehouse.gov website. As of last November, the robots.txt file was nearly 2300 lines long. (Here’s a copy of it.) In my opinion, such materials should be freely archivable for researchers and historians. It’s appalling that search engines or the Internet Archive would be asked to stay away.
President Obama has been in office for barely more than an hour, and the Whitehouse.gov website has already been updated. And the robots.txt file? Not 2000 lines long. It’s only two lines long:
User-agent: *
Disallow: /includes/
Much better.
UPDATE: The EFF has a posting on presidential orders and memoranda regarding transparency and openess in government. H/T to BoingBoing via Twitter.