President Obama and White House robots
I’ve written before about the Bush administration’s use of the robots exclusion standard to tell search engines and web archives to stay away from chunks of the Whitehouse.gov website. As of last November, the robots.txt file was nearly 2300 lines long. (Here’s a copy of it.) In my opinion, such materials should be freely archivable […]