I’ve written before about the Bush administration’s use of the robots exclusion standard to tell search engines and web archives to stay away from chunks of the Whitehouse.gov website. As of last November, the robots.txt file was nearly 2300 lines long. (Here’s a copy of it.) In my opinion, such materials should be freely archivable for researchers and historians. It’s appalling that search engines or the Internet Archive would be asked to stay away.