Re: Byte ranges (actually robots)

Kee Hinckley (nazgul@utopia.com)
Thu, 25 May 1995 11:26:05 +0500


At 9:21 AM 5/25/95, Gavin Nicol wrote:
>>So what to do? If we recognize the user-agent as a known robot, we give it
>>a flat presentation of the structure. The catch of course, is that there
>>is no standard way to recognize a robot. Some convention in the UserAgent
>>field would be sufficient, but....
>
>Isn't there a standard permissions file that robots can fetch?

Yes, but that just let's me tell them what to stay away from. The issue
here is not that they should stay away, but that I should recognize them.
The permissions file is not fetched everytime they come around, and the
spacing between visits tends to be long (to reduce load on any one system),
and I can't just flag anyone who fetches robot.txt as a robot, since of
course a person may fetch it too.

Kee Hinckley Utopia Inc. - Cyberspace Architects=81 617/721-6100
nazgul@utopia.com http://www.utopia.com/

I'm not sure which upsets me more: that people are so unwilling to accept
responsibility for their own actions, or that they are so eager to regulate
everyone else's.