Re: Byte ranges (actually robots)
Gavin Nicol (gtn@ebt.com)
Thu, 25 May 1995 16:01:38 +0500
>>Isn't there a standard permissions file that robots can fetch?
>
>Yes, but that just let's me tell them what to stay away from. The issue
>here is not that they should stay away, but that I should recognize them.
Fair enough. I'd also like to recognise them, and some kind of
convention in User-Agent or somesuch seems desirable.
Putting a robot on our server is a great way of stress-testing it ;-)