This gives the minimum and maximum robots.txtCrawl-delay values to allow; values found outside this range will
be changed to the appropriate minimum or maximum. -1 means no limit.
The defaults are 0 and 10. These values can be used to set reasonable
bounds to sites' Crawl-delay values.
Note that a Crawl-delay seen (modified to these limits) is only
used if Robots robots.txt is Y, and overrides Walk
Delay (here). Thus, to use the greater of
robots.txtCrawl-delay or Walk Delay (e.g. if the
walker is bandwidth-limited, and sites' robots.txt delays are
to be followed), keep Min equal to Walk Delay. To always
use Walk Delay but still respect other robots.txt
directives (i.e. just ignore Crawl-delay), keep Min and
Max equal to Walk Delay.
