org.archive.crawler.datamodel
Class RobotsDirectives

java.lang.Object
  org.archive.crawler.datamodel.RobotsDirectives

All Implemented Interfaces:

java.io.Serializable

public class RobotsDirectives
extends java.lang.Object
implements java.io.Serializable

Represents the directives that apply to a user-agent (or set of user-agents)

See Also:

Serialized Form

Field Summary

(package private) java.util.concurrent.ConcurrentSkipListSet<java.lang.String>	allows
(package private) float	crawlDelay
(package private) java.util.concurrent.ConcurrentSkipListSet<java.lang.String>	disallows

Constructor Summary

RobotsDirectives()

Method Summary

void	addAllow(java.lang.String path)
void	addDisallow(java.lang.String path)
boolean	allows(java.lang.String path)
float	getCrawlDelay()
protected int	longestPrefixLength(java.util.concurrent.ConcurrentSkipListSet<java.lang.String> prefixSet, java.lang.String str)
void	setCrawlDelay(float i)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

disallows

java.util.concurrent.ConcurrentSkipListSet<java.lang.String> disallows

allows

java.util.concurrent.ConcurrentSkipListSet<java.lang.String> allows

crawlDelay

float crawlDelay

Constructor Detail

RobotsDirectives

public RobotsDirectives()

Method Detail

allows

public boolean allows(java.lang.String path)

longestPrefixLength

protected int longestPrefixLength(java.util.concurrent.ConcurrentSkipListSet<java.lang.String> prefixSet,
                                  java.lang.String str)

Parameters:

prefixSet -

str -

Returns:

length of longest entry in prefixSet that prefixes str, or zero if no entry prefixes str

addDisallow

public void addDisallow(java.lang.String path)

addAllow

public void addAllow(java.lang.String path)

setCrawlDelay

public void setCrawlDelay(float i)

getCrawlDelay

public float getCrawlDelay()