org.archive.crawler.prefetch
Class Preselector

java.lang.Object
  javax.management.Attribute
      org.archive.crawler.settings.Type
          org.archive.crawler.settings.ComplexType
              org.archive.crawler.settings.ModuleType
                  org.archive.crawler.framework.Processor
                      org.archive.crawler.framework.Scoper
                          org.archive.crawler.prefetch.Preselector

All Implemented Interfaces:

java.io.Serializable, javax.management.DynamicMBean, FetchStatusCodes

public class Preselector
extends Scoper
implements FetchStatusCodes

If set to recheck the crawl's scope, gives a yes/no on whether a CrawlURI should be processed at all. If not, its status will be marked OUT_OF_SCOPE and the URI will skip directly to the first "postprocessor".

Author:

gojomo

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.archive.crawler.settings.ComplexType

ComplexType.MBeanAttributeInfoIterator

Field Summary

static java.lang.String	ATTR_ALLOW_BY_REGEXP indicator allowing all matching URIs
static java.lang.String	ATTR_BLOCK_ALL indicator allowing all URIs (of a given host, typically) to be blocked at this step
static java.lang.String	ATTR_BLOCK_BY_REGEXP indicator allowing all matching URIs to be blocked at this step
static java.lang.String	ATTR_RECHECK_SCOPE whether to reapply crawl scope at this step

Fields inherited from class org.archive.crawler.framework.Scoper

ATTR_OVERRIDE_LOGGER_ENABLED

Fields inherited from class org.archive.crawler.framework.Processor

ATTR_DECIDE_RULES, ATTR_ENABLED, attrDecideRules

Fields inherited from class org.archive.crawler.settings.ComplexType

definition, definitionMap

Fields inherited from interface org.archive.crawler.datamodel.FetchStatusCodes

S_BLOCKED_BY_CUSTOM_PROCESSOR, S_BLOCKED_BY_QUOTA, S_BLOCKED_BY_RUNTIME_LIMIT, S_BLOCKED_BY_USER, S_CONNECT_FAILED, S_CONNECT_LOST, S_DEEMED_CHAFF, S_DEEMED_NOT_FOUND, S_DEFERRED, S_DELETED_BY_USER, S_DNS_SUCCESS, S_DOMAIN_PREREQUISITE_FAILURE, S_DOMAIN_UNRESOLVABLE, S_GETBYNAME_SUCCESS, S_OTHER_PREREQUISITE_FAILURE, S_OUT_OF_SCOPE, S_PREREQUISITE_UNSCHEDULABLE_FAILURE, S_PROCESSING_THREAD_KILLED, S_ROBOTS_PRECLUDED, S_ROBOTS_PREREQUISITE_FAILURE, S_RUNTIME_EXCEPTION, S_SERIOUS_ERROR, S_TIMEOUT, S_TOO_MANY_EMBED_HOPS, S_TOO_MANY_LINK_HOPS, S_TOO_MANY_RETRIES, S_UNATTEMPTED, S_UNFETCHABLE_URI, S_UNQUEUEABLE

Constructor Summary

Preselector(java.lang.String name)
Constructor.

Method Summary

protected void

innerProcess(CrawlURI curi) Classes subclassing this one should override this method to perform their custom actions on the CrawlURI.

Methods inherited from class org.archive.crawler.framework.Scoper

finalTasks, initialTasks, isInScope, isOverrideLogger, outOfScope

Methods inherited from class org.archive.crawler.framework.Processor

checkForInterrupt, getController, getDecideRule, getDefaultNextProcessor, innerRejectProcess, isContentToProcess, isEnabled, isExpectedMimeType, isHttpTransactionContentToProcess, kickUpdate, process, report, rulesAccept, rulesAccept, setDefaultNextProcessor, spawn

Methods inherited from class org.archive.crawler.settings.ModuleType

addElement, listUsedFiles

Methods inherited from class org.archive.crawler.settings.ComplexType

addElementToDefinition, checkValue, earlyInitialize, getAbsoluteName, getAttribute, getAttribute, getAttribute, getAttributeInfo, getAttributeInfo, getAttributeInfoIterator, getAttributes, getDataContainerRecursive, getDataContainerRecursive, getDefaultValue, getDescription, getElementFromDefinition, getLegalValues, getLocalAttribute, getMBeanInfo, getMBeanInfo, getParent, getPreservedFields, getSettingsHandler, getUncheckedAttribute, getValue, globalSettings, invoke, isInitialized, isOverridden, iterator, removeElementFromDefinition, setAsOrder, setAttribute, setAttribute, setAttributes, setDescription, setPreservedFields, toString, unsetAttribute

Methods inherited from class org.archive.crawler.settings.Type

addConstraint, equals, getConstraints, getLegalValueType, isExpertSetting, isOverrideable, isTransient, setExpertSetting, setLegalValueType, setOverrideable, setTransient

Methods inherited from class javax.management.Attribute

getName, hashCode

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

ATTR_RECHECK_SCOPE

public static final java.lang.String ATTR_RECHECK_SCOPE

whether to reapply crawl scope at this step

See Also:

Constant Field Values

ATTR_BLOCK_ALL

public static final java.lang.String ATTR_BLOCK_ALL

indicator allowing all URIs (of a given host, typically) to be blocked at this step

See Also:

Constant Field Values

ATTR_BLOCK_BY_REGEXP

public static final java.lang.String ATTR_BLOCK_BY_REGEXP

indicator allowing all matching URIs to be blocked at this step

See Also:

Constant Field Values

ATTR_ALLOW_BY_REGEXP

public static final java.lang.String ATTR_ALLOW_BY_REGEXP

indicator allowing all matching URIs

See Also:

Constant Field Values

Constructor Detail

Preselector

public Preselector(java.lang.String name)

Constructor.

Parameters:

name - Name of this processor.

Method Detail

innerProcess

protected void innerProcess(CrawlURI curi)

Description copied from class: Processor

Classes subclassing this one should override this method to perform their custom actions on the CrawlURI.

Overrides:

innerProcess in class Processor

Parameters:

curi - The CrawlURI being processed.