com.norconex.collector.http.sitemap.impl

Class StandardSitemapResolver

java.lang.Object

com.norconex.collector.http.sitemap.impl.StandardSitemapResolver

All Implemented Interfaces:

ISitemapResolver

public class StandardSitemapResolver
extends Object
implements ISitemapResolver

Implementation of ISitemapResolver as per sitemap.xml standard defined at http://www.sitemaps.org/protocol.html.

Sitemaps are only resolved if they have not been resolved already for the same URL "root" (the protocol, host and possible port).

The Sitemap specifications dictates that a sitemap.xml file defined in a sub-directory applies only to URLs found in that sub-directory and its children. This behavior is respected by default. Setting lenient to true no longer honors this restriction.

Since 2.3.0

Paths relative to URL roots can be specified and an attempt will be made to load and parse any sitemap found at those locations for each root URLs encountered (except for "start URLs" sitemaps, see below). Default paths are /sitemap.xml and /sitemap_index.xml. Setting null or an empty path array on setSitemapPaths(String...) will prevent attempts to locate sitemaps and only sitemaps found in robots.txt or defined as start URLs will be considered.

Sitemaps can be specified as "start URLs" (defined in HttpCrawlerConfig.getStartSitemapURLs()). Sitemaps defined that way will be the only ones resolved for the root URL they represent (sitemap paths or sitemaps defined in robots.txt won't apply).

Sitemaps are first stored in a local temporary file before being parsed. The tempDir constructor argument is used as the location where to store those files. When null, the system temporary directory is used, as returned by FileUtils.getTempDirectoryPath().

Author:

Pascal Essiembre

Field Summary

Fields

Modifier and Type	Field and Description
static String[]	DEFAULT_SITEMAP_PATHS

Constructor Summary

Constructors

Constructor and Description
StandardSitemapResolver(File tempDir, SitemapStore sitemapStore)

Method Summary

Modifier and Type	Method and Description
boolean	equals(Object other)
long	getFrom()
String[]	getSitemapLocations() Deprecated. Since 2.3.0, use HttpCrawlerConfig.getStartSitemapURLs()
String[]	getSitemapPaths() Gets the URL paths, relative to the URL root, from which to try locate and resolve sitemaps.
File	getTempDir() Gets the directory where temporary sitemap files are written.
int	hashCode()
boolean	isEscalateErrors()
boolean	isLenient()
void	resolveSitemaps(HttpClient httpClient, String urlRoot, String[] sitemapLocations, SitemapURLAdder sitemapURLAdder, boolean startURLs) Resolves the sitemap instructions for a URL "root" (e.g.
void	setEscalateErrors(boolean escalateErrors)
void	setFrom(long from)
void	setLenient(boolean lenient)
void	setSitemapLocations(String... sitemapLocations) Deprecated. Since 2.3.0, use HttpCrawlerConfig.setStartSitemapURLs(String[])
void	setSitemapPaths(String... sitemapPaths) Sets the URL paths, relative to the URL root, from which to try locate and resolve sitemaps.
void	setTempDir(File tempDir) Sets the directory where temporary sitemap files are written.
void	stop() Stops any ongoing sitemap resolution.
String	toString()

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

DEFAULT_SITEMAP_PATHS

public static final String[] DEFAULT_SITEMAP_PATHS

Constructor Detail

StandardSitemapResolver

public StandardSitemapResolver(File tempDir,
                               SitemapStore sitemapStore)

Method Detail

getSitemapPaths

public String[] getSitemapPaths()

Gets the URL paths, relative to the URL root, from which to try locate and resolve sitemaps. Default paths are "/sitemap.xml" and "/sitemap-index.xml".

Returns:

sitemap paths.

Since:

2.3.0

setSitemapPaths

public void setSitemapPaths(String... sitemapPaths)

Sets the URL paths, relative to the URL root, from which to try locate and resolve sitemaps.

Parameters:

sitemapPaths - sitemap paths.

Since:

2.3.0

resolveSitemaps

public void resolveSitemaps(HttpClient httpClient,
                            String urlRoot,
                            String[] sitemapLocations,
                            SitemapURLAdder sitemapURLAdder,
                            boolean startURLs)

Description copied from interface: ISitemapResolver

Resolves the sitemap instructions for a URL "root" (e.g. http://www.example.com).

Specified by:

resolveSitemaps in interface ISitemapResolver

Parameters:

httpClient - the http client to use to stream Internet files if needed

urlRoot - the URL root for which to resolve the sitemap

sitemapLocations - sitemap locations to resolve

sitemapURLAdder - where to store retrieved site map URLs

startURLs - whether the sitemapLocations provided (if any) are start URLs (defined in HttpCrawlerConfig.getStartSitemapURLs())

getSitemapLocations

@Deprecated
public String[] getSitemapLocations()

Deprecated. Since 2.3.0, use HttpCrawlerConfig.getStartSitemapURLs()

Get sitemap locations.

Returns:

sitemap locations

setSitemapLocations

public void setSitemapLocations(String... sitemapLocations)

Deprecated. Since 2.3.0, use HttpCrawlerConfig.setStartSitemapURLs(String[])

Set sitemap locations.

Parameters:

sitemapLocations - sitemap locations

isLenient

public boolean isLenient()

setLenient

public void setLenient(boolean lenient)

getFrom

public long getFrom()

setFrom

public void setFrom(long from)

isEscalateErrors

public boolean isEscalateErrors()

setEscalateErrors

public void setEscalateErrors(boolean escalateErrors)

getTempDir

public File getTempDir()

Gets the directory where temporary sitemap files are written.

Returns:

directory

Since:

2.3.0

setTempDir

public void setTempDir(File tempDir)

Sets the directory where temporary sitemap files are written.

Parameters:

tempDir - directory

Since:

2.3.0

stop

public void stop()

Description copied from interface: ISitemapResolver

Stops any ongoing sitemap resolution. Some sitemaps can be huge, and they may take a while to process. Upon the crawler receiving a stop request, this method will be invoked and implementors should try to exit cleanly without much delay.

Specified by:

stop in interface ISitemapResolver

equals

public boolean equals(Object other)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object

toString

public String toString()

Overrides:

toString in class Object