Package | Description |
---|---|
com.norconex.collector.http.robot |
Modifier and Type | Method and Description |
---|---|
IRobotsTxtFilter[] |
RobotsTxt.getAllowFilters()
Gets "Allow" filters.
|
IRobotsTxtFilter[] |
RobotsTxt.getDisallowFilters()
Gets "Disallow" filters.
|
IRobotsTxtFilter[] |
RobotsTxt.getFilters()
Gets all filters.
|
Constructor and Description |
---|
RobotsTxt(IRobotsTxtFilter[] filters) |
RobotsTxt(IRobotsTxtFilter[] filters,
float crawlDelay) |
RobotsTxt(IRobotsTxtFilter[] filters,
String[] sitemapLocations) |
RobotsTxt(IRobotsTxtFilter[] filters,
String[] sitemapLocations,
float crawlDelay) |
Copyright © 2009–2020 Norconex Inc.. All rights reserved.