| Package | Description |
|---|---|
| com.norconex.collector.http.robot |
| Modifier and Type | Method and Description |
|---|---|
IRobotsTxtFilter[] |
RobotsTxt.getAllowFilters()
Gets "Allow" filters.
|
IRobotsTxtFilter[] |
RobotsTxt.getDisallowFilters()
Gets "Disallow" filters.
|
IRobotsTxtFilter[] |
RobotsTxt.getFilters()
Gets all filters.
|
| Constructor and Description |
|---|
RobotsTxt(IRobotsTxtFilter[] filters) |
RobotsTxt(IRobotsTxtFilter[] filters,
float crawlDelay) |
RobotsTxt(IRobotsTxtFilter[] filters,
String[] sitemapLocations) |
RobotsTxt(IRobotsTxtFilter[] filters,
String[] sitemapLocations,
float crawlDelay) |
Copyright © 2009–2020 Norconex Inc.. All rights reserved.