public interface IRobotsTxtProvider
Modifier and Type | Method and Description |
---|---|
RobotsTxt |
getRobotsTxt(HttpClient httpClient,
String url,
String userAgent)
Gets robots.txt rules.
|
RobotsTxt getRobotsTxt(HttpClient httpClient, String url, String userAgent)
httpClient
- the http client to grab robots.txturl
- the URL to derive the robots.txt fromuserAgent
- the User-Agent to match ourselves with the robot rulesCopyright © 2009–2020 Norconex Inc.. All rights reserved.