Authorizations
Bearer authentication header of the form Bearer <token>, where <token> is your auth token.
Body
application/json
The URL to crawl
Example:
"https://example.com"
Maximum number of pages to crawl (1-1000)
Required range:
1 <= x <= 1000Example:
100
Paths to exclude from crawling
Example:
["/blog/*", "/private/*"]Whether to allow external links
Example:
true
Paths to include in crawling
Example:
["/blog/*", "/private/*"]Response
Crawl job has been created successfully.
Available options:
cancelled, completed, failed, scraping