Docs · Intelligence

Robots compliance check

Is this path allowed for a given user agent?

POST /v1/intel/robots

Reads the target site's robots.txt and answers whether a URL is allowed for the user agent you specify.

Parameters

Name Type Required Default Description
url string yes — URL to check.
user_agent string no * User agent string to check against.

Request

curl -X POST https://api.datasonar.dev/v1/intel/robots \
  -H "Authorization: Bearer osk_..." \
  -d '{"url": "https://anthropic.com/pricing", "user_agent": "*"}'

Response

{
  "status": "success",
  "target_url": "https://anthropic.com/pricing",
  "user_agent": "*",
  "is_allowed": false,
  "robots_url": "https://anthropic.com/robots.txt"
}

Related

POST /v1/intel/sitemap
Sitemap unroll
POST /v1/crawl
Full-site crawl