Three primitives — Fetch, Map, Crawl — powered by the same stealth engine. Get an API key, send a request, get clean data.
Get an API key from /api-keys, then run:
curl -X POST http://localhost/api/v1/fetch \
-H "x-api-key: $GYD_API_KEY" \
-H "Content-Type: application/json" \
-d '{"urls": ["https://example.com"]}'Send your API key as the x-api-key header on every request. Keys are scoped per workspace and revocable from the dashboard.
Base URL: http://localhost/api/v1
Returns the rendered HTML and clean Markdown for one or more URLs.
curl -X POST http://localhost/api/v1/fetch \
-H "x-api-key: $GYD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"urls": ["https://example.com"],
"proxy": { "type": "datacenter" }
}'Walks sitemaps and robots.txt to return every reachable URL on a domain.
curl -X POST http://localhost/api/v1/map \
-H "x-api-key: $GYD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"urls": ["https://example.com"],
"proxy": { "type": "residential" }
}'Discovers URLs from the seed page and fetches each one through the same stealth pipeline as /fetch. Returns clean Markdown per page plus a downloadable bundle.
Note: max_depth is currently capped at 1. Multi-depth traversal is on the roadmap.
curl -X POST http://localhost/api/v1/crawl \
-H "x-api-key: $GYD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"urls": ["https://example.com"],
"max_depth": 1,
"max_pages": 100,
"proxy": { "type": "datacenter" }
}'All three endpoints return a job_id that you poll until status is completed or failed.
curl http://localhost/api/v1/fetch/<job_id> \ -H "x-api-key: $GYD_API_KEY"
Every POST returns a queued job descriptor. Single-URL requests get convenience fields job_id and poll_url; batch requests use the plural job_ids + poll_urls arrays.
{
"service": "fetch",
"status": "queued",
"count": 1,
"accepted_count": 1,
"rejected_count": 0,
"duplicates_removed": 0,
"rejected": [],
"job_ids": ["batch_abc123"],
"poll_urls": ["/api/v1/fetch/batch_abc123"],
"job_id": "batch_abc123",
"poll_url": "/api/v1/fetch/batch_abc123"
}Safe-retry any POST by sending an Idempotency-Key header. Repeated calls with the same key return the original response and don't double-charge or duplicate work. Keys are scoped per API key and expire after 24 hours.
curl -X POST http://localhost/api/v1/fetch \
-H "x-api-key: $GYD_API_KEY" \
-H "Idempotency-Key: my-unique-request-id" \
-H "Content-Type: application/json" \
-d '{"urls": ["https://example.com"]}'Skip polling — pass webhook_url in your request body and we'll POST the completion payload to your endpoint. Add webhook_secret to receive an x-gyd-signature: sha256=<hex> HMAC header so you can verify the call came from us.
curl -X POST http://localhost/api/v1/fetch \
-H "x-api-key: $GYD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"urls": ["https://example.com"],
"webhook_url": "https://your-server.com/gyd-hook",
"webhook_secret": "your-shared-hmac-secret"
}'Retries with exponential backoff: 2s → 8s → 32s. Webhook is fire-and-forget — your endpoint failures don't fail the original job.
Always verify x-gyd-signature before trusting the payload — it confirms the request came from us and wasn't replayed or tampered with. Use a constant-time comparison (crypto.timingSafeEqual / hmac.compare_digest) to avoid timing attacks.
Node.js (Express)
import crypto from "crypto";
// Express handler. Use raw-body so the HMAC matches what we signed.
app.post("/gyd-hook", express.raw({ type: "application/json" }), (req, res) => {
const signature = req.header("x-gyd-signature") || "";
const expected = "sha256=" + crypto
.createHmac("sha256", process.env.GYD_WEBHOOK_SECRET)
.update(req.body) // raw Buffer
.digest("hex");
if (!crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(expected))) {
return res.status(401).send("invalid signature");
}
const payload = JSON.parse(req.body.toString("utf8"));
// ... handle payload (job_id, service, status, outputs, ...)
res.json({ ok: true });
});Python (Flask)
import hmac, hashlib, os
from flask import Flask, request, abort
app = Flask(__name__)
@app.post("/gyd-hook")
def gyd_hook():
signature = request.headers.get("x-gyd-signature", "")
expected = "sha256=" + hmac.new(
os.environ["GYD_WEBHOOK_SECRET"].encode(),
request.get_data(), # raw bytes
hashlib.sha256,
).hexdigest()
if not hmac.compare_digest(signature, expected):
abort(401)
payload = request.get_json()
# ... handle payload
return {"ok": True}Errors return a JSON body with error and an HTTP status code:
| Status | Meaning |
|---|---|
| 400 | Invalid request body or unsupported parameter. |
| 401 | Missing x-api-key or unauthenticated dashboard session. |
| 402 | Out of credits — top up from the dashboard. |
| 403 | Invalid API key, or key lacks the scope for this endpoint (fetch / map / crawl). |
| 404 | Job ID not found or expired. |
| 429 | Rate limit exceeded — retry with exponential backoff. |
| 5xx | Server-side error — safe to retry. |
The full API contract is available as an OpenAPI 3.0 spec. Import it into Postman, Insomnia, or your editor of choice for autocomplete and request testing.
Download openapi.v1.yamlHave questions or want SDK support for your language? Get in touch.