Watching changes: feed + webhooks

CrawlPipe gives you two ways to know when a document (or a specific khoản inside it) changes: poll the changes feed, or subscribe to webhooks. The feed is always the source of truth — webhooks are a latency optimization on top of it.

The changes feed

GET /v1/legal/changes (1 unit, Builder+) is an append-only, cursor-paginated event log ordered strictly by id ascending.

curl "https://api.crawlpipe.com/v1/legal/changes?since=0&limit=100" \
-H "Authorization: Bearer dp_live_a1b2c3d4e5f6…"

import requests

cursor = "0"  # 0 = genesis, replay full history
resp = requests.get(
  "https://api.crawlpipe.com/v1/legal/changes",
  params={"since": cursor, "limit": 100},
  headers={"Authorization": "Bearer dp_live_a1b2c3d4e5f6…"},
)
body = resp.json()
events = body["data"]["events"]
next_cursor = body["meta"]["pagination"]["next_cursor"]

let cursor = "0";
const res = await fetch(
`https://api.crawlpipe.com/v1/legal/changes?since=${cursor}&limit=100`,
{ headers: { Authorization: "Bearer dp_live_a1b2c3d4e5f6…" } },
);
const { data, meta } = await res.json();
cursor = meta.pagination.next_cursor ?? cursor;

Persist meta.pagination.next_cursor and pass it back as since on your next poll. since=0 replays full history from genesis (paged, not one giant response) — use it once to backfill, then switch to incremental polling with the last cursor you saw.

Each event:

{
  "id": 12345,
  "type": "record.updated",
  "subtype": "units_amended",
  "occurred_at": "2026-08-01T04:12:09Z",
  "record": { "id": "…", "natural_key": "vn:tt:2026:18:btc", "doc_number": "18/2026/TT-BTC", "title": "…" },
  "affected_units": ["d3.k2"],
  "diff": { "…": "structured unit-level diff" }
}

diff is structural (added/removed/changed unit paths and metadata deltas), not a raw text diff — small enough to keep in the feed. To read the new text, fetch GET /v1/legal/documents/{id} for the affected affected_units paths.

Webhooks

Create a subscription once (portal UI or POST /v1/webhooks), and CrawlPipe pushes matching events to your endpoint as they happen — usually within seconds of the feed entry, not on your polling interval.

POST /v1/webhooks
{
  "url": "https://example.com/hooks/crawlpipe",
  "domain": "legal-vn",
  "event_types": ["record.updated"],
  "filters": { "doc_types": ["thong_tu"], "fields": ["thue"] }
}

The response includes secret (whsec_…) exactly once — store it, it's what you use to verify deliveries.

Delivery payload

{
  "id": "evt_000000012345",
  "type": "record.updated",
  "subtype": "units_amended",
  "domain": "legal-vn",
  "occurred_at": "2026-08-01T04:12:09Z",
  "data": {
    "record_id": "0198…",
    "natural_key": "vn:tt:2026:18:btc",
    "doc_number": "18/2026/TT-BTC",
    "affected_units": ["d3.k2"],
    "diff": { "…": "…" },
    "links": { "document": "https://api.crawlpipe.com/v1/legal/documents/0198…" }
  }
}

Verifying `X-CrawlPipe-Signature`

Every delivery carries X-CrawlPipe-Signature: t={unix_ts},v1={hex_hmac} where hex_hmac = hex(hmac_sha256(secret, "{t}.{raw_body}")). Recompute it and compare in constant time; also reject anything with clock skew over 5 minutes (replay guard).

# illustrative only — verification happens in your server code, not curl
echo -n "${t}.${body}" | openssl dgst -sha256 -hmac "${secret}"

import hashlib
import hmac
import time

def verify(secret: str, header: str, raw_body: bytes) -> bool:
  parts = dict(p.split("=", 1) for p in header.split(","))
  ts, sig = int(parts["t"]), parts["v1"]

  if abs(time.time() - ts) > 300:
      return False

  expected = hmac.new(
      secret.encode(), f"{ts}.{raw_body.decode()}".encode(), hashlib.sha256
  ).hexdigest()
  return hmac.compare_digest(expected, sig)

import { createHmac, timingSafeEqual } from "node:crypto";

function verify(secret: string, header: string, rawBody: string): boolean {
const parts = Object.fromEntries(header.split(",").map((p) => p.split("=")));
const ts = Number(parts.t);
if (Math.abs(Date.now() / 1000 - ts) > 300) return false;

const expected = createHmac("sha256", secret)
  .update(`${ts}.${rawBody}`)
  .digest("hex");
return timingSafeEqual(Buffer.from(expected), Buffer.from(parts.v1));
}

Guarantees