agenthub · knowledge.crawl
Knowledge Crawl
Fetch every same-host page reachable from a starting URL, breadth-first, up to a hard page cap. The front of the ingestion chain.
Why the node earns its place
The first of five nodes that turn a website into searchable knowledge: crawl, filter, chunk, embed, store. It emits a URL, title and text per page, ready for the rest of the chain.
The page cap is not a suggestion — it is what keeps a crawl bounded on a site whose link graph is effectively infinite.
How it works at run time
- 1Fetch the start URL, then follow same-host links breadth-first.
- 2Stop at
max_pages; abandon any page that exceeds the per-page timeout. - 3Emit the pages collected.
What you wire
Inputs
- urlstring
Outputs
- pagesarray
Configuration
| Key | Type | Default & options | What it does |
|---|---|---|---|
| max_pages | number | default 10 | Hard cap on pages crawled. Defaults to 10 for slice 1. |
| timeout_seconds | number | default 15 | Per-page HTTP timeout. |
What usually goes wrong
Watch for this
No JavaScript is rendered. A marketing site built as a heavy single-page app returns empty bodies — the crawl appears to succeed and produces nothing usable. Check the extracted text, not just the page count.
This node fetches URLs an operator supplies and follows redirects, so some deployments deliberately do not register it at all. If it is missing from your builder, that is why.
Behaviour & provenance