Skip to content
Foldspace Docs
Talk to an engineer

Knowledge

Crawl a website or help center

Path: Agent Studio → Knowledge → Sources → + Add Source → Crawl Website or Help Center

Crawling is the fastest way to onboard an existing public knowledge base, docs portal, or support site in one step. You give it a starting URL, and it indexes the whole site, following links to pull in every page it finds. It re-crawls on a schedule, so published changes flow in without manual updates.

Reach for it when the canonical content is already live on your own domain and is publicly reachable.

Crawl, single page, or API connector?

Section titled “Crawl, single page, or API connector?”

Three source types bring in web content, and they aren’t interchangeable:

  • Crawl indexes a whole public site from a starting URL. Best when the content lives on your domain and you want all of it.
  • Add a single web page brings in exactly one public page without crawling around it. Use it for a specific page, or to top up a crawl with a page that lives elsewhere. See Adding knowledge sources.
  • Help center connectors (Intercom, Zendesk) authenticate over the platform API instead of crawling HTML. Use these when your help center sits behind a customer login, or when you want structured, reliable syncing rather than scraping rendered pages.

The quick rule: if your help center is public, crawling works; if it’s gated, use the API connector.

Set up a crawl

Section titled “Set up a crawl”
  1. Click + Add Source and choose Crawl Website or Help Center.
  2. Enter the starting URL. The crawler follows links from here, so point it at the top of the section you want, for example the root of your help center or docs.
  3. Start the crawl. The first pass indexes what it finds and adds each page to the article list.

After crawling

Section titled “After crawling”

Each crawled page appears in the main article list with its source URL and Synced date. Confirm the crawl pulled in what you expected before you rely on the agent to answer from it. See Validating & improving.

The crawler re-runs on a schedule, so edits to the live site flow in automatically. If a page isn’t being retrieved even though it was crawled, that’s a retrieval issue rather than a source one. See Pointers.

Section titled “Related”