> ## Documentation Index
> Fetch the complete documentation index at: https://docs.open.cx/llms.txt
> Use this file to discover all available pages before exploring further.

# Website

> The Website source crawls a domain you control and ingests its pages into the AI's training index.

The **Website** source crawls a domain you control and ingests its pages into the AI's training index. Use it when the content you want the AI to know about is reachable at a public URL — a help center, a product docs site, a marketing pages surface — and you don't have a direct integration for the system that serves it.

## When to use the crawler vs a direct integration

| Situation                                                           | Use                                                                                                                                                                 |
| ------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| You publish your help center through Zendesk, Intercom, or Front    | The dedicated [Zendesk](/knowledge/sources/zendesk/overview) / [Intercom](/knowledge/sources/intercom/overview) / [Front](/knowledge/sources/front/overview) source |
| Your content lives in Confluence, Notion, GitBook, or Freshdesk     | The dedicated source for that tool                                                                                                                                  |
| Your content is reachable at a public URL and nothing above applies | The crawler                                                                                                                                                         |
| The site is gated behind auth                                       | Not today — the crawler only fetches public URLs                                                                                                                    |

<Warning>
  Don't run both. If you've already connected Zendesk (or any other direct source), don't also crawl the public help-center URL for the same content. The AI will double-index every article and surface duplicates in retrieval.
</Warning>

## What the crawler does

* Kicks off a crawl against the URL you provide.
* Follows links within the domain up to `page_limit` pages (default **100**, max **5000**).
* Extracts main content as Markdown, skipping navigation chrome and binary assets.
* Tracks a content hash per page — re-crawls only re-index pages whose content changed.
* Re-runs on a schedule you pick (`crawl_interval_hours`, default **168 hours / 7 days**).
* Exposes every discovered page so you can exclude individual URLs, re-include ones you excluded, or force-resync one page.

***

## Related Documentation

<CardGroup cols={2}>
  <Card title="Connect a website" icon="plug" href="/knowledge/sources/website/connect">
    URL, limits, include/exclude paths, crawl interval.
  </Card>

  <Card title="Troubleshooting" icon="wrench" href="/knowledge/sources/website/troubleshooting">
    Stuck crawls, locale scoping, pages not indexing.
  </Card>

  <Card title="Crawl API" icon="code" href="/api-reference/crawl">
    Programmatic control of datasources, crawls, and pages.
  </Card>

  <Card title="Connect a knowledge source" icon="brain" href="/knowledge/sources/overview">
    Decision matrix for all sources.
  </Card>
</CardGroup>