On this page
Site knowledge
Last updated
How vTilt crawls and indexes your website for AI — HTTP crawl plus session-recording fallback, markdown chunks, semantic search, and dashboard review.
Site knowledge is vTilt's per-project index of your website content. It crawls public pages over HTTP, falls back to session recordings for auth-gated or JavaScript-heavy routes, and stores clean markdown chunks with embeddings so AI surfaces can answer product, pricing, and policy questions with citations — not guesses.
Site knowledge is opt-in per project — turn it on under Project → Site Knowledge when you want crawling and AI context from your website.
#Why it exists
Structured page content is ground truth for everything downstream:
| Goal | What site knowledge provides |
|---|---|
| Accurate AI answers | Widget chat, MCP agents, and summarizers cite indexed copy instead of hallucinating |
| Behavior in context | Session summaries attach page content for pathnames a visitor actually opened |
| Ad and landing-page review | Agents compare campaign copy with what /pricing or /landing actually says |
| Operator visibility | Review what was indexed, which source was used, and re-crawl on demand |
Site knowledge stores what your site says today (one row per pathname in site_pages) plus append-on-change version history (site_page_versions) so you can correlate layout/copy changes with metrics — see Limits below.
#Where indexed content is used
| Surface | How it consumes site knowledge |
|---|---|
MCP (site-knowledge toolset) | search_site_knowledge, get_site_page, list_site_pages, list_site_references, get_site_reference, get_site_overview — see Site knowledge (MCP) |
| Widget chat | Site overview (synopsis/features) plus top relevant chunks (and the visitor’s current page when known) pre-injected into the system prompt; search_site_knowledge available as a tool. Turn on under Channels → Website chat (requires Indexing → Use indexed pages in AI) |
| AI memory | Session and conversation summarizers attach === WEBSITE KNOWLEDGE === (and === BUSINESS CONTEXT ===) for visited pathnames via buildSiteContextBlock |
| AI Canvas | Person context includes page chunks for URLs the person opened |
| Campaign AI fields | Generated business context pulls from indexed site chunks |
| VQL | Query site_pages, site_knowledge_chunks, site_page_elements, site_page_versions, and site_overview with semantic :embed('text') |
| References (dashboard) | Upload briefs/PDFs under Site Knowledge → References so RAG includes operator files alongside crawled pages |
Tip: Site knowledge answers questions about your website. For vTilt SDK install steps or MCP auth, use search_docs — not site knowledge.
#How crawling works
vTilt runs a hybrid crawl on a schedule and on manual trigger:
Discover URLs → HTTP fetch → (recording fallback if needed) → merge → chunk → embed → store#1. URL discovery
Discovery runs in two flows that build a priority-ranked URL list:
| Flow | Source | Role |
|---|---|---|
| Usage | Pageviews | Traffic-weighted pathnames from the last 7 days — high-traffic pages first |
| Usage | Session recordings | URLs seen in replay metadata — catches routes HTTP alone misses |
| Structure | Sitemap | Broad coverage from sitemap.xml (and nested indexes) |
| Structure | Link-following | Follows internal links from your landing page — reveals real navigation and pages missing from the sitemap |
| Structure | llms.txt / llms-full.txt | Links from AI-oriented site maps |
| Structure | Seed paths | /, /pricing, /about as bootstrap |
The usage flow captures what your visitors actually see (including auth-gated pages, via recordings); the structure flow maps your public, link-reachable pages. A pathname found by both is kept once and prioritized.
Near-duplicate pages are sampled, not exhaustively crawled. When many pages share a template but differ only by an id (for example thousands of /property/12345 detail pages), the structure flow keeps a representative sample rather than every one — pages your visitors actually open are always kept.
Only hosts on your project's verified domains are fetched. Private IPs and off-domain URLs are blocked (SSRF guard).
#2. HTTP fetch
For each discovered URL, vTilt:
- Fetches the page with
VtiltSiteCrawler/1.0user agent (15 s timeout) - Honors robots.txt
Disallowrules - Parses HTML with JSDOM — title, meta tags, Open Graph, JSON-LD, forms, links
- Converts the body to clean markdown (headings, lists, links, tables preserved) via a Readability-vs-full-DOM dual run with quality fallback
Note: HTTP crawl does not execute JavaScript. When a public page comes back thin or JavaScript-heavy and headless rendering is configured, vTilt re-renders it with a headless browser (Cloudflare Browser Run) to capture the real content before falling back to recordings.
#Bot protection / Crawl access
Some hosts (for example Vercel Bot Protection or Cloudflare Bot Fight Mode) challenge non-browser clients and return a Security Checkpoint page — often with HTTP 200. Without access configured, Site Knowledge can index that interstitial instead of your real content.
Under Site Knowledge → Settings → Crawl access, choose a path that matches your host:
| Option | What to do |
|---|---|
| Allowlist our crawler (preferred) | In Vercel Firewall → Rules, or Cloudflare WAF (Pro+ Super Bot Fight Mode), add a Bypass / Skip rule when User-Agent contains VtiltSiteCrawler. No secret is stored in vTilt. |
| Cloudflare Free Bot Fight Mode | Custom Skip rules cannot bypass Free Bot Fight Mode. Prefer upgrading to Super Bot Fight Mode, use session-recording fallback for gated pages, or wait for published egress IPs. |
| Vercel protection bypass | In Vercel → Project Settings → Deployment Protection → Protection Bypass for Automation, copy the secret and paste it into Crawl access. vTilt sends it as the x-vercel-protection-bypass header on crawl requests. The secret is encrypted at rest and never shown again after save. |
The first-run Site Knowledge setup also shows a Before you index callout with the crawler User-Agent and a link to Crawl access.
After saving a bypass secret (or updating a firewall rule), re-index from Pages so protected URLs refresh.
Warning: If bot protection still blocks the crawler, vTilt detects checkpoint pages and does not index them. Failed pages and the crawl status control deep-link to Settings → Crawl access.
After a successful crawl, vTilt regenerates the project site overview (synopsis + taxonomy). If that refresh fails, indexed pages remain healthy and About this site offers Regenerate.
#3. Recording fallback
When HTTP returns 401/403, is auth-gated, or the extracted body is under ~500 characters, vTilt samples up to five session recordings for that pathname and rebuilds the page from rrweb snapshots (FullSnapshot + DOM mutations). The same markdown engine used for recording transcripts drops nav/chrome and keeps substantive content.
Recording fallback requires:
- Session replay enabled on the project
- Real visitor sessions that reached the pathname (with a FullSnapshot in the recording)
Indexed pages from this path are marked source: recording or hybrid.
#4. Merge, skip, and persist
- HTTP and recording extracts are merged into one canonical page per pathname
- A content hash skips re-embedding when nothing changed (saves cost on daily crawls)
- Sections split on markdown heading boundaries with full heading ancestry (
Pricing > Enterprise) - Chunks (~2 000 characters) are embedded with
text-embedding-3-small(1536 dimensions) - Rows written to ClickHouse:
site_pages,site_page_chunks - After each crawl, an LLM generates a project site overview (synopsis + taxonomy) in
site_overview(best-effort — never fails the crawl; failures surface as a dashboard warning and About → Regenerate)
#Crawl schedule
| Mode | Schedule | Scope |
|---|---|---|
| Usage discovery | Daily (03:00 UTC) | Finds pageview + recording URLs (what visitors actually see) |
| Structure discovery | Weekly (Sunday 04:00 UTC) | Sitemap + link-following + llms.txt (the site's public link graph) |
| Crawling | Continuous (every few minutes) | Works through discovered URLs by priority; unchanged pages are skipped |
| Manual | Re-crawl now in dashboard | Runs both discovery flows, then indexes the pages in the background |
Discovery queues the URLs to crawl; crawling then works through that queue continuously and picks up where it left off, so large sites finish reliably across multiple runs rather than being cut off by a single time-boxed pass. Default cap: 500 pages per crawl (configurable in project settings).
Re-crawl now returns immediately and indexes in the background — the dashboard shows a live progress bar (pages processed, pending, failed). If a very large site does not finish in one pass, a Resume button appears to continue; the background schedule also picks up the remainder. Auto-crawls run per project on your chosen frequency (daily or weekly) and can be paused independently with the auto-crawl toggle without disabling the whole feature.
Under Site Knowledge → Settings → Indexing, you can add excluded paths (prefix, glob, or regex) so matching URLs are not crawled and are omitted from AI retrieval. On Pages, select one or more discovered paths and choose Exclude selected — the same rules apply, with an Excluded status pill on matching rows. Saving rules that match already-indexed pages prompts you to remove those pages from the index.
New projects land on the Site Knowledge setup step after creation: enter your website URL to start indexing. Indexing runs in the background and typically takes a few minutes.
#Output format
The crawl pipeline is optimized for AI retrieval, not raw HTML archival.
| Layer | Format | Stored? | Purpose |
|---|---|---|---|
| Raw HTML | Full HTTP response | No | Transient input to the extractor |
| Markdown chunks | Clean text with heading paths | Yes → site_page_chunks | Semantic search, RAG, MCP citations |
| Page metadata | Title, meta, JSON-LD types, page type, auth flag | Yes → site_pages | Filters, list views, overview |
| Page summary | Short synopsis per pathname | Yes → site_pages.summary | Quick context |
| Site overview | LLM synopsis + taxonomy JSON | Yes → site_overview | High-level orientation for agents |
| Embeddings | 1536-d vectors | Yes | search_site_knowledge, VQL :embed() |
| Media / screenshots | Desktop + mobile full-page PNGs per public page | Optional → site_pages / versions | Off by default; enable Capture screenshots in settings (two viewports per page) |
#Chunk kinds
Each chunk carries a chunk_kind for filtering:
| Kind | Typical content |
|---|---|
body | Main page copy |
pricing | Pricing sections (auto-detected from pathname or content) |
faq | FAQ-style Q&A blocks |
form | Form labels and fields |
meta | Meta description, Open Graph |
structured_data | JSON-LD blocks |
nav | Navigation labels (when extracted) |
ui_label | Short UI strings |
#Crawl sources (per page)
source | Meaning |
|---|---|
http | Public HTTP crawl only |
recording | Extracted from session recordings only |
hybrid | Both HTTP and recording contributed |
#Dashboard
Project → Site Knowledge is a table of indexed pages. Configuration lives in a Settings modal, and clicking a row opens a full page detail view.
| Control | Where | Action |
|---|---|---|
| Indexed pages table | Main area | Search pathnames, filter by source, click a row to open its detail page |
| Status | Toolbar | Live page count, auth-gated count, last discovery, or crawl progress while a run is active |
| Settings | Site Knowledge → Settings → Indexing / Crawl access | Website domain, AI usage, crawl schedule, exclusions; Crawl access for WAF allowlist / Vercel bypass; full-site re-crawl on Pages |
| Page detail | Row click (own page) | Overview (summary, desktop/mobile screenshots, metadata), Content, and History tabs; per-page re-crawl |
#Prerequisites
- Verified domains — add and verify your site host under project settings so the crawler is allowed to fetch it
- Session replay — enable for auth-gated or SPA pages that HTTP cannot render
- Traffic — pageviews and recordings drive URL discovery; low-traffic pages may not be indexed until a full crawl or sitemap picks them up
#Security
- Crawl fetches only verified project domains; private IP ranges are blocked
- robots.txt is respected
- Recording extraction inherits replay masking and block rules from capture settings
- Indexed content is tenant-scoped — agents never cross projects
#Limits
| Topic | Current behavior |
|---|---|
| JavaScript rendering | Public pages that return a thin body are re-rendered with a headless browser (Cloudflare Browser Run) when configured; otherwise SPAs rely on recording fallback or server-rendered HTML |
| Version history | Append-on-change versions in site_page_versions; dashboard History tab + MCP list_page_versions / get_page_version / diff_page_versions |
| Event-timestamp DOM | "What did the page look like at click time?" uses recording analysis, not the crawl index — see AI intelligence (recording_summaries, recording_chunks) |
| Raw HTML archive | Not persisted after extraction |
| Near-duplicate templates | Pages sharing a template that differ only by an id are sampled, not crawled exhaustively (pages visitors open are always kept) |
#Related
- Site knowledge (MCP) — tools, VQL tables, and example agent prompts
- AI intelligence — AI memory, recording summaries, and
:embed()search - Session recording — enable replay for auth-gated page indexing
- MCP server — connect an AI client to your project