Site knowledge

Site knowledge is vTilt's per-project index of your website content. It crawls public pages over HTTP, falls back to session recordings for auth-gated or JavaScript-heavy routes, and stores clean markdown chunks with embeddings so AI surfaces can answer product, pricing, and policy questions with citations — not guesses.

Site knowledge is opt-in per project — turn it on under Project → Site Knowledge when you want crawling and AI context from your website.

#Why it exists

Structured page content is ground truth for everything downstream:

Site knowledge stores what your site says today (one row per pathname in site_pages) plus append-on-change version history (site_page_versions) so you can correlate layout/copy changes with metrics — see Limits below.

#Where indexed content is used

Tip: Site knowledge answers questions about your website. For vTilt SDK install steps or MCP auth, use search_docs — not site knowledge.

#How crawling works

vTilt runs a hybrid crawl on a schedule and on manual trigger:

Discover URLs → HTTP fetch → (recording fallback if needed) → merge → chunk → embed → store

#1. URL discovery

Discovery runs in two flows that build a priority-ranked URL list:

The usage flow captures what your visitors actually see (including auth-gated pages, via recordings); the structure flow maps your public, link-reachable pages. A pathname found by both is kept once and prioritized.

Near-duplicate pages are sampled, not exhaustively crawled. When many pages share a template but differ only by an id (for example thousands of /property/12345 detail pages), the structure flow keeps a representative sample rather than every one — pages your visitors actually open are always kept.

Only hosts on your project's verified domains are fetched. Private IPs and off-domain URLs are blocked (SSRF guard).

#2. HTTP fetch

For each discovered URL, vTilt:

1. Fetches the page with VtiltSiteCrawler/1.0 user agent (15 s timeout)
2. Honors robots.txt Disallow rules
3. Parses HTML with JSDOM — title, meta tags, Open Graph, JSON-LD, forms, links
4. Converts the body to clean markdown (headings, lists, links, tables preserved) via a Readability-vs-full-DOM dual run with quality fallback

Note: HTTP crawl does not execute JavaScript. When a public page comes back thin or JavaScript-heavy and headless rendering is configured, vTilt re-renders it with a headless browser (Cloudflare Browser Run) to capture the real content before falling back to recordings.

#3. Recording fallback

When HTTP returns 401/403, is auth-gated, or the extracted body is under ~500 characters, vTilt samples up to five session recordings for that pathname and rebuilds the page from rrweb snapshots (FullSnapshot + DOM mutations). The same markdown engine used for recording transcripts drops nav/chrome and keeps substantive content.

Recording fallback requires:

• Session replay enabled on the project
• Real visitor sessions that reached the pathname (with a FullSnapshot in the recording)

Indexed pages from this path are marked source: recording or hybrid.

#4. Merge, skip, and persist

• HTTP and recording extracts are merged into one canonical page per pathname
• A content hash skips re-embedding when nothing changed (saves cost on daily crawls)
• Sections split on markdown heading boundaries with full heading ancestry (Pricing > Enterprise)
• Chunks (~2 000 characters) are embedded with text-embedding-3-small (1536 dimensions)
• Rows written to ClickHouse: site_pages, site_page_chunks
• After each crawl, an LLM generates a project site overview (synopsis + taxonomy) in site_overview (best-effort — never fails the crawl)

#Crawl schedule

Discovery queues the URLs to crawl; crawling then works through that queue continuously and picks up where it left off, so large sites finish reliably across multiple runs rather than being cut off by a single time-boxed pass. Default cap: 500 pages per crawl (configurable in project settings).

Re-crawl now returns immediately and indexes in the background — the dashboard shows a live progress bar (pages processed, pending, failed). If a very large site does not finish in one pass, a Resume button appears to continue; the background schedule also picks up the remainder. Auto-crawls run per project on your chosen frequency (daily or weekly) and can be paused independently with the auto-crawl toggle without disabling the whole feature.

New projects land on the Site Knowledge setup step after creation: enter your website URL to start indexing. Indexing runs in the background and typically takes a few minutes.

#Output format

The crawl pipeline is optimized for AI retrieval, not raw HTML archival.

#Chunk kinds

Each chunk carries a chunk_kind for filtering:

#Crawl sources (per page)

#Dashboard

Project → Site Knowledge is a table of indexed pages. Configuration lives in a Settings modal, and clicking a row opens a full page detail view.

#Prerequisites

1. Verified domains — add and verify your site host under project settings so the crawler is allowed to fetch it
2. Session replay — enable for auth-gated or SPA pages that HTTP cannot render
3. Traffic — pageviews and recordings drive URL discovery; low-traffic pages may not be indexed until a full crawl or sitemap picks them up

#Security

• Crawl fetches only verified project domains; private IP ranges are blocked
• robots.txt is respected
• Recording extraction inherits replay masking and block rules from capture settings
• Indexed content is tenant-scoped — agents never cross projects

#Limits

#Related

• Site knowledge (MCP) — tools, VQL tables, and example agent prompts
• AI intelligence — AI memory, recording summaries, and :embed() search
• Session recording — enable replay for auth-gated page indexing
• MCP server — connect an AI client to your project