Discovery

Whether your site is found — by search engines, by AI assistants, and by people who want it on their phone. Both halves are things the site does by itself.

“Does ChatGPT find my site?” has the same answer as “does Google find my site?”: yes, by construction. Content is stored as Schema.org, the vocabulary search engines and AI assistants read. A product page says it is a Product — name, price, image, description — in the fields a machine expects. One file renders the page and the structured data, so the two cannot disagree.

The machine views

Next to the pages, the site serves a set of addresses for machines, generated from the live content on request: the map for assistants (/llms.txt), the index of the content (/data.json, or /data.md — the same index as a list; ?q= searches either — ranked hits with a snippet, paged with a cursor, never the full record), every entity at its own address under /data/<type>/, the sitemap (/sitemap.xml for crawlers; /sitemap.json and /sitemap.md the same page register for code and for reading — every page with its type, file and name, no bodies: the light way to know what pages exist), the feed and robots.txt. Nothing is exported, cached or maintained by hand; unpublished drafts never appear. The engine’s own documentation is not among them: these docs are read through the MCP (search { in: "docs" }, fetch "docs") — as are the site’s own rules in guide/, which are never served. The one list of every claimed address, what each is and what you may change, is What your site serves; /admin shows the same addresses under Discover, for a person who wants to look. Here: how they are read.

The discovery hierarchy

Knowledge first, then breadth, then depth — the order an agent reads in is the MCP (how to write here), the guide (this site’s own rules, fetch "guide"), the docs (how CTXR works, search { in: "docs" }), the index (/data: what is here), the sitemap (where). robots.txt comes first for crawlers: one absolute sitemap line per language, the content signals, and pointers to the MCP, the wall, the index and llms.txt. llms.txt is the first line of discovery proper: a heading and summary from the organisation, the curated sections in your order, a closing Pages section with every page the curation skipped — so the index is complete however you curated — and the deeper routes. A page needs a description to be listed: llms.txt invents nothing, so a page without one stays out and the audit says so.

The index is the depth. /data.json is one Schema.org DataCatalog: one Dataset per type, each describing its field shape and linking to the type’s own index at /data/<type>.json; /data.md is the same index as a list — per type a table of identifier, name, address and one line — for a reader that wants to know what is there before it parses anything. The extension chooses the form, never a request header, so an address always means one thing. Searching (?q=<words>, on the whole index or one type, in either form; per language on that language’s own index, /nl/data.json) answers with hits, not records, and every hit is a member of the index searched: an entity the language lacks is never found there, whatever words it shares, while a page written in another language is found where its slug is and an anchor in every language. A hit carries @type, identifier (the file), name and a snippet around the words, plus @id and url where the entity is a page or an anchor — the page’s address the register’s, so of two entities that claim one slug only the first carries it — and an entity without a page is found all the same. Hits are ranked in tiers: an exact phrase first, then matches in names and identifiers, then in descriptions, then in bodies; a lower tier never outweighs a higher one. numberOfItems is the total, and a ReadAction “next page” carries an opaque cursor while there is more — the page stays schema; in the .md form the hits are a table and the cursor a Next line. A cursor belongs to its words, its scope (the whole index or one type) and its language, and to the index as it was: anywhere else it is refused, after a publish it says so and the search starts over. Precision is the next step, and it is a file: /data/<type>/<identifier>.json is the entity as it lies on disk — every language inside it, references as stubs, no @id — the same bytes the MCP’s fetch returns; /data/<type>/<identifier>.md is its reading, the fields in the page’s language as front matter and the body as prose. The leaf is the source, the folder is the reading. An agent on the MCP gets the same hits from search without opening a URL, lang choosing the language — and search reaches further than the URL does: in: "docs" for how CTXR works, in: "theme", "assets" or "guide" for the site’s own files, behind that root’s read right.

The two forms are for two readers. The .json form is for code: complete, typed, parseable. The .md form is for a mind: the same facts, fewer tokens, readable as they are. An agent reads the list, searches, and fetches the one file it needs.

The index changes with every publish — remember it only with its dateModified, and ask conditionally (304) before trusting a kept copy. The docs change rarely; an agent with a memory may keep what it read.

What the site tells AI about itself

A CTXR site declares, in the standard way, that its content may be used for search, as input for AI answers, and for training AI.

This stance is not configurable today. If your organisation needs to withhold content from AI training, a CTXR site as it stands is not the right fit for that content, and it is better to know that now than later.

The machine views never contain anything a visitor cannot see: drafts, the inbox, keys and history are outside them.

What you control

The views write themselves from the data; the data shapes findability.

  • Titles and descriptions in plain language, an image with a caption, an address that says what the page is. The engine caps no length; search and AI readers do, so keep titles compact and descriptions to about a sentence.
  • The curated sections of llms.txt (SITE_LLMS_SECTIONS) — what a reader sees first: a section reads a type (every entity of it, from the index) or a list you keep by hand. Curation steers order and emphasis, never existence.
  • Item lists as the spine — a page in a list is a line in llms.txt and a member of the index for free.
  • The discovery links in the head, pointing readers at the feed and the index.

Ask your agent “how does this page look to an AI” and it reads the machine view back to you.

Your site as an app

A CTXR site can be installed on a phone or laptop from the browser, opens with its own icon, and keeps working without a connection. Nothing is built, submitted to a store or maintained separately: the manifest (/manifest.json), the service worker (/sw.js) and the offline page (/offline) are engine paths, derived from the site like the sitemap is. Why it is built this way: An app by default.

Installing

A visitor adds the site to their home screen from the browser’s menu; /install shows the steps per browser. The site pushes no install prompt — the browser knows when someone returns often enough to want one.

Installing is a property of the device: no account, nothing about the person sent anywhere, undone by removing the icon. See Anonymous by default.

Offline

Once installed, the site keeps a copy of its pages on the device — what the visitor has opened, and once a day the rest of the site from its own sitemap — and shows a proper offline page for anything it lacks. With a connection the live page wins and the copy is refreshed. Nothing syncs the other way: reading needs no account.

What comes from your site

The app’s name, icon and colours come from what the site already says about itself: the organisation’s name, the site’s icon, the theme colours. Change them in the site and the app follows. A designer can shape the install page or the manifest further; a site that never touches them is a complete app already.

What it is not

Not a native app: no push notifications, camera access or app-store presence. The trade: no store review, no second codebase, no update cycle, and a visitor who wants the site on their phone has it in two taps.