Search Engines & AI

Google and Bing run a page's JavaScript before they read it, but many of the robots that collect pages for AI tools (GPTBot, ClaudeBot and others) do not: they read the HTML the server sends and nothing more. A site that builds its words in the browser is close to blank to them. This page explains what Domma CMS sends, what still needs JavaScript, and how to check a page yourself.

What every robot sees

Every public page is built on the server. The HTML that arrives already holds:

• the page's words, headings, pictures and links, including blocks, cards, tabs, accordions, collections, blog posts and the Docs and manual pages;
• the title, description and canonical address, and the share card tags for Facebook, X and LinkedIn (set them in the page editor's SEO tab or in Tools > SEO);
• structured data (JSON-LD) saying what the page is - a web page, a blog post, who runs the site - plus the breadcrumb trail;
• the links in the navbar and footer, with their sub-menus, the copyright line and your social links - written as plain links that only robots and screen readers notice, and replaced by the real navbar and footer as the page loads;
• every entry of a collection with search, filters or paging, and what each form is for (its title and description).

/sitemap.xml lists every published page, and /robots.txt lets every robot in except to /admin/ and /api/. Drafts, private pages and pages marked noindex are left out of the sitemap. A few public parts of /api/ are let through - the ones a page reads while it loads, such as a form's questions - so Google, which runs a page's JavaScript, can draw the page as a visitor sees it. Nothing that needs a sign-in is.

llms.txt for AI assistants

/llms.txt is a short Markdown list of the same pages as the sitemap (and your blog posts), each with its title, full address and description, under your site's name and description. It is the format AI assistants look for. Write a description for each page in the page editor's SEO tab and it appears here too.

It is on by default. Switch it off in Tools > SEO > Settings ("Publish /llms.txt for AI assistants"). While "Ask every search engine to stay away" is on, it is withdrawn as well.

Kept out of the way of the page

• Scripts load at the end of the page, after the words. The ones that can wait carry defer, so the browser finishes reading the page first.
• The CMS's own scripts come from your own site, never another company's servers, so no version changes under you without an update. DOMPurify, the library that cleans HTML in the browser, ships with the CMS at a fixed version.
• Pages are compressed and, in production, held in a cache, so a robot gets a quick answer.

What still needs JavaScript

These parts are drawn in the browser, so a robot that runs no JavaScript does not see them:

• Form questions - the form's title and description are in the page, but the questions appear once the page has loaded. Say what the form is for in the text around it too.
• Large interactive collections - a collection set to load its pages from the server (mode="server", or one too big to send whole) sends its first page of entries; the rest arrive as the visitor pages through.
• Site search results, the Calendar's month view and other Tool screens that load live data.

Robots also find your pages through the sitemap and llms.txt, so a page left out of the menus is still found.

Check a page yourself

1. Open the page and choose View page source (Ctrl+U). That is what a robot that runs no JavaScript reads.
2. Search it (Ctrl+F) for a sentence from the page. If it is there, every robot can read it.
3. Run the audit in Tools > SEO for titles, descriptions, headings and alt text across the whole site.