Say hello
Menu
All notes
NoteHatim Rih

Serve llms.txt from an Astro endpoint, built from your posts

Astrollms.txtSEO

Add an endpoint at src/pages/llms.txt.ts that builds the file from your content collections, with the same filter as your RSS feed. Astro writes it as a static llms.txt at build time, a post shows up there the day it’s published and never while it’s a draft, and nobody edits the file by hand. This is hrih.si’s, shortened:

import type {APIRoute} from 'astro';
import {getCollection} from 'astro:content';
import {writingUrl, noteUrl, listedNotes} from '../lib/content';

const line = (title: string, url: string, text: string) => `- [${title}](${url}): ${text}`;

export const GET: APIRoute = async ({site}) => {
  const abs = (path: string) => new URL(path, site).href;
  const posts = (await getCollection('writing', ({data}) => !data.draft && !data.sample))
    .sort((a, b) => b.data.date.valueOf() - a.data.date.valueOf());
  const notes = await listedNotes();
  const text = [
    '# hrih.si', '',
    '> Hatim Rih builds software and writes about AI and the work of programming.', '',
    '## Articles', ...posts.map(post => line(post.data.title, abs(writingUrl(post)), post.data.description)), '',
    '## Notes', ...notes.map(note => line(note.data.title, abs(noteUrl(note)), note.data.description)), '',
    '## Optional', line('Feed', abs('/rss.xml'), 'every essay and article in full, newest first'), '',
  ].join('\n');
  return new Response(text, {headers: {'Content-Type': 'text/plain; charset=utf-8'}});
};

The shape comes from llmstxt.org: a title, a one-line summary as a quote, then sections of links with a short description each. A section called Optional holds what a tool can skip when it’s short of room. The full file also has essays and articles under separate headings, the projects, and an About section.

Reuse the feed’s filter

The posts come through the same query as /rss.xml, and the notes through listedNotes(), the function the notes page uses. Whatever keeps a draft out of the feed keeps it out of llms.txt, and there’s no second list to forget.

Stop the release when it misses a post

Since 3 November, hrih.si’s release script reads the built file and stops if any item in either feed, or any project page in the sitemap, isn’t in it:

const llms = readFileSync('build/llms.txt', 'utf8');
const published = [
  ...['build/rss.xml', 'build/notes/rss.xml'].flatMap(file =>
    [...readFileSync(file, 'utf8').matchAll(/<item>[\s\S]*?<link>([^<]+)<\/link>/g)].map(match => match[1])),
  ...[...readFileSync('build/sitemap-0.xml', 'utf8').matchAll(/<loc>([^<]+\/projects\/[^<]+\/)<\/loc>/g)].map(match => match[1]),
];
const unlisted = published.filter(url => !llms.includes(`](${url})`));
if (!published.length || unlisted.length) throw new Error(`build/llms.txt is missing ${unlisted.join(', ')}`);

Check what AI crawlers get

robots.txt can allow everything while a firewall or a CDN rule still blocks a crawler, so ask as the crawler:

curl -s -o /dev/null -w "%{http_code} %{content_type}\n" -A "GPTBot" https://hrih.si/llms.txt
curl -s -o /dev/null -w "%{http_code} %{content_type}\n" -A "ClaudeBot" https://hrih.si/llms.txt

Both answer 200 text/plain; charset=utf-8.

What it’s worth

Google’s guide to its AI features says Search doesn’t use llms.txt, and that keeping one neither helps nor hurts. Between 18 and 31 October, Umami counted one visit to hrih.si from claude.ai, against 131 from google.com. It isn’t a traffic source. The file costs nothing to keep current, so I keep it for the assistants and agents that look for one, and Chrome’s Lighthouse checks for it in its agentic browsing audits.