Hatim RihWhy your LLM summaries credit other people's posts to your site
Take your publication’s name out of the summary prompt, tell the model never to mention it, and give it the story’s site and author. Rubric’s two summary prompts said each summary was written “for Hrih News”, the name its pipeline still goes by, and 65 of the 1,027 summaries in the editions of 24 to 31 October named it. I took the name out on 31 October, and none of the 209 summaries since has used it.
What the model wrote
The usual openings were “Hrih News covers Doozy, an AI agent that answers questions in Slack threads…” and “Hrih News reports on Timestral…”. Three say outright that someone else’s post was on Hrih News. “A post on Hrih News describes a plan to use nixpkgs’ dependency chain as a blueprint for formally verifying…” is about a post on theorem.dev, and posts on brunborg.com and vhamolya.com got the same opening on 30 October. Every edition had some, between 4 and 12 a day.
On Rubric, a summary’s first sentence is also the story page’s meta description, so a wrong credit can end up in search results.
The line that did it
The summaries for stories ranked 21 and below come from this system prompt. The change, in the sentences that matter:
- You write a one-paragraph summary, at most 50 words, of one story for Hrih News, a daily edition for people who build technology products and companies.
+ You write a one-paragraph summary, at most 50 words, of one story for a daily tech edition read by people who build technology products and companies.
+ Never mention the edition or say the story appeared in it; when the source matters, name its publisher or author.The prompt for the top 20 opened the same way and got the same two changes, the second as one more rule in its list.
The model only knew one publication
The two prompts behaved very differently, and the user messages show why. The top 20 get the source:
`Title: ${item.title}\nSite: ${siteName(item)}${item.author ? `\nAuthor: ${item.author}` : ""}\nURL: ${item.canonical_url}\n\n${body}`The shorter summaries get the title and the text, nothing else:
`Title: ${it.title}\n\n${truncateToTokens(text ?? "", maxTokens).text || "(no text available)"}`Of the 65, one was in the top 20 (of 160 summaries) and 64 below it (of 867). With no site or author to go on, the only publication the model had seen was the one in its instructions, so it used that. “For Hrih News” said who the summary was for, and the model read it as where the story came from.
After the change
| Editions | Summaries | Naming the edition |
|---|---|---|
| 24 to 31 October | 1,027 | 65 |
| 1 November | 59 | 0 |
| 2 November | 43 | 0 |
| 3 November | 107 | 0 |
The change only reaches new summaries, so the 65 already published still say it. And the shorter summaries still don’t get a site or an author, so they can’t name the publisher unless the text does. Passing both, as the top 20’s call does, would close that gap too.