RSS knowledge items: recency-based retrieval and configurable expiry

timstoop commented

2026-03-30 07:32:03 +00:00

(Migrated from codeberg.org)

We have been using OmniMem with an RSS feed setup to surface relevant articles during sessions, and we love the concept. After some real-world use, we ran into two gaps that limit how useful it is in practice, and we wanted to raise them before building anything, to make sure a contribution would be welcome and that the approach fits your vision.

How we are using it

We configure RSS feeds covering topics relevant to our work (Kubernetes, security, SRE, etc.) and rely on recall() and the briefing's new-knowledge section to surface relevant articles. The idea is to have a lightweight "what's new in my field" capability without leaving Claude Code.

The mental model we are working towards

RSS feeds are a firehose: most items are not relevant and should expire automatically. The knowledge namespace should become a curated collection, where an item that turns out to be genuinely useful gets promoted to permanent status. So the flow would be:

RSS ingests articles with a short expiry (e.g. 30 days).
During a session, the LLM surfaces a relevant article via recall or a recency tool.
If it is genuinely useful, the user or LLM clears expires_at, and it stays in the knowledge namespace indefinitely.
Everything else expires quietly.

This keeps the knowledge namespace lean and meaningful rather than a growing pile of half-relevant RSS history.

Problem 1: No way to query by recency

The recall pipeline applies recency decay (a scoring penalty after 90 days), but there is no way to ask "show me knowledge items from the last 24 hours" or "what was ingested today." The briefing's _get_new_knowledge() does a 7-day scan internally, but it is not exposed as a standalone tool and is capped at 10 items with no topic or feed filtering.

In practice this means we cannot use OmniMem as a daily news digest. We have to use recall() with a topic query and hope recent articles bubble up, but older items with strong vector similarity can outrank newer ones even when recency is what matters.

Problem 2: RSS-ingested items accumulate without bound

RSS-ingested knowledge items (those with feed_name set) have no expiry and no automatic cleanup. The 20-article-per-feed limit applies per ingestion run, not in total. With daily ingestion across multiple active feeds, these items accumulate indefinitely. Knowledge items are also not included in the existing auto-maintenance deduplication, which only covers episodic.

What we would suggest

Expiry via expires_at field (application-managed, not Valkey TTL)

At ingest time, set an expires_at field on each RSS-ingested item: created_at + MAX_KNOWLEDGE_AGE_DAYS (configurable env var, e.g. default 30 days). During auto-maintenance, scan RSS-ingested knowledge items and archive or delete any where expires_at has passed.

To promote an article to permanent status, the user or LLM simply clears expires_at. Manually stored knowledge items via remember(namespace="knowledge") would never have expires_at set in the first place, so they are unaffected.

This keeps lifecycle management entirely within the application, consistent with the existing maintenance pattern, and makes cleanup visible and auditable rather than silently disappearing via a Valkey-level TTL.

A recent_knowledge MCP tool

A standalone tool that returns knowledge items filtered by created_at or published_at within a given window (e.g. last 1/7/30 days), optionally filtered by feed_name or topics. This would make the "daily news digest" use case work directly without relying on recall scoring to surface recent articles.

Questions for you

Does this fit how you intended the knowledge namespace to be used?
Would a contribution covering these two features be welcome?
Any preference on whether recent_knowledge should be a separate tool or an additional parameter on an existing one?
Any concerns about the expires_at field approach, or preference for a different retention mechanism?

We are happy to build this if you are open to it, and would follow the existing patterns (fake stores in tests, no Docker in test suite, auto-maintenance hook for cleanup). Just wanted to align on the design first.

Fallback

If this is not the direction you want to take the RSS integration, we could always use a separate RSS reader and selectively push relevant articles into OmniMem via remember(). That said, it would reduce the value of the built-in RSS worker significantly, since the main appeal is having ingestion, embedding, and recall all in one place.

We have been using OmniMem with an RSS feed setup to surface relevant articles during sessions, and we love the concept. After some real-world use, we ran into two gaps that limit how useful it is in practice, and we wanted to raise them before building anything, to make sure a contribution would be welcome and that the approach fits your vision. **How we are using it** We configure RSS feeds covering topics relevant to our work (Kubernetes, security, SRE, etc.) and rely on `recall()` and the briefing's new-knowledge section to surface relevant articles. The idea is to have a lightweight "what's new in my field" capability without leaving Claude Code. **The mental model we are working towards** RSS feeds are a firehose: most items are not relevant and should expire automatically. The knowledge namespace should become a curated collection, where an item that turns out to be genuinely useful gets promoted to permanent status. So the flow would be: 1. RSS ingests articles with a short expiry (e.g. 30 days). 2. During a session, the LLM surfaces a relevant article via recall or a recency tool. 3. If it is genuinely useful, the user or LLM clears `expires_at`, and it stays in the knowledge namespace indefinitely. 4. Everything else expires quietly. This keeps the knowledge namespace lean and meaningful rather than a growing pile of half-relevant RSS history. **Problem 1: No way to query by recency** The recall pipeline applies recency *decay* (a scoring penalty after 90 days), but there is no way to ask "show me knowledge items from the last 24 hours" or "what was ingested today." The briefing's `_get_new_knowledge()` does a 7-day scan internally, but it is not exposed as a standalone tool and is capped at 10 items with no topic or feed filtering. In practice this means we cannot use OmniMem as a daily news digest. We have to use `recall()` with a topic query and hope recent articles bubble up, but older items with strong vector similarity can outrank newer ones even when recency is what matters. **Problem 2: RSS-ingested items accumulate without bound** RSS-ingested knowledge items (those with `feed_name` set) have no expiry and no automatic cleanup. The 20-article-per-feed limit applies per ingestion *run*, not in total. With daily ingestion across multiple active feeds, these items accumulate indefinitely. Knowledge items are also not included in the existing auto-maintenance deduplication, which only covers episodic. **What we would suggest** **Expiry via `expires_at` field (application-managed, not Valkey TTL)** At ingest time, set an `expires_at` field on each RSS-ingested item: `created_at + MAX_KNOWLEDGE_AGE_DAYS` (configurable env var, e.g. default 30 days). During auto-maintenance, scan RSS-ingested knowledge items and archive or delete any where `expires_at` has passed. To promote an article to permanent status, the user or LLM simply clears `expires_at`. Manually stored knowledge items via `remember(namespace="knowledge")` would never have `expires_at` set in the first place, so they are unaffected. This keeps lifecycle management entirely within the application, consistent with the existing maintenance pattern, and makes cleanup visible and auditable rather than silently disappearing via a Valkey-level TTL. **A `recent_knowledge` MCP tool** A standalone tool that returns knowledge items filtered by `created_at` or `published_at` within a given window (e.g. last 1/7/30 days), optionally filtered by `feed_name` or `topics`. This would make the "daily news digest" use case work directly without relying on recall scoring to surface recent articles. **Questions for you** - Does this fit how you intended the knowledge namespace to be used? - Would a contribution covering these two features be welcome? - Any preference on whether `recent_knowledge` should be a separate tool or an additional parameter on an existing one? - Any concerns about the `expires_at` field approach, or preference for a different retention mechanism? We are happy to build this if you are open to it, and would follow the existing patterns (fake stores in tests, no Docker in test suite, auto-maintenance hook for cleanup). Just wanted to align on the design first. **Fallback** If this is not the direction you want to take the RSS integration, we could always use a separate RSS reader and selectively push relevant articles into OmniMem via `remember()`. That said, it would reduce the value of the built-in RSS worker significantly, since the main appeal is having ingestion, embedding, and recall all in one place.

ric_harvey commented

2026-03-30 09:46:26 +00:00

(Migrated from codeberg.org)

This sounds an excellent addition. Let's do it, will we implement a way for a user to mark knowledge as useful so it doesn't expire?

One of the main things I've been doing is following a git rss feed for releases, this means the agent "should" always know the latest version.

I have other news also like you are describing, and things like new features in products is handy.

This context might help with shaping the reader.

This sounds an excellent addition. Let's do it, will we implement a way for a user to mark knowledge as useful so it doesn't expire? One of the main things I've been doing is following a git rss feed for releases, this means the agent "should" always know the latest version. I have other news also like you are describing, and things like new features in products is handy. This context might help with shaping the reader.

ric_harvey commented

2026-03-30 10:14:28 +00:00

(Migrated from codeberg.org)

I've created a new branch for us to work against:

https://codeberg.org/ric_harvey/omnimem/src/branch/feat/rss-reader-improvements

this will help me with merging other code if you can merge/pull request against that branch please

I've created a new branch for us to work against: https://codeberg.org/ric_harvey/omnimem/src/branch/feat/rss-reader-improvements this will help me with merging other code if you can merge/pull request against that branch please

timstoop commented

2026-03-30 17:53:36 +00:00

(Migrated from codeberg.org)

So I mainly worked on it from the POV of an LLM using an MCP, as although the dashboard looks nice, I seldom use it. Would that be good enough for you as well? How were you handling archiving/deletion of old knowledge? Or are you just keeping it around forever?

ric_harvey commented

2026-03-31 15:44:00 +00:00

(Migrated from codeberg.org)

all merged into v3.13.0 thanks for the contribution

Rows
Columns

RSS knowledge items: recency-based retrieval and configurable expiry #6