105 lines
4.9 KiB
Markdown
105 lines
4.9 KiB
Markdown
|
# feedsignal
|
|||
|
|
|
|||
|
|
A self-hosted RSS reader that uses a local Ollama LLM (via [rig](https://github.com/0xPlaygrounds/rig))
|
|||
|
|
to surface the articles actually worth reading out of a large set of subscribed feeds, and learns
|
|||
|
|
from what you read (and how long, and what you dismiss) to get better at that over time.
|
|||
|
|
|
|||
|
|
## Architecture
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
crates/
|
|||
|
|
core/ domain models (Article, Feed, ReadingEvent) + the topic-affinity
|
|||
|
|
learning engine + the final relevance-scoring formula. No I/O.
|
|||
|
|
db/ SQLite (sqlx) persistence: feeds, articles, the append-only
|
|||
|
|
reading_events log, topic_affinities, preferences.
|
|||
|
|
feeds/ RSS/Atom fetching + parsing (feed-rs).
|
|||
|
|
llm/ rig + local Ollama: embeddings for the cheap first-pass filter,
|
|||
|
|
chat completion for the relevance judgment on the shortlist.
|
|||
|
|
web/ axum + Dioxus fullstack UI. `server` feature = native binary
|
|||
|
|
(DB, feed polling, LLM calls, scheduler); `web` feature = the
|
|||
|
|
WASM client shipped to the browser. No separate JS/npm stack.
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### Two-stage relevance filtering
|
|||
|
|
|
|||
|
|
Running the LLM on every article from every feed doesn't scale locally. Instead:
|
|||
|
|
|
|||
|
|
1. Every new article is embedded (`nomic-embed-text` by default) and scored against your
|
|||
|
|
preference-profile embedding — cheap, runs on all of them.
|
|||
|
|
2. Only articles above `EMBEDDING_SHORTLIST_THRESHOLD` (`crates/web/src/server/pipeline.rs`)
|
|||
|
|
go to the LLM (`llama3.1` by default) for an actual relevance judgment with a rationale.
|
|||
|
|
3. `feedsignal_core::scoring::score_article` blends LLM score (dominant when present),
|
|||
|
|
embedding score (fallback / floor), and topic affinity (bounded nudge) into `final_score`,
|
|||
|
|
which is what the UI ranks by.
|
|||
|
|
|
|||
|
|
### Learning from reading behavior
|
|||
|
|
|
|||
|
|
Every interaction (impression, open, dwell time, star, dismiss) is appended to an immutable
|
|||
|
|
`reading_events` log — never mutated, so the model can be recomputed or retuned later without
|
|||
|
|
losing data (`crates/db/migrations/0001_init.sql`).
|
|||
|
|
|
|||
|
|
From that log, `feedsignal_core::affinity`:
|
|||
|
|
|
|||
|
|
- Computes an **engagement score** per article (0.0–1.0) from whether it was opened, how long
|
|||
|
|
you spent relative to its estimated read time, and explicit star/dismiss signals.
|
|||
|
|
- Compares engagement to what the pipeline *predicted* (`final_score` at ingest time) to get a
|
|||
|
|
**surprise** signal — did you engage more or less than expected?
|
|||
|
|
- Nudges the affinity of that article's topics by `learning_rate * surprise`, clamped to
|
|||
|
|
`[-1, 1]`, so "the model rated this low but I read the whole thing" measurably shifts future
|
|||
|
|
scoring, and "the model rated this high but I dismissed it unread" pulls the other way.
|
|||
|
|
- Decays all affinities toward zero once a day so stale interests fade instead of anchoring
|
|||
|
|
the model forever.
|
|||
|
|
|
|||
|
|
This is a from-scratch numeric signal engine, deliberately not an LLM-rewritten prose "taste
|
|||
|
|
profile" — the topic scores are inspectable, and the update rule is simple enough to reason
|
|||
|
|
about and tune. See `crates/core/src/affinity.rs` for the full implementation and unit tests.
|
|||
|
|
|
|||
|
|
## Status
|
|||
|
|
|
|||
|
|
This is a scaffold, not a working app yet. What's real and compiles (`cargo check`/`cargo test`
|
|||
|
|
pass for every crate, both the `server` and `wasm32-unknown-unknown` targets of `feedsignal-web`):
|
|||
|
|
|
|||
|
|
- The domain model, DB schema/migrations, and the topic-affinity engine (with tests).
|
|||
|
|
- Feed fetching/parsing.
|
|||
|
|
- Ollama embedding + chat-completion calls via rig (against real rig-core 0.42 APIs).
|
|||
|
|
- The scoring pipeline shape and a cron-scheduled job wiring (`tokio-cron-scheduler`).
|
|||
|
|
- A minimal Dioxus UI that lists ranked articles and lets you dismiss one.
|
|||
|
|
|
|||
|
|
What's *not* wired up yet (left as the natural next steps):
|
|||
|
|
|
|||
|
|
- Feed subscription management (add/remove feed URLs) — `feedsignal_feeds::fetch_feed` exists
|
|||
|
|
but nothing calls it on a schedule yet.
|
|||
|
|
- The `preferences` table (your free-text "what I care about" description) isn't read by the
|
|||
|
|
pipeline yet — `pipeline.rs` has a `TODO` where it belongs.
|
|||
|
|
- Dwell-time capture from the browser (needs a small bit of JS/visibility-API glue on the
|
|||
|
|
article view, or a "mark read" action, to backfill `dwell_seconds` on the `opened` event).
|
|||
|
|
- Full article content extraction (currently only the feed's summary/content field is used;
|
|||
|
|
no readability-style scraping of the linked page).
|
|||
|
|
|
|||
|
|
## Running it
|
|||
|
|
|
|||
|
|
Requires [Ollama](https://ollama.com) running locally with the models pulled:
|
|||
|
|
|
|||
|
|
```sh
|
|||
|
|
ollama pull nomic-embed-text
|
|||
|
|
ollama pull llama3.1
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
And the [Dioxus CLI](https://dioxuslabs.com/learn/0.7/getting_started/) (`dx`) to build/serve
|
|||
|
|
the fullstack app — `dx` also has an `dx add`/component-scaffolding workflow worth using instead
|
|||
|
|
of hand-rolling UI pieces as this grows:
|
|||
|
|
|
|||
|
|
```sh
|
|||
|
|
cargo install dioxus-cli
|
|||
|
|
cd crates/web
|
|||
|
|
dx serve
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
The native crates (`core`, `db`, `feeds`, `llm`) build with plain `cargo check`/`cargo test` and
|
|||
|
|
don't need `dx` at all.
|
|||
|
|
|
|||
|
|
## CI
|
|||
|
|
|
|||
|
|
`.forgejo/workflows/ci.yml` runs on the self-hosted Forgejo runner: checks the native crates,
|
|||
|
|
both feature-sets of `feedsignal-web` (server + wasm client), and runs tests.
|