Default LLM scoring stage to a model already pulled locally
Some checks failed
CI / check (push) Failing after 11s
Some checks failed
CI / check (push) Failing after 11s
Was hardcoded to llama3.1, which isn't actually on this machine. Switched to gemma4-e4b (5.0 GB, already pulled) as a speed/quality balance for a call made once per shortlisted article on every pipeline run.
This commit is contained in:
parent
1a4a72fc27
commit
2b61c80143
2 changed files with 7 additions and 4 deletions
|
|
@ -27,7 +27,10 @@ Running the LLM on every article from every feed doesn't scale locally. Instead:
|
||||||
1. Every new article is embedded (`nomic-embed-text` by default) and scored against your
|
1. Every new article is embedded (`nomic-embed-text` by default) and scored against your
|
||||||
preference-profile embedding — cheap, runs on all of them.
|
preference-profile embedding — cheap, runs on all of them.
|
||||||
2. Only articles above `EMBEDDING_SHORTLIST_THRESHOLD` (`crates/web/src/server/pipeline.rs`)
|
2. Only articles above `EMBEDDING_SHORTLIST_THRESHOLD` (`crates/web/src/server/pipeline.rs`)
|
||||||
go to the LLM (`llama3.1` by default) for an actual relevance judgment with a rationale.
|
go to the LLM (`gemma4-e4b` by default — chosen because it's already pulled locally and a
|
||||||
|
reasonable size/speed tradeoff for a call made once per shortlisted article on every
|
||||||
|
pipeline run; swap it in `crates/web/src/server.rs` for a bigger/smaller model as needed)
|
||||||
|
for an actual relevance judgment with a rationale.
|
||||||
3. `feedsignal_core::scoring::score_article` blends LLM score (dominant when present),
|
3. `feedsignal_core::scoring::score_article` blends LLM score (dominant when present),
|
||||||
embedding score (fallback / floor), and topic affinity (bounded nudge) into `final_score`,
|
embedding score (fallback / floor), and topic affinity (bounded nudge) into `final_score`,
|
||||||
which is what the UI ranks by.
|
which is what the UI ranks by.
|
||||||
|
|
@ -82,7 +85,7 @@ Requires [Ollama](https://ollama.com) running locally with the models pulled:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
ollama pull nomic-embed-text
|
ollama pull nomic-embed-text
|
||||||
ollama pull llama3.1
|
ollama pull gemma4-e4b
|
||||||
```
|
```
|
||||||
|
|
||||||
And the [Dioxus CLI](https://dioxuslabs.com/learn/0.7/getting_started/) (`dx`) to build/serve
|
And the [Dioxus CLI](https://dioxuslabs.com/learn/0.7/getting_started/) (`dx`) to build/serve
|
||||||
|
|
|
||||||
|
|
@ -23,8 +23,8 @@ async fn ensure_background_jobs_started(db: Arc<Db>) {
|
||||||
BACKGROUND_JOBS
|
BACKGROUND_JOBS
|
||||||
.get_or_init(|| async {
|
.get_or_init(|| async {
|
||||||
// TODO: move base_url/model names to config/env once there's a
|
// TODO: move base_url/model names to config/env once there's a
|
||||||
// settings story; hardcoded to the common local Ollama defaults.
|
// settings story; hardcoded to models already pulled locally.
|
||||||
let llm = Arc::new(Llm::new("http://localhost:11434", "nomic-embed-text", 768, "llama3.1").expect("failed to construct ollama client"));
|
let llm = Arc::new(Llm::new("http://localhost:11434", "nomic-embed-text", 768, "gemma4-e4b").expect("failed to construct ollama client"));
|
||||||
if let Err(err) = pipeline::start_scheduler(db, llm).await {
|
if let Err(err) = pipeline::start_scheduler(db, llm).await {
|
||||||
tracing::error!(?err, "failed to start background job scheduler");
|
tracing::error!(?err, "failed to start background job scheduler");
|
||||||
}
|
}
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue