ff9f9455a6
All checks were successful
CI / test (pull_request) Successful in 11m5s
A flat 20K-char cutoff (copied from the fetched-web-page limit) silently dropped everything past the first ~20KB of a larger file, and even within the cutoff, embedding a whole multi-page document as one vector made retrieval coarse — the vector just averages out whatever topics the document covers. Split each file into ~1500-char chunks via text-splitter (recursive semantic-boundary splitting: paragraph > sentence > word, never mid-word) and embed each chunk as its own document, tagged with its source and part number. This removes the practical size ceiling — a large file chunks the same way a short one does — and sharpens retrieval by letting it surface the specific passage relevant to a query. It also incidentally caps the worst-case retrieval payload: 5 chunks now tops out around 7500 chars versus the old worst case of 5 full 20K-char documents. Verified against a live Ollama nomic-embed-text pull: a short document still embeds as a single chunk, unchanged from before.
29 lines
871 B
TOML
29 lines
871 B
TOML
[package]
|
|
name = "deep_research"
|
|
version = "0.1.0"
|
|
edition = "2024"
|
|
|
|
# Not part of a Cargo workspace — this crate is the whole repo. Declared
|
|
# explicitly (rather than just omitting it) so that checking this repo out
|
|
# as a nested git worktree (as this project's own workflow does) can't
|
|
# accidentally pick up an ancestor directory's workspace manifest.
|
|
[workspace]
|
|
|
|
[dependencies]
|
|
anyhow = "1.0.104"
|
|
chrono = "0.4.45"
|
|
clap = { version = "4", features = ["derive"] }
|
|
futures = "0.3"
|
|
indicatif = "0.18.6"
|
|
reqwest = { version = "0.13.4", features = ["query", "json"] }
|
|
rig = "0.41.0"
|
|
schemars = "1"
|
|
scraper = "0.27"
|
|
serde = { version = "1.0.229", features = ["derive"] }
|
|
text-splitter = "0.32"
|
|
tokio = { version = "1.53.1", features = ["full"] }
|
|
tracing = "0.1"
|
|
tracing-subscriber = { version = "0.3.23", features = ["env-filter"] }
|
|
|
|
[dev-dependencies]
|
|
serde_json = "1"
|