Decide how the glossary data grows: file layout, loading and storage #34

New issue

Open

opened 2026-10-16 13:04:12 +03:00 by olli-agent · 0 comments

olli-agent commented

2026-10-16 13:04:12 +03:00

Collaborator

The whole glossary is one hand-edited file, public/glossary/graph.json. This issue is about deciding, before it hurts, how the glossary's data should grow: as more terms in the same file, as one file per term, or with a different store.

Where it stands (2026-10-16)

224 terms, 478 edges and 68 abbreviations, in 7,500 lines: 303 KB raw, 84 KB gzipped, about 375 bytes gzipped per term.
Four readers load the whole file. The glossary page (glossary.js) and the quiz page (quiz.js) fetch it in the browser, the server loads it into memory at startup (internal/glossary, GRAPH_PATH), and CI's check-glossary.mjs reads it.
The only query anything makes is "this term and its direct neighbours".

What breaks first

Editing. One file means every glossary branch touches the same sorted node list and edge list. Three of four PRs on 2026-10-16 changed it, and two branches adding terms at the same time will conflict. This is already close.
Download size. The page downloads every term before it shows one. At the current rate, 1,000 terms is about 375 KB gzipped, which is fine; around 5,000 terms (about 1.9 MB) the first load starts to suffer.
Queries. Nothing needs more than direct neighbours today. Paths between terms ("how does X relate to Y") would be the first feature needing traversal, and Go can do that in memory for tens of thousands of nodes.

Options

Split the source, keep the output. One file per term (glossary/terms/<id>.json with its outgoing edges), plus topics, edge types and abbreviations in their own files. A small script assembles graph.json, which CI runs and checks; the source becomes the reviewed part. This removes the merge conflicts and keeps the page and server unchanged. It adds a build step for the glossary, which AGENTS.md currently rules out for public/; ROADMAP's web/ phase is the natural place for it.
Load in parts. Serve a small index (id, label, kind, tags) up front and each term's card from /api/glossary/<id> or per-term files. Worth it only when download size, not editing, is the problem.
PostgreSQL. Already running for the quiz. Terms and edges fit two tables, and recursive CTEs cover path queries. This makes sense if the glossary ever changes at runtime (suggestions, an editing UI), but it moves the source of truth out of Git review.
A graph database (Neo4j, Memgraph, Apache AGE on PostgreSQL). Built for millions of edges and deep traversals. At this scale it would add a server to a static-first site for queries Go answers from memory, against AGENTS.md's rule that a dependency is added only when the platform can't reasonably do the job. Re-evaluate if multi-hop queries become central and the graph reaches tens of thousands of terms. Apache AGE would be the least disruptive choice, because it is a PostgreSQL extension rather than a new server.

Proposal

Split the source when the next large batch of terms lands, or after the first real merge conflict, whichever comes first. Revisit loading in parts at about 1,000 terms. Treat a database, graph or relational, as a later decision driven by a feature that needs runtime changes or deep queries, not by term count.

Business vocabulary (B2B, ERP, SaaS, …) is wanted too. It belongs in the same graph under a new topic, not a separate file, because the links between the two are the value: ERP to Monolith, SaaS to Serverless. Two things to settle first: every card still needs an edge that holds everywhere, and the daily quiz round would include business terms unless it leaves that topic out or #29 lands first.

The whole glossary is one hand-edited file, `public/glossary/graph.json`. This issue is about deciding, before it hurts, how the glossary's data should grow: as more terms in the same file, as one file per term, or with a different store. ## Where it stands (2026-10-16) - 224 terms, 478 edges and 68 abbreviations, in 7,500 lines: 303 KB raw, 84 KB gzipped, about 375 bytes gzipped per term. - Four readers load the whole file. The glossary page (`glossary.js`) and the quiz page (`quiz.js`) fetch it in the browser, the server loads it into memory at startup (`internal/glossary`, `GRAPH_PATH`), and CI's `check-glossary.mjs` reads it. - The only query anything makes is "this term and its direct neighbours". ## What breaks first 1. **Editing.** One file means every glossary branch touches the same sorted node list and edge list. Three of four PRs on 2026-10-16 changed it, and two branches adding terms at the same time will conflict. This is already close. 2. **Download size.** The page downloads every term before it shows one. At the current rate, 1,000 terms is about 375 KB gzipped, which is fine; around 5,000 terms (about 1.9 MB) the first load starts to suffer. 3. **Queries.** Nothing needs more than direct neighbours today. Paths between terms ("how does X relate to Y") would be the first feature needing traversal, and Go can do that in memory for tens of thousands of nodes. ## Options - **Split the source, keep the output.** One file per term (`glossary/terms/<id>.json` with its outgoing edges), plus topics, edge types and abbreviations in their own files. A small script assembles `graph.json`, which CI runs and checks; the source becomes the reviewed part. This removes the merge conflicts and keeps the page and server unchanged. It adds a build step for the glossary, which AGENTS.md currently rules out for `public/`; ROADMAP's `web/` phase is the natural place for it. - **Load in parts.** Serve a small index (id, label, kind, tags) up front and each term's card from `/api/glossary/<id>` or per-term files. Worth it only when download size, not editing, is the problem. - **PostgreSQL.** Already running for the quiz. Terms and edges fit two tables, and recursive CTEs cover path queries. This makes sense if the glossary ever changes at runtime (suggestions, an editing UI), but it moves the source of truth out of Git review. - **A graph database** (Neo4j, Memgraph, Apache AGE on PostgreSQL). Built for millions of edges and deep traversals. At this scale it would add a server to a static-first site for queries Go answers from memory, against AGENTS.md's rule that a dependency is added only when the platform can't reasonably do the job. Re-evaluate if multi-hop queries become central and the graph reaches tens of thousands of terms. Apache AGE would be the least disruptive choice, because it is a PostgreSQL extension rather than a new server. ## Proposal Split the source when the next large batch of terms lands, or after the first real merge conflict, whichever comes first. Revisit loading in parts at about 1,000 terms. Treat a database, graph or relational, as a later decision driven by a feature that needs runtime changes or deep queries, not by term count. ## Related: topics outside tech Business vocabulary (B2B, ERP, SaaS, …) is wanted too. It belongs in the same graph under a new topic, not a separate file, because the links between the two are the value: ERP to Monolith, SaaS to Serverless. Two things to settle first: every card still needs an edge that holds everywhere, and the daily quiz round would include business terms unless it leaves that topic out or #29 lands first.

olli-agent referenced this issue

2026-10-16 13:13:27 +03:00

Glossary terms to add in the next batch #36

olli-agent referenced this issue from a commit

2026-10-16 13:46:21 +03:00

Split the glossary into one file per term, built in CI

olli-agent referenced this issue

2026-10-16 13:46:36 +03:00

Split the glossary into one file per term, built in CI #37