198 lines
7.3 KiB
Markdown
198 lines
7.3 KiB
Markdown
# LLM Analysis — Subject-Matter Tag Clouds (spec §4)
|
|
|
|
**Task:** `t_950780e0` — Ollama bill analysis + subject-matter tag clouds.
|
|
**Depends on:** `t_2736f40d` (correlation lane / relational DB schema).
|
|
**Python:** 3.9.6. No new third-party dependencies (stdlib `urllib`).
|
|
|
|
---
|
|
|
|
## Architecture
|
|
|
|
```
|
|
config.yaml::llm ──Config.llm──▶ scraper/llm.py ──▶ bill_tags table
|
|
│ (db/daily_file.sqlite)
|
|
│ │
|
|
▼ ▼
|
|
fallback tagger out/tag_cloud.json
|
|
(zero-network, keyword
|
|
/ short_title extraction)
|
|
```
|
|
|
|
### Modules
|
|
|
|
| File | Purpose |
|
|
|---|---|
|
|
| `scraper/llm.py` | Ollama HTTP client, deterministic fallback tagger, `bill_tags` DDL, `build_tag_cloud()`, `analyze()` orchestrator. |
|
|
| `scraper/analyze_cli.py` | `analyze` subcommand (guarded `register()` + `handle_analyze()`); wired additively into `scraper/cli.py`. |
|
|
| `config.yaml` § `llm:` | Endpoint + model knobs (append; http/cache/alerts/sources untouched). |
|
|
| `scraper/config.py` | `Config.llm` property; `_DEFAULTS["llm"]` fallbacks so the package survives a missing section. |
|
|
|
|
---
|
|
|
|
## Configuration
|
|
|
|
Append to `config.yaml` (do NOT touch `http`, `cache`, `alerts`, `sources`):
|
|
|
|
```yaml
|
|
llm:
|
|
enabled: false # false → deterministic fallback tagger (no server)
|
|
backend: ollama
|
|
endpoint: "http://localhost:11434" # configurable; default local Ollama
|
|
model: "llama3" # configurable; empty → fallback
|
|
default_tags: 8 # max tags emitted per bill
|
|
timeout_seconds: 30
|
|
```
|
|
|
|
`--list-config` prints the resolved settings:
|
|
|
|
```
|
|
$ python -m scraper analyze --list-config
|
|
llm:
|
|
enabled: False
|
|
backend: ollama
|
|
endpoint: http://localhost:11434
|
|
model: llama3
|
|
default_tags: 8
|
|
timeout_seconds: 30
|
|
```
|
|
|
|
---
|
|
|
|
## Deterministic degradation
|
|
|
|
The `analyze` path selects Ollama vs. fallback:
|
|
|
|
| Trigger | Path | `generated_by` | reason |
|
|
|---|---|---|---|
|
|
| `enabled: false` | fallback | `fallback` | `disabled` |
|
|
| `model` empty | fallback | `fallback` | `no model configured` |
|
|
| endpoint reachable, model returns valid tags | ollama | `ollama:<model>` | `enabled=true, model=<m>` |
|
|
| endpoint unreachable / connection refused | fallback | `fallback` | `unreachable: URLError` |
|
|
| Ollama returns empty body | fallback | `fallback` | `ollama empty → fallback` |
|
|
| Ollama returns unparseable response | fallback | `fallback` | `unreachable: ValueError` |
|
|
|
|
Each bill tags row carries its own `generated_by` (`fallback` | `ollama:<model>`),
|
|
so a mixed run (some bills ollama, some degraded) is honestly recorded without
|
|
fabrication. The `analyze()` summary's `path` field reports the dominant path
|
|
(`fallback` wins on ties — conservative/honest).
|
|
|
|
---
|
|
|
|
## Storage: `bill_tags`
|
|
|
|
```sql
|
|
CREATE TABLE IF NOT EXISTS bill_tags (
|
|
id INTEGER PRIMARY KEY,
|
|
bill_id TEXT NOT NULL,
|
|
tag TEXT NOT NULL,
|
|
source TEXT, -- 'short_title' | 'title+taxonomy' | 'ollama'
|
|
generated_by TEXT NOT NULL, -- 'fallback' | 'ollama:llama3' | …
|
|
generated_at TEXT,
|
|
UNIQUE (bill_id, tag, generated_by)
|
|
);
|
|
```
|
|
|
|
- Re-runs with the same `generated_by` add no rows (idempotent).
|
|
- The flat `daily_file_records` table and its view are untouched.
|
|
- `--rebuild` drops all `bill_tags` before rewriting.
|
|
|
|
---
|
|
|
|
## Usage
|
|
|
|
### Default run (fallback, zero network)
|
|
|
|
```bash
|
|
$ python -m scraper analyze --rebuild
|
|
=== analyze ===
|
|
bills_tagged : 156
|
|
unique_tags : 118
|
|
path : fallback
|
|
generated_by : fallback
|
|
reason : disabled
|
|
rows_written : 260
|
|
out : /Users/kyle/Documents/cali_scraper/out/tag_cloud.json
|
|
top tags :
|
|
20 budget
|
|
14 bill related budget
|
|
9 health care
|
|
8 education
|
|
8 environment
|
|
7 child welfare
|
|
7 employment
|
|
7 energy
|
|
6 housing
|
|
5 insurance
|
|
```
|
|
|
|
### Use live Ollama (flip enabled in config.yaml, or `--force-llm`)
|
|
|
|
```bash
|
|
$ python -m scraper analyze --force-llm
|
|
```
|
|
|
|
When Ollama is unreachable (default when no server is running), the run
|
|
silently falls back to the built-in tagger and logs a warning per bill.
|
|
|
|
---
|
|
|
|
## Tag-cloud output
|
|
|
|
`out/tag_cloud.json` — deterministic, diff-stable:
|
|
|
|
```json
|
|
{
|
|
"tag_count": 118,
|
|
"bill_count": 156,
|
|
"generated_by": null,
|
|
"tags": [
|
|
{"tag": "budget", "count": 20, "bills": 20},
|
|
{"tag": "health care", "count": 9, "bills": 9},
|
|
{"tag": "education", "count": 8, "bills": 8},
|
|
…
|
|
]
|
|
}
|
|
```
|
|
|
|
`generated_by: null` aggregates all rows (both ollama and fallback); pass
|
|
`--top N` to cap the list.
|
|
|
|
---
|
|
|
|
## Tests
|
|
|
|
`tests/test_llm.py` — 12 tests, all offline (no real network):
|
|
|
|
| Test | Coverage |
|
|
|---|---|
|
|
| `test_fallback_when_disabled_is_zero_network` | (a) `enabled=false`: `urlopen` is never called (`monkeypatch` asserts this). |
|
|
| `test_tag_bill_fallback_never_invents` | Fallback always produces a signal; keyword tags are extracted correctly. |
|
|
| `test_mocked_ollama_parsed_and_stored` | (b) Canned OpenAI-compatible dict fed via injectable `post`; `generated_by='ollama:llama3'` confirmed in DB. |
|
|
| `test_ollama_parse_json_and_fenced` | JSON-in-fence parsing; bare JSON parsing; degraded fallback splitting. |
|
|
| `test_unreachable_endpoint_falls_back_no_crash` | (c) Canned `post` raises `URLError`; all bills tagged, `generated_by='fallback'`, `reason='unreachable: …'`. |
|
|
| `test_mixed_run_reports_fallback_path` | Ollama empty → per-bill fallback; `dominant='fallback'`. |
|
|
| `test_build_tag_cloud_aggregates` | Cloud is sorted desc; JSON file shape correct. |
|
|
| `test_bill_tags_table_is_additive_and_idempotent` | Second analyze adds 0 rows; flat table untouched. |
|
|
| `test_ensure_tag_table_does_not_touch_flat` | `CREATE … IF NOT EXISTS` for `bill_tags` adds no column to `daily_file_records`. |
|
|
| `test_analyze_registers_without_clobbering` | `analyze` subcommand doesn't remove sibling commands (`parse`/`migrate`/`ingest`). |
|
|
| `test_config_llm_section_is_additive` | `config.llm` is loadable; `http`/`sources` still present. |
|
|
| `test_handle_analyze_end_to_end` | CLI handler returns 0; `out/tag_cloud.json` written and parseable. |
|
|
|
|
Full suite: 106 passed, 2 skipped, 1 xfailed. Coverage: 92.0% ≥ 85 gate.
|
|
|
|
---
|
|
|
|
## Contract compliance
|
|
|
|
| Requirement | Where honoured |
|
|
|---|---|
|
|
| Python 3.9.6 | `from __future__ import annotations`; f-strings; no 3.10+ syntax. |
|
|
| Configurable endpoint + model | `cfg["endpoint"]`, `cfg["model"]` read from `config.llm`; never hardcoded in the tagger. |
|
|
| Stdlib `urllib` (no new dep) | `_urllib_post()` uses `urllib.request.Request` + `urlopen`. |
|
|
| Deterministic fallback | `tag_bill_fallback()` runs when `enabled=false` / unreachable / empty model answer. |
|
|
| Zero-neg degradation proven | `monkeypatch.setattr(urllib.request, "urlopen", _deny)` — asserts `urlopen` never called. |
|
|
| `bill_tags(bill_id, tag, source, generated_by)` | DDL matches (plus `id`, `generated_at` for idempotency + audit trail). |
|
|
| `out/tag_cloud.json` | Written by `write_tag_cloud()` via `analyze()`. |
|
|
| CLI additive + guarded | `analyze_cli.register(sub)` inside `try/except` in `cli.build_parser()`; `handle_analyze` added to the handlers dict. |
|
|
| pytest green, ≥ 85 coverage | 106 passed / 92.0% coverage. |
|