Initial commit
This commit is contained in:
kfj001 committed
commit
5f2db08b43
100 files changed
+16985
No files matched your search
+197
@@ -0,0 +1,197 @@
|
||||
# LLM Analysis — Subject-Matter Tag Clouds (spec §4)
|
||||
|
||||
**Task:** `t_950780e0` — Ollama bill analysis + subject-matter tag clouds.
|
||||
**Depends on:** `t_2736f40d` (correlation lane / relational DB schema).
|
||||
**Python:** 3.9.6. No new third-party dependencies (stdlib `urllib`).
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
config.yaml::llm ──Config.llm──▶ scraper/llm.py ──▶ bill_tags table
|
||||
│ (db/daily_file.sqlite)
|
||||
│ │
|
||||
▼ ▼
|
||||
fallback tagger out/tag_cloud.json
|
||||
(zero-network, keyword
|
||||
/ short_title extraction)
|
||||
```
|
||||
|
||||
### Modules
|
||||
|
||||
| File | Purpose |
|
||||
|---|---|
|
||||
| `scraper/llm.py` | Ollama HTTP client, deterministic fallback tagger, `bill_tags` DDL, `build_tag_cloud()`, `analyze()` orchestrator. |
|
||||
| `scraper/analyze_cli.py` | `analyze` subcommand (guarded `register()` + `handle_analyze()`); wired additively into `scraper/cli.py`. |
|
||||
| `config.yaml` § `llm:` | Endpoint + model knobs (append; http/cache/alerts/sources untouched). |
|
||||
| `scraper/config.py` | `Config.llm` property; `_DEFAULTS["llm"]` fallbacks so the package survives a missing section. |
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
Append to `config.yaml` (do NOT touch `http`, `cache`, `alerts`, `sources`):
|
||||
|
||||
```yaml
|
||||
llm:
|
||||
enabled: false # false → deterministic fallback tagger (no server)
|
||||
backend: ollama
|
||||
endpoint: "http://localhost:11434" # configurable; default local Ollama
|
||||
model: "llama3" # configurable; empty → fallback
|
||||
default_tags: 8 # max tags emitted per bill
|
||||
timeout_seconds: 30
|
||||
```
|
||||
|
||||
`--list-config` prints the resolved settings:
|
||||
|
||||
```
|
||||
$ python -m scraper analyze --list-config
|
||||
llm:
|
||||
enabled: False
|
||||
backend: ollama
|
||||
endpoint: http://localhost:11434
|
||||
model: llama3
|
||||
default_tags: 8
|
||||
timeout_seconds: 30
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Deterministic degradation
|
||||
|
||||
The `analyze` path selects Ollama vs. fallback:
|
||||
|
||||
| Trigger | Path | `generated_by` | reason |
|
||||
|---|---|---|---|
|
||||
| `enabled: false` | fallback | `fallback` | `disabled` |
|
||||
| `model` empty | fallback | `fallback` | `no model configured` |
|
||||
| endpoint reachable, model returns valid tags | ollama | `ollama:<model>` | `enabled=true, model=<m>` |
|
||||
| endpoint unreachable / connection refused | fallback | `fallback` | `unreachable: URLError` |
|
||||
| Ollama returns empty body | fallback | `fallback` | `ollama empty → fallback` |
|
||||
| Ollama returns unparseable response | fallback | `fallback` | `unreachable: ValueError` |
|
||||
|
||||
Each bill tags row carries its own `generated_by` (`fallback` | `ollama:<model>`),
|
||||
so a mixed run (some bills ollama, some degraded) is honestly recorded without
|
||||
fabrication. The `analyze()` summary's `path` field reports the dominant path
|
||||
(`fallback` wins on ties — conservative/honest).
|
||||
|
||||
---
|
||||
|
||||
## Storage: `bill_tags`
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS bill_tags (
|
||||
id INTEGER PRIMARY KEY,
|
||||
bill_id TEXT NOT NULL,
|
||||
tag TEXT NOT NULL,
|
||||
source TEXT, -- 'short_title' | 'title+taxonomy' | 'ollama'
|
||||
generated_by TEXT NOT NULL, -- 'fallback' | 'ollama:llama3' | …
|
||||
generated_at TEXT,
|
||||
UNIQUE (bill_id, tag, generated_by)
|
||||
);
|
||||
```
|
||||
|
||||
- Re-runs with the same `generated_by` add no rows (idempotent).
|
||||
- The flat `daily_file_records` table and its view are untouched.
|
||||
- `--rebuild` drops all `bill_tags` before rewriting.
|
||||
|
||||
---
|
||||
|
||||
## Usage
|
||||
|
||||
### Default run (fallback, zero network)
|
||||
|
||||
```bash
|
||||
$ python -m scraper analyze --rebuild
|
||||
=== analyze ===
|
||||
bills_tagged : 156
|
||||
unique_tags : 118
|
||||
path : fallback
|
||||
generated_by : fallback
|
||||
reason : disabled
|
||||
rows_written : 260
|
||||
out : /Users/kyle/Documents/cali_scraper/out/tag_cloud.json
|
||||
top tags :
|
||||
20 budget
|
||||
14 bill related budget
|
||||
9 health care
|
||||
8 education
|
||||
8 environment
|
||||
7 child welfare
|
||||
7 employment
|
||||
7 energy
|
||||
6 housing
|
||||
5 insurance
|
||||
```
|
||||
|
||||
### Use live Ollama (flip enabled in config.yaml, or `--force-llm`)
|
||||
|
||||
```bash
|
||||
$ python -m scraper analyze --force-llm
|
||||
```
|
||||
|
||||
When Ollama is unreachable (default when no server is running), the run
|
||||
silently falls back to the built-in tagger and logs a warning per bill.
|
||||
|
||||
---
|
||||
|
||||
## Tag-cloud output
|
||||
|
||||
`out/tag_cloud.json` — deterministic, diff-stable:
|
||||
|
||||
```json
|
||||
{
|
||||
"tag_count": 118,
|
||||
"bill_count": 156,
|
||||
"generated_by": null,
|
||||
"tags": [
|
||||
{"tag": "budget", "count": 20, "bills": 20},
|
||||
{"tag": "health care", "count": 9, "bills": 9},
|
||||
{"tag": "education", "count": 8, "bills": 8},
|
||||
…
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
`generated_by: null` aggregates all rows (both ollama and fallback); pass
|
||||
`--top N` to cap the list.
|
||||
|
||||
---
|
||||
|
||||
## Tests
|
||||
|
||||
`tests/test_llm.py` — 12 tests, all offline (no real network):
|
||||
|
||||
| Test | Coverage |
|
||||
|---|---|
|
||||
| `test_fallback_when_disabled_is_zero_network` | (a) `enabled=false`: `urlopen` is never called (`monkeypatch` asserts this). |
|
||||
| `test_tag_bill_fallback_never_invents` | Fallback always produces a signal; keyword tags are extracted correctly. |
|
||||
| `test_mocked_ollama_parsed_and_stored` | (b) Canned OpenAI-compatible dict fed via injectable `post`; `generated_by='ollama:llama3'` confirmed in DB. |
|
||||
| `test_ollama_parse_json_and_fenced` | JSON-in-fence parsing; bare JSON parsing; degraded fallback splitting. |
|
||||
| `test_unreachable_endpoint_falls_back_no_crash` | (c) Canned `post` raises `URLError`; all bills tagged, `generated_by='fallback'`, `reason='unreachable: …'`. |
|
||||
| `test_mixed_run_reports_fallback_path` | Ollama empty → per-bill fallback; `dominant='fallback'`. |
|
||||
| `test_build_tag_cloud_aggregates` | Cloud is sorted desc; JSON file shape correct. |
|
||||
| `test_bill_tags_table_is_additive_and_idempotent` | Second analyze adds 0 rows; flat table untouched. |
|
||||
| `test_ensure_tag_table_does_not_touch_flat` | `CREATE … IF NOT EXISTS` for `bill_tags` adds no column to `daily_file_records`. |
|
||||
| `test_analyze_registers_without_clobbering` | `analyze` subcommand doesn't remove sibling commands (`parse`/`migrate`/`ingest`). |
|
||||
| `test_config_llm_section_is_additive` | `config.llm` is loadable; `http`/`sources` still present. |
|
||||
| `test_handle_analyze_end_to_end` | CLI handler returns 0; `out/tag_cloud.json` written and parseable. |
|
||||
|
||||
Full suite: 106 passed, 2 skipped, 1 xfailed. Coverage: 92.0% ≥ 85 gate.
|
||||
|
||||
---
|
||||
|
||||
## Contract compliance
|
||||
|
||||
| Requirement | Where honoured |
|
||||
|---|---|
|
||||
| Python 3.9.6 | `from __future__ import annotations`; f-strings; no 3.10+ syntax. |
|
||||
| Configurable endpoint + model | `cfg["endpoint"]`, `cfg["model"]` read from `config.llm`; never hardcoded in the tagger. |
|
||||
| Stdlib `urllib` (no new dep) | `_urllib_post()` uses `urllib.request.Request` + `urlopen`. |
|
||||
| Deterministic fallback | `tag_bill_fallback()` runs when `enabled=false` / unreachable / empty model answer. |
|
||||
| Zero-neg degradation proven | `monkeypatch.setattr(urllib.request, "urlopen", _deny)` — asserts `urlopen` never called. |
|
||||
| `bill_tags(bill_id, tag, source, generated_by)` | DDL matches (plus `id`, `generated_at` for idempotency + audit trail). |
|
||||
| `out/tag_cloud.json` | Written by `write_tag_cloud()` via `analyze()`. |
|
||||
| CLI additive + guarded | `analyze_cli.register(sub)` inside `try/except` in `cli.build_parser()`; `handle_analyze` added to the handlers dict. |
|
||||
| pytest green, ≥ 85 coverage | 106 passed / 92.0% coverage. |
|
||||
Reference in new issue
Block a user