LLM Analysis — Subject-Matter Tag Clouds (spec §4)

Task: t_950780e0 — Ollama bill analysis + subject-matter tag clouds. Depends on: t_2736f40d (correlation lane / relational DB schema). Python: 3.9.6. No new third-party dependencies (stdlib urllib).

Architecture

config.yaml::llm  ──Config.llm──▶  scraper/llm.py  ──▶  bill_tags table
                                     │                   (db/daily_file.sqlite)
                                     │                     │
                                     ▼                     ▼
                              fallback tagger         out/tag_cloud.json
                          (zero-network, keyword
                           / short_title extraction)

Modules

File	Purpose
`scraper/llm.py`	Ollama HTTP client, deterministic fallback tagger, `bill_tags` DDL, `build_tag_cloud()`, `analyze()` orchestrator.
`scraper/analyze_cli.py`	`analyze` subcommand (guarded `register()` + `handle_analyze()`); wired additively into `scraper/cli.py`.
`config.yaml` § `llm:`	Endpoint + model knobs (append; http/cache/alerts/sources untouched).
`scraper/config.py`	`Config.llm` property; `_DEFAULTS["llm"]` fallbacks so the package survives a missing section.

Configuration

Append to config.yaml (do NOT touch http, cache, alerts, sources):

llm:
    enabled: false             # false → deterministic fallback tagger (no server)
    backend: ollama
    endpoint: "http://localhost:11434"    # configurable; default local Ollama
    model: "llama3"            # configurable; empty → fallback
    default_tags: 8            # max tags emitted per bill
    timeout_seconds: 30

--list-config prints the resolved settings:

$ python -m scraper analyze --list-config
llm:
    enabled: False
    backend: ollama
    endpoint: http://localhost:11434
    model: llama3
    default_tags: 8
    timeout_seconds: 30

Deterministic degradation

The analyze path selects Ollama vs. fallback:

Trigger	Path	`generated_by`	reason
`enabled: false`	fallback	`fallback`	`disabled`
`model` empty	fallback	`fallback`	`no model configured`
endpoint reachable, model returns valid tags	ollama	`ollama:<model>`	`enabled=true, model=<m>`
endpoint unreachable / connection refused	fallback	`fallback`	`unreachable: URLError`
Ollama returns empty body	fallback	`fallback`	`ollama empty → fallback`
Ollama returns unparseable response	fallback	`fallback`	`unreachable: ValueError`

Each bill tags row carries its own generated_by (fallback | ollama:<model>), so a mixed run (some bills ollama, some degraded) is honestly recorded without fabrication. The analyze() summary's path field reports the dominant path (fallback wins on ties — conservative/honest).

Storage: `bill_tags`

CREATE TABLE IF NOT EXISTS bill_tags (
    id           INTEGER PRIMARY KEY,
    bill_id      TEXT NOT NULL,
    tag          TEXT NOT NULL,
    source       TEXT,               -- 'short_title' | 'title+taxonomy' | 'ollama'
    generated_by TEXT NOT NULL,      -- 'fallback' | 'ollama:llama3' | …
    generated_at TEXT,
    UNIQUE (bill_id, tag, generated_by)
);

Re-runs with the same generated_by add no rows (idempotent).
The flat daily_file_records table and its view are untouched.
--rebuild drops all bill_tags before rewriting.

Usage

Default run (fallback, zero network)

$ python -m scraper analyze --rebuild
=== analyze ===
  bills_tagged : 156
  unique_tags     : 118
  path            : fallback
  generated_by    : fallback
  reason          : disabled
  rows_written    : 260
  out            : /Users/kyle/Documents/cali_scraper/out/tag_cloud.json
  top tags        :
      20  budget
      14  bill related budget
       9  health care
       8  education
       8  environment
       7  child welfare
       7  employment
       7  energy
       6  housing
       5  insurance

Use live Ollama (flip enabled in config.yaml, or `--force-llm`)

$ python -m scraper analyze --force-llm

When Ollama is unreachable (default when no server is running), the run silently falls back to the built-in tagger and logs a warning per bill.

out/tag_cloud.json — deterministic, diff-stable:

{
  "tag_count": 118,
  "bill_count": 156,
  "generated_by": null,
  "tags": [
    {"tag": "budget", "count": 20, "bills": 20},
    {"tag": "health care", "count": 9, "bills": 9},
    {"tag": "education", "count": 8, "bills": 8},
    …
  ]
}

generated_by: null aggregates all rows (both ollama and fallback); pass --top N to cap the list.

Tests

tests/test_llm.py — 12 tests, all offline (no real network):

Test	Coverage
`test_fallback_when_disabled_is_zero_network`	(a) `enabled=false`: `urlopen` is never called (`monkeypatch` asserts this).
`test_tag_bill_fallback_never_invents`	Fallback always produces a signal; keyword tags are extracted correctly.
`test_mocked_ollama_parsed_and_stored`	(b) Canned OpenAI-compatible dict fed via injectable `post`; `generated_by='ollama:llama3'` confirmed in DB.
`test_ollama_parse_json_and_fenced`	JSON-in-fence parsing; bare JSON parsing; degraded fallback splitting.
`test_unreachable_endpoint_falls_back_no_crash`	(c) Canned `post` raises `URLError`; all bills tagged, `generated_by='fallback'`, `reason='unreachable: …'`.
`test_mixed_run_reports_fallback_path`	Ollama empty → per-bill fallback; `dominant='fallback'`.
`test_build_tag_cloud_aggregates`	Cloud is sorted desc; JSON file shape correct.
`test_bill_tags_table_is_additive_and_idempotent`	Second analyze adds 0 rows; flat table untouched.
`test_ensure_tag_table_does_not_touch_flat`	`CREATE … IF NOT EXISTS` for `bill_tags` adds no column to `daily_file_records`.
`test_analyze_registers_without_clobbering`	`analyze` subcommand doesn't remove sibling commands (`parse`/`migrate`/`ingest`).
`test_config_llm_section_is_additive`	`config.llm` is loadable; `http`/`sources` still present.
`test_handle_analyze_end_to_end`	CLI handler returns 0; `out/tag_cloud.json` written and parseable.

Full suite: 106 passed, 2 skipped, 1 xfailed. Coverage: 92.0% ≥ 85 gate.

Contract compliance

Requirement	Where honoured
Python 3.9.6	`from __future__ import annotations`; f-strings; no 3.10+ syntax.
Configurable endpoint + model	`cfg["endpoint"]`, `cfg["model"]` read from `config.llm`; never hardcoded in the tagger.
Stdlib `urllib` (no new dep)	`_urllib_post()` uses `urllib.request.Request` + `urlopen`.
Deterministic fallback	`tag_bill_fallback()` runs when `enabled=false` / unreachable / empty model answer.
Zero-neg degradation proven	`monkeypatch.setattr(urllib.request, "urlopen", _deny)` — asserts `urlopen` never called.
`bill_tags(bill_id, tag, source, generated_by)`	DDL matches (plus `id`, `generated_at` for idempotency + audit trail).
`out/tag_cloud.json`	Written by `write_tag_cloud()` via `analyze()`.
CLI additive + guarded	`analyze_cli.register(sub)` inside `try/except` in `cli.build_parser()`; `handle_analyze` added to the handlers dict.
pytest green, ≥ 85 coverage	106 passed / 92.0% coverage.

7.3 KiB Raw Permalink Blame History