cali_scraper/scraper/dashboard/DASHBOARD.md

Legislative Dashboard

This is the anonymous + logged-in dashboard for cali_scraper, implementing spec section 5 (design documents: design/API_CONTRACT.md, design/DESIGN_DASHBOARD.md). It is served by a stdlib http.server, backed by the SQLite relational DB and the tag-cloud JSON, and produces the frozen API-contract response shapes.

Architecture

scraper/dashboard/
  __init__.py       # package docstring + version
  ranking.py        # pure I/O-free float-to-top scoring (tested in isolation)
  context.py        # RelevanceContext from user prefs
  data.py           # read-only SQLite access, tag enrichment, committee lookup
  prefs.py          # JSON prefs store (watch/topic toggles, dev auth)
  resolve.py        # ZIP -> district two-tier resolution
  api.py            # API_CONTRACT handlers (no sockets) + 6 empty states
  server.py         # http.server ThreadingHTTPServer + static file routing
  dashboard_cli.py  # guarded CLI entry via cli.py subcommand registration
  static/
    index.html      # frontend shell
    style.css       # visual system
    app.js          # fetch client, anon/personal rendering, ZIP resolve, watch/opt-in

How it fits the CLI

scraper/cli.py wires in the dashboard with the same guarded pattern as analyze_cli and relate_cli:

try:
    from .dashboard import dashboard_cli as _dcli
    _dcli.register(sub)
except Exception:
    pass

A broken or missing dashboard module never breaks parse, correlate, or analyze.

Run it

python3 -m scraper dashboard --port 8733
# browse http://127.0.0.1:8733/

API

Method	Path	Auth	Purpose
GET	/api/v1/bills	any	Bill list, personal / anonymous
GET	/api/v1/tags	any	Tag cloud
GET	/api/v1/legislators	any	Legislators
GET	/api/v1/districts	any	Districts by chamber/zip
GET	/api/v1/resolve	any	ZIP -> district resolution
POST	/api/v1/prefs	auth	Update user prefs
POST	/api/v1/auth/register	none	Dev auth placeholder
POST	/api/v1/prefs/watch	auth	Add/remove watched legislator
POST	/api/v1/prefs/topics	auth	Add/remove opted-in topic

Float-to-top ranking

Personal view applies a pure scoring function (ranking.py) over the bill list:

• Watched-legislator match (author) = +3
• Watched-legislator's committee match = +2
• Opted-in tag match = +2 × (number of matched tags)
• Home-chamber match = +1 (optional)
• Bills with score > 0 are moved to the front with a "Relevant to you" marker and a "Floated: ..." reason line
• Zero-scored bills remain in their date-grouped order below

Forward signal: "what's next" (scheduled_events, schema v2)

The daily-file bill queue is a backward snapshot — leginfo's bill status and bill history are past-action logs, so they cannot answer "what is scheduled next." The only genuine future-dated signal is the chambers' "upcoming events" calendars (assembly.ca.gov/events/upcoming, senate.ca.gov/calendar). These are fetched (python3 -m scraper ingest --store-events), parsed into the scheduled_events table (relational schema v2), and surfaced on /bills as two new envelope keys:

• next_legislative_day — {date, kind, chamber, title, source} (the earliest event on/after as_of), or {date: null, recess: true} when nothing is scheduled.
• scheduled_events — the next 7 days of chamber events.

The "Upcoming" filter is a forward view: with upcoming=1 and no explicit as_of, the bill queue floors at today, so a stale/older daily-file snapshot can no longer present itself as "upcoming." Omitting upcoming (the "All" / history view) still returns every row.

The dashboard renders a "Next legislative day" line in the freshness band and a "What's next · next 7 days" band. A recess (empty calendar) is an honest empty state, not an error.

Tests

python3 -m pytest tests/test_dashboard_ranking.py tests/test_dashboard_api.py \
  tests/test_parse_events.py

55 tests (12 ranking + 25 API + 18 parse-events), all passing. Pure ranking logic is tested in isolation; API handlers are tested with no sockets via the handler(req) -> (status, body) pattern; a live ThreadingHTTPServer smoke test confirms static + API endpoints serve. The event parser is tested against inline fixtures plus the saved samples/assembly_events_*.html pages.

DASHBOARD.md

This document. The full UX specification is design/DESIGN_DASHBOARD.md and the frozen API contract is design/API_CONTRACT.md.

Renders

• out/dashboard_render_anon.png — anonymous dashboard
• out/dashboard_render_personal.png — personalized dashboard (ZIP 90210, watch Schiavo, opt-in budget topic)

Both are data-faithful snapshots from the live API, rendered via macOS qlmanage (no headless browser available in this environment; the browser remote-debugging permission popup is a user-level security dialog that cannot be auto-approved).

Known limitations / next steps

• out/tag_cloud.json must exist for tags endpoint (118 tags as of 2026-09-06).
• 0 committee_members roster — committee lookup degrades gracefully when no roster data is supplied.
• Dev auth is a JSON-file local account, not production auth.
• ZIP -> district uses a documented demo dataset until roster ETL lands.
• Forward signal gap: leginfo's bill status/history are backward-only (past actions + status narrative, no future dates), so there is no per-bill "next step" label. The dashboard's forward answer comes solely from the chamber events calendars (scheduled_events).
• Recess empty state: while the Assembly (and at times the Senate) is in final recess, /events/upcoming legitimately returns 0 events. The "What's next" band and next_legislative_day then show an honest "recess / unknown" state rather than fabricating a date. This is expected, not a parsing failure.
• The 2026-08-31 daily-file snapshot is analysis-only; it is not re-surfaced as "upcoming" because the upcoming view floors at today.