kfj001
03eeb6bb62
The daily-file bill queue is a backward snapshot, so "upcoming" could only resurface the latest past day (the 08-31 snapshot read as today's queue) and there was no "what is scheduled next." leginfo bill status/history are backward-only, so the genuine future-dated signal is the chambers' "upcoming events" calendars. - api.py: floor the "upcoming" view at *today* (not the DB max date) when no as_of is pinned; surface next_legislative_day + scheduled_events (next 7 days) on /bills. Add _today_iso / _next_legislative_day / _scheduled_events, all degrading to a recess marker / [] (never 5xx). - relational.py: schema v2 adds the scheduled_events table + store_scheduled_events (idempotent per-day refresh), load_scheduled_events, next_legislative_day. - parsers.py: parse_events_page + classify_event_kind for the chamber events calendar; skip empty-title rows (decompose the <time> before the fallback). - ingest.py + cli.py: --store-events lane (cache-first, recess-safe) that parses the calendar HTML into scheduled_events. - sources.py + config.yaml: register assembly_events / senate_events (kind: events, empty_is_ok: true). - run_ingest.sh: run the events lane after the daily-file lane as non-fatal (skippable via CALI_SCRAPER_SKIP_EVENTS=1); daily-file exit still gates cron. - frontend: "Next legislative day" line + "What's next · next 7 days" band with honest recess empty states. - DASHBOARD.md: document the forward signal + known limitations (leginfo backward-only gap, recess empty state, 08-31 snapshot is analysis-only). - tests: 5 new API tests + 18 parse-events tests; 170 pass, 92% coverage.
150 lines
6.3 KiB
Markdown
150 lines
6.3 KiB
Markdown
# Legislative Dashboard
|
||
|
||
This is the anonymous + logged-in dashboard for `cali_scraper`, implementing
|
||
spec section 5 (design documents: `design/API_CONTRACT.md`,
|
||
`design/DESIGN_DASHBOARD.md`). It is served by a stdlib `http.server`,
|
||
backed by the SQLite relational DB and the tag-cloud JSON, and produces the
|
||
frozen API-contract response shapes.
|
||
|
||
## Architecture
|
||
|
||
```
|
||
scraper/dashboard/
|
||
__init__.py # package docstring + version
|
||
ranking.py # pure I/O-free float-to-top scoring (tested in isolation)
|
||
context.py # RelevanceContext from user prefs
|
||
data.py # read-only SQLite access, tag enrichment, committee lookup
|
||
prefs.py # JSON prefs store (watch/topic toggles, dev auth)
|
||
resolve.py # ZIP -> district two-tier resolution
|
||
api.py # API_CONTRACT handlers (no sockets) + 6 empty states
|
||
server.py # http.server ThreadingHTTPServer + static file routing
|
||
dashboard_cli.py # guarded CLI entry via cli.py subcommand registration
|
||
static/
|
||
index.html # frontend shell
|
||
style.css # visual system
|
||
app.js # fetch client, anon/personal rendering, ZIP resolve, watch/opt-in
|
||
```
|
||
|
||
### How it fits the CLI
|
||
|
||
`scraper/cli.py` wires in the dashboard with the same guarded pattern as
|
||
`analyze_cli` and `relate_cli`:
|
||
|
||
```python
|
||
try:
|
||
from .dashboard import dashboard_cli as _dcli
|
||
_dcli.register(sub)
|
||
except Exception:
|
||
pass
|
||
```
|
||
|
||
A broken or missing dashboard module never breaks `parse`, `correlate`, or
|
||
`analyze`.
|
||
|
||
### Run it
|
||
|
||
```bash
|
||
python3 -m scraper dashboard --port 8733
|
||
# browse http://127.0.0.1:8733/
|
||
```
|
||
|
||
## API
|
||
|
||
| Method | Path | Auth | Purpose |
|
||
|--------|------------------------|-------|--------------------------------------|
|
||
| GET | `/api/v1/bills` | any | Bill list, personal / anonymous |
|
||
| GET | `/api/v1/tags` | any | Tag cloud |
|
||
| GET | `/api/v1/legislators` | any | Legislators |
|
||
| GET | `/api/v1/districts` | any | Districts by chamber/zip |
|
||
| GET | `/api/v1/resolve` | any | ZIP -> district resolution |
|
||
| POST | `/api/v1/prefs` | auth | Update user prefs |
|
||
| POST | `/api/v1/auth/register`| none | Dev auth placeholder |
|
||
| POST | `/api/v1/prefs/watch` | auth | Add/remove watched legislator |
|
||
| POST | `/api/v1/prefs/topics` | auth | Add/remove opted-in topic |
|
||
|
||
### Float-to-top ranking
|
||
|
||
Personal view applies a pure scoring function (`ranking.py`) over the bill
|
||
list:
|
||
|
||
- **Watched-legislator match (author)** = +3
|
||
- **Watched-legislator's committee match** = +2
|
||
- **Opted-in tag match** = +2 × (number of matched tags)
|
||
- **Home-chamber match** = +1 (optional)
|
||
- Bills with score > 0 are moved to the front with a "Relevant to you"
|
||
marker and a "Floated: ..." reason line
|
||
- Zero-scored bills remain in their date-grouped order below
|
||
|
||
## Forward signal: "what's next" (scheduled_events, schema v2)
|
||
|
||
The daily-file bill queue is a *backward* snapshot — leginfo's bill status and
|
||
bill history are past-action logs, so they cannot answer "what is scheduled
|
||
next." The only genuine future-dated signal is the chambers' "upcoming events"
|
||
calendars (`assembly.ca.gov/events/upcoming`, `senate.ca.gov/calendar`). These
|
||
are fetched (`python3 -m scraper ingest --store-events`), parsed into the
|
||
`scheduled_events` table (relational schema v2), and surfaced on `/bills` as two
|
||
new envelope keys:
|
||
|
||
- `next_legislative_day` — `{date, kind, chamber, title, source}` (the earliest
|
||
event on/after `as_of`), or `{date: null, recess: true}` when nothing is
|
||
scheduled.
|
||
- `scheduled_events` — the next 7 days of chamber events.
|
||
|
||
The "Upcoming" filter is a *forward* view: with `upcoming=1` and no explicit
|
||
`as_of`, the bill queue floors at **today**, so a stale/older daily-file
|
||
snapshot can no longer present itself as "upcoming." Omitting `upcoming`
|
||
(the "All" / history view) still returns every row.
|
||
|
||
The dashboard renders a "Next legislative day" line in the freshness band and a
|
||
"What's next · next 7 days" band. A recess (empty calendar) is an *honest*
|
||
empty state, not an error.
|
||
|
||
## Tests
|
||
|
||
```bash
|
||
python3 -m pytest tests/test_dashboard_ranking.py tests/test_dashboard_api.py \
|
||
tests/test_parse_events.py
|
||
```
|
||
|
||
55 tests (12 ranking + 25 API + 18 parse-events), all passing. Pure ranking
|
||
logic is tested in isolation; API handlers are tested with no sockets via the
|
||
`handler(req) -> (status, body)` pattern; a live `ThreadingHTTPServer` smoke test
|
||
confirms static + API endpoints serve. The event parser is tested against
|
||
inline fixtures plus the saved `samples/assembly_events_*.html` pages.
|
||
|
||
## DASHBOARD.md
|
||
|
||
This document. The full UX specification is `design/DESIGN_DASHBOARD.md`
|
||
and the frozen API contract is `design/API_CONTRACT.md`.
|
||
|
||
## Renders
|
||
|
||
- `out/dashboard_render_anon.png` — anonymous dashboard
|
||
- `out/dashboard_render_personal.png` — personalized dashboard (ZIP 90210,
|
||
watch Schiavo, opt-in budget topic)
|
||
|
||
Both are data-faithful snapshots from the live API, rendered via macOS
|
||
`qlmanage` (no headless browser available in this environment; the browser
|
||
remote-debugging permission popup is a user-level security dialog that
|
||
cannot be auto-approved).
|
||
|
||
## Known limitations / next steps
|
||
|
||
- `out/tag_cloud.json` must exist for tags endpoint (118 tags as of
|
||
2026-09-06).
|
||
- `0 committee_members` roster — committee lookup degrades gracefully when
|
||
no roster data is supplied.
|
||
- Dev auth is a JSON-file local account, not production auth.
|
||
- ZIP -> district uses a documented demo dataset until roster ETL lands.
|
||
- **Forward signal gap:** leginfo's bill status/history are *backward-only*
|
||
(past actions + status narrative, no future dates), so there is no per-bill
|
||
"next step" label. The dashboard's forward answer comes solely from the
|
||
chamber events calendars (`scheduled_events`).
|
||
- **Recess empty state:** while the Assembly (and at times the Senate) is in
|
||
final recess, `/events/upcoming` legitimately returns **0 events**. The
|
||
"What's next" band and `next_legislative_day` then show an honest "recess /
|
||
unknown" state rather than fabricating a date. This is expected, not a
|
||
parsing failure.
|
||
- The 2026-08-31 daily-file snapshot is analysis-only; it is *not* re-surfaced
|
||
as "upcoming" because the upcoming view floors at today.
|