feat(dashboard): add forward "what's next" signal + fix upcoming floor
The daily-file bill queue is a backward snapshot, so "upcoming" could only resurface the latest past day (the 08-31 snapshot read as today's queue) and there was no "what is scheduled next." leginfo bill status/history are backward-only, so the genuine future-dated signal is the chambers' "upcoming events" calendars. - api.py: floor the "upcoming" view at *today* (not the DB max date) when no as_of is pinned; surface next_legislative_day + scheduled_events (next 7 days) on /bills. Add _today_iso / _next_legislative_day / _scheduled_events, all degrading to a recess marker / [] (never 5xx). - relational.py: schema v2 adds the scheduled_events table + store_scheduled_events (idempotent per-day refresh), load_scheduled_events, next_legislative_day. - parsers.py: parse_events_page + classify_event_kind for the chamber events calendar; skip empty-title rows (decompose the <time> before the fallback). - ingest.py + cli.py: --store-events lane (cache-first, recess-safe) that parses the calendar HTML into scheduled_events. - sources.py + config.yaml: register assembly_events / senate_events (kind: events, empty_is_ok: true). - run_ingest.sh: run the events lane after the daily-file lane as non-fatal (skippable via CALI_SCRAPER_SKIP_EVENTS=1); daily-file exit still gates cron. - frontend: "Next legislative day" line + "What's next · next 7 days" band with honest recess empty states. - DASHBOARD.md: document the forward signal + known limitations (leginfo backward-only gap, recess empty state, 08-31 snapshot is analysis-only). - tests: 5 new API tests + 18 parse-events tests; 170 pass, 92% coverage.
This commit is contained in:
1 parent
1919d43bf0
commit
03eeb6bb62
16 files changed
+941
-70
No files matched your search
@@ -75,16 +75,42 @@ list:
|
||||
marker and a "Floated: ..." reason line
|
||||
- Zero-scored bills remain in their date-grouped order below
|
||||
|
||||
## Forward signal: "what's next" (scheduled_events, schema v2)
|
||||
|
||||
The daily-file bill queue is a *backward* snapshot — leginfo's bill status and
|
||||
bill history are past-action logs, so they cannot answer "what is scheduled
|
||||
next." The only genuine future-dated signal is the chambers' "upcoming events"
|
||||
calendars (`assembly.ca.gov/events/upcoming`, `senate.ca.gov/calendar`). These
|
||||
are fetched (`python3 -m scraper ingest --store-events`), parsed into the
|
||||
`scheduled_events` table (relational schema v2), and surfaced on `/bills` as two
|
||||
new envelope keys:
|
||||
|
||||
- `next_legislative_day` — `{date, kind, chamber, title, source}` (the earliest
|
||||
event on/after `as_of`), or `{date: null, recess: true}` when nothing is
|
||||
scheduled.
|
||||
- `scheduled_events` — the next 7 days of chamber events.
|
||||
|
||||
The "Upcoming" filter is a *forward* view: with `upcoming=1` and no explicit
|
||||
`as_of`, the bill queue floors at **today**, so a stale/older daily-file
|
||||
snapshot can no longer present itself as "upcoming." Omitting `upcoming`
|
||||
(the "All" / history view) still returns every row.
|
||||
|
||||
The dashboard renders a "Next legislative day" line in the freshness band and a
|
||||
"What's next · next 7 days" band. A recess (empty calendar) is an *honest*
|
||||
empty state, not an error.
|
||||
|
||||
## Tests
|
||||
|
||||
```bash
|
||||
python3 -m pytest tests/test_dashboard_ranking.py tests/test_dashboard_api.py
|
||||
python3 -m pytest tests/test_dashboard_ranking.py tests/test_dashboard_api.py \
|
||||
tests/test_parse_events.py
|
||||
```
|
||||
|
||||
32 tests (12 ranking + 20 API), all passing. Pure ranking logic is tested in
|
||||
isolation; API handlers are tested with no sockets via the
|
||||
55 tests (12 ranking + 25 API + 18 parse-events), all passing. Pure ranking
|
||||
logic is tested in isolation; API handlers are tested with no sockets via the
|
||||
`handler(req) -> (status, body)` pattern; a live `ThreadingHTTPServer` smoke test
|
||||
confirms static + API endpoints serve.
|
||||
confirms static + API endpoints serve. The event parser is tested against
|
||||
inline fixtures plus the saved `samples/assembly_events_*.html` pages.
|
||||
|
||||
## DASHBOARD.md
|
||||
|
||||
@@ -110,3 +136,14 @@ cannot be auto-approved).
|
||||
no roster data is supplied.
|
||||
- Dev auth is a JSON-file local account, not production auth.
|
||||
- ZIP -> district uses a documented demo dataset until roster ETL lands.
|
||||
- **Forward signal gap:** leginfo's bill status/history are *backward-only*
|
||||
(past actions + status narrative, no future dates), so there is no per-bill
|
||||
"next step" label. The dashboard's forward answer comes solely from the
|
||||
chamber events calendars (`scheduled_events`).
|
||||
- **Recess empty state:** while the Assembly (and at times the Senate) is in
|
||||
final recess, `/events/upcoming` legitimately returns **0 events**. The
|
||||
"What's next" band and `next_legislative_day` then show an honest "recess /
|
||||
unknown" state rather than fabricating a date. This is expected, not a
|
||||
parsing failure.
|
||||
- The 2026-08-31 daily-file snapshot is analysis-only; it is *not* re-surfaced
|
||||
as "upcoming" because the upcoming view floors at today.
|
||||
Reference in new issue
Block a user