feat(dashboard): add forward "what's next" signal + fix upcoming floor

The daily-file bill queue is a backward snapshot, so "upcoming" could only
resurface the latest past day (the 08-31 snapshot read as today's queue) and
there was no "what is scheduled next." leginfo bill status/history are
backward-only, so the genuine future-dated signal is the chambers' "upcoming
events" calendars.

- api.py: floor the "upcoming" view at *today* (not the DB max date) when no
  as_of is pinned; surface next_legislative_day + scheduled_events (next 7
  days) on /bills. Add _today_iso / _next_legislative_day / _scheduled_events,
  all degrading to a recess marker / [] (never 5xx).
- relational.py: schema v2 adds the scheduled_events table +
  store_scheduled_events (idempotent per-day refresh), load_scheduled_events,
  next_legislative_day.
- parsers.py: parse_events_page + classify_event_kind for the chamber events
  calendar; skip empty-title rows (decompose the <time> before the fallback).
- ingest.py + cli.py: --store-events lane (cache-first, recess-safe) that
  parses the calendar HTML into scheduled_events.
- sources.py + config.yaml: register assembly_events / senate_events
  (kind: events, empty_is_ok: true).
- run_ingest.sh: run the events lane after the daily-file lane as non-fatal
  (skippable via CALI_SCRAPER_SKIP_EVENTS=1); daily-file exit still gates cron.
- frontend: "Next legislative day" line + "What's next · next 7 days" band
  with honest recess empty states.
- DASHBOARD.md: document the forward signal + known limitations (leginfo
  backward-only gap, recess empty state, 08-31 snapshot is analysis-only).
- tests: 5 new API tests + 18 parse-events tests; 170 pass, 92% coverage.
This commit is contained in:
kfj001 committed 2026-09-06 19:50:02 -07:00
1 parent 1919d43bf0
commit 03eeb6bb62
16 files changed
+941 -70

No files matched your search

+21 -4
View File
@@ -38,14 +38,31 @@ echo "==================================================================="
echo "[run_ingest] ${DATE} root=${ROOT} py=${PY}"
echo "==================================================================="
# -u: unbuffered; the CLI prints the human-readable summary.
# Lane 1: the daily-file bill queue (backward snapshot). -u: unbuffered; prints
# the human-readable summary.
"${PY}" -u -m scraper ingest --date "${DATE}" ${FORCE}
EXIT=$?
case "${EXIT}" in
0) echo "[run_ingest] DONE ${DATE} (ok / cached / no publication -- all expected)" ;;
1) echo "[run_ingest] FAILED ${DATE} see cache/request_log.jsonl for the attempt trail" >&2 ;;
*) echo "[run_ingest] ERROR ${DATE} (exit ${EXIT})" >&2 ;;
0) echo "[run_ingest] DONE ${DATE} daily-file (ok / cached / no publication -- all expected)" ;;
1) echo "[run_ingest] FAILED ${DATE} daily-file see cache/request_log.jsonl for the attempt trail" >&2 ;;
*) echo "[run_ingest] ERROR ${DATE} daily-file (exit ${EXIT})" >&2 ;;
esac
# Lane 2: the chamber "upcoming events" calendar -> scheduled_events table
# (the forward-looking "what's next" signal; the daily file alone is backward).
# A recess (empty calendar) is a no-op, not a failure. CALI_SCRAPER_SKIP_EVENTS=
# 1 disables this lane. The daily-file EXIT still governs cron mail: events are
# best-effort and degrade to an empty forward band rather than page.
if [ "${CALI_SCRAPER_SKIP_EVENTS:-0}" != "1" ]; then
"${PY}" -u -m scraper ingest --date "${DATE}" ${FORCE} --store-events \
--db "db/daily_file.sqlite"
EVT=$?
case "${EVT}" in
0) echo "[run_ingest] DONE ${DATE} events (ok / recess / cached)" ;;
1) echo "[run_ingest] WARNING ${DATE} events ingest hiccup (non-fatal; see request_log)" >&2 ;;
*) echo "[run_ingest] WARNING ${DATE} events unexpected exit ${EVT} (non-fatal)" >&2 ;;
esac
fi
exit "${EXIT}"