feat(dashboard): add forward "what's next" signal + fix upcoming floor
The daily-file bill queue is a backward snapshot, so "upcoming" could only resurface the latest past day (the 08-31 snapshot read as today's queue) and there was no "what is scheduled next." leginfo bill status/history are backward-only, so the genuine future-dated signal is the chambers' "upcoming events" calendars. - api.py: floor the "upcoming" view at *today* (not the DB max date) when no as_of is pinned; surface next_legislative_day + scheduled_events (next 7 days) on /bills. Add _today_iso / _next_legislative_day / _scheduled_events, all degrading to a recess marker / [] (never 5xx). - relational.py: schema v2 adds the scheduled_events table + store_scheduled_events (idempotent per-day refresh), load_scheduled_events, next_legislative_day. - parsers.py: parse_events_page + classify_event_kind for the chamber events calendar; skip empty-title rows (decompose the <time> before the fallback). - ingest.py + cli.py: --store-events lane (cache-first, recess-safe) that parses the calendar HTML into scheduled_events. - sources.py + config.yaml: register assembly_events / senate_events (kind: events, empty_is_ok: true). - run_ingest.sh: run the events lane after the daily-file lane as non-fatal (skippable via CALI_SCRAPER_SKIP_EVENTS=1); daily-file exit still gates cron. - frontend: "Next legislative day" line + "What's next · next 7 days" band with honest recess empty states. - DASHBOARD.md: document the forward signal + known limitations (leginfo backward-only gap, recess empty state, 08-31 snapshot is analysis-only). - tests: 5 new API tests + 18 parse-events tests; 170 pass, 92% coverage.
This commit is contained in:
1 parent
1919d43bf0
commit
03eeb6bb62
16 files changed
+941
-70
No files matched your search
+21
-4
@@ -38,14 +38,31 @@ echo "==================================================================="
|
||||
echo "[run_ingest] ${DATE} root=${ROOT} py=${PY}"
|
||||
echo "==================================================================="
|
||||
|
||||
# -u: unbuffered; the CLI prints the human-readable summary.
|
||||
# Lane 1: the daily-file bill queue (backward snapshot). -u: unbuffered; prints
|
||||
# the human-readable summary.
|
||||
"${PY}" -u -m scraper ingest --date "${DATE}" ${FORCE}
|
||||
EXIT=$?
|
||||
|
||||
case "${EXIT}" in
|
||||
0) echo "[run_ingest] DONE ${DATE} (ok / cached / no publication -- all expected)" ;;
|
||||
1) echo "[run_ingest] FAILED ${DATE} see cache/request_log.jsonl for the attempt trail" >&2 ;;
|
||||
*) echo "[run_ingest] ERROR ${DATE} (exit ${EXIT})" >&2 ;;
|
||||
0) echo "[run_ingest] DONE ${DATE} daily-file (ok / cached / no publication -- all expected)" ;;
|
||||
1) echo "[run_ingest] FAILED ${DATE} daily-file see cache/request_log.jsonl for the attempt trail" >&2 ;;
|
||||
*) echo "[run_ingest] ERROR ${DATE} daily-file (exit ${EXIT})" >&2 ;;
|
||||
esac
|
||||
|
||||
# Lane 2: the chamber "upcoming events" calendar -> scheduled_events table
|
||||
# (the forward-looking "what's next" signal; the daily file alone is backward).
|
||||
# A recess (empty calendar) is a no-op, not a failure. CALI_SCRAPER_SKIP_EVENTS=
|
||||
# 1 disables this lane. The daily-file EXIT still governs cron mail: events are
|
||||
# best-effort and degrade to an empty forward band rather than page.
|
||||
if [ "${CALI_SCRAPER_SKIP_EVENTS:-0}" != "1" ]; then
|
||||
"${PY}" -u -m scraper ingest --date "${DATE}" ${FORCE} --store-events \
|
||||
--db "db/daily_file.sqlite"
|
||||
EVT=$?
|
||||
case "${EVT}" in
|
||||
0) echo "[run_ingest] DONE ${DATE} events (ok / recess / cached)" ;;
|
||||
1) echo "[run_ingest] WARNING ${DATE} events ingest hiccup (non-fatal; see request_log)" >&2 ;;
|
||||
*) echo "[run_ingest] WARNING ${DATE} events unexpected exit ${EVT} (non-fatal)" >&2 ;;
|
||||
esac
|
||||
fi
|
||||
|
||||
exit "${EXIT}"
|
||||
Reference in new issue
Block a user