kfj001
03eeb6bb62
The daily-file bill queue is a backward snapshot, so "upcoming" could only resurface the latest past day (the 08-31 snapshot read as today's queue) and there was no "what is scheduled next." leginfo bill status/history are backward-only, so the genuine future-dated signal is the chambers' "upcoming events" calendars. - api.py: floor the "upcoming" view at *today* (not the DB max date) when no as_of is pinned; surface next_legislative_day + scheduled_events (next 7 days) on /bills. Add _today_iso / _next_legislative_day / _scheduled_events, all degrading to a recess marker / [] (never 5xx). - relational.py: schema v2 adds the scheduled_events table + store_scheduled_events (idempotent per-day refresh), load_scheduled_events, next_legislative_day. - parsers.py: parse_events_page + classify_event_kind for the chamber events calendar; skip empty-title rows (decompose the <time> before the fallback). - ingest.py + cli.py: --store-events lane (cache-first, recess-safe) that parses the calendar HTML into scheduled_events. - sources.py + config.yaml: register assembly_events / senate_events (kind: events, empty_is_ok: true). - run_ingest.sh: run the events lane after the daily-file lane as non-fatal (skippable via CALI_SCRAPER_SKIP_EVENTS=1); daily-file exit still gates cron. - frontend: "Next legislative day" line + "What's next · next 7 days" band with honest recess empty states. - DASHBOARD.md: document the forward signal + known limitations (leginfo backward-only gap, recess empty state, 08-31 snapshot is analysis-only). - tests: 5 new API tests + 18 parse-events tests; 170 pass, 92% coverage.
69 lines
3.0 KiB
Bash
Executable File
69 lines
3.0 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Cron-compatible wrapper for the California Daily File ingestion layer.
|
|
#
|
|
# Install (nightly at 02:00 PT -- see install notes below) with:
|
|
# crontab -e ->
|
|
# 0 2 * * * /Users/kyle/Documents/cali_scraper/scripts/run_ingest.sh
|
|
#
|
|
# Behavior:
|
|
# * Runs `python -m scraper ingest` for BOTH chambers for "today" (local date),
|
|
# unless CALI_SCRAPER_DATE overrides it (YYYY-MM-DD).
|
|
# * --force is NOT used by default: a date already in the local raw cache is
|
|
# served from disk with zero network (the "re-parse doesn't re-hit" req).
|
|
# * Exit code is 0 on ok / cached / expected-404 (no publication that day),
|
|
# 1 on a real failure (antibot / transient / unexpected 404) so cron mail
|
|
# fires only for problems worth paging on.
|
|
# * Every request is appended to cache/request_log.jsonl.
|
|
|
|
set -uo pipefail
|
|
|
|
# Resolve the repo root regardless of the cron caller's working directory.
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
|
|
cd "${ROOT}" || { echo "[run_ingest] cannot cd to ${ROOT}" >&2; exit 2; }
|
|
|
|
# Pick the Python interpreter (prefer a project venv if present).
|
|
PY="${PYTHON:-python3}"
|
|
|
|
# Date to ingest: CALI_SCRAPER_DATE or today in the local (PT) timezone.
|
|
DATE="${CALI_SCRAPER_DATE:-$(date +%F)}"
|
|
|
|
# Force a network refresh even if cached? CALI_SCRAPER_FORCE=1 enables --force.
|
|
FORCE=""
|
|
if [ "${CALI_SCRAPER_FORCE:-0}" = "1" ]; then
|
|
FORCE="--force"
|
|
fi
|
|
|
|
echo "==================================================================="
|
|
echo "[run_ingest] ${DATE} root=${ROOT} py=${PY}"
|
|
echo "==================================================================="
|
|
|
|
# Lane 1: the daily-file bill queue (backward snapshot). -u: unbuffered; prints
|
|
# the human-readable summary.
|
|
"${PY}" -u -m scraper ingest --date "${DATE}" ${FORCE}
|
|
EXIT=$?
|
|
|
|
case "${EXIT}" in
|
|
0) echo "[run_ingest] DONE ${DATE} daily-file (ok / cached / no publication -- all expected)" ;;
|
|
1) echo "[run_ingest] FAILED ${DATE} daily-file see cache/request_log.jsonl for the attempt trail" >&2 ;;
|
|
*) echo "[run_ingest] ERROR ${DATE} daily-file (exit ${EXIT})" >&2 ;;
|
|
esac
|
|
|
|
# Lane 2: the chamber "upcoming events" calendar -> scheduled_events table
|
|
# (the forward-looking "what's next" signal; the daily file alone is backward).
|
|
# A recess (empty calendar) is a no-op, not a failure. CALI_SCRAPER_SKIP_EVENTS=
|
|
# 1 disables this lane. The daily-file EXIT still governs cron mail: events are
|
|
# best-effort and degrade to an empty forward band rather than page.
|
|
if [ "${CALI_SCRAPER_SKIP_EVENTS:-0}" != "1" ]; then
|
|
"${PY}" -u -m scraper ingest --date "${DATE}" ${FORCE} --store-events \
|
|
--db "db/daily_file.sqlite"
|
|
EVT=$?
|
|
case "${EVT}" in
|
|
0) echo "[run_ingest] DONE ${DATE} events (ok / recess / cached)" ;;
|
|
1) echo "[run_ingest] WARNING ${DATE} events ingest hiccup (non-fatal; see request_log)" >&2 ;;
|
|
*) echo "[run_ingest] WARNING ${DATE} events unexpected exit ${EVT} (non-fatal)" >&2 ;;
|
|
esac
|
|
fi
|
|
|
|
exit "${EXIT}"
|