feat(dashboard): add forward "what's next" signal + fix upcoming floor
The daily-file bill queue is a backward snapshot, so "upcoming" could only resurface the latest past day (the 08-31 snapshot read as today's queue) and there was no "what is scheduled next." leginfo bill status/history are backward-only, so the genuine future-dated signal is the chambers' "upcoming events" calendars. - api.py: floor the "upcoming" view at *today* (not the DB max date) when no as_of is pinned; surface next_legislative_day + scheduled_events (next 7 days) on /bills. Add _today_iso / _next_legislative_day / _scheduled_events, all degrading to a recess marker / [] (never 5xx). - relational.py: schema v2 adds the scheduled_events table + store_scheduled_events (idempotent per-day refresh), load_scheduled_events, next_legislative_day. - parsers.py: parse_events_page + classify_event_kind for the chamber events calendar; skip empty-title rows (decompose the <time> before the fallback). - ingest.py + cli.py: --store-events lane (cache-first, recess-safe) that parses the calendar HTML into scheduled_events. - sources.py + config.yaml: register assembly_events / senate_events (kind: events, empty_is_ok: true). - run_ingest.sh: run the events lane after the daily-file lane as non-fatal (skippable via CALI_SCRAPER_SKIP_EVENTS=1); daily-file exit still gates cron. - frontend: "Next legislative day" line + "What's next · next 7 days" band with honest recess empty states. - DASHBOARD.md: document the forward signal + known limitations (leginfo backward-only gap, recess empty state, 08-31 snapshot is analysis-only). - tests: 5 new API tests + 18 parse-events tests; 170 pass, 92% coverage.
This commit is contained in:
1 parent
1919d43bf0
commit
03eeb6bb62
16 files changed
+941
-70
No files matched your search
+21
-3
@@ -32,10 +32,21 @@ def cmd_ingest(args) -> int:
|
||||
try:
|
||||
ing = Ingestor(config_path=args.config, force=args.force)
|
||||
sources = _sources_arg(args)
|
||||
result = ing.ingest(date, sources=sources)
|
||||
if getattr(args, "store_events", False):
|
||||
# Forward-looking chamber calendar -> scheduled_events table. Reuses
|
||||
# the same cache-first pipeline; a recess (empty calendar) is a
|
||||
# no-op, not a failure. Default to the events sources when none are
|
||||
# named (the daily-file sources are a different lane).
|
||||
if not sources:
|
||||
sources = [n for n, s in ing.cfg.sources.items()
|
||||
if s.get("kind") == "events"]
|
||||
result = ing.ingest_events(date, sources=sources,
|
||||
db_path=getattr(args, "db", None))
|
||||
else:
|
||||
result = ing.ingest(date, sources=sources)
|
||||
ing.close()
|
||||
except SystemExit as exc:
|
||||
# config / argparse errors surface here
|
||||
# config / argparse errors surface here
|
||||
print("ERROR: %s" % exc, file=sys.stderr)
|
||||
return int(exc.code or 2)
|
||||
|
||||
@@ -172,11 +183,18 @@ def build_parser() -> argparse.ArgumentParser:
|
||||
pi.add_argument("--date", help="YYYY-MM-DD (default: today)")
|
||||
pi.add_argument("--source",
|
||||
help="Chamber name(s), comma-separated "
|
||||
"(assembly_daily_file, senate_daily_file). Default: all.")
|
||||
"(assembly_daily_file, senate_daily_file). Default: all.")
|
||||
pi.add_argument("--force", action="store_true",
|
||||
help="Re-fetch even if already cached.")
|
||||
pi.add_argument("--json", action="store_true",
|
||||
help="Emit the result as JSON.")
|
||||
pi.add_argument("--store-events", dest="store_events", action="store_true",
|
||||
help="Fetch + parse the chamber 'upcoming events' calendar "
|
||||
"and store it in the scheduled_events table "
|
||||
"(forward-looking; the daily-file queue is backward).")
|
||||
pi.add_argument("--db", dest="db", default=None,
|
||||
help="SQLite path for --store-events "
|
||||
"(default: the relational lane's db/daily_file.sqlite).")
|
||||
|
||||
pl = sub.add_parser("list", help="List configured sources + URLs")
|
||||
pc = sub.add_parser("check", help="Cache status for a date (no network)")
|
||||
|
||||
Reference in new issue
Block a user