feat(dashboard): add forward "what's next" signal + fix upcoming floor

The daily-file bill queue is a backward snapshot, so "upcoming" could only
resurface the latest past day (the 08-31 snapshot read as today's queue) and
there was no "what is scheduled next." leginfo bill status/history are
backward-only, so the genuine future-dated signal is the chambers' "upcoming
events" calendars.

- api.py: floor the "upcoming" view at *today* (not the DB max date) when no
  as_of is pinned; surface next_legislative_day + scheduled_events (next 7
  days) on /bills. Add _today_iso / _next_legislative_day / _scheduled_events,
  all degrading to a recess marker / [] (never 5xx).
- relational.py: schema v2 adds the scheduled_events table +
  store_scheduled_events (idempotent per-day refresh), load_scheduled_events,
  next_legislative_day.
- parsers.py: parse_events_page + classify_event_kind for the chamber events
  calendar; skip empty-title rows (decompose the <time> before the fallback).
- ingest.py + cli.py: --store-events lane (cache-first, recess-safe) that
  parses the calendar HTML into scheduled_events.
- sources.py + config.yaml: register assembly_events / senate_events
  (kind: events, empty_is_ok: true).
- run_ingest.sh: run the events lane after the daily-file lane as non-fatal
  (skippable via CALI_SCRAPER_SKIP_EVENTS=1); daily-file exit still gates cron.
- frontend: "Next legislative day" line + "What's next · next 7 days" band
  with honest recess empty states.
- DASHBOARD.md: document the forward signal + known limitations (leginfo
  backward-only gap, recess empty state, 08-31 snapshot is analysis-only).
- tests: 5 new API tests + 18 parse-events tests; 170 pass, 92% coverage.
This commit is contained in:
kfj001 committed 2026-09-06 19:50:02 -07:00
1 parent 1919d43bf0
commit 03eeb6bb62
16 files changed
+941 -70

No files matched your search

+60 -4
View File
@@ -97,20 +97,76 @@ class Ingestor:
timespec="seconds"))
self.log.record(event)
# -- main API --
# -- main API --
def ingest(self, date_iso, source=None, sources=None):
names = sources or ([source] if source else self.cfg.source_names())
result = IngestResult(date=date_iso)
for name in names:
if name not in self.cfg.sources:
raise ValueError(
"Unknown source '%s'. Known: %s"
% (name, self.cfg.source_names()))
"Unknown source '%s'. Known: %s"
% (name, self.cfg.source_names()))
cr = self._ingest_chamber(name, date_iso)
result.chambers.append(cr)
return result
# -- per-chamber --
# -- chamber events forward source (scheduled_events, schema v2) --
def ingest_events(self, date_iso, source=None, sources=None, db_path=None):
"""Fetch + cache + parse + store the chamber "upcoming events" source.
Like :meth:`ingest`, this is cache-first (a date already on disk is served
with no network) and reuses the same source planner (the events source is
``kind: events``). It parses the calendar HTML into scheduled-event rows
and stores them in the ``scheduled_events`` table.
``db_path`` defaults to the relational lane DEFAULT_DB so the dashboard
can read what this writes. A recess (an empty calendar) is a legitimate
no-op, not an error: nothing is cleared and nothing is stored.
"""
from . import relational
from . import parsers
db = db_path or relational.DEFAULT_DB
relational.migrate(db) # ensure the scheduled_events table exists
names = sources or ([source] if source else self.cfg.source_names())
result = IngestResult(date=date_iso)
for name in names:
cfg = self.cfg.source(name)
if cfg.get("kind") != "events":
raise ValueError(
"ingest_events needs an events source; %r is kind %r"
% (name, cfg.get("kind")))
cr = self._ingest_chamber(name, date_iso)
events = []
if cr.status == "ok" and cr.path is not None:
try:
html = cr.path.read_text(encoding="utf-8",
errors="replace")
except OSError as e:
html = None
cr.note += " | cannot read cached page: %s" % e
if html:
base = getattr(cr, "url", "") or ""
events = parsers.parse_events_page(
html, chamber=cfg.get("chamber") or cr.source, source=
name, base_url=base,)
# Store when a page was read (fresh or served from cache). A fresh
# timestamp lets a re-published calendar refresh its day on the next
# run while an unchanged read stays a no-op. An empty parse (recess)
# clears no days, so existing scheduled rows are preserved.
when = datetime.datetime.now(datetime.timezone.utc).strftime(
"%Y-%m-%dT%H:%M:%SZ")
summary = relational.store_scheduled_events(events, path=db,
fetched_at=when)
cr.note += (" | events: %d stored, %d stale cleared, %d total"
% (summary["stored"], summary["cleared_stale"],
summary["total"]))
result.chambers.append(cr)
return result
# -- per-chamber --
def _ingest_chamber(self, name, date_iso):
cfg = self.cfg.source(name)
plan = build_source_plan(name, date_iso, cfg)