feat(dashboard): add forward "what's next" signal + fix upcoming floor
The daily-file bill queue is a backward snapshot, so "upcoming" could only resurface the latest past day (the 08-31 snapshot read as today's queue) and there was no "what is scheduled next." leginfo bill status/history are backward-only, so the genuine future-dated signal is the chambers' "upcoming events" calendars. - api.py: floor the "upcoming" view at *today* (not the DB max date) when no as_of is pinned; surface next_legislative_day + scheduled_events (next 7 days) on /bills. Add _today_iso / _next_legislative_day / _scheduled_events, all degrading to a recess marker / [] (never 5xx). - relational.py: schema v2 adds the scheduled_events table + store_scheduled_events (idempotent per-day refresh), load_scheduled_events, next_legislative_day. - parsers.py: parse_events_page + classify_event_kind for the chamber events calendar; skip empty-title rows (decompose the <time> before the fallback). - ingest.py + cli.py: --store-events lane (cache-first, recess-safe) that parses the calendar HTML into scheduled_events. - sources.py + config.yaml: register assembly_events / senate_events (kind: events, empty_is_ok: true). - run_ingest.sh: run the events lane after the daily-file lane as non-fatal (skippable via CALI_SCRAPER_SKIP_EVENTS=1); daily-file exit still gates cron. - frontend: "Next legislative day" line + "What's next · next 7 days" band with honest recess empty states. - DASHBOARD.md: document the forward signal + known limitations (leginfo backward-only gap, recess empty state, 08-31 snapshot is analysis-only). - tests: 5 new API tests + 18 parse-events tests; 170 pass, 92% coverage.
This commit is contained in:
1 parent
1919d43bf0
commit
03eeb6bb62
16 files changed
+941
-70
No files matched your search
+60
-4
@@ -97,20 +97,76 @@ class Ingestor:
|
||||
timespec="seconds"))
|
||||
self.log.record(event)
|
||||
|
||||
# -- main API --
|
||||
# -- main API --
|
||||
def ingest(self, date_iso, source=None, sources=None):
|
||||
names = sources or ([source] if source else self.cfg.source_names())
|
||||
result = IngestResult(date=date_iso)
|
||||
for name in names:
|
||||
if name not in self.cfg.sources:
|
||||
raise ValueError(
|
||||
"Unknown source '%s'. Known: %s"
|
||||
% (name, self.cfg.source_names()))
|
||||
"Unknown source '%s'. Known: %s"
|
||||
% (name, self.cfg.source_names()))
|
||||
cr = self._ingest_chamber(name, date_iso)
|
||||
result.chambers.append(cr)
|
||||
return result
|
||||
|
||||
# -- per-chamber --
|
||||
# -- chamber events forward source (scheduled_events, schema v2) --
|
||||
def ingest_events(self, date_iso, source=None, sources=None, db_path=None):
|
||||
"""Fetch + cache + parse + store the chamber "upcoming events" source.
|
||||
|
||||
Like :meth:`ingest`, this is cache-first (a date already on disk is served
|
||||
with no network) and reuses the same source planner (the events source is
|
||||
``kind: events``). It parses the calendar HTML into scheduled-event rows
|
||||
and stores them in the ``scheduled_events`` table.
|
||||
|
||||
``db_path`` defaults to the relational lane DEFAULT_DB so the dashboard
|
||||
can read what this writes. A recess (an empty calendar) is a legitimate
|
||||
no-op, not an error: nothing is cleared and nothing is stored.
|
||||
"""
|
||||
from . import relational
|
||||
from . import parsers
|
||||
|
||||
db = db_path or relational.DEFAULT_DB
|
||||
relational.migrate(db) # ensure the scheduled_events table exists
|
||||
names = sources or ([source] if source else self.cfg.source_names())
|
||||
result = IngestResult(date=date_iso)
|
||||
for name in names:
|
||||
cfg = self.cfg.source(name)
|
||||
if cfg.get("kind") != "events":
|
||||
raise ValueError(
|
||||
"ingest_events needs an events source; %r is kind %r"
|
||||
% (name, cfg.get("kind")))
|
||||
cr = self._ingest_chamber(name, date_iso)
|
||||
|
||||
events = []
|
||||
if cr.status == "ok" and cr.path is not None:
|
||||
try:
|
||||
html = cr.path.read_text(encoding="utf-8",
|
||||
errors="replace")
|
||||
except OSError as e:
|
||||
html = None
|
||||
cr.note += " | cannot read cached page: %s" % e
|
||||
if html:
|
||||
base = getattr(cr, "url", "") or ""
|
||||
events = parsers.parse_events_page(
|
||||
html, chamber=cfg.get("chamber") or cr.source, source=
|
||||
name, base_url=base,)
|
||||
|
||||
# Store when a page was read (fresh or served from cache). A fresh
|
||||
# timestamp lets a re-published calendar refresh its day on the next
|
||||
# run while an unchanged read stays a no-op. An empty parse (recess)
|
||||
# clears no days, so existing scheduled rows are preserved.
|
||||
when = datetime.datetime.now(datetime.timezone.utc).strftime(
|
||||
"%Y-%m-%dT%H:%M:%SZ")
|
||||
summary = relational.store_scheduled_events(events, path=db,
|
||||
fetched_at=when)
|
||||
cr.note += (" | events: %d stored, %d stale cleared, %d total"
|
||||
% (summary["stored"], summary["cleared_stale"],
|
||||
summary["total"]))
|
||||
result.chambers.append(cr)
|
||||
return result
|
||||
|
||||
# -- per-chamber --
|
||||
def _ingest_chamber(self, name, date_iso):
|
||||
cfg = self.cfg.source(name)
|
||||
plan = build_source_plan(name, date_iso, cfg)
|
||||
|
||||
Reference in new issue
Block a user