feat(dashboard): add forward "what's next" signal + fix upcoming floor
The daily-file bill queue is a backward snapshot, so "upcoming" could only resurface the latest past day (the 08-31 snapshot read as today's queue) and there was no "what is scheduled next." leginfo bill status/history are backward-only, so the genuine future-dated signal is the chambers' "upcoming events" calendars. - api.py: floor the "upcoming" view at *today* (not the DB max date) when no as_of is pinned; surface next_legislative_day + scheduled_events (next 7 days) on /bills. Add _today_iso / _next_legislative_day / _scheduled_events, all degrading to a recess marker / [] (never 5xx). - relational.py: schema v2 adds the scheduled_events table + store_scheduled_events (idempotent per-day refresh), load_scheduled_events, next_legislative_day. - parsers.py: parse_events_page + classify_event_kind for the chamber events calendar; skip empty-title rows (decompose the <time> before the fallback). - ingest.py + cli.py: --store-events lane (cache-first, recess-safe) that parses the calendar HTML into scheduled_events. - sources.py + config.yaml: register assembly_events / senate_events (kind: events, empty_is_ok: true). - run_ingest.sh: run the events lane after the daily-file lane as non-fatal (skippable via CALI_SCRAPER_SKIP_EVENTS=1); daily-file exit still gates cron. - frontend: "Next legislative day" line + "What's next · next 7 days" band with honest recess empty states. - DASHBOARD.md: document the forward signal + known limitations (leginfo backward-only gap, recess empty state, 08-31 snapshot is analysis-only). - tests: 5 new API tests + 18 parse-events tests; 170 pass, 92% coverage.
This commit is contained in:
1 parent
1919d43bf0
commit
03eeb6bb62
16 files changed
+941
-70
No files matched your search
+21
-6
@@ -16,18 +16,33 @@ SENATE_PDF_RE = re.compile(
|
||||
|
||||
@dataclass
|
||||
class SourcePlan:
|
||||
source: str # "assembly_daily_file" | "senate_daily_file"
|
||||
date: str # requested YYYY-MM-DD
|
||||
kind: str # "html" | "pdf"
|
||||
primary_url: str # URL that is actually fetched
|
||||
ext: str # cache extension ("html" | "pdf")
|
||||
source: str # "assembly_daily_file" | "senate_daily_file"
|
||||
date: str # requested YYYY-MM-DD
|
||||
kind: str # "html" | "pdf" | "events"
|
||||
primary_url: str # URL that is actually fetched
|
||||
ext: str # cache extension ("html" | "pdf")
|
||||
pdf_url: Optional[str] = None
|
||||
as_of: Optional[str] = None # date the live page itself represents
|
||||
as_of: Optional[str] = None # date the live page itself represents
|
||||
extra_urls: list = field(default_factory=list)
|
||||
|
||||
|
||||
def build_source_plan(name: str, date_iso: str, src_cfg: dict) -> SourcePlan:
|
||||
kind = src_cfg.get("kind")
|
||||
if kind == "events":
|
||||
# The chamber "upcoming events" calendar is a live HTML page (a
|
||||
# forward-looking source, distinct from the daily file). Its URL is
|
||||
# constant; the date only drives the cache key (the page may list
|
||||
# future-dated events, none of which is the --date value itself).
|
||||
url = src_cfg.get("url") or src_cfg.get("index_url")
|
||||
return SourcePlan(
|
||||
source=name,
|
||||
date=date_iso,
|
||||
kind="events",
|
||||
primary_url=url,
|
||||
ext="html",
|
||||
as_of=date_iso,
|
||||
)
|
||||
|
||||
if kind == "html":
|
||||
# Assembly: constant URL, always the current live file.
|
||||
return SourcePlan(
|
||||
|
||||
Reference in new issue
Block a user