feat(dashboard): add forward "what's next" signal + fix upcoming floor

The daily-file bill queue is a backward snapshot, so "upcoming" could only resurface the latest past day (the 08-31 snapshot read as today's queue) and there was no "what is scheduled next." leginfo bill status/history are backward-only, so the genuine future-dated signal is the chambers' "upcoming events" calendars. - api.py: floor the "upcoming" view at *today* (not the DB max date) when no as_of is pinned; surface next_legislative_day + scheduled_events (next 7 days) on /bills. Add _today_iso / _next_legislative_day / _scheduled_events, all degrading to a recess marker / [] (never 5xx). - relational.py: schema v2 adds the scheduled_events table + store_scheduled_events (idempotent per-day refresh), load_scheduled_events, next_legislative_day. - parsers.py: parse_events_page + classify_event_kind for the chamber events calendar; skip empty-title rows (decompose the <time> before the fallback). - ingest.py + cli.py: --store-events lane (cache-first, recess-safe) that parses the calendar HTML into scheduled_events. - sources.py + config.yaml: register assembly_events / senate_events (kind: events, empty_is_ok: true). - run_ingest.sh: run the events lane after the daily-file lane as non-fatal (skippable via CALI_SCRAPER_SKIP_EVENTS=1); daily-file exit still gates cron. - frontend: "Next legislative day" line + "What's next · next 7 days" band with honest recess empty states. - DASHBOARD.md: document the forward signal + known limitations (leginfo backward-only gap, recess empty state, 08-31 snapshot is analysis-only). - tests: 5 new API tests + 18 parse-events tests; 170 pass, 92% coverage.
2026-09-06 19:50:02 -07:00
1 parent 1919d43bf0
commit 03eeb6bb62
@@ -32,10 +32,21 @@ def cmd_ingest(args) -> int:
    try:
        ing = Ingestor(config_path=args.config, force=args.force)
        sources = _sources_arg(args)
-        result = ing.ingest(date, sources=sources)
+        if getattr(args, "store_events", False):
+              # Forward-looking chamber calendar -> scheduled_events table. Reuses
+              # the same cache-first pipeline; a recess (empty calendar) is a
+              # no-op, not a failure. Default to the events sources when none are
+              # named (the daily-file sources are a different lane).
+            if not sources:
+                sources = [n for n, s in ing.cfg.sources.items()
+                           if s.get("kind") == "events"]
+            result = ing.ingest_events(date, sources=sources,
+                                       db_path=getattr(args, "db", None))
+        else:
+            result = ing.ingest(date, sources=sources)
        ing.close()
    except SystemExit as exc:
-              # config / argparse errors surface here
+            # config / argparse errors surface here
        print("ERROR: %s" % exc, file=sys.stderr)
        return int(exc.code or 2)

@@ -172,11 +183,18 @@ def build_parser() -> argparse.ArgumentParser:
    pi.add_argument("--date", help="YYYY-MM-DD (default: today)")
    pi.add_argument("--source",
                     help="Chamber name(s), comma-separated "
-                           "(assembly_daily_file, senate_daily_file). Default: all.")
+                            "(assembly_daily_file, senate_daily_file). Default: all.")
    pi.add_argument("--force", action="store_true",
                     help="Re-fetch even if already cached.")
    pi.add_argument("--json", action="store_true",
                     help="Emit the result as JSON.")
+    pi.add_argument("--store-events", dest="store_events", action="store_true",
+                     help="Fetch + parse the chamber 'upcoming events' calendar "
+                            "and store it in the scheduled_events table "
+                            "(forward-looking; the daily-file queue is backward).")
+    pi.add_argument("--db", dest="db", default=None,
+                     help="SQLite path for --store-events "
+                            "(default: the relational lane's db/daily_file.sqlite).")

    pl = sub.add_parser("list", help="List configured sources + URLs")
    pc = sub.add_parser("check", help="Cache status for a date (no network)")
@@ -75,16 +75,42 @@ list:
  marker and a "Floated: ..." reason line
 - Zero-scored bills remain in their date-grouped order below

+## Forward signal: "what's next" (scheduled_events, schema v2)
+
+The daily-file bill queue is a *backward* snapshot — leginfo's bill status and
+bill history are past-action logs, so they cannot answer "what is scheduled
+next." The only genuine future-dated signal is the chambers' "upcoming events"
+calendars (`assembly.ca.gov/events/upcoming`, `senate.ca.gov/calendar`). These
+are fetched (`python3 -m scraper ingest --store-events`), parsed into the
+`scheduled_events` table (relational schema v2), and surfaced on `/bills` as two
+new envelope keys:
+
+- `next_legislative_day` — `{date, kind, chamber, title, source}` (the earliest
+    event on/after `as_of`), or `{date: null, recess: true}` when nothing is
+    scheduled.
+- `scheduled_events` — the next 7 days of chamber events.
+
+The "Upcoming" filter is a *forward* view: with `upcoming=1` and no explicit
+`as_of`, the bill queue floors at **today**, so a stale/older daily-file
+snapshot can no longer present itself as "upcoming." Omitting `upcoming`
+(the "All" / history view) still returns every row.
+
+The dashboard renders a "Next legislative day" line in the freshness band and a
+"What's next · next 7 days" band. A recess (empty calendar) is an *honest*
+empty state, not an error.
+
 ## Tests

 ```bash
-python3 -m pytest tests/test_dashboard_ranking.py tests/test_dashboard_api.py
+python3 -m pytest tests/test_dashboard_ranking.py tests/test_dashboard_api.py \
+  tests/test_parse_events.py
 ```

-32 tests (12 ranking + 20 API), all passing.  Pure ranking logic is tested in
-isolation; API handlers are tested with no sockets via the
+55 tests (12 ranking + 25 API + 18 parse-events), all passing.  Pure ranking
+logic is tested in isolation; API handlers are tested with no sockets via the
 `handler(req) -> (status, body)` pattern; a live `ThreadingHTTPServer` smoke test
-confirms static + API endpoints serve.
+confirms static + API endpoints serve. The event parser is tested against
+inline fixtures plus the saved `samples/assembly_events_*.html` pages.

 ## DASHBOARD.md

@@ -110,3 +136,14 @@ cannot be auto-approved).
  no roster data is supplied.
 - Dev auth is a JSON-file local account, not production auth.
 - ZIP -> district uses a documented demo dataset until roster ETL lands.
+- **Forward signal gap:** leginfo's bill status/history are *backward-only*
+  (past actions + status narrative, no future dates), so there is no per-bill
+  "next step" label. The dashboard's forward answer comes solely from the
+  chamber events calendars (`scheduled_events`).
+- **Recess empty state:** while the Assembly (and at times the Senate) is in
+  final recess, `/events/upcoming` legitimately returns **0 events**. The
+  "What's next" band and `next_legislative_day` then show an honest "recess /
+  unknown" state rather than fabricating a date. This is expected, not a
+  parsing failure.
+- The 2026-08-31 daily-file snapshot is analysis-only; it is *not* re-surfaced
+  as "upcoming" because the upcoming view floors at today.
@@ -13,6 +13,7 @@ from . import data as _data
 from . import ranking
 from . import resolve as _resolve
 from . import bill_status
+from .. import relational
 from .data import (load_tag_cloud, list_events,
                   load_legislators, committee_roster_for)
 from .ranking import hearing_sort, full_queue, personal_band
@@ -117,17 +118,25 @@ def _truthy(v):

 def handle_bills(req):
    q = req.query
+     # "Upcoming" is a FORWARD view: when the caller asks for upcoming without
+     # pinning an as_of, floor it at *today*, not the most-recent data date.
+     # Flooring at today is the fix for "upcoming shows the latest PAST day";
+     # a None as_of would fall back to the DB max date and re-introduce it.
+    upcoming = _truthy(q.get("upcoming"))
+    as_of = q.get("as_of")
+    if upcoming and not as_of:
+        as_of = _today_iso()
    events = list_events(
-        db_path=req.svc.db_path,
-        upcoming=_truthy(q.get("upcoming")),
-        from_date=q.get("from") or q.get("date_from"),
-        to_date=q.get("to") or q.get("date_to"),
-        chamber=q.get("chamber"),
-        tags=q_list("tags", q),
-        q=q.get("q"),
-        author=q.get("author"),
-        as_of=q.get("as_of"),
-    )
+         db_path=req.svc.db_path,
+         upcoming=upcoming,
+         from_date=q.get("from") or q.get("date_from"),
+         to_date=q.get("to") or q.get("date_to"),
+         chamber=q.get("chamber"),
+         tags=q_list("tags", q),
+         q=q.get("q"),
+         author=q.get("author"),
+         as_of=as_of,
+     )
    if _truthy(q.get("personal")):
        uid, err = req.require_auth()
        if err:
@@ -143,7 +152,7 @@ def handle_bills(req):
            events = full_queue(events)
        else:
            events = personal_band(
-                events, int(q.get("band_n") or ranking.PERSONAL_BAND_N))
+                 events, int(q.get("band_n") or ranking.PERSONAL_BAND_N))
    session = _detect_session(events, req.svc.db_path)
    items = [_client_bill(e) for e in events]
    groups = {}
@@ -151,16 +160,76 @@ def handle_bills(req):
        key = e.get("date") or "TBD"
        groups[key] = groups.get(key, 0) + 1
    chamber_avail = _chamber_availability(items, req.svc.db_path)
+     # The bill queue is a backward daily-file snapshot; the chamber "upcoming
+     # events" calendar (scheduled_events, schema v2) is the forward-looking
+     # signal. Surface "next legislative day" + the next-7-days events so a bill
+     # like AB-1099 shows what is scheduled next, not just the past. A None
+     # as_of (the "All"/history view) still answers "next" from today onward.
+    floor = as_of if as_of else _today_iso()
+    next_day = _next_legislative_day(req.svc.db_path, floor, q.get("chamber"))
+    scheduled = _scheduled_events(req.svc.db_path, floor, q.get("chamber"))
    body = {
-        "session": session,
-        "as_of": q.get("as_of") or _now_iso(),
-        "chamber_availability": chamber_avail,
-        "items": items,
-        "groups": groups,
-    }
+         "session": session,
+         "as_of": as_of or floor,
+         "chamber_availability": chamber_avail,
+         "next_legislative_day": next_day,
+         "scheduled_events": scheduled,
+         "items": items,
+         "groups": groups,
+     }
    return 200, body


+def _today_iso() -> str:
+    import datetime
+    return datetime.date.today().isoformat()
+
+
+def _next_legislative_day(db_path, as_of, chamber=None):
+    """Earliest scheduled chamber event on/after as_of (DESIGN section 6.1).
+
+    Degrades gracefully: when the scheduled_events table is absent (pre-v2 DB)
+    or empty for the window (a recess), returns ``{"date": None, "recess":
+    True}`` so the UI can show "next legislative day unknown". Never raises.
+    """
+    try:
+        return relational.next_legislative_day(db_path, as_of=as_of,
+                                                chamber=chamber)
+    except Exception:    # noqa: BLE001 - forward source must not break /bills
+        return {"date": None, "recess": True}
+
+
+def _scheduled_events(db_path, as_of, chamber=None, window_days=7):
+    """Near-term scheduled chamber events (the "next 7 days" band).
+
+    Returns events from as_of through as_of+window_days, de-duplicated by
+    (date, kind, title). Degrades to [] when the table is absent / empty (a
+    recess). Never raises.
+    """
+    try:
+        import datetime
+        start = _today_iso() if not as_of else str(as_of)
+        d0 = datetime.date.fromisoformat(start[:10])
+        end = (d0 + datetime.timedelta(days=window_days)).isoformat()
+        evs = relational.load_scheduled_events(db_path, as_of=start,
+                                                chamber=chamber)
+        seen = set()
+        out = []
+        for e in evs:
+            if e.get("date") and e["date"] > end:
+                continue
+            k = (e.get("date"), e.get("kind"), e.get("title"))
+            if k in seen:
+                continue
+            seen.add(k)
+            out.append(e)
+        out.sort(key=lambda e: (str(e.get("date") or "z"),
+                                str(e.get("kind") or ""),
+                                str(e.get("title") or "")))
+        return out
+    except Exception:    # noqa: BLE001
+        return []
+
 def _client_bill(ev):
    out = {k: ev.get(k) for k in _data._EVENT_FIELDS}
    out["hearing_sort"] = hearing_sort(ev.get("hearing_time"))
@@ -175,16 +175,74 @@
        wireRows(container);
    }

-    function niceDate(iso) {
-        try {
-            var d = new Date(iso + "T00:00:00");
-            return d.toLocaleDateString(undefined, {
-                weekday: "short", month: "short", day: "numeric",
-                year: "numeric" });
-        } catch (e) { return iso; }
-    }
+     function niceDate(iso) {
+         try {
+             var d = new Date(iso + "T00:00:00");
+             return d.toLocaleDateString(undefined, {
+                 weekday: "short", month: "short", day: "numeric",
+                 year: "numeric" });
+          } catch (e) { return iso; }
+      }

-    /* ---------- empty states (DESIGN §6) ---------- */
+      /* ---------- forward "what's next" (scheduled_events, schema v2) -------
+         The daily-file queue is a backward snapshot; the chamber "upcoming
+         events" calendar is the only genuine future-dated signal. A recess
+         (recess: true / empty) is an honest empty state, not an error.        */
+     function eventRowHTML(ev) {
+         var url = ev.source_url || "";
+         var kindLabel = ev.kind && ev.kind !== "OTHER"
+              ? esc(ev.kind)
+              : (ev.chamber ? ev.chamber : "");
+         return '<div class="ev-row">'
+              + '<span class="ev-date">' + esc(niceDate(ev.date || ""))
+                   + "</span>"
+              + '<span class="ev-main">'
+                   + esc(ev.title || "(untitled)")
+                   + (url
+                        ? " <a href='" + esc(url) + "' target='_blank'"
+                            + " rel='noopener'>details →</a>"
+                        : "")
+              + "</span>"
+              + (kindLabel ? '<span class="chip ev-kind">'
+                   + kindLabel + "</span>" : "")
+              + "</div>";
+      }
+
+     function renderNextDay(nd) {
+         var e = el("freshness-next-day");
+         if (!e) return;
+         if (!nd || (nd.recess && !nd.date)) {
+             e.textContent = "Next legislative day: unknown (recess)";
+             e.style.color = "var(--warn)";
+             return;
+         }
+         e.textContent = "Next legislative day: "
+              + (nd.date ? niceDate(nd.date) : "unknown");
+         e.style.color = "var(--muted)";
+      }
+
+     function renderNextBand(events) {
+         var list = el("next-list");
+         var empty = el("next-empty");
+         var count = el("next-count");
+         if (!list || !empty || !count) return;
+         if (!events.length) {
+             list.innerHTML = "";
+             count.textContent = "";
+             empty.innerHTML =
+                  "<h3>Nothing scheduled in the next 7 days</h3>"
+                   + "<p>The chambers are in recess. This band fills in as the "
+                   + "next hearings are posted.</p>";
+             empty.hidden = false;
+             return;
+         }
+         empty.hidden = true;
+         count.textContent = events.length
+              + (events.length === 1 ? " event" : " events");
+         list.innerHTML = events.map(eventRowHTML).join("");
+      }
+
+      /* ---------- empty states (DESIGN §6) ---------- */
    function showEmpty(kind, container) {
        var el = el(container);
        var h = "";
@@ -250,6 +308,9 @@
        el("freshness-chamber").style.color =
             c.SENATE === "available" ? "var(--muted)" : "var(--warn)";

+         renderNextDay(env.next_legislative_day);
+         renderNextBand(env.scheduled_events || []);
+
        var items = env.items || [];
        state.allItems = items;
        var band = items.filter(function (i) {
@@ -33,10 +33,11 @@
 <main id="app">
    <!-- §9 status / context band -->
    <section class="freshness" id="freshness" aria-live="polite">
-        <span id="freshness-updated">Loading…</span>
-        <span id="freshness-session" class="badge"></span>
-        <span id="freshness-chamber" class="note"></span>
-        <button type="button" id="btn-refresh" class="ghost">Refresh</button>
+          <span id="freshness-updated">Loading…</span>
+          <span id="freshness-session" class="badge"></span>
+          <span id="freshness-chamber" class="note"></span>
+          <span id="freshness-next-day" class="note"></span>
+          <button type="button" id="btn-refresh" class="ghost">Refresh</button>
    </section>

    <!-- Controls bar -->
@@ -64,9 +65,17 @@
            <input type="checkbox" id="f-reason" title="Show why each
                floated bill is in your view">
        </label>
-    </section>
+      </section>

-    <!-- Logged-in personal controls (ZIP + watch/opt-in) -->
+      <!-- What's next: forward-looking chamber events calendar (scheduled_events) -->
+      <section class="band next-band" id="next-band" aria-label="Next 7 days">
+          <h2 class="band-title">What's next · next 7 days
+              <span id="next-count" class="count"></span></h2>
+          <div id="next-list" class="list next-list"></div>
+          <div id="next-empty" class="empty-state" hidden></div>
+      </section>
+
+      <!-- Logged-in personal controls (ZIP + watch/opt-in) -->
    <section class="personal-controls" id="personal-controls" hidden>
        <div class="zip-lookup">
            <label for="zip-input">Where do you live?</label>
@@ -139,6 +139,19 @@ button.chip-toggle { min-width: 120px; }
 .band-title .count { color: var(--ink); font-weight: 700; }
 #personal-band .band-title { color: var(--accent); }

+/* ---- forward "what's next" band (scheduled_events) ---- */
+.next-band { background: #fbfcff; }
+.next-band .band-title { color: var(--accent); }
+.ev-row {
+    display: grid; grid-template-columns: 120px 1fr auto;
+    gap: 10px; align-items: baseline;
+    padding: 8px 8px 8px 0; border-bottom: 1px solid var(--border);
+}
+.ev-date { color: var(--muted); font-size: 13px; white-space: nowrap; }
+.ev-main { font-weight: 600; }
+.chip.ev-kind { background: #eef3fb; color: var(--asm); border-color: #cdd9ee; }
+#freshness-next-day { white-space: nowrap; }
+
 .day-heading {
   font-size: 13px; color: var(--muted); margin: 14px 0 6px;
   padding-bottom: 4px; border-bottom: 1px solid var(--border); }
@@ -97,20 +97,76 @@ class Ingestor:
                 timespec="seconds"))
        self.log.record(event)

-     # -- main API --
+    # -- main API --
    def ingest(self, date_iso, source=None, sources=None):
        names = sources or ([source] if source else self.cfg.source_names())
        result = IngestResult(date=date_iso)
        for name in names:
            if name not in self.cfg.sources:
                raise ValueError(
-                     "Unknown source '%s'. Known: %s"
-                     % (name, self.cfg.source_names()))
+                      "Unknown source '%s'. Known: %s"
+                      % (name, self.cfg.source_names()))
            cr = self._ingest_chamber(name, date_iso)
            result.chambers.append(cr)
        return result

-     # -- per-chamber --
+    # -- chamber events forward source (scheduled_events, schema v2) --
+    def ingest_events(self, date_iso, source=None, sources=None, db_path=None):
+        """Fetch + cache + parse + store the chamber "upcoming events" source.
+
+     Like :meth:`ingest`, this is cache-first (a date already on disk is served
+      with no network) and reuses the same source planner (the events source is
+       ``kind: events``). It parses the calendar HTML into scheduled-event rows
+       and stores them in the ``scheduled_events`` table.
+
+      ``db_path`` defaults to the relational lane DEFAULT_DB so the dashboard
+       can read what this writes. A recess (an empty calendar) is a legitimate
+       no-op, not an error: nothing is cleared and nothing is stored.
+        """
+        from . import relational
+        from . import parsers
+
+        db = db_path or relational.DEFAULT_DB
+        relational.migrate(db)        # ensure the scheduled_events table exists
+        names = sources or ([source] if source else self.cfg.source_names())
+        result = IngestResult(date=date_iso)
+        for name in names:
+            cfg = self.cfg.source(name)
+            if cfg.get("kind") != "events":
+                raise ValueError(
+                     "ingest_events needs an events source; %r is kind %r"
+                     % (name, cfg.get("kind")))
+            cr = self._ingest_chamber(name, date_iso)
+
+            events = []
+            if cr.status == "ok" and cr.path is not None:
+                try:
+                    html = cr.path.read_text(encoding="utf-8",
+                                            errors="replace")
+                except OSError as e:
+                    html = None
+                    cr.note += " | cannot read cached page: %s" % e
+                if html:
+                    base = getattr(cr, "url", "") or ""
+                    events = parsers.parse_events_page(
+                         html, chamber=cfg.get("chamber") or cr.source, source=
+                         name, base_url=base,)
+
+            # Store when a page was read (fresh or served from cache). A fresh
+            # timestamp lets a re-published calendar refresh its day on the next
+            # run while an unchanged read stays a no-op. An empty parse (recess)
+            # clears no days, so existing scheduled rows are preserved.
+            when = datetime.datetime.now(datetime.timezone.utc).strftime(
+                  "%Y-%m-%dT%H:%M:%SZ")
+            summary = relational.store_scheduled_events(events, path=db,
+                                                       fetched_at=when)
+            cr.note += (" | events: %d stored, %d stale cleared, %d total"
+                         % (summary["stored"], summary["cleared_stale"],
+                           summary["total"]))
+            result.chambers.append(cr)
+        return result
+
+      # -- per-chamber --
    def _ingest_chamber(self, name, date_iso):
        cfg = self.cfg.source(name)
        plan = build_source_plan(name, date_iso, cfg)
@@ -315,3 +315,109 @@ def parse(input_path: str, *, house: str = "assembly",
            doc_records = [r for r in doc_records if r.record_type != "hearing"]
        records.extend(doc_records)
    return records
+
+# -------------------------------------------------------------------
+# Chamber "upcoming events" calendar (scheduled_events source, v2)
+#
+# The chamber calendar (assembly.ca.gov/events/upcoming, senate.ca.gov/
+# calendar) is a Drupal "views" table (<table class="cols-3"> with Date /
+# Title / Video columns). Each row is a scheduled hearing / floor session --
+# the only *forward-looking* signal (the daily file is a backward snapshot).
+# Rows are stored in scheduled_events (relational.store_scheduled_events).
+# An empty page (a recess) is a legitimate empty result, not an error.
+# -------------------------------------------------------------------
+
+# Title keywords that classify an event's kind (best-effort, never crash).
+_EV_KIND_PATTERNS = (
+    (re.compile(r"\bfloor\b", re.I), "FLOOR"),
+    (re.compile(r"\bcommittee\b", re.I), "COMMITTEE"),
+)
+
+
+def classify_event_kind(title):
+    """Map an event title to a kind: FLOOR / COMMITTEE / OTHER."""
+    t = title or ""
+    for pat, kind in _EV_KIND_PATTERNS:
+        if pat.search(t):
+            return kind
+    return "OTHER"
+
+
+def parse_events_page(html, chamber="assembly", source="assembly_events",
+                      base_url="", parser="lxml"):
+    """Parse a chamber events calendar page into scheduled-event dicts.
+
+    Returns a list of dicts with keys: date (ISO YYYY-MM-DD), chamber
+    (upper), kind (FLOOR|COMMITTEE|OTHER), title, source_url. An empty /
+    recess page yields []. Rows without a resolvable date are skipped
+    (logged). Header rows (<th>) are ignored.
+    """
+    from bs4 import BeautifulSoup
+
+    engine = parser if parser in ("lxml", "html.parser") else "html.parser"
+    try:
+        soup = BeautifulSoup(html, engine)
+    except Exception as e:   # noqa: BLE001
+        log.error("events: failed to parse document (%s): %s",
+                  type(e).__name__, e)
+        return []
+
+    out = []
+     # Fall back to any <tr> carrying a <time> cell when table.cols-3 is
+     # absent, so a layout change still yields rows rather than a silent
+     # empty page.
+    base = (base_url or "").rstrip("/")
+    host = base.rsplit("/", 2)[0] if base else "https://www.assembly.ca.gov"
+    tables = soup.select("table.cols-3")
+    rows = [tr for tbl in tables for tr in tbl.find_all("tr")]
+    if not rows:
+        rows = [tr for tr in soup.find_all("tr") if tr.find("time")]
+    for i, tr in enumerate(rows):
+         # Skip header rows (column labels render as <th>).
+        if tr.find("th"):
+            continue
+        try:
+            time_el = tr.find("time")
+            if time_el is None:
+                raise ValueError("event row has no date cell")
+             # Prefer the datetime attr: "2026-09-01T00:00:00-07:00".
+            dt = time_el.get("datetime") or ""
+            date = dt[:10] if len(dt) >= 10 and dt[4:5] == "-" else None
+            if not date:
+                date = normalise_date(time_el.get_text(strip=True))
+
+            title_td = tr.select_one("td.views-field-title")
+            if title_td is not None:
+                title = title_td.get_text(" ", strip=True)
+            else:
+                t = tr.find("time")
+                if t is not None:
+                    t.decompose()
+                title = tr.get_text(" ", strip=True)
+            if not date or not title:
+                raise ValueError("event row has no date or title")
+
+              # source_url: the row's own link (event / video detail) when
+              # present, else the calendar page (base_url) as provenance.
+            a = tr.find("a", href=True)
+            source_url = None
+            if a is not None:
+                href = (a.get("href") or "").strip()
+                if href.startswith("http"):
+                    source_url = href
+                elif href.startswith("/"):
+                    source_url = host + href
+            if not source_url:
+                source_url = base or None
+
+            out.append({
+                "date": date,
+                "chamber": chamber.strip().upper(),
+                "kind": classify_event_kind(title),
+                "title": title,
+                "source_url": source_url,
+                "source": source,
+             })
+        except Exception as e:   # noqa: BLE001
+            log.warning("skip malformed event row #%d: %s", i, e)
+    return out
@@ -51,7 +51,11 @@ DEFAULT_DB = os.path.join(_REPO_ROOT, "db", "daily_file.sqlite")
 FLAT_TABLE = "daily_file_records"

 #: Schema version stamped into ``schema_migrations`` by :func:`migrate`.
-SCHEMA_VERSION = 1
+# v1: bills/committees/members/committee_members/bill_committees + flat view.
+# v2: adds scheduled_events (the chamber "upcoming events" forward source — the
+#    only genuine future-dated signal; see design/API_CONTRACT.md §5 + the
+#    "what's next" work). Kept OUT of the frozen flat daily_file_records view.
+SCHEMA_VERSION = 2

 #: The flat columns QA + downstream rely on. Imported so the flat-preserving
 #: view is byte-for-byte aligned with the contract the rest of the pipeline
@@ -211,9 +215,27 @@ def _create_schema(conn: sqlite3.Connection) -> None:
            applied_at  TEXT,
            note        TEXT,
            PRIMARY KEY (version)
-        );
-        """
-    )
+         );
+
+        -- v2: the chamber "upcoming events" forward source. This is the ONLY
+        -- genuine *future-dated* signal (the daily file is a backward snapshot;
+        -- leginfo bill status/history is backward-only). It holds the next
+        -- scheduled floor/committee events parsed from the chamber events
+        -- calendar (assembly.ca.gov/events, senate.ca.gov/calendar). It is
+        -- deliberately OUT of the frozen flat daily_file_records view.
+        CREATE TABLE IF NOT EXISTS scheduled_events (
+            id          INTEGER PRIMARY KEY,
+            date        TEXT NOT NULL,            -- ISO YYYY-MM-DD (event day)
+            chamber     TEXT NOT NULL,            -- ASSEMBLY | SENATE
+            kind        TEXT,                     -- FLOOR | COMMITTEE | OTHER
+            title       TEXT NOT NULL,            -- human event title
+            source_url  TEXT,                     -- where the event came from
+            fetched_at  TEXT,                     -- when this page was read
+            source      TEXT,                     -- assembly_events | senate_events
+            UNIQUE (date, chamber, kind, title, source)
+         );
+      """
+      )
    # Flat-preserving view: a drop-in mirror of the table QA/UI read. Column
    # order and row set are identical to daily_file_records, ordered
    # deterministically so the "flat contract unchanged" test is stable.
@@ -229,13 +251,13 @@ def _create_schema(conn: sqlite3.Connection) -> None:

 def _stamp_migration(conn: sqlite3.Connection, version: int = SCHEMA_VERSION) -> None:
    conn.execute(
-        "INSERT OR IGNORE INTO schema_migrations (version, applied_at, note) "
-        "VALUES (?, ?, ?)",
-        (version,
-         __import__("datetime").date.today().isoformat(),
-         "relational schema: bills/committees/members/committee_members/"
-            "bill_committees + flat view"),
-    )
+         "INSERT OR IGNORE INTO schema_migrations (version, applied_at, note) "
+          "VALUES (?, ?, ?)",
+         (version,
+          __import__("datetime").date.today().isoformat(),
+          "relational schema: bills/committees/members/committee_members/"
+             "bill_committees + flat view + scheduled_events (v2)"),
+       )
    conn.commit()


@@ -514,12 +536,132 @@ def sample_selects(path: str = DEFAULT_DB, limit: int = 5) -> Dict[str, Any]:
                "JOIN committees c ON c.name_key = cm.committee "
                "JOIN members m ON m.name_norm = cm.member "
                "ORDER BY c.name_raw, m.full_name LIMIT ?", (limit,)),
-            "committees_without_roster": q(
-                "SELECT name_raw, chamber, session FROM committees c "
-                "WHERE NOT EXISTS (SELECT 1 FROM committee_members cm "
-                "  WHERE cm.committee = c.name_key "
-                "  AND cm.chamber = c.chamber "
-                "  AND cm.session = c.session)"),
+      "committees_without_roster": q(
+                  "SELECT name_raw, chamber, session FROM committees c "
+                  "WHERE NOT EXISTS (SELECT 1 FROM committee_members cm "
+                  "  WHERE cm.committee = c.name_key "
+                  "  AND cm.chamber = c.chamber "
+                  "  AND cm.session = c.session)"),
        }
    finally:
        conn.close()
+
+
+# ---------------------------------------------------------------------------
+# scheduled_events: the chamber "upcoming events" forward source (v2)
+# ---------------------------------------------------------------------------
+
+def _scheduled_events_table(conn: sqlite3.Connection, name: str = "scheduled_events") -> bool:
+    cur = conn.execute(
+         "SELECT 1 FROM sqlite_master WHERE type='table' AND name=?",
+           (name,))
+    return cur.fetchone() is not None
+
+
+def store_scheduled_events(events: Iterable[Dict[str, Any]],
+                            path: str = DEFAULT_DB,
+                            fetched_at: str = "") -> Dict[str, Any]:
+    """Insert parsed chamber events into ``scheduled_events`` idempotently.
+
+    Each ``event`` is a dict with keys ``date`` (ISO), ``chamber`` (upper),
+    ``kind`` (``FLOOR``/``COMMITTEE``/``OTHER``), ``title``, ``source_url``,
+    ``source``. The natural key ``(date, chamber, kind, title, source)`` makes
+    re-runs a no-op (``INSERT OR IGNORE``), so the existing-cron behaviour
+    ("re-parse doesn't re-hit") holds: a second parse of the same calendar day
+    adds no rows. Ensures the table exists (via :func:`_create_schema`). Returns
+    a small summary ``{stored, cleared_stale, total, fetched_at}``.
+    """
+    conn = connect(path)
+    try:
+        _create_schema(conn)      # CREATE IF NOT EXISTS; cheap + idempotent
+        # Drop a stale day's rows first so a re-published calendar (a moved or
+        # renamed hearing) reflects the latest read rather than a frozen
+        # snapshot. Only that day is cleared; other days are untouched.
+        days = {str(e.get("date")) for e in events if e.get("date")}
+        deleted = 0
+        for d in days:
+            cur = conn.execute(
+                "DELETE FROM scheduled_events WHERE date=? AND fetched_at<>?",
+                (d, fetched_at))
+            deleted += cur.rowcount
+        rows = []
+        for e in events:
+            if not e.get("date") or not e.get("title"):
+                continue
+            rows.append((
+                str(e["date"]).strip(),
+                (e.get("chamber") or "unknown").strip().upper(),
+                (e.get("kind") or "OTHER").strip().upper() or "OTHER",
+                str(e["title"]).strip(),
+                e.get("source_url") or None,
+                fetched_at or None,
+                (e.get("source") or "assembly_events").strip()
+                or "assembly_events",
+            ))
+        if rows:
+            cur = conn.executemany(
+                "INSERT OR IGNORE INTO scheduled_events "
+                "(date, chamber, kind, title, source_url, fetched_at, "
+                "source) VALUES (?,?,?,?,?,?,?)",
+                rows)
+            stored = cur.rowcount
+        else:
+            stored = 0
+        conn.commit()
+        total = conn.execute(
+            "SELECT COUNT(*) FROM scheduled_events").fetchone()[0]
+        return {"stored": stored, "cleared_stale": deleted,
+                "total": total, "fetched_at": fetched_at or None}
+    finally:
+        conn.close()
+
+
+def load_scheduled_events(path: str = DEFAULT_DB,
+                          as_of: Optional[str] = None,
+                          chamber: Optional[str] = None,
+                          limit: int = 200) -> List[Dict[str, Any]]:
+    """Return scheduled (chamber) events, optionally future-dated from ``as_of``.
+
+    A ``None``/absent ``as_of`` keeps every event; a date string keeps
+    ``date >= as_of`` (a T-1 event is excluded, matching API_CONTRACT §5).
+    Degrades to ``[]`` when the table does not exist (pre-v2 DBs / fresh schema).
+    """
+    conn = connect(path, foreign_keys=False)
+    try:
+        if not _scheduled_events_table(conn):
+            return []
+        sql = ("SELECT date, chamber, kind, title, source_url, fetched_at, "
+               "source FROM scheduled_events WHERE 1=1")
+        clauses: list = []
+        args: list = []
+        if as_of:
+            clauses.append("date >= ?")
+            args.append(str(as_of))
+        if chamber:
+            clauses.append("UPPER(chamber) = ?")
+            args.append(chamber.strip().upper())
+        if clauses:
+            sql += " AND " + " AND ".join(clauses)
+        sql += " ORDER BY date, chamber, kind, title LIMIT ?"
+        args.append(int(limit))
+        rows = conn.execute(sql, args).fetchall()
+    finally:
+        conn.close()
+    return [dict(r) for r in rows]
+
+
+def next_legislative_day(path: str = DEFAULT_DB,
+                          as_of: Optional[str] = None,
+                          chamber: Optional[str] = None) -> Dict[str, Any]:
+    """The earliest scheduled event on/after ``as_of`` (DESIGN §6.1 "next
+    legislative day"), or a ``{date: None, recess: True}`` marker when nothing is
+    scheduled (a genuine recess).
+
+    This is the dashboard's "what's next" answer when the daily-file bill queue
+    has no future rows: it surfaces the chamber events calendar. ``recess`` is
+    True when there is no event after ``as_of``.
+    """
+    events = load_scheduled_events(path, as_of=as_of, chamber=chamber, limit=1)
+    if not events:
+        return {"date": None, "recess": True}
+    return {**events[0], "recess": False}
@@ -16,18 +16,33 @@ SENATE_PDF_RE = re.compile(

@dataclass
 class SourcePlan:
-    source: str                  # "assembly_daily_file" | "senate_daily_file"
-    date: str                    # requested YYYY-MM-DD
-    kind: str                    # "html" | "pdf"
-    primary_url: str             # URL that is actually fetched
-    ext: str                     # cache extension ("html" | "pdf")
+    source: str                   # "assembly_daily_file" | "senate_daily_file"
+    date: str                     # requested YYYY-MM-DD
+    kind: str                     # "html" | "pdf" | "events"
+    primary_url: str              # URL that is actually fetched
+    ext: str                      # cache extension ("html" | "pdf")
    pdf_url: Optional[str] = None
-    as_of: Optional[str] = None      # date the live page itself represents
+    as_of: Optional[str] = None       # date the live page itself represents
    extra_urls: list = field(default_factory=list)


 def build_source_plan(name: str, date_iso: str, src_cfg: dict) -> SourcePlan:
    kind = src_cfg.get("kind")
+    if kind == "events":
+        # The chamber "upcoming events" calendar is a live HTML page (a
+        # forward-looking source, distinct from the daily file). Its URL is
+        # constant; the date only drives the cache key (the page may list
+        # future-dated events, none of which is the --date value itself).
+        url = src_cfg.get("url") or src_cfg.get("index_url")
+        return SourcePlan(
+             source=name,
+             date=date_iso,
+             kind="events",
+             primary_url=url,
+             ext="html",
+             as_of=date_iso,
+          )
+
    if kind == "html":
        # Assembly: constant URL, always the current live file.
        return SourcePlan(