# California Legislature Daily File — Parse & Ingestion Spec **Research task:** `t_a171f4d1` **Date:** 2026-09-04 (PDT, current 2025-2026 regular session) **Verified against live pages** and **archived to `./samples/`**. This document is the single source of truth for the ingestion task (`t_f595542f`), the parser task (`t_fb8081e7`), and the RDB schema task (`t_8e5a3821`). A developer should be able to implement a parser from this document without revisiting the live pages, using the samples in `./samples/`. --- ## TL;DR — the most important finding - **Assembly Daily File = fully server-rendered HTML.** Three data tables with semantic CSS classes and `data-sort` attributes hold pre-cleaned canonical values (dates without the `-###` suffix, trimmed committee/author names). Parse offline with BeautifulSoup + the CSS selectors below; no JS required. - **Senate Daily File = PDF only.** `senate.ca.gov/dailyfile` is a Vue pdf-viewer wrapper with **no parseable data table**. The publication exists only as a PDF, and `senate.ca.gov` wraps the download path in an **antibot challenge** (HTTP 403 with a 67-byte HTML body on programmatic fetch, verified 2026-09-04). The Senate daily file must be ingested via a PDF path (pdfplumber / PyMuPDF) with antibot mitigation, **or** enriched from leginfo's parallel `billResultsClient` / `billHistoryClient` views (see alt sources). - **Neither `?date=` nor `?dailyfile=…` is addressable.** The Assembly page always renders the **current live** daily file — there is no historical-date query parameter. "Date" is a data value (read from the hearing table), not a URL parameter. - **Cross-chamber bills:** an Assembly bill (AB…) can appear on a Senate daily file and vice-versa. The "chamber" field means **origin**, not publishing house. --- ## 1. Canonical URLs ### 1.1 Assembly Daily File | Item | URL | |---|---| | Canonical (HTML, always current) | `` | | Alias (redirects to canonical) | `https://assembly.ca.gov/schedules-publications/daily-file` | | Alias | `https://assembly.ca.gov/dailyfile/pdfpublications` | | Legacy help page | `https://leginfo.ca.gov/help/dayfileh.html` | | Live sample | `samples/assembly_dailyfile_latest.html` (2026-09-04, 401 192 bytes) | Properties: - `requires_js=false` — data lives in static `` markup. - `antibot=false` — plain `requests` GET with a sensible `User-Agent` works. - `content-type: text/html; charset=UTF-8`. - No date parameter. `?date=2025-08-31` and `?dailyfile=2026-08-31` were probed and **do not select a historical file** — the page always returns the current live daily file. ### 1.2 Senate Daily File | Item | URL | |---|---| | HTML index (human page, JS-rendered pdf viewer) | `https://www.senate.ca.gov/dailyfile` | | Canonical (per ``) | `https://www.senate.ca.gov/publications/senate-daily-file` | | PDF download base | `https://senate.ca.gov/dailyfile/publications/download/` | | Live index sample | `samples/senate_dailyfile_index.html` (34 137 bytes) | | Verified PDF fragment | `SEN_COM_2025_R_20260902.PDF` (200 when served; 403 under antibot for this scanner) | PDF filename pattern: ``` SEN_{TYPE}_{year}_R_{YYYYMMDD}.PDF ``` - `{TYPE}` = `R` (regular) or `COM` (committee daily file). Historical variants: `SEN_ESS_*` for extraordinary/special sessions. - `{year}` = the legislative session start year (e.g., `2025` for 2025-2026). - `{YYYYMMDD}` = the publication date. **Critical:** Not every calendar day has a published file (weekends, non-legislative days). 404 on the PDF is **expected**, NOT an error — the ingestion layer must distinguish "404 no-file" from "5xx transient". --- ## 2. Parameterisation | Source | Parameter | Scheme | Notes | |---|---|---|---| | Assembly Daily File | `date` | — | No URL param. As-of date is read from the hearing table. | | Senate Daily File | `date` + `session_year` + `type` | `SEN_{TYPE}_{year}_R_{YYYYMMDD}.PDF` | Filename-encoded. | | Bill detail (enrichment) | — | `bill_id` | Leginfo canonical bill page; see §3. | | Committee rosters | — | Static pages | No per-date param; refreshed periodically. | --- ## 3. Assembly Daily File HTML structure Three `
` elements, in document order. Each represents a distinct record type. Every data cell is a `td` with a **specific class** and, where applicable, a **`data-sort`** attribute carrying the clean canonical value. Prefer `data-sort` over `textContent` in extraction — it strips the display suffix (`'08/31/26-001'` → `08/31/26`), leading zeros, and the like. ### 3.1 Committee Hearings - **Table selector:** `table.committee-hearing-table` - **Row selector:** `tr[class*='committee-hearing-details']` Verified on 2026-09-04: **4 hearing rows** in the current file. | Field | Selector | Best value source | |---|---|---| | `date` | `td.committee_hearing-date` | `attr:data-sort` (e.g. `"08/31/26"`; raw `"08/31/26-001"`) | | `time` | `td.committee_hearing-time` | `text` (e.g. `"9:30am"`, `"Upon Call of the Chair"`) | | `committee` | `td.committee_hearing-name` | `attr:data-sort` | | `location` | `td.committee_hearing-location` | `attr:data-sort` | | `agenda_id` | `td.committee_hearing-menu a[href^='/api/dailyfile/agenda/']` | `attr:href` — Drupal AJAX id | | `ics_url` | `td.committee_hearing-menu a[href^='/api/ics/generate']` | `attr:href` | Example row (verbatim): ```html ``` XPath equivalents: ``` //tr[contains(@class,'committee-hearing-details')]//td[contains(@class,'committee_hearing-date')]/@data-sort //tr[contains(@class,'committee-hearing-details')]//td[contains(@class,'committee_hearing-name')]/@data-sort //tr[contains(@class,'committee-hearing-details')]//td[contains(@class,'committee_hearing-menu')]//a[starts-with(@href,'/api/dailyfile/agenda/')]/@href ``` ### 3.2 Bills on the Floor (`floor-bills-table`) - **Table selector:** `table.floor-bills-table` - **Row selector:** `tr.class='floor-bill-details'` Verified on 2026-09-04: **156 rows**. | Field | Selector | Best value source | |---|---|---| | `bill_number` | `td.floor_bill-measure a` | `attr:href` (parse bill_id; §4) | | `bill_display` | `td.floor_bill-measure a` | `text` (e.g. `"AB 28"`) | | `supplement` | `td.floor_bill-measure` | `text` after `
` (e.g. `"Supplemental 2"`) | | `author` | `td.floor_bill-author` | `attr:data-sort` | | `title` | `td.floor_bill-subject` | `text` (short title / description) | | `location` | `td.floor_bill-location` | `attr:data-sort` | | `item_number` | `td.floor_bill-item` | `text` (zero-padded; strip to int) | | `chamber` | derived | from `bill_id` prefix | | `date` | **derived** | Not on the row; context is the hearing/date section | Example row (verbatim, first row): ```html ``` **Note on `td.floor_bill-view-details`:** contains a `` modal DOM widget (close button, SVG, "Details" panel), NOT a data field. **Ignore it.** XPath equivalents: ``` //tr[contains(@class,'floor-bill-details')]//td[contains(@class,'floor_bill-measure')]/a/@href //tr[contains(@class,'floor-bill-details')]//td[contains(@class,'floor_bill-author')]/@data-sort //tr[contains(@class,'floor-bill-details')]//td[contains(@class,'floor_bill-subject')]/text() //tr[contains(@class,'floor-bill-details')]//td[contains(@class,'floor_bill-item')]/text() ``` ### 3.3 Bills in Committee (`committee-bills-table`) - **Table selector:** `table.committee-bills-table` - **Row selector:** `tr[class*='committee-bills-details']` Verified on 2026-09-04: **1 row** (the current live file has few committee bills — the count varies by day). | Field | Selector | Best value source | |---|---|---| | `bill_number` | `td.committee_bill-number a` | `attr:href` (parse bill_id; §4) | | `author` | `td.committee_bill-author` | `attr:data-sort` | | `title` | `td.committee_bill-subject` | `text` | | `committee` | `td.committee_bill-committee` | `attr:data-sort` | Example row (verbatim, only row): ```html ``` --- ## 4. Bill ID encoding (universal across all daily-file links) Every bill link on the Assembly (and Senate) daily file points to a single canonical pattern: ``` https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id={bill_id} ``` `bill_id` decomposes with the regex: ```regex ^(?P\d{8})\d(?PAB|SB|ACR|SCR|HR|SJR|ABR|SCR\d+)(?P\d+)$ ``` | Component | Example | Meaning | |---|---|---| | `session` | `20252026` | `YYYYYYYY` = `YYYY-YYYY` regular session (e.g. `2025-2026`) | | middle `0` | `0` | constant pad between session and chamber | | `chamber` | `AB` | `AB`/`ACR`/`HR`/`ABR` = Assembly-origin; `SB`/`SCR`/`SJR` = Senate-origin | | `number` | `28` | 1–4 digits; **zero-pad to 4** when writing back (`AB 28` ↔ `AB 0028`) | Worked example: ``` bill_id = 202520260AB28 session = 2025-2026 chamber = AB number = 28 display = "AB 28" # or "AB 0028" with 4-digit zero-pad ``` This bill_id is the join key to: - `billNavClient.xhtml?bill_id=…` → status, current version, author(s), short title - `billHistoryClient.xhtml?bill_id=…` → full action-level history, committee referrals - `billResultsClient.xhtml?…` → floor / committee agenda views (JS-rendered; use JSON variants) --- ## 5. Senate Daily File — PDF parsing The Senate Daily File is a **PDF**. There is no HTML table equivalent with the same structure. Two ingestion options: ### 5a. PDF path (primary) 1. Resolve the current filename from the index HTML (`samples/senate_dailyfile_index.html`): ``` # Find the most recent PDF link on the index page grep -oE '/dailyfile/publications/download/SEN[A-Z_]*_[0-9]{4}_R_[0-9]{8}\.PDF' … ``` On 2026-09-04 the only referenced PDF is `SEN_COM_2025_R_20260902.PDF`. 2. Fetch the PDF: ``` https://www.senate.ca.gov/dailyfile/publications/download/SEN_COM_2025_R_20260902.PDF ``` **Antibot mitigation required.** Without a browser challenge solver, this returns `HTTP 403` with a 67-byte HTML body. See `./skills/blocked-page-recovery` and `t_f595542f` for the mitigation plan. 3. Extract text (pdfplumber / PyMuPDF), then apply layout heuristics: - Section headers: `BILLS ON THE FLOOR`, `COMMITTEE HEARINGS`, `BILLS IN COMMITTEE`, `HELD AT DESK`, `INACTIVE FILE`, `OFFICERS OF THE SENATE`, `SENATE ORDER OF BUSINESS`. - Bill-number regex at start of line: ```regex ^(AB|SB|ACR|SCR|HR|SJR|ABR)\s+(\d+) ``` - Following lines carry author, subject, and (for committeee sections) committee name (`COMMITTEE OF `). ### 5b. Leginfo alternative (recommended as backup / enrichment) `billResultsClient/xhtml` and `billHistoryClient/xhtml` (see §6) give a structured representation of Senate floor items and committee actions. They are JS-rendered; use the underlying JSON endpoints (documented by leginfo; exposed by `billResultsClient.xhtml?…`) for programmatic fetch. Sample: `samples/leginfo_assembly_floor_2026-04-16.html` (title: "California Legislative Information"; results injected by JS). --- ## 6. Correlation sources (for §2.5 of the parent — bills ↔ committees ↔ members) | Concern | URL | Sample | |---|---|---| | Assembly committee rosters | `https://www.assembly.ca.gov/committees` | — | | Senate committee rosters | `https://www.senate.ca.gov/committees` | `samples/senate_committees_index.html` | | Assembly members | `https://www.assembly.ca.gov/assemblymembers` | — | | Find-my-rep by address | `https://www.assembly.ca.gov/assemblymembers/find-my-rep` and `https://findyourrep.legislature.ca.gov/` | — | | Bill detail (author / title / status / committee history) | `https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id={bill_id}` | `samples/leginfo_assembly_floor_2026-04-16.html` | | Bill committee history (action-level) | `https://leginfo.legislature.ca.gov/faces/billHistoryClient.xhtml?bill_id={bill_id}` | — | ### 6.1 Committee rosters — key observation Senate committee slugs are **URL-encoded names** (no slugs or IDs; the slug IS the committee name). Example from 2026-09-04: ``` /committees/selectcommitteeonhydrogenenergy /committees/selectcommitteeoneconomicdevelopmentandtechnologicalinnovation /committees/senateselectcommitteeonolderlgbtqcalifornians /committees/offices-caucuses ``` Normalize by title-casing and stripping non-alphanumerics; cross-reference against the `data-sort` value in `td.committee_hearing-name` and `td.committee_bill-committee`. ### 6.2 Bill detail — structure observed From `samples/leginfo_assembly_floor_2026-04-16.html` (a bill-detail page for `202520260AB28`): - `

Bill Text

` — page title - `.bill_title_yr` — short title + session year (e.g. "Solid waste landfills…") - Author and committee fields live in the bill-info block (exact selectors to re-verify at parse-time; this page is enrichment, NOT the daily file source). --- ## 7. Normalised output schema (for `t_fb8081e7`) One JSON-Lines record per daily-file entry: ```json { "id": "sha1(house|bill_number|date|committee)", "date": "2026-09-02", "house": "ASSEMBLY", // or SENATE "source": "ASSEMBLY_DAILY_FILE", // | SENATE_DAILY_FILE_PDF | LEGINFO_BILL_RESULTS "bill_number": "AB 28", "bill_id": "202520260AB28", "session": "2025-2026", "chamber_origin": "ASSEMBLY", // AB-origin vs SB-origin "author": "Schiavo et al.", "short_title": "An act relating to solid waste, and making an appropriation therefor.", "description": "An act relating to solid waste, and making an appropriation therefor.", "committee": null, // null for floor items; e.g. "Emergency Management" for committee items "hearing_time": null, // "9:30am" / "Upon Call of the Chair" for committee-hearings rows "location": "State Capitol, Room 444", "item_number": 32, // int, un-padded "supplement": 2, // optional, from "Supplemental 2" "ics_url": "https://www.assembly.ca.gov/api/ics/generate?name=Budget&start=…", // where applicable "agenda_id": "20361" // where applicable } ``` Field-to-row mapping: - Floor bill rows: `bill_number` ← `floor_bill-measure a href`, `author` ← `floor_bill-author data-sort`, `title` ← `floor_bill-subject text`, `location` ← `floor_bill-location data-sort`, `item_number` ← `floor_bill-item text` (un-pad). `committee` = null, `hearing_time` = null (unless the bill also appears as a committee item on the same file). - Committee bill rows: same fields plus `committee` ← `committee_bill-committee data-sort`, `item_number` = null. - Committee hearing rows: no bill — record the hearing metadata itself (date, time, committee, location, `agenda_id`, `ics_url`). - Senate (from PDF or leginfo alt): same schema, `source=SENATE_DAILY_FILE_PDF` or `LEGINFO_BILL_RESULTS`. --- ## 8. Pitfalls & risk register 1. **Antibot on Senate PDFs.** `senate.ca.gov/dailyfile/publications/download/…` behind a `form.antibot` challenge. Programmatic `requests`/browser JS fetch yields `HTTP 403` with a 67-byte HTML body (verified 2026-09-04). Mitigation: headless browser with a cookie/session pre-warm, or route through the `blocked-page-recovery` skill. 2. **No historical date param.** The Assembly page is always current-live. "Date" is read from data (hearing table), not written into the URL. 3. **404 ≠ error.** Non-legislative days legitimately return 404 on the Senate PDF index. The ingestion layer must classify 404 as "no file today" and 5xx / retry-able as a transient error. 4. **Session boundary.** When the session changes in January, the URL pattern (`bill_id` = `YYYYYYYY` + …) stays the same but bills renumber. No code change required — the `session` field in the output reflects the current value read from `bill_id`. 5. **Data-sort > textContent.** Always prefer the `data-sort` attribute; it is the clean canonical value. `textContent` carries display suffixes and leading zeros. 6. **Modal widget in floor-bill rows.** `td.floor_bill-view-details` holds a ``; do NOT extract it as a field. 7. **Cross-chamber bills.** `AB`-prefilled rows can appear on a Senate daily file and vice-versa. `chamber_origin` means *origin*, not publishing house. 8. **Supplement tracking.** A bill can have multiple supplements (`Supplemental 2`); the `supplement` field is the edition. 9. **JS-rendered leginfo pages.** `billResultsClient` and `billHistoryClient` need a headless browser or JSON endpoint; the static `
08/31/26 Upon Call of the Chair Budget 1021 O Street, Room 1100
AB 28
Supplemental 2		 Schiavo et al. An act relating to solid waste, and making an appropriation therefor. Unfinished Business – Concurrence In Senate Amendments 32 …
SB 795 Archuleta Air pollution: gasoline vapor control systems: ethanol blends. Emergency Management
` they render is NOT in the initial HTML. --- ## 9. Sample archive ``` samples/ assembly_dailyfile_latest.html # 401 192 bytes, 2026-09-04 # 4 hearing rows, 156 floor-bill rows, 1 committee-bills row senate_dailyfile_index.html # 34 137 bytes, pdf-viewer index # links to SEN_COM_2025_R_20260902.PDF senate_committees_index.html # 58 642 bytes, Senate committee slugs leginfo_assembly_floor_2026-04-16.html # 163 825 bytes, leginfo results page # billResultsClient, JS-rendered ``` These are the offline test fixtures for `t_fb8081e7`. Every sample was fetched via `curl` (assembly index, senate index, committee index, leginfo floor) or `browser_exec` fetch (leginfo floor). The Senate PDF sample is NOT archived because it is behind antibot — that is itself a documented finding. See §10 for the acceptance test harness the parser task should run against these files. --- ## 10. Acceptance test harness (for `t_fb8081e7`) ```yaml - id: asm_hearings input: samples/assembly_dailyfile_latest.html selector: tr[class*='committee-hearing-details'] expect_min_rows: 1 verified: 4 - id: asm_floor_bills input: samples/assembly_dailyfile_latest.html selector: tr.class='floor-bill-details' expect_min_rows: 100 expect_fields: [bill_number, author, short_title, item_number] verified: 156 - id: asm_committee_bills input: samples/assembly_dailyfile_latest.html selector: tr[class*='committee-bills-details'] expect_min_rows: 1 verified: 1 - id: bill_id_parse regex: "^(?P\d{8})\d(?PAB|SB|ACR|SCR|HR|SJR|ABR)(?P\d+)$" example_url: "https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=202520260AB28" expect: {session: "2025-2026", chamber: "AB", number: 28, display: "AB 28"} verified: pass - id: senate_pdf_index input: samples/senate_dailyfile_index.html expect: "at least one /dailyfile/publications/download/*.PDF reference" verified: 1 (SEN_COM_2025_R_20260902.PDF) ``` --- **Downstream handoff:** - `t_f595542f` (ingestion): read `daily_file_sources.yaml` for URL patterns + parameters + antibot flags. Add a Senate PDF download step with a `blocked-page-recovery` fallback for 403s. - `t_fb8081e7` (parser): the §3 and §5 tables are the exact selector / XPath / regex contracts. Use `data-sort` over `textContent`. Skip `td.floor_bill-view-details`. - `t_8e5a3821` (RDB schema): the §7 per-record schema is the target. Foreign keys to `committee_roster`, `bill_detail`, `bill_history` tables (see §6).