0 Fork 0

Files

T

2026-09-06 10:18:26 -07:00

21 KiB

Raw Blame History

California Legislature Daily File — Parse & Ingestion Spec

Research task: t_a171f4d1
Date: 2026-09-04 (PDT, current 2025-2026 regular session)
Verified against live pages and archived to ./samples/.

This document is the single source of truth for the ingestion task (t_f595542f), the parser task (t_fb8081e7), and the RDB schema task (t_8e5a3821). A developer should be able to implement a parser from this document without revisiting the live pages, using the samples in ./samples/.

TL;DR — the most important finding

Assembly Daily File = fully server-rendered HTML. Three data tables with semantic CSS classes and data-sort attributes hold pre-cleaned canonical values (dates without the -### suffix, trimmed committee/author names). Parse offline with BeautifulSoup + the CSS selectors below; no JS required.
Senate Daily File = PDF only. senate.ca.gov/dailyfile is a Vue pdf-viewer wrapper with no parseable data table. The publication exists only as a PDF, and senate.ca.gov wraps the download path in an antibot challenge (HTTP 403 with a 67-byte HTML body on programmatic fetch, verified 2026-09-04). The Senate daily file must be ingested via a PDF path (pdfplumber / PyMuPDF) with antibot mitigation, or enriched from leginfo's parallel billResultsClient / billHistoryClient views (see alt sources).
Neither ?date= nor ?dailyfile=… is addressable. The Assembly page always renders the current live daily file — there is no historical-date query parameter. "Date" is a data value (read from the hearing table), not a URL parameter.
Cross-chamber bills: an Assembly bill (AB…) can appear on a Senate daily file and vice-versa. The "chamber" field means origin, not publishing house.

1. Canonical URLs

1.1 Assembly Daily File

Item	URL
Canonical (HTML, always current)	`<https://www.assembly.ca.gov/schedules-publications/daily-file>`
Alias (redirects to canonical)	`https://assembly.ca.gov/schedules-publications/daily-file`
Alias	`https://assembly.ca.gov/dailyfile/pdfpublications`
Legacy help page	`https://leginfo.ca.gov/help/dayfileh.html`
Live sample	`samples/assembly_dailyfile_latest.html` (2026-09-04, 401 192 bytes)

Properties:

requires_js=false — data lives in static <table> markup.
antibot=false — plain requests GET with a sensible User-Agent works.
content-type: text/html; charset=UTF-8.
No date parameter. ?date=2025-08-31 and ?dailyfile=2026-08-31 were probed and do not select a historical file — the page always returns the current live daily file.

1.2 Senate Daily File

Item	URL
HTML index (human page, JS-rendered pdf viewer)	`https://www.senate.ca.gov/dailyfile`
Canonical (per `<link rel=canonical>`)	`https://www.senate.ca.gov/publications/senate-daily-file`
PDF download base	`https://senate.ca.gov/dailyfile/publications/download/`
Live index sample	`samples/senate_dailyfile_index.html` (34 137 bytes)
Verified PDF fragment	`SEN_COM_2025_R_20260902.PDF` (200 when served; 403 under antibot for this scanner)

PDF filename pattern:

SEN_{TYPE}_{year}_R_{YYYYMMDD}.PDF

{TYPE} = R (regular) or COM (committee daily file). Historical variants: SEN_ESS_* for extraordinary/special sessions.
{year} = the legislative session start year (e.g., 2025 for 2025-2026).
{YYYYMMDD} = the publication date.

Critical: Not every calendar day has a published file (weekends, non-legislative days). 404 on the PDF is expected, NOT an error — the ingestion layer must distinguish "404 no-file" from "5xx transient".

2. Parameterisation

Source	Parameter	Scheme	Notes
Assembly Daily File	`date`	—	No URL param. As-of date is read from the hearing table.
Senate Daily File	`date` + `session_year` + `type`	`SEN_{TYPE}_{year}_R_{YYYYMMDD}.PDF`	Filename-encoded.
Bill detail (enrichment)	—	`bill_id`	Leginfo canonical bill page; see §3.
Committee rosters	—	Static pages	No per-date param; refreshed periodically.

3. Assembly Daily File HTML structure

Three <table> elements, in document order. Each represents a distinct record type. Every data cell is a td with a specific class and, where applicable, a data-sort attribute carrying the clean canonical value.

Prefer data-sort over textContent in extraction — it strips the display suffix ('08/31/26-001' → 08/31/26), leading zeros, and the like.

3.1 Committee Hearings

Table selector: table.committee-hearing-table
Row selector: tr[class*='committee-hearing-details']

Verified on 2026-09-04: 4 hearing rows in the current file.

Field	Selector	Best value source
`date`	`td.committee_hearing-date`	`attr:data-sort` (e.g. `"08/31/26"`; raw `"08/31/26-001"`)
`time`	`td.committee_hearing-time`	`text` (e.g. `"9:30am"`, `"Upon Call of the Chair"`)
`committee`	`td.committee_hearing-name`	`attr:data-sort`
`location`	`td.committee_hearing-location`	`attr:data-sort`
`agenda_id`	`td.committee_hearing-menu a[href^='/api/dailyfile/agenda/']`	`attr:href` — Drupal AJAX id
`ics_url`	`td.committee_hearing-menu a[href^='/api/ics/generate']`	`attr:href`

Example row (verbatim):

<tr class=" committee-hearing-details">
  <td class="committee_hearing-date" data-sort="08/31/26-001">08/31/26</td>
  <td class="committee_hearing-time">Upon Call of the Chair</td>
  <td class="committee_hearing-name" data-sort="Budget">Budget</td>
  <td class="committee_hearing-location" data-sort="1021 O Street, Room 1100">1021 O Street, Room 1100</td>
  <td class="committee_hearing-menu">
    <div class="was-dropdown" data-once="wasDropdown">
      <button class="was-dropdown-toggle hearing-options" aria-haspopup="true" aria-expanded="false">…</button>
      <div class="was-dropdown-menu" role="menu" aria-hidden="true">
        <ul>
          <li role="none"><a href="/api/ics/generate?name=Budget&amp;start=20260831T000000&amp;end=20260901T000000&amp;location=1021%20O%20Street%2C%20Room%201100" role="menuitem">Add to Calendar</a></li>
          <li role="none"><a href="/api/dailyfile/agenda/20361" role="menuitem" class="use-ajax">View Agenda</a></li>
        </ul>
      </div>
    </div>
  </td>
</tr>

XPath equivalents:

//tr[contains(@class,'committee-hearing-details')]//td[contains(@class,'committee_hearing-date')]/@data-sort
//tr[contains(@class,'committee-hearing-details')]//td[contains(@class,'committee_hearing-name')]/@data-sort
//tr[contains(@class,'committee-hearing-details')]//td[contains(@class,'committee_hearing-menu')]//a[starts-with(@href,'/api/dailyfile/agenda/')]/@href

3.2 Bills on the Floor (`floor-bills-table`)

Table selector: table.floor-bills-table
Row selector: tr.class='floor-bill-details'

Verified on 2026-09-04: 156 rows.

Field	Selector	Best value source
`bill_number`	`td.floor_bill-measure a`	`attr:href` (parse bill_id; §4)
`bill_display`	`td.floor_bill-measure a`	`text` (e.g. `"AB 28"`)
`supplement`	`td.floor_bill-measure`	`text` after `<br>` (e.g. `"Supplemental 2"`)
`author`	`td.floor_bill-author`	`attr:data-sort`
`title`	`td.floor_bill-subject`	`text` (short title / description)
`location`	`td.floor_bill-location`	`attr:data-sort`
`item_number`	`td.floor_bill-item`	`text` (zero-padded; strip to int)
`chamber`	derived	from `bill_id` prefix
`date`	derived	Not on the row; context is the hearing/date section

Example row (verbatim, first row):

<tr class="floor-bill-details">
  <td class="floor_bill-measure" data-sort="AB 00028">
    <a href="https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=202520260AB28" target="_blank">AB 28</a><br>Supplemental 2</td>
  <td class="floor_bill-author" data-sort="Schiavo et al.">Schiavo et al.</td>
  <td class="floor_bill-subject">An act relating to solid waste, and making an appropriation therefor.</td>
  <td class="floor_bill-location" data-sort="Unfinished Business – Concurrence In Senate Amendments">Unfinished Business – Concurrence In Senate Amendments</td>
  <td class="floor_bill-item" data-sort="0000032">32</td>
  <td class="floor_bill-view-details">
    <!-- Modal Dialog --><dialog class="was-dialog floor_bill__modal" data-once="wasDialog">…</dialog>
  </td>
</tr>

Note on td.floor_bill-view-details: contains a <dialog> modal DOM widget (close button, SVG, "Details" panel), NOT a data field. Ignore it.

XPath equivalents:

//tr[contains(@class,'floor-bill-details')]//td[contains(@class,'floor_bill-measure')]/a/@href
//tr[contains(@class,'floor-bill-details')]//td[contains(@class,'floor_bill-author')]/@data-sort
//tr[contains(@class,'floor-bill-details')]//td[contains(@class,'floor_bill-subject')]/text()
//tr[contains(@class,'floor-bill-details')]//td[contains(@class,'floor_bill-item')]/text()

3.3 Bills in Committee (`committee-bills-table`)

Table selector: table.committee-bills-table
Row selector: tr[class*='committee-bills-details']

Verified on 2026-09-04: 1 row (the current live file has few committee bills — the count varies by day).

Field	Selector	Best value source
`bill_number`	`td.committee_bill-number a`	`attr:href` (parse bill_id; §4)
`author`	`td.committee_bill-author`	`attr:data-sort`
`title`	`td.committee_bill-subject`	`text`
`committee`	`td.committee_bill-committee`	`attr:data-sort`

Example row (verbatim, only row):

<tr class="committee-bills-details">
  <td class="committee_bill-number" data-sort="SB 00795">
    <a href="https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=202520260SB795" target="_blank">SB 795</a></td>
  <td class="committee_bill-author" data-sort="Archuleta">Archuleta</td>
  <td class="committee_bill-subject">Air pollution: gasoline vapor control systems: ethanol blends.</td>
  <td class="committee_bill-committee" data-sort="Emergency Management">Emergency Management</td>
</tr>

4. Bill ID encoding (universal across all daily-file links)

Every bill link on the Assembly (and Senate) daily file points to a single canonical pattern:

https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id={bill_id}

bill_id decomposes with the regex:

^(?P<session>\d{8})\d(?P<chamber>AB|SB|ACR|SCR|HR|SJR|ABR|SCR\d+)(?P<number>\d+)$

Component	Example	Meaning
`session`	`20252026`	`YYYYYYYY` = `YYYY-YYYY` regular session (e.g. `2025-2026`)
middle `0`	`0`	constant pad between session and chamber
`chamber`	`AB`	`AB`/`ACR`/`HR`/`ABR` = Assembly-origin; `SB`/`SCR`/`SJR` = Senate-origin
`number`	`28`	1–4 digits; zero-pad to 4 when writing back (`AB 28` ↔ `AB 0028`)

Worked example:

bill_id = 202520260AB28
  session = 2025-2026
  chamber = AB
  number  = 28
  display = "AB 28"    # or "AB 0028" with 4-digit zero-pad

This bill_id is the join key to:

billNavClient.xhtml?bill_id=… → status, current version, author(s), short title
billHistoryClient.xhtml?bill_id=… → full action-level history, committee referrals
billResultsClient.xhtml?… → floor / committee agenda views (JS-rendered; use JSON variants)

5. Senate Daily File — PDF parsing

The Senate Daily File is a PDF. There is no HTML table equivalent with the same structure. Two ingestion options:

5a. PDF path (primary)

Resolve the current filename from the index HTML (samples/senate_dailyfile_index.html):
```
# Find the most recent PDF link on the index page
grep -oE '/dailyfile/publications/download/SEN[A-Z_]*_[0-9]{4}_R_[0-9]{8}\.PDF' …
```
On 2026-09-04 the only referenced PDF is SEN_COM_2025_R_20260902.PDF.
Fetch the PDF:
```
https://www.senate.ca.gov/dailyfile/publications/download/SEN_COM_2025_R_20260902.PDF
```
Antibot mitigation required. Without a browser challenge solver, this returns HTTP 403 with a 67-byte HTML body. See ./skills/blocked-page-recovery and t_f595542f for the mitigation plan.
Extract text (pdfplumber / PyMuPDF), then apply layout heuristics:
- Section headers: BILLS ON THE FLOOR, COMMITTEE HEARINGS, BILLS IN COMMITTEE, HELD AT DESK, INACTIVE FILE, OFFICERS OF THE SENATE, SENATE ORDER OF BUSINESS.
- Bill-number regex at start of line:
```
^(AB|SB|ACR|SCR|HR|SJR|ABR)\s+(\d+)
```
- Following lines carry author, subject, and (for committeee sections) committee name (COMMITTEE OF <NAME>).

5b. Leginfo alternative (recommended as backup / enrichment)

billResultsClient/xhtml and billHistoryClient/xhtml (see §6) give a structured representation of Senate floor items and committee actions. They are JS-rendered; use the underlying JSON endpoints (documented by leginfo; exposed by billResultsClient.xhtml?…) for programmatic fetch. Sample: samples/leginfo_assembly_floor_2026-04-16.html (title: "California Legislative Information"; results injected by JS).

6. Correlation sources (for §2.5 of the parent — bills ↔ committees ↔ members)

Concern	URL	Sample
Assembly committee rosters	`https://www.assembly.ca.gov/committees`	—
Senate committee rosters	`https://www.senate.ca.gov/committees`	`samples/senate_committees_index.html`
Assembly members	`https://www.assembly.ca.gov/assemblymembers`	—
Find-my-rep by address	`https://www.assembly.ca.gov/assemblymembers/find-my-rep` and `https://findyourrep.legislature.ca.gov/`	—
Bill detail (author / title / status / committee history)	`https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id={bill_id}`	`samples/leginfo_assembly_floor_2026-04-16.html`
Bill committee history (action-level)	`https://leginfo.legislature.ca.gov/faces/billHistoryClient.xhtml?bill_id={bill_id}`	—

6.1 Committee rosters — key observation

Senate committee slugs are URL-encoded names (no slugs or IDs; the slug IS the committee name). Example from 2026-09-04:

/committees/selectcommitteeonhydrogenenergy
/committees/selectcommitteeoneconomicdevelopmentandtechnologicalinnovation
/committees/senateselectcommitteeonolderlgbtqcalifornians
/committees/offices-caucuses

Normalize by title-casing and stripping non-alphanumerics; cross-reference against the data-sort value in td.committee_hearing-name and td.committee_bill-committee.

6.2 Bill detail — structure observed

From samples/leginfo_assembly_floor_2026-04-16.html (a bill-detail page for 202520260AB28):

<h1 id="bill_nav_title">Bill Text</h1> — page title
.bill_title_yr — short title + session year (e.g. "Solid waste landfills…")
Author and committee fields live in the bill-info block (exact selectors to re-verify at parse-time; this page is enrichment, NOT the daily file source).

7. Normalised output schema (for `t_fb8081e7`)

One JSON-Lines record per daily-file entry:

{
  "id": "sha1(house|bill_number|date|committee)",
  "date": "2026-09-02",
  "house": "ASSEMBLY",              // or SENATE
  "source": "ASSEMBLY_DAILY_FILE",  // | SENATE_DAILY_FILE_PDF | LEGINFO_BILL_RESULTS
  "bill_number": "AB 28",
  "bill_id": "202520260AB28",
  "session": "2025-2026",
  "chamber_origin": "ASSEMBLY",     // AB-origin vs SB-origin
  "author": "Schiavo et al.",
  "short_title": "An act relating to solid waste, and making an appropriation therefor.",
  "description": "An act relating to solid waste, and making an appropriation therefor.",
  "committee": null,                 // null for floor items; e.g. "Emergency Management" for committee items
  "hearing_time": null,             // "9:30am" / "Upon Call of the Chair" for committee-hearings rows
  "location": "State Capitol, Room 444",
  "item_number": 32,               // int, un-padded
  "supplement": 2,                 // optional, from "Supplemental 2"
  "ics_url": "https://www.assembly.ca.gov/api/ics/generate?name=Budget&start=…",  // where applicable
  "agenda_id": "20361"             // where applicable
}

Field-to-row mapping:

Floor bill rows: bill_number ← floor_bill-measure a href, author ← floor_bill-author data-sort, title ← floor_bill-subject text, location ← floor_bill-location data-sort, item_number ← floor_bill-item text (un-pad). committee = null, hearing_time = null (unless the bill also appears as a committee item on the same file).
Committee bill rows: same fields plus committee ← committee_bill-committee data-sort, item_number = null.
Committee hearing rows: no bill — record the hearing metadata itself (date, time, committee, location, agenda_id, ics_url).
Senate (from PDF or leginfo alt): same schema, source=SENATE_DAILY_FILE_PDF or LEGINFO_BILL_RESULTS.

8. Pitfalls & risk register

Antibot on Senate PDFs. senate.ca.gov/dailyfile/publications/download/… behind a form.antibot challenge. Programmatic requests/browser JS fetch yields HTTP 403 with a 67-byte HTML body (verified 2026-09-04). Mitigation: headless browser with a cookie/session pre-warm, or route through the blocked-page-recovery skill.
No historical date param. The Assembly page is always current-live. "Date" is read from data (hearing table), not written into the URL.
404 ≠ error. Non-legislative days legitimately return 404 on the Senate PDF index. The ingestion layer must classify 404 as "no file today" and 5xx / retry-able as a transient error.
Session boundary. When the session changes in January, the URL pattern (bill_id = YYYYYYYY + …) stays the same but bills renumber. No code change required — the session field in the output reflects the current value read from bill_id.
Data-sort > textContent. Always prefer the data-sort attribute; it is the clean canonical value. textContent carries display suffixes and leading zeros.
Modal widget in floor-bill rows. td.floor_bill-view-details holds a <dialog>; do NOT extract it as a field.
Cross-chamber bills. AB-prefilled rows can appear on a Senate daily file and vice-versa. chamber_origin means origin, not publishing house.
Supplement tracking. A bill can have multiple supplements (Supplemental 2); the supplement field is the edition.
JS-rendered leginfo pages. billResultsClient and billHistoryClient need a headless browser or JSON endpoint; the static <table> they render is NOT in the initial HTML.

9. Sample archive

samples/
  assembly_dailyfile_latest.html          # 401 192 bytes, 2026-09-04
                                         # 4 hearing rows, 156 floor-bill rows, 1 committee-bills row
  senate_dailyfile_index.html            # 34 137 bytes, pdf-viewer index
                                         # links to SEN_COM_2025_R_20260902.PDF
  senate_committees_index.html           # 58 642 bytes, Senate committee slugs
  leginfo_assembly_floor_2026-04-16.html # 163 825 bytes, leginfo results page
                                         # billResultsClient, JS-rendered

These are the offline test fixtures for t_fb8081e7. Every sample was fetched via curl (assembly index, senate index, committee index, leginfo floor) or browser_exec fetch (leginfo floor). The Senate PDF sample is NOT archived because it is behind antibot — that is itself a documented finding. See §10 for the acceptance test harness the parser task should run against these files.

10. Acceptance test harness (for `t_fb8081e7`)

- id: asm_hearings
  input: samples/assembly_dailyfile_latest.html
  selector: tr[class*='committee-hearing-details']
  expect_min_rows: 1
  verified: 4

- id: asm_floor_bills
  input: samples/assembly_dailyfile_latest.html
  selector: tr.class='floor-bill-details'
  expect_min_rows: 100
  expect_fields: [bill_number, author, short_title, item_number]
  verified: 156

- id: asm_committee_bills
  input: samples/assembly_dailyfile_latest.html
  selector: tr[class*='committee-bills-details']
  expect_min_rows: 1
  verified: 1

- id: bill_id_parse
  regex: "^(?P<session>\d{8})\d(?P<chamber>AB|SB|ACR|SCR|HR|SJR|ABR)(?P<number>\d+)$"
  example_url: "https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=202520260AB28"
  expect: {session: "2025-2026", chamber: "AB", number: 28, display: "AB 28"}
  verified: pass

- id: senate_pdf_index
  input: samples/senate_dailyfile_index.html
  expect: "at least one /dailyfile/publications/download/*.PDF reference"
  verified: 1 (SEN_COM_2025_R_20260902.PDF)

Downstream handoff:

t_f595542f (ingestion): read daily_file_sources.yaml for URL patterns + parameters + antibot flags. Add a Senate PDF download step with a blocked-page-recovery fallback for 403s.
t_fb8081e7 (parser): the §3 and §5 tables are the exact selector / XPath / regex contracts. Use data-sort over textContent. Skip td.floor_bill-view-details.
t_8e5a3821 (RDB schema): the §7 per-record schema is the target. Foreign keys to committee_roster, bill_detail, bill_history tables (see §6).

21 KiB Raw Blame History Unescape Escape