WesternTechnologies/cali_scraper
2
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial commit

Browse Source
This commit is contained in:
kfj001 committed 2026-09-06 10:18:26 -07:00
commit 5f2db08b43
100 files changed
+16985
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

DAILY_FILE_PARSE_SPEC.md
+485
View File
@@ -0,0 +1,485 @@
# California Legislature Daily File — Parse & Ingestion Spec
**Research task:** `t_a171f4d1`
**Date:** 2026-09-04 (PDT, current 2025-2026 regular session)
**Verified against live pages** and **archived to `./samples/`**.
This document is the single source of truth for the ingestion
task (`t_f595542f`), the parser task (`t_fb8081e7`), and the RDB schema
task (`t_8e5a3821`). A developer should be able to implement a parser from
this document without revisiting the live pages, using the samples in
`./samples/`.
---
## TL;DR — the most important finding
- **Assembly Daily File = fully server-rendered HTML.** Three data tables with
semantic CSS classes and `data-sort` attributes hold pre-cleaned canonical
values (dates without the `-###` suffix, trimmed committee/author names).
Parse offline with BeautifulSoup + the CSS selectors below; no JS required.
- **Senate Daily File = PDF only.** `senate.ca.gov/dailyfile` is a Vue pdf-viewer
wrapper with **no parseable data table**. The publication exists only as a
PDF, and `senate.ca.gov` wraps the download path in an **antibot challenge**
(HTTP 403 with a 67-byte HTML body on programmatic fetch, verified 2026-09-04).
The Senate daily file must be ingested via a PDF path (pdfplumber / PyMuPDF)
with antibot mitigation, **or** enriched from leginfo's parallel
`billResultsClient` / `billHistoryClient` views (see alt sources).
- **Neither `?date=` nor `?dailyfile=…` is addressable.** The Assembly page
always renders the **current live** daily file — there is no historical-date
query parameter. "Date" is a data value (read from the hearing table), not a
URL parameter.
- **Cross-chamber bills:** an Assembly bill (AB…) can appear on a Senate daily
file and vice-versa. The "chamber" field means **origin**, not publishing house.
---
## 1. Canonical URLs
### 1.1 Assembly Daily File
| Item | URL |
|---|---|
| Canonical (HTML, always current) | `<https://www.assembly.ca.gov/schedules-publications/daily-file>` |
| Alias (redirects to canonical) | `https://assembly.ca.gov/schedules-publications/daily-file` |
| Alias | `https://assembly.ca.gov/dailyfile/pdfpublications` |
| Legacy help page | `https://leginfo.ca.gov/help/dayfileh.html` |
| Live sample | `samples/assembly_dailyfile_latest.html` (2026-09-04, 401 192 bytes) |
Properties:
- `requires_js=false` — data lives in static `<table>` markup.
- `antibot=false` — plain `requests` GET with a sensible `User-Agent` works.
- `content-type: text/html; charset=UTF-8`.
- No date parameter. `?date=2025-08-31` and `?dailyfile=2026-08-31` were
probed and **do not select a historical file** — the page always returns the
current live daily file.
### 1.2 Senate Daily File
| Item | URL |
|---|---|
| HTML index (human page, JS-rendered pdf viewer) | `https://www.senate.ca.gov/dailyfile` |
| Canonical (per `<link rel=canonical>`) | `https://www.senate.ca.gov/publications/senate-daily-file` |
| PDF download base | `https://senate.ca.gov/dailyfile/publications/download/` |
| Live index sample | `samples/senate_dailyfile_index.html` (34 137 bytes) |
| Verified PDF fragment | `SEN_COM_2025_R_20260902.PDF` (200 when served; 403 under antibot for this scanner) |
PDF filename pattern:
```
SEN_{TYPE}_{year}_R_{YYYYMMDD}.PDF
```
- `{TYPE}` = `R` (regular) or `COM` (committee daily file). Historical variants:
`SEN_ESS_*` for extraordinary/special sessions.
- `{year}` = the legislative session start year (e.g., `2025` for 2025-2026).
- `{YYYYMMDD}` = the publication date.
**Critical:** Not every calendar day has a published file (weekends,
non-legislative days). 404 on the PDF is **expected**, NOT an error —
the ingestion layer must distinguish "404 no-file" from "5xx transient".
---
## 2. Parameterisation
| Source | Parameter | Scheme | Notes |
|---|---|---|---|
| Assembly Daily File | `date` | — | No URL param. As-of date is read from the hearing table. |
| Senate Daily File | `date` + `session_year` + `type` | `SEN_{TYPE}_{year}_R_{YYYYMMDD}.PDF` | Filename-encoded. |
| Bill detail (enrichment) | — | `bill_id` | Leginfo canonical bill page; see §3. |
| Committee rosters | — | Static pages | No per-date param; refreshed periodically. |
---
## 3. Assembly Daily File HTML structure
Three `<table>` elements, in document order. Each represents a distinct record
type. Every data cell is a `td` with a **specific class** and, where
applicable, a **`data-sort`** attribute carrying the clean canonical value.
Prefer `data-sort` over `textContent` in extraction — it strips the display
suffix (`'08/31/26-001'` → `08/31/26`), leading zeros, and the like.
### 3.1 Committee Hearings
- **Table selector:** `table.committee-hearing-table`
- **Row selector:** `tr[class*='committee-hearing-details']`
Verified on 2026-09-04: **4 hearing rows** in the current file.
| Field | Selector | Best value source |
|---|---|---|
| `date` | `td.committee_hearing-date` | `attr:data-sort` (e.g. `"08/31/26"`; raw `"08/31/26-001"`) |
| `time` | `td.committee_hearing-time` | `text` (e.g. `"9:30am"`, `"Upon Call of the Chair"`) |
| `committee` | `td.committee_hearing-name` | `attr:data-sort` |
| `location` | `td.committee_hearing-location` | `attr:data-sort` |
| `agenda_id` | `td.committee_hearing-menu a[href^='/api/dailyfile/agenda/']` | `attr:href` — Drupal AJAX id |
| `ics_url` | `td.committee_hearing-menu a[href^='/api/ics/generate']` | `attr:href` |
Example row (verbatim):
```html
<tr class=" committee-hearing-details">
<td class="committee_hearing-date" data-sort="08/31/26-001">08/31/26</td>
<td class="committee_hearing-time">Upon Call of the Chair</td>
<td class="committee_hearing-name" data-sort="Budget">Budget</td>
<td class="committee_hearing-location" data-sort="1021 O Street, Room 1100">1021 O Street, Room 1100</td>
<td class="committee_hearing-menu">
<div class="was-dropdown" data-once="wasDropdown">
<button class="was-dropdown-toggle hearing-options" aria-haspopup="true" aria-expanded="false">…</button>
<div class="was-dropdown-menu" role="menu" aria-hidden="true">
<ul>
<li role="none"><a href="/api/ics/generate?name=Budget&amp;start=20260831T000000&amp;end=20260901T000000&amp;location=1021%20O%20Street%2C%20Room%201100" role="menuitem">Add to Calendar</a></li>
<li role="none"><a href="/api/dailyfile/agenda/20361" role="menuitem" class="use-ajax">View Agenda</a></li>
</ul>
</div>
</div>
</td>
</tr>
```
XPath equivalents:
```
//tr[contains(@class,'committee-hearing-details')]//td[contains(@class,'committee_hearing-date')]/@data-sort
//tr[contains(@class,'committee-hearing-details')]//td[contains(@class,'committee_hearing-name')]/@data-sort
//tr[contains(@class,'committee-hearing-details')]//td[contains(@class,'committee_hearing-menu')]//a[starts-with(@href,'/api/dailyfile/agenda/')]/@href
```
### 3.2 Bills on the Floor (`floor-bills-table`)
- **Table selector:** `table.floor-bills-table`
- **Row selector:** `tr.class='floor-bill-details'`
Verified on 2026-09-04: **156 rows**.
| Field | Selector | Best value source |
|---|---|---|
| `bill_number` | `td.floor_bill-measure a` | `attr:href` (parse bill_id; §4) |
| `bill_display` | `td.floor_bill-measure a` | `text` (e.g. `"AB 28"`) |
| `supplement` | `td.floor_bill-measure` | `text` after `<br>` (e.g. `"Supplemental 2"`) |
| `author` | `td.floor_bill-author` | `attr:data-sort` |
| `title` | `td.floor_bill-subject` | `text` (short title / description) |
| `location` | `td.floor_bill-location` | `attr:data-sort` |
| `item_number` | `td.floor_bill-item` | `text` (zero-padded; strip to int) |
| `chamber` | derived | from `bill_id` prefix |
| `date` | **derived** | Not on the row; context is the hearing/date section |
Example row (verbatim, first row):
```html
<tr class="floor-bill-details">
<td class="floor_bill-measure" data-sort="AB 00028">
<a href="https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=202520260AB28" target="_blank">AB 28</a><br>Supplemental 2</td>
<td class="floor_bill-author" data-sort="Schiavo et al.">Schiavo et al.</td>
<td class="floor_bill-subject">An act relating to solid waste, and making an appropriation therefor.</td>
<td class="floor_bill-location" data-sort="Unfinished Business – Concurrence In Senate Amendments">Unfinished Business – Concurrence In Senate Amendments</td>
<td class="floor_bill-item" data-sort="0000032">32</td>
<td class="floor_bill-view-details">
<!-- Modal Dialog --><dialog class="was-dialog floor_bill__modal" data-once="wasDialog">…</dialog>
</td>
</tr>
```
**Note on `td.floor_bill-view-details`:** contains a `<dialog>` modal DOM
widget (close button, SVG, "Details" panel), NOT a data field. **Ignore it.**
XPath equivalents:
```
//tr[contains(@class,'floor-bill-details')]//td[contains(@class,'floor_bill-measure')]/a/@href
//tr[contains(@class,'floor-bill-details')]//td[contains(@class,'floor_bill-author')]/@data-sort
//tr[contains(@class,'floor-bill-details')]//td[contains(@class,'floor_bill-subject')]/text()
//tr[contains(@class,'floor-bill-details')]//td[contains(@class,'floor_bill-item')]/text()
```
### 3.3 Bills in Committee (`committee-bills-table`)
- **Table selector:** `table.committee-bills-table`
- **Row selector:** `tr[class*='committee-bills-details']`
Verified on 2026-09-04: **1 row** (the current live file has few committee
bills — the count varies by day).
| Field | Selector | Best value source |
|---|---|---|
| `bill_number` | `td.committee_bill-number a` | `attr:href` (parse bill_id; §4) |
| `author` | `td.committee_bill-author` | `attr:data-sort` |
| `title` | `td.committee_bill-subject` | `text` |
| `committee` | `td.committee_bill-committee` | `attr:data-sort` |
Example row (verbatim, only row):
```html
<tr class="committee-bills-details">
<td class="committee_bill-number" data-sort="SB 00795">
<a href="https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=202520260SB795" target="_blank">SB 795</a></td>
<td class="committee_bill-author" data-sort="Archuleta">Archuleta</td>
<td class="committee_bill-subject">Air pollution: gasoline vapor control systems: ethanol blends.</td>
<td class="committee_bill-committee" data-sort="Emergency Management">Emergency Management</td>
</tr>
```
---
## 4. Bill ID encoding (universal across all daily-file links)
Every bill link on the Assembly (and Senate) daily file points to a single
canonical pattern:
```
https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id={bill_id}
```
`bill_id` decomposes with the regex:
```regex
^(?P<session>\d{8})\d(?P<chamber>AB|SB|ACR|SCR|HR|SJR|ABR|SCR\d+)(?P<number>\d+)$
```
| Component | Example | Meaning |
|---|---|---|
| `session` | `20252026` | `YYYYYYYY` = `YYYY-YYYY` regular session (e.g. `2025-2026`) |
| middle `0` | `0` | constant pad between session and chamber |
| `chamber` | `AB` | `AB`/`ACR`/`HR`/`ABR` = Assembly-origin; `SB`/`SCR`/`SJR` = Senate-origin |
| `number` | `28` | 1–4 digits; **zero-pad to 4** when writing back (`AB 28` ↔ `AB 0028`) |
Worked example:
```
bill_id = 202520260AB28
session = 2025-2026
chamber = AB
number = 28
display = "AB 28" # or "AB 0028" with 4-digit zero-pad
```
This bill_id is the join key to:
- `billNavClient.xhtml?bill_id=…` → status, current version, author(s), short title
- `billHistoryClient.xhtml?bill_id=…` → full action-level history, committee referrals
- `billResultsClient.xhtml?…` → floor / committee agenda views (JS-rendered; use JSON variants)
---
## 5. Senate Daily File — PDF parsing
The Senate Daily File is a **PDF**. There is no HTML table equivalent with the
same structure. Two ingestion options:
### 5a. PDF path (primary)
1. Resolve the current filename from the index HTML
(`samples/senate_dailyfile_index.html`):
```
# Find the most recent PDF link on the index page
grep -oE '/dailyfile/publications/download/SEN[A-Z_]*_[0-9]{4}_R_[0-9]{8}\.PDF' …
```
On 2026-09-04 the only referenced PDF is `SEN_COM_2025_R_20260902.PDF`.
2. Fetch the PDF:
```
https://www.senate.ca.gov/dailyfile/publications/download/SEN_COM_2025_R_20260902.PDF
```
**Antibot mitigation required.** Without a browser challenge solver, this
returns `HTTP 403` with a 67-byte HTML body. See `./skills/blocked-page-recovery`
and `t_f595542f` for the mitigation plan.
3. Extract text (pdfplumber / PyMuPDF), then apply layout heuristics:
- Section headers: `BILLS ON THE FLOOR`, `COMMITTEE HEARINGS`,
`BILLS IN COMMITTEE`, `HELD AT DESK`, `INACTIVE FILE`,
`OFFICERS OF THE SENATE`, `SENATE ORDER OF BUSINESS`.
- Bill-number regex at start of line:
```regex
^(AB|SB|ACR|SCR|HR|SJR|ABR)\s+(\d+)
```
- Following lines carry author, subject, and (for committeee sections)
committee name (`COMMITTEE OF <NAME>`).
### 5b. Leginfo alternative (recommended as backup / enrichment)
`billResultsClient/xhtml` and `billHistoryClient/xhtml` (see §6) give a
structured representation of Senate floor items and committee actions. They
are JS-rendered; use the underlying JSON endpoints (documented by leginfo;
exposed by `billResultsClient.xhtml?…`) for programmatic fetch. Sample:
`samples/leginfo_assembly_floor_2026-04-16.html` (title: "California
Legislative Information"; results injected by JS).
---
## 6. Correlation sources (for §2.5 of the parent — bills ↔ committees ↔ members)
| Concern | URL | Sample |
|---|---|---|
| Assembly committee rosters | `https://www.assembly.ca.gov/committees` | — |
| Senate committee rosters | `https://www.senate.ca.gov/committees` | `samples/senate_committees_index.html` |
| Assembly members | `https://www.assembly.ca.gov/assemblymembers` | — |
| Find-my-rep by address | `https://www.assembly.ca.gov/assemblymembers/find-my-rep` and `https://findyourrep.legislature.ca.gov/` | — |
| Bill detail (author / title / status / committee history) | `https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id={bill_id}` | `samples/leginfo_assembly_floor_2026-04-16.html` |
| Bill committee history (action-level) | `https://leginfo.legislature.ca.gov/faces/billHistoryClient.xhtml?bill_id={bill_id}` | — |
### 6.1 Committee rosters — key observation
Senate committee slugs are **URL-encoded names** (no slugs or IDs; the slug IS
the committee name). Example from 2026-09-04:
```
/committees/selectcommitteeonhydrogenenergy
/committees/selectcommitteeoneconomicdevelopmentandtechnologicalinnovation
/committees/senateselectcommitteeonolderlgbtqcalifornians
/committees/offices-caucuses
```
Normalize by title-casing and stripping non-alphanumerics; cross-reference
against the `data-sort` value in `td.committee_hearing-name` and
`td.committee_bill-committee`.
### 6.2 Bill detail — structure observed
From `samples/leginfo_assembly_floor_2026-04-16.html` (a bill-detail page for
`202520260AB28`):
- `<h1 id="bill_nav_title">Bill Text</h1>` — page title
- `.bill_title_yr` — short title + session year (e.g. "Solid waste landfills…")
- Author and committee fields live in the bill-info block (exact selectors to
re-verify at parse-time; this page is enrichment, NOT the daily file source).
---
## 7. Normalised output schema (for `t_fb8081e7`)
One JSON-Lines record per daily-file entry:
```json
{
"id": "sha1(house|bill_number|date|committee)",
"date": "2026-09-02",
"house": "ASSEMBLY", // or SENATE
"source": "ASSEMBLY_DAILY_FILE", // | SENATE_DAILY_FILE_PDF | LEGINFO_BILL_RESULTS
"bill_number": "AB 28",
"bill_id": "202520260AB28",
"session": "2025-2026",
"chamber_origin": "ASSEMBLY", // AB-origin vs SB-origin
"author": "Schiavo et al.",
"short_title": "An act relating to solid waste, and making an appropriation therefor.",
"description": "An act relating to solid waste, and making an appropriation therefor.",
"committee": null, // null for floor items; e.g. "Emergency Management" for committee items
"hearing_time": null, // "9:30am" / "Upon Call of the Chair" for committee-hearings rows
"location": "State Capitol, Room 444",
"item_number": 32, // int, un-padded
"supplement": 2, // optional, from "Supplemental 2"
"ics_url": "https://www.assembly.ca.gov/api/ics/generate?name=Budget&start=…", // where applicable
"agenda_id": "20361" // where applicable
}
```
Field-to-row mapping:
- Floor bill rows: `bill_number` ← `floor_bill-measure a href`, `author` ← `floor_bill-author data-sort`, `title` ← `floor_bill-subject text`, `location` ← `floor_bill-location data-sort`, `item_number` ← `floor_bill-item text` (un-pad). `committee` = null, `hearing_time` = null (unless the bill also appears as a committee item on the same file).
- Committee bill rows: same fields plus `committee` ← `committee_bill-committee data-sort`, `item_number` = null.
- Committee hearing rows: no bill — record the hearing metadata itself (date, time, committee, location, `agenda_id`, `ics_url`).
- Senate (from PDF or leginfo alt): same schema, `source=SENATE_DAILY_FILE_PDF` or `LEGINFO_BILL_RESULTS`.
---
## 8. Pitfalls & risk register
1. **Antibot on Senate PDFs.** `senate.ca.gov/dailyfile/publications/download/…`
behind a `form.antibot` challenge. Programmatic `requests`/browser JS fetch
yields `HTTP 403` with a 67-byte HTML body (verified 2026-09-04). Mitigation:
headless browser with a cookie/session pre-warm, or route through the
`blocked-page-recovery` skill.
2. **No historical date param.** The Assembly page is always current-live.
"Date" is read from data (hearing table), not written into the URL.
3. **404 ≠ error.** Non-legislative days legitimately return 404 on the Senate
PDF index. The ingestion layer must classify 404 as "no file today" and
5xx / retry-able as a transient error.
4. **Session boundary.** When the session changes in January, the URL pattern
(`bill_id` = `YYYYYYYY` + …) stays the same but bills renumber. No code
change required — the `session` field in the output reflects the current
value read from `bill_id`.
5. **Data-sort > textContent.** Always prefer the `data-sort` attribute; it
is the clean canonical value. `textContent` carries display suffixes and
leading zeros.
6. **Modal widget in floor-bill rows.** `td.floor_bill-view-details` holds a
`<dialog>`; do NOT extract it as a field.
7. **Cross-chamber bills.** `AB`-prefilled rows can appear on a Senate daily
file and vice-versa. `chamber_origin` means *origin*, not publishing house.
8. **Supplement tracking.** A bill can have multiple supplements
(`Supplemental 2`); the `supplement` field is the edition.
9. **JS-rendered leginfo pages.** `billResultsClient` and `billHistoryClient`
need a headless browser or JSON endpoint; the static `<table>` they render
is NOT in the initial HTML.
---
## 9. Sample archive
```
samples/
assembly_dailyfile_latest.html # 401 192 bytes, 2026-09-04
# 4 hearing rows, 156 floor-bill rows, 1 committee-bills row
senate_dailyfile_index.html # 34 137 bytes, pdf-viewer index
# links to SEN_COM_2025_R_20260902.PDF
senate_committees_index.html # 58 642 bytes, Senate committee slugs
leginfo_assembly_floor_2026-04-16.html # 163 825 bytes, leginfo results page
# billResultsClient, JS-rendered
```
These are the offline test fixtures for `t_fb8081e7`. Every sample was
fetched via `curl` (assembly index, senate index, committee index, leginfo
floor) or `browser_exec` fetch (leginfo floor). The Senate PDF sample is NOT
archived because it is behind antibot — that is itself a documented finding.
See §10 for the acceptance test harness the parser task should run against
these files.
---
## 10. Acceptance test harness (for `t_fb8081e7`)
```yaml
- id: asm_hearings
input: samples/assembly_dailyfile_latest.html
selector: tr[class*='committee-hearing-details']
expect_min_rows: 1
verified: 4
- id: asm_floor_bills
input: samples/assembly_dailyfile_latest.html
selector: tr.class='floor-bill-details'
expect_min_rows: 100
expect_fields: [bill_number, author, short_title, item_number]
verified: 156
- id: asm_committee_bills
input: samples/assembly_dailyfile_latest.html
selector: tr[class*='committee-bills-details']
expect_min_rows: 1
verified: 1
- id: bill_id_parse
regex: "^(?P<session>\d{8})\d(?P<chamber>AB|SB|ACR|SCR|HR|SJR|ABR)(?P<number>\d+)$"
example_url: "https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=202520260AB28"
expect: {session: "2025-2026", chamber: "AB", number: 28, display: "AB 28"}
verified: pass
- id: senate_pdf_index
input: samples/senate_dailyfile_index.html
expect: "at least one /dailyfile/publications/download/*.PDF reference"
verified: 1 (SEN_COM_2025_R_20260902.PDF)
```
---
**Downstream handoff:**
- `t_f595542f` (ingestion): read `daily_file_sources.yaml` for URL patterns +
parameters + antibot flags. Add a Senate PDF download step with a
`blocked-page-recovery` fallback for 403s.
- `t_fb8081e7` (parser): the §3 and §5 tables are the exact selector / XPath
/ regex contracts. Use `data-sort` over `textContent`. Skip
`td.floor_bill-view-details`.
- `t_8e5a3821` (RDB schema): the §7 per-record schema is the target. Foreign
keys to `committee_roster`, `bill_detail`, `bill_history` tables (see §6).