cali_scraper/DAILY_FILE_PARSE_SPEC.md

# California Legislature Daily File — Parse & Ingestion Spec

**Research task:** `t_a171f4d1`  
**Date:** 2026-09-04 (PDT, current 2025-2026 regular session)  
**Verified against live pages** and **archived to `./samples/`**.

This document is the single source of truth for the ingestion
task (`t_f595542f`), the parser task (`t_fb8081e7`), and the RDB schema
task (`t_8e5a3821`). A developer should be able to implement a parser from
this document without revisiting the live pages, using the samples in
`./samples/`.

---

## TL;DR — the most important finding

- **Assembly Daily File = fully server-rendered HTML.** Three data tables with
  semantic CSS classes and `data-sort` attributes hold pre-cleaned canonical
  values (dates without the `-###` suffix, trimmed committee/author names).
  Parse offline with BeautifulSoup + the CSS selectors below; no JS required.
- **Senate Daily File = PDF only.** `senate.ca.gov/dailyfile` is a Vue pdf-viewer
  wrapper with **no parseable data table**. The publication exists only as a
  PDF, and `senate.ca.gov` wraps the download path in an **antibot challenge**
  (HTTP 403 with a 67-byte HTML body on programmatic fetch, verified 2026-09-04).
  The Senate daily file must be ingested via a PDF path (pdfplumber / PyMuPDF)
  with antibot mitigation, **or** enriched from leginfo's parallel
  `billResultsClient` / `billHistoryClient` views (see alt sources).
- **Neither `?date=` nor `?dailyfile=…` is addressable.** The Assembly page
  always renders the **current live** daily file — there is no historical-date
  query parameter. "Date" is a data value (read from the hearing table), not a
  URL parameter.
- **Cross-chamber bills:** an Assembly bill (AB…) can appear on a Senate daily
  file and vice-versa. The "chamber" field means **origin**, not publishing house.

---

## 1. Canonical URLs

### 1.1 Assembly Daily File

| Item | URL |
|---|---|
| Canonical (HTML, always current) | `<https://www.assembly.ca.gov/schedules-publications/daily-file>` |
| Alias (redirects to canonical) | `https://assembly.ca.gov/schedules-publications/daily-file` |
| Alias | `https://assembly.ca.gov/dailyfile/pdfpublications` |
| Legacy help page | `https://leginfo.ca.gov/help/dayfileh.html` |
| Live sample | `samples/assembly_dailyfile_latest.html` (2026-09-04, 401 192 bytes) |

Properties:

- `requires_js=false` — data lives in static `<table>` markup.
- `antibot=false` — plain `requests` GET with a sensible `User-Agent` works.
- `content-type: text/html; charset=UTF-8`.
- No date parameter. `?date=2025-08-31` and `?dailyfile=2026-08-31` were
  probed and **do not select a historical file** — the page always returns the
  current live daily file.

### 1.2 Senate Daily File

| Item | URL |
|---|---|
| HTML index (human page, JS-rendered pdf viewer) | `https://www.senate.ca.gov/dailyfile` |
| Canonical (per `<link rel=canonical>`) | `https://www.senate.ca.gov/publications/senate-daily-file` |
| PDF download base | `https://senate.ca.gov/dailyfile/publications/download/` |
| Live index sample | `samples/senate_dailyfile_index.html` (34 137 bytes) |
| Verified PDF fragment | `SEN_COM_2025_R_20260902.PDF` (200 when served; 403 under antibot for this scanner) |

PDF filename pattern:

```
SEN_{TYPE}_{year}_R_{YYYYMMDD}.PDF
```

- `{TYPE}` = `R` (regular) or `COM` (committee daily file). Historical variants:
  `SEN_ESS_*` for extraordinary/special sessions.
- `{year}` = the legislative session start year (e.g., `2025` for 2025-2026).
- `{YYYYMMDD}` = the publication date.

**Critical:** Not every calendar day has a published file (weekends,
non-legislative days). 404 on the PDF is **expected**, NOT an error —
the ingestion layer must distinguish "404 no-file" from "5xx transient".

---

## 2. Parameterisation

| Source | Parameter | Scheme | Notes |
|---|---|---|---|
| Assembly Daily File | `date` | — | No URL param. As-of date is read from the hearing table. |
| Senate Daily File | `date` + `session_year` + `type` | `SEN_{TYPE}_{year}_R_{YYYYMMDD}.PDF` | Filename-encoded. |
| Bill detail (enrichment) | — | `bill_id` | Leginfo canonical bill page; see §3. |
| Committee rosters | — | Static pages | No per-date param; refreshed periodically. |

---

## 3. Assembly Daily File HTML structure

Three `<table>` elements, in document order. Each represents a distinct record
type. Every data cell is a `td` with a **specific class** and, where
applicable, a **`data-sort`** attribute carrying the clean canonical value.

Prefer `data-sort` over `textContent` in extraction — it strips the display
suffix (`'08/31/26-001'` → `08/31/26`), leading zeros, and the like.

### 3.1 Committee Hearings

- **Table selector:** `table.committee-hearing-table`
- **Row selector:** `tr[class*='committee-hearing-details']`

Verified on 2026-09-04: **4 hearing rows** in the current file.

| Field | Selector | Best value source |
|---|---|---|
| `date` | `td.committee_hearing-date` | `attr:data-sort` (e.g. `"08/31/26"`; raw `"08/31/26-001"`) |
| `time` | `td.committee_hearing-time` | `text` (e.g. `"9:30am"`, `"Upon Call of the Chair"`) |
| `committee` | `td.committee_hearing-name` | `attr:data-sort` |
| `location` | `td.committee_hearing-location` | `attr:data-sort` |
| `agenda_id` | `td.committee_hearing-menu a[href^='/api/dailyfile/agenda/']` | `attr:href` — Drupal AJAX id |
| `ics_url` | `td.committee_hearing-menu a[href^='/api/ics/generate']` | `attr:href` |

Example row (verbatim):

```html
<tr class=" committee-hearing-details">
  <td class="committee_hearing-date" data-sort="08/31/26-001">08/31/26</td>
  <td class="committee_hearing-time">Upon Call of the Chair</td>
  <td class="committee_hearing-name" data-sort="Budget">Budget</td>
  <td class="committee_hearing-location" data-sort="1021 O Street, Room 1100">1021 O Street, Room 1100</td>
  <td class="committee_hearing-menu">
    <div class="was-dropdown" data-once="wasDropdown">
      <button class="was-dropdown-toggle hearing-options" aria-haspopup="true" aria-expanded="false">…</button>
      <div class="was-dropdown-menu" role="menu" aria-hidden="true">
        <ul>
          <li role="none"><a href="/api/ics/generate?name=Budget&amp;start=20260831T000000&amp;end=20260901T000000&amp;location=1021%20O%20Street%2C%20Room%201100" role="menuitem">Add to Calendar</a></li>
          <li role="none"><a href="/api/dailyfile/agenda/20361" role="menuitem" class="use-ajax">View Agenda</a></li>
        </ul>
      </div>
    </div>
  </td>
</tr>
```

XPath equivalents:

```
//tr[contains(@class,'committee-hearing-details')]//td[contains(@class,'committee_hearing-date')]/@data-sort
//tr[contains(@class,'committee-hearing-details')]//td[contains(@class,'committee_hearing-name')]/@data-sort
//tr[contains(@class,'committee-hearing-details')]//td[contains(@class,'committee_hearing-menu')]//a[starts-with(@href,'/api/dailyfile/agenda/')]/@href
```

### 3.2 Bills on the Floor (`floor-bills-table`)

- **Table selector:** `table.floor-bills-table`
- **Row selector:** `tr.class='floor-bill-details'`

Verified on 2026-09-04: **156 rows**.

| Field | Selector | Best value source |
|---|---|---|
| `bill_number` | `td.floor_bill-measure a` | `attr:href` (parse bill_id; §4) |
| `bill_display` | `td.floor_bill-measure a` | `text` (e.g. `"AB 28"`) |
| `supplement` | `td.floor_bill-measure` | `text` after `<br>` (e.g. `"Supplemental 2"`) |
| `author` | `td.floor_bill-author` | `attr:data-sort` |
| `title` | `td.floor_bill-subject` | `text` (short title / description) |
| `location` | `td.floor_bill-location` | `attr:data-sort` |
| `item_number` | `td.floor_bill-item` | `text` (zero-padded; strip to int) |
| `chamber` | derived | from `bill_id` prefix |
| `date` | **derived** | Not on the row; context is the hearing/date section |

Example row (verbatim, first row):

```html
<tr class="floor-bill-details">
  <td class="floor_bill-measure" data-sort="AB 00028">
    <a href="https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=202520260AB28" target="_blank">AB 28</a><br>Supplemental 2</td>
  <td class="floor_bill-author" data-sort="Schiavo et al.">Schiavo et al.</td>
  <td class="floor_bill-subject">An act relating to solid waste, and making an appropriation therefor.</td>
  <td class="floor_bill-location" data-sort="Unfinished Business – Concurrence In Senate Amendments">Unfinished Business – Concurrence In Senate Amendments</td>
  <td class="floor_bill-item" data-sort="0000032">32</td>
  <td class="floor_bill-view-details">
    <!-- Modal Dialog --><dialog class="was-dialog floor_bill__modal" data-once="wasDialog">…</dialog>
  </td>
</tr>
```

**Note on `td.floor_bill-view-details`:** contains a `<dialog>` modal DOM
widget (close button, SVG, "Details" panel), NOT a data field. **Ignore it.**

XPath equivalents:

```
//tr[contains(@class,'floor-bill-details')]//td[contains(@class,'floor_bill-measure')]/a/@href
//tr[contains(@class,'floor-bill-details')]//td[contains(@class,'floor_bill-author')]/@data-sort
//tr[contains(@class,'floor-bill-details')]//td[contains(@class,'floor_bill-subject')]/text()
//tr[contains(@class,'floor-bill-details')]//td[contains(@class,'floor_bill-item')]/text()
```

### 3.3 Bills in Committee (`committee-bills-table`)

- **Table selector:** `table.committee-bills-table`
- **Row selector:** `tr[class*='committee-bills-details']`

Verified on 2026-09-04: **1 row** (the current live file has few committee
bills — the count varies by day).

| Field | Selector | Best value source |
|---|---|---|
| `bill_number` | `td.committee_bill-number a` | `attr:href` (parse bill_id; §4) |
| `author` | `td.committee_bill-author` | `attr:data-sort` |
| `title` | `td.committee_bill-subject` | `text` |
| `committee` | `td.committee_bill-committee` | `attr:data-sort` |

Example row (verbatim, only row):

```html
<tr class="committee-bills-details">
  <td class="committee_bill-number" data-sort="SB 00795">
    <a href="https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=202520260SB795" target="_blank">SB 795</a></td>
  <td class="committee_bill-author" data-sort="Archuleta">Archuleta</td>
  <td class="committee_bill-subject">Air pollution: gasoline vapor control systems: ethanol blends.</td>
  <td class="committee_bill-committee" data-sort="Emergency Management">Emergency Management</td>
</tr>
```

---

## 4. Bill ID encoding (universal across all daily-file links)

Every bill link on the Assembly (and Senate) daily file points to a single
canonical pattern:

```
https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id={bill_id}
```

`bill_id` decomposes with the regex:

```regex
^(?P<session>\d{8})\d(?P<chamber>AB|SB|ACR|SCR|HR|SJR|ABR|SCR\d+)(?P<number>\d+)$
```

| Component | Example | Meaning |
|---|---|---|
| `session` | `20252026` | `YYYYYYYY` = `YYYY-YYYY` regular session (e.g. `2025-2026`) |
| middle `0` | `0` | constant pad between session and chamber |
| `chamber` | `AB` | `AB`/`ACR`/`HR`/`ABR` = Assembly-origin; `SB`/`SCR`/`SJR` = Senate-origin |
| `number` | `28` | 1–4 digits; **zero-pad to 4** when writing back (`AB 28` ↔ `AB 0028`) |

Worked example:

```
bill_id = 202520260AB28
  session = 2025-2026
  chamber = AB
  number  = 28
  display = "AB 28"    # or "AB 0028" with 4-digit zero-pad
```

This bill_id is the join key to:

- `billNavClient.xhtml?bill_id=…` → status, current version, author(s), short title
- `billHistoryClient.xhtml?bill_id=…` → full action-level history, committee referrals
- `billResultsClient.xhtml?…` → floor / committee agenda views (JS-rendered; use JSON variants)

---

## 5. Senate Daily File — PDF parsing

The Senate Daily File is a **PDF**. There is no HTML table equivalent with the
same structure. Two ingestion options:

### 5a. PDF path (primary)

1. Resolve the current filename from the index HTML
   (`samples/senate_dailyfile_index.html`):
   ```
   # Find the most recent PDF link on the index page
   grep -oE '/dailyfile/publications/download/SEN[A-Z_]*_[0-9]{4}_R_[0-9]{8}\.PDF' …
   ```
   On 2026-09-04 the only referenced PDF is `SEN_COM_2025_R_20260902.PDF`.

2. Fetch the PDF:
   ```
   https://www.senate.ca.gov/dailyfile/publications/download/SEN_COM_2025_R_20260902.PDF
   ```
   **Antibot mitigation required.** Without a browser challenge solver, this
   returns `HTTP 403` with a 67-byte HTML body. See `./skills/blocked-page-recovery`
   and `t_f595542f` for the mitigation plan.

3. Extract text (pdfplumber / PyMuPDF), then apply layout heuristics:
   - Section headers: `BILLS ON THE FLOOR`, `COMMITTEE HEARINGS`,
     `BILLS IN COMMITTEE`, `HELD AT DESK`, `INACTIVE FILE`,
     `OFFICERS OF THE SENATE`, `SENATE ORDER OF BUSINESS`.
   - Bill-number regex at start of line:
     ```regex
     ^(AB|SB|ACR|SCR|HR|SJR|ABR)\s+(\d+)
     ```
   - Following lines carry author, subject, and (for committeee sections)
     committee name (`COMMITTEE OF <NAME>`).

### 5b. Leginfo alternative (recommended as backup / enrichment)

`billResultsClient/xhtml` and `billHistoryClient/xhtml` (see §6) give a
structured representation of Senate floor items and committee actions. They
are JS-rendered; use the underlying JSON endpoints (documented by leginfo;
exposed by `billResultsClient.xhtml?…`) for programmatic fetch. Sample:
`samples/leginfo_assembly_floor_2026-04-16.html` (title: "California
Legislative Information"; results injected by JS).

---

## 6. Correlation sources (for §2.5 of the parent — bills ↔ committees ↔ members)

| Concern | URL | Sample |
|---|---|---|
| Assembly committee rosters | `https://www.assembly.ca.gov/committees` | — |
| Senate committee rosters | `https://www.senate.ca.gov/committees` | `samples/senate_committees_index.html` |
| Assembly members | `https://www.assembly.ca.gov/assemblymembers` | — |
| Find-my-rep by address | `https://www.assembly.ca.gov/assemblymembers/find-my-rep` and `https://findyourrep.legislature.ca.gov/` | — |
| Bill detail (author / title / status / committee history) | `https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id={bill_id}` | `samples/leginfo_assembly_floor_2026-04-16.html` |
| Bill committee history (action-level) | `https://leginfo.legislature.ca.gov/faces/billHistoryClient.xhtml?bill_id={bill_id}` | — |

### 6.1 Committee rosters — key observation

Senate committee slugs are **URL-encoded names** (no slugs or IDs; the slug IS
the committee name). Example from 2026-09-04:

```
/committees/selectcommitteeonhydrogenenergy
/committees/selectcommitteeoneconomicdevelopmentandtechnologicalinnovation
/committees/senateselectcommitteeonolderlgbtqcalifornians
/committees/offices-caucuses
```

Normalize by title-casing and stripping non-alphanumerics; cross-reference
against the `data-sort` value in `td.committee_hearing-name` and
`td.committee_bill-committee`.

### 6.2 Bill detail — structure observed

From `samples/leginfo_assembly_floor_2026-04-16.html` (a bill-detail page for
`202520260AB28`):

- `<h1 id="bill_nav_title">Bill Text</h1>` — page title
- `.bill_title_yr` — short title + session year (e.g. "Solid waste landfills…")
- Author and committee fields live in the bill-info block (exact selectors to
  re-verify at parse-time; this page is enrichment, NOT the daily file source).

---

## 7. Normalised output schema (for `t_fb8081e7`)

One JSON-Lines record per daily-file entry:

```json
{
  "id": "sha1(house|bill_number|date|committee)",
  "date": "2026-09-02",
  "house": "ASSEMBLY",              // or SENATE
  "source": "ASSEMBLY_DAILY_FILE",  // | SENATE_DAILY_FILE_PDF | LEGINFO_BILL_RESULTS
  "bill_number": "AB 28",
  "bill_id": "202520260AB28",
  "session": "2025-2026",
  "chamber_origin": "ASSEMBLY",     // AB-origin vs SB-origin
  "author": "Schiavo et al.",
  "short_title": "An act relating to solid waste, and making an appropriation therefor.",
  "description": "An act relating to solid waste, and making an appropriation therefor.",
  "committee": null,                 // null for floor items; e.g. "Emergency Management" for committee items
  "hearing_time": null,             // "9:30am" / "Upon Call of the Chair" for committee-hearings rows
  "location": "State Capitol, Room 444",
  "item_number": 32,               // int, un-padded
  "supplement": 2,                 // optional, from "Supplemental 2"
  "ics_url": "https://www.assembly.ca.gov/api/ics/generate?name=Budget&start=…",  // where applicable
  "agenda_id": "20361"             // where applicable
}
```

Field-to-row mapping:

- Floor bill rows: `bill_number` ← `floor_bill-measure a href`, `author` ← `floor_bill-author data-sort`, `title` ← `floor_bill-subject text`, `location` ← `floor_bill-location data-sort`, `item_number` ← `floor_bill-item text` (un-pad). `committee` = null, `hearing_time` = null (unless the bill also appears as a committee item on the same file).
- Committee bill rows: same fields plus `committee` ← `committee_bill-committee data-sort`, `item_number` = null.
- Committee hearing rows: no bill — record the hearing metadata itself (date, time, committee, location, `agenda_id`, `ics_url`).
- Senate (from PDF or leginfo alt): same schema, `source=SENATE_DAILY_FILE_PDF` or `LEGINFO_BILL_RESULTS`.

---

## 8. Pitfalls & risk register

1. **Antibot on Senate PDFs.** `senate.ca.gov/dailyfile/publications/download/…`
   behind a `form.antibot` challenge. Programmatic `requests`/browser JS fetch
   yields `HTTP 403` with a 67-byte HTML body (verified 2026-09-04). Mitigation:
   headless browser with a cookie/session pre-warm, or route through the
   `blocked-page-recovery` skill.
2. **No historical date param.** The Assembly page is always current-live.
   "Date" is read from data (hearing table), not written into the URL.
3. **404 ≠ error.** Non-legislative days legitimately return 404 on the Senate
   PDF index. The ingestion layer must classify 404 as "no file today" and
   5xx / retry-able as a transient error.
4. **Session boundary.** When the session changes in January, the URL pattern
   (`bill_id` = `YYYYYYYY` + …) stays the same but bills renumber. No code
   change required — the `session` field in the output reflects the current
   value read from `bill_id`.
5. **Data-sort > textContent.** Always prefer the `data-sort` attribute; it
   is the clean canonical value. `textContent` carries display suffixes and
   leading zeros.
6. **Modal widget in floor-bill rows.** `td.floor_bill-view-details` holds a
   `<dialog>`; do NOT extract it as a field.
7. **Cross-chamber bills.** `AB`-prefilled rows can appear on a Senate daily
   file and vice-versa. `chamber_origin` means *origin*, not publishing house.
8. **Supplement tracking.** A bill can have multiple supplements
   (`Supplemental 2`); the `supplement` field is the edition.
9. **JS-rendered leginfo pages.** `billResultsClient` and `billHistoryClient`
   need a headless browser or JSON endpoint; the static `<table>` they render
   is NOT in the initial HTML.

---

## 9. Sample archive

```
samples/
  assembly_dailyfile_latest.html          # 401 192 bytes, 2026-09-04
                                         # 4 hearing rows, 156 floor-bill rows, 1 committee-bills row
  senate_dailyfile_index.html            # 34 137 bytes, pdf-viewer index
                                         # links to SEN_COM_2025_R_20260902.PDF
  senate_committees_index.html           # 58 642 bytes, Senate committee slugs
  leginfo_assembly_floor_2026-04-16.html # 163 825 bytes, leginfo results page
                                         # billResultsClient, JS-rendered
```

These are the offline test fixtures for `t_fb8081e7`. Every sample was
fetched via `curl` (assembly index, senate index, committee index, leginfo
floor) or `browser_exec` fetch (leginfo floor). The Senate PDF sample is NOT
archived because it is behind antibot — that is itself a documented finding.
See §10 for the acceptance test harness the parser task should run against
these files.

---

## 10. Acceptance test harness (for `t_fb8081e7`)

```yaml
- id: asm_hearings
  input: samples/assembly_dailyfile_latest.html
  selector: tr[class*='committee-hearing-details']
  expect_min_rows: 1
  verified: 4

- id: asm_floor_bills
  input: samples/assembly_dailyfile_latest.html
  selector: tr.class='floor-bill-details'
  expect_min_rows: 100
  expect_fields: [bill_number, author, short_title, item_number]
  verified: 156

- id: asm_committee_bills
  input: samples/assembly_dailyfile_latest.html
  selector: tr[class*='committee-bills-details']
  expect_min_rows: 1
  verified: 1

- id: bill_id_parse
  regex: "^(?P<session>\d{8})\d(?P<chamber>AB|SB|ACR|SCR|HR|SJR|ABR)(?P<number>\d+)$"
  example_url: "https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=202520260AB28"
  expect: {session: "2025-2026", chamber: "AB", number: 28, display: "AB 28"}
  verified: pass

- id: senate_pdf_index
  input: samples/senate_dailyfile_index.html
  expect: "at least one /dailyfile/publications/download/*.PDF reference"
  verified: 1 (SEN_COM_2025_R_20260902.PDF)
```

---

**Downstream handoff:**

- `t_f595542f` (ingestion): read `daily_file_sources.yaml` for URL patterns +
  parameters + antibot flags. Add a Senate PDF download step with a
  `blocked-page-recovery` fallback for 403s.
- `t_fb8081e7` (parser): the §3 and §5 tables are the exact selector / XPath
  / regex contracts. Use `data-sort` over `textContent`. Skip
  `td.floor_bill-view-details`.
- `t_8e5a3821` (RDB schema): the §7 per-record schema is the target. Foreign
  keys to `committee_roster`, `bill_detail`, `bill_history` tables (see §6).