102 lines
4.3 KiB
Markdown
102 lines
4.3 KiB
Markdown
# Correlation — bills ↔ committees ↔ members
|
|
|
|
## Commands
|
|
|
|
```bash
|
|
# Step 1: parse cached HTML into the flat table (parser lane — already done)
|
|
python -m scraper parse --input cache/raw --db db/daily_file.sqlite
|
|
|
|
# Step 2: create the relational schema (idempotent)
|
|
python -m scraper migrate --db db/daily_file.sqlite
|
|
|
|
# Step 3: derive bill<->committee<->member relations from the flat table
|
|
python -m scraper correlate --db db/daily_file.sqlite
|
|
|
|
# Optional: backfill committee membership from a roster JSON file
|
|
python -m scraper correlate --db db/daily_file.sqlite --roster rosters.json
|
|
|
|
# Or: one-shot — migrate then correlate in process order
|
|
python -m scraper migrate --db db/daily_file.sqlite && \
|
|
python -m scraper correlate --db db/daily_file.sqlite
|
|
```
|
|
|
|
## What correlate derives
|
|
|
|
| Source column / field | Target table | Rule |
|
|
|---------------------------------|----------------------|---------------------------------------------------|
|
|
| `bill_id` (any row) | `bills` | One bill row per unique `(bill_id, session, chamber_origin)` |
|
|
| `committee` on hearing / committee-bill rows | `committees` | One committee row per unique slug |
|
|
| `author LIKE 'Committee on …'` | `committees` | Strips the "Committee on " prefix; same slug rules |
|
|
| `bill_id AND committee` non-null | `bill_committees` | The "reviewed by" edge: bill was acted on by this committee |
|
|
| `author` = "Name" or "Name et al." (not committee) | `members` | `kind = 'author-seed'`; real legislator name, never a guess |
|
|
| roster JSON `members[]` | `committee_members` | `kind = 'roster'`; only when a roster is supplied |
|
|
|
|
## Never-fabricate rule
|
|
|
|
* `committee_members` is populated **only** from a supplied roster file
|
|
(`--roster rosters.json`).
|
|
* When a committee seen in the daily file has **no roster**, it is still
|
|
created in `committees` but its membership is left **empty**.
|
|
* A WARN is logged for each committee missing a roster.
|
|
* No member is ever invented. `test_committee_without_roster_has_no_members`
|
|
asserts `COUNT(*) = 0` for a committee with no roster.
|
|
|
|
The roster JSON format:
|
|
|
|
```json
|
|
[
|
|
{
|
|
"committee": "Emergency Management",
|
|
"session": "2025-2026",
|
|
"chamber": "senate",
|
|
"members": [
|
|
{"name": "Perez", "role": "Chair"},
|
|
{"name": "Davis", "role": "Member"}
|
|
]
|
|
}
|
|
]
|
|
```
|
|
|
|
## Roster keying
|
|
|
|
`load_rosters()` returns a dict keyed by
|
|
`(slug, session, chamber)` — three fields, not just the slug — so
|
|
a committee appearing in two different `(session, chamber)` contexts
|
|
keeps separate membership sets instead of clobbering.
|
|
|
|
`_ingest_relations()` builds a `roster_slugs` set from that dict. A
|
|
committee `(slug, session, chamber)` slot is reported "missing a roster"
|
|
when **no roster block covers that slug at all**, regardless of
|
|
`(session, chamber)`. This allows one roster file to backfill
|
|
multiple slots for the same slug.
|
|
|
|
## Idempotency
|
|
|
|
| Operation | Idempotency mechanism |
|
|
|--------------|------------------------------------------------|
|
|
| `migrate()` | All DDL is `CREATE … IF NOT EXISTS`; |
|
|
| | `schema_migrations` uses `INSERT OR IGNORE` |
|
|
| `correlate()`| All `INSERT` is `INSERT OR IGNORE` on natural |
|
|
| | keys; re-running produces zero new rows |
|
|
| `--rebuild` | `DELETE FROM` all 5 derived tables, then re- |
|
|
| | correlate — a clean deterministic rebuild |
|
|
|
|
## Roster-driven slot matching
|
|
|
|
The chamber axis defaults to `chamber_origin` (for bill rows, the
|
|
origin house of the bill) and falls back to `house` (the publishing
|
|
chamber) when `chamber_origin` is absent. This matters for hearing
|
|
rows, which have `chamber_origin = NULL` but `house = 'assembly'` —
|
|
without the fallback the hearing slot would get `chamber = 'unknown'`
|
|
and never match a roster entry with `chamber = 'assembly'`.
|
|
|
|
## CLI registration
|
|
|
|
`scraper/cli.py` is a shared hotspot co-owned by the ingestion lane
|
|
(`ingest`/`list`/`check`) and the parser lane (`parse`). This lane
|
|
registers `migrate` and `correlate` **additively**, through
|
|
`scraper/relate_cli.py::register()` called in a `try/except` block
|
|
in `cli.py::build_parser()` — exactly the same guarded pattern that
|
|
`parse_cli.py::register()` uses. The new subcommand names
|
|
(`migrate`, `correlate`) do not collide with any sibling owner.
|