136 lines
6.1 KiB
Markdown
136 lines
6.1 KiB
Markdown
# Relational Schema — California Legislature Daily File
|
|
|
|
## Purpose
|
|
|
|
The pipeline stores its canonical output as a **flat** `daily_file_records`
|
|
SQLite table (one `BillAction` per row, 20 columns). QA and downstream
|
|
consumers depend on that contract. This schema **extends** — never mutates
|
|
— the flat table with a normalised relational structure that strongly
|
|
relates **bills ↔ committees ↔ committee members**, per spec §2.5.
|
|
|
|
## Design principle: preserve the flat contract
|
|
|
|
The flat table `daily_file_records` is preserved byte-for-byte.
|
|
A `CREATE TABLE IF NOT EXISTS` creates it only when absent;
|
|
`migrate()` never `ALTER`s or `DROP`s it.
|
|
|
|
The `v_daily_file_records` **view** reproduces the table's exact column
|
|
set and row set, so any existing or future consumer reading the view
|
|
behaves identically to one reading the table directly.
|
|
|
|
## Tables
|
|
|
|
All DDL uses `CREATE TABLE IF NOT EXISTS` + `PRAGMA foreign_keys = ON`.
|
|
Natural-key `UNIQUE` constraints make re-runs idempotent
|
|
(`INSERT OR IGNORE`).
|
|
|
|
### `bills`
|
|
|
|
| Column | Type | Constraint |
|
|
|----------------|-----------------|-----------------------|
|
|
| id | INTEGER | PK |
|
|
| bill_id | TEXT | NOT NULL, part of UNI |
|
|
| bill_number | TEXT | |
|
|
| session | TEXT | NOT NULL DEFAULT '' |
|
|
| chamber_origin | TEXT | NOT NULL DEFAULT 'unknown' |
|
|
| house | TEXT | |
|
|
| author | TEXT | |
|
|
| short_title | TEXT | |
|
|
| first_seen | TEXT | |
|
|
| source | TEXT | |
|
|
| **UNIQUE** | | `(bill_id, session, chamber_origin)` |
|
|
|
|
Derived from `daily_file_records` rows where `bill_id IS NOT NULL`.
|
|
|
|
### `committees`
|
|
|
|
| Column | Type | Constraint |
|
|
|--------------|-----------------|---------------------------|
|
|
| id | INTEGER | PK |
|
|
| name_key | TEXT | NOT NULL, part of UNI |
|
|
| name_raw | TEXT | NOT NULL |
|
|
| session | TEXT | NOT NULL DEFAULT '' |
|
|
| chamber | TEXT | NOT NULL DEFAULT 'unknown' |
|
|
| hearing_time | TEXT | |
|
|
| location | TEXT | |
|
|
| source | TEXT | |
|
|
| **UNIQUE** | | `(name_key, session, chamber)` |
|
|
|
|
Derived from two sources:
|
|
* `committee` field on hearing/committee-bill rows
|
|
* `author` field starting with "Committee on …" (committee-authored bills)
|
|
|
|
`name_key` = `slug(name)` — lowercased, non-alphanumerics replaced with
|
|
hyphens — so different raw spellings map to the same committee.
|
|
|
|
### `members`
|
|
|
|
| Column | Type | Constraint |
|
|
|------------|-----------------|-------------------------------|
|
|
| id | INTEGER | PK |
|
|
| name_norm | TEXT | NOT NULL, part of UNI |
|
|
| full_name | TEXT | |
|
|
| chamber | TEXT | NOT NULL DEFAULT 'unknown' |
|
|
| kind | TEXT | NOT NULL DEFAULT 'roster' |
|
|
| source | TEXT | |
|
|
| **UNIQUE** | | `(name_norm, chamber, kind)` |
|
|
|
|
Populated by two **real, non-fabricated** sources:
|
|
* `kind = 'author-seed'` — a bill author is a real legislator;
|
|
the leading name before " et al." is the legislator.
|
|
* `kind = 'roster'` — a committee-roster JSON file.
|
|
|
|
### `committee_members`
|
|
|
|
| Column | Type | Constraint |
|
|
|-----------|-----------------|----------------------------------|
|
|
| id | INTEGER | PK |
|
|
| committee | TEXT | NOT NULL, part of UNI (FK → committees.name_key) |
|
|
| member | TEXT | NOT NULL, part of UNI (FK → members.name_norm) |
|
|
| session | TEXT | NOT NULL DEFAULT '' |
|
|
| chamber | TEXT | NOT NULL DEFAULT 'unknown' |
|
|
| role | TEXT | |
|
|
| source | TEXT | |
|
|
| **UNIQUE**| | `(committee, member, session, chamber)` |
|
|
|
|
### `bill_committees`
|
|
|
|
| Column | Type | Constraint |
|
|
|-----------|-----------------|---------------------------------------|
|
|
| id | INTEGER | PK |
|
|
| bill_id | TEXT | NOT NULL, part of UNI (FK → bills.bill_id) |
|
|
| committee | TEXT | NOT NULL, part of UNI (FK → committees.name_key) |
|
|
| session | TEXT | NOT NULL DEFAULT '' |
|
|
| chamber | TEXT | NOT NULL DEFAULT 'unknown' |
|
|
| source | TEXT | |
|
|
| **UNIQUE**| | `(bill_id, committee, session, chamber)` |
|
|
|
|
### `schema_migrations`
|
|
|
|
| Column | Type | Constraint |
|
|
|------------|-----------|----------------|
|
|
| version | INTEGER | PK |
|
|
| applied_at | TEXT | |
|
|
| note | TEXT | |
|
|
|
|
Stamped at each `migrate()` run.
|
|
|
|
### `v_daily_file_records` (view)
|
|
|
|
A `CREATE VIEW` that mirrors `daily_file_records` exactly — same column
|
|
names, same column order, same row set — so no consumer must change its
|
|
query from `daily_file_records` to `v_daily_file_records`.
|
|
|
|
## Foreign keys
|
|
|
|
`PRAGMA foreign_keys = ON` is set on every connection opened by
|
|
`relational.connect()`. The FK constraints are declarative in DDL
|
|
but are only enforced when FK pragma is active (SQLite behaviour).
|
|
|
|
## Never-fabricate rule
|
|
|
|
Committees with no matching roster record have an **empty** member set
|
|
in `committee_members` and a WARN is logged by name. No members are
|
|
invented. This is testable: `test_committee_without_roster_has_no_members`
|
|
asserts `COUNT(*) = 0` for a committee that has no roster.
|