Initial commit
This commit is contained in:
kfj001 committed
commit
5f2db08b43
100 files changed
+16985
No files matched your search
@@ -0,0 +1,135 @@
|
||||
# Relational Schema — California Legislature Daily File
|
||||
|
||||
## Purpose
|
||||
|
||||
The pipeline stores its canonical output as a **flat** `daily_file_records`
|
||||
SQLite table (one `BillAction` per row, 20 columns). QA and downstream
|
||||
consumers depend on that contract. This schema **extends** — never mutates
|
||||
— the flat table with a normalised relational structure that strongly
|
||||
relates **bills ↔ committees ↔ committee members**, per spec §2.5.
|
||||
|
||||
## Design principle: preserve the flat contract
|
||||
|
||||
The flat table `daily_file_records` is preserved byte-for-byte.
|
||||
A `CREATE TABLE IF NOT EXISTS` creates it only when absent;
|
||||
`migrate()` never `ALTER`s or `DROP`s it.
|
||||
|
||||
The `v_daily_file_records` **view** reproduces the table's exact column
|
||||
set and row set, so any existing or future consumer reading the view
|
||||
behaves identically to one reading the table directly.
|
||||
|
||||
## Tables
|
||||
|
||||
All DDL uses `CREATE TABLE IF NOT EXISTS` + `PRAGMA foreign_keys = ON`.
|
||||
Natural-key `UNIQUE` constraints make re-runs idempotent
|
||||
(`INSERT OR IGNORE`).
|
||||
|
||||
### `bills`
|
||||
|
||||
| Column | Type | Constraint |
|
||||
|----------------|-----------------|-----------------------|
|
||||
| id | INTEGER | PK |
|
||||
| bill_id | TEXT | NOT NULL, part of UNI |
|
||||
| bill_number | TEXT | |
|
||||
| session | TEXT | NOT NULL DEFAULT '' |
|
||||
| chamber_origin | TEXT | NOT NULL DEFAULT 'unknown' |
|
||||
| house | TEXT | |
|
||||
| author | TEXT | |
|
||||
| short_title | TEXT | |
|
||||
| first_seen | TEXT | |
|
||||
| source | TEXT | |
|
||||
| **UNIQUE** | | `(bill_id, session, chamber_origin)` |
|
||||
|
||||
Derived from `daily_file_records` rows where `bill_id IS NOT NULL`.
|
||||
|
||||
### `committees`
|
||||
|
||||
| Column | Type | Constraint |
|
||||
|--------------|-----------------|---------------------------|
|
||||
| id | INTEGER | PK |
|
||||
| name_key | TEXT | NOT NULL, part of UNI |
|
||||
| name_raw | TEXT | NOT NULL |
|
||||
| session | TEXT | NOT NULL DEFAULT '' |
|
||||
| chamber | TEXT | NOT NULL DEFAULT 'unknown' |
|
||||
| hearing_time | TEXT | |
|
||||
| location | TEXT | |
|
||||
| source | TEXT | |
|
||||
| **UNIQUE** | | `(name_key, session, chamber)` |
|
||||
|
||||
Derived from two sources:
|
||||
* `committee` field on hearing/committee-bill rows
|
||||
* `author` field starting with "Committee on …" (committee-authored bills)
|
||||
|
||||
`name_key` = `slug(name)` — lowercased, non-alphanumerics replaced with
|
||||
hyphens — so different raw spellings map to the same committee.
|
||||
|
||||
### `members`
|
||||
|
||||
| Column | Type | Constraint |
|
||||
|------------|-----------------|-------------------------------|
|
||||
| id | INTEGER | PK |
|
||||
| name_norm | TEXT | NOT NULL, part of UNI |
|
||||
| full_name | TEXT | |
|
||||
| chamber | TEXT | NOT NULL DEFAULT 'unknown' |
|
||||
| kind | TEXT | NOT NULL DEFAULT 'roster' |
|
||||
| source | TEXT | |
|
||||
| **UNIQUE** | | `(name_norm, chamber, kind)` |
|
||||
|
||||
Populated by two **real, non-fabricated** sources:
|
||||
* `kind = 'author-seed'` — a bill author is a real legislator;
|
||||
the leading name before " et al." is the legislator.
|
||||
* `kind = 'roster'` — a committee-roster JSON file.
|
||||
|
||||
### `committee_members`
|
||||
|
||||
| Column | Type | Constraint |
|
||||
|-----------|-----------------|----------------------------------|
|
||||
| id | INTEGER | PK |
|
||||
| committee | TEXT | NOT NULL, part of UNI (FK → committees.name_key) |
|
||||
| member | TEXT | NOT NULL, part of UNI (FK → members.name_norm) |
|
||||
| session | TEXT | NOT NULL DEFAULT '' |
|
||||
| chamber | TEXT | NOT NULL DEFAULT 'unknown' |
|
||||
| role | TEXT | |
|
||||
| source | TEXT | |
|
||||
| **UNIQUE**| | `(committee, member, session, chamber)` |
|
||||
|
||||
### `bill_committees`
|
||||
|
||||
| Column | Type | Constraint |
|
||||
|-----------|-----------------|---------------------------------------|
|
||||
| id | INTEGER | PK |
|
||||
| bill_id | TEXT | NOT NULL, part of UNI (FK → bills.bill_id) |
|
||||
| committee | TEXT | NOT NULL, part of UNI (FK → committees.name_key) |
|
||||
| session | TEXT | NOT NULL DEFAULT '' |
|
||||
| chamber | TEXT | NOT NULL DEFAULT 'unknown' |
|
||||
| source | TEXT | |
|
||||
| **UNIQUE**| | `(bill_id, committee, session, chamber)` |
|
||||
|
||||
### `schema_migrations`
|
||||
|
||||
| Column | Type | Constraint |
|
||||
|------------|-----------|----------------|
|
||||
| version | INTEGER | PK |
|
||||
| applied_at | TEXT | |
|
||||
| note | TEXT | |
|
||||
|
||||
Stamped at each `migrate()` run.
|
||||
|
||||
### `v_daily_file_records` (view)
|
||||
|
||||
A `CREATE VIEW` that mirrors `daily_file_records` exactly — same column
|
||||
names, same column order, same row set — so no consumer must change its
|
||||
query from `daily_file_records` to `v_daily_file_records`.
|
||||
|
||||
## Foreign keys
|
||||
|
||||
`PRAGMA foreign_keys = ON` is set on every connection opened by
|
||||
`relational.connect()`. The FK constraints are declarative in DDL
|
||||
but are only enforced when FK pragma is active (SQLite behaviour).
|
||||
|
||||
## Never-fabricate rule
|
||||
|
||||
Committees with no matching roster record have an **empty** member set
|
||||
in `committee_members` and a WARN is logged by name. No members are
|
||||
invented. This is testable: `test_committee_without_roster_has_no_members`
|
||||
asserts `COUNT(*) = 0` for a committee that has no roster.
|
||||
Reference in new issue
Block a user