cali_scraper/SCHEMA.md

Relational Schema — California Legislature Daily File

Purpose

The pipeline stores its canonical output as a flat daily_file_records SQLite table (one BillAction per row, 20 columns). QA and downstream consumers depend on that contract. This schema extends — never mutates — the flat table with a normalised relational structure that strongly relates bills ↔ committees ↔ committee members, per spec §2.5.

Design principle: preserve the flat contract

The flat table daily_file_records is preserved byte-for-byte. A CREATE TABLE IF NOT EXISTS creates it only when absent; migrate() never ALTERs or DROPs it.

The v_daily_file_records view reproduces the table's exact column set and row set, so any existing or future consumer reading the view behaves identically to one reading the table directly.

Tables

All DDL uses CREATE TABLE IF NOT EXISTS + PRAGMA foreign_keys = ON. Natural-key UNIQUE constraints make re-runs idempotent (INSERT OR IGNORE).

bills

Column	Type	Constraint
id	INTEGER	PK
bill_id	TEXT	NOT NULL, part of UNI
bill_number	TEXT
session	TEXT	NOT NULL DEFAULT ''
chamber_origin	TEXT	NOT NULL DEFAULT 'unknown'
house	TEXT
author	TEXT
short_title	TEXT
first_seen	TEXT
source	TEXT
UNIQUE		(bill_id, session, chamber_origin)

Derived from daily_file_records rows where bill_id IS NOT NULL.

committees

Column	Type	Constraint
id	INTEGER	PK
name_key	TEXT	NOT NULL, part of UNI
name_raw	TEXT	NOT NULL
session	TEXT	NOT NULL DEFAULT ''
chamber	TEXT	NOT NULL DEFAULT 'unknown'
hearing_time	TEXT
location	TEXT
source	TEXT
UNIQUE		(name_key, session, chamber)

Derived from two sources:

• committee field on hearing/committee-bill rows
• author field starting with "Committee on …" (committee-authored bills)

name_key = slug(name) — lowercased, non-alphanumerics replaced with hyphens — so different raw spellings map to the same committee.

members

Column	Type	Constraint
id	INTEGER	PK
name_norm	TEXT	NOT NULL, part of UNI
full_name	TEXT
chamber	TEXT	NOT NULL DEFAULT 'unknown'
kind	TEXT	NOT NULL DEFAULT 'roster'
source	TEXT
UNIQUE		(name_norm, chamber, kind)

Populated by two real, non-fabricated sources:

• kind = 'author-seed' — a bill author is a real legislator; the leading name before " et al." is the legislator.
• kind = 'roster' — a committee-roster JSON file.

committee_members

Column	Type	Constraint
id	INTEGER	PK
committee	TEXT	NOT NULL, part of UNI (FK → committees.name_key)
member	TEXT	NOT NULL, part of UNI (FK → members.name_norm)
session	TEXT	NOT NULL DEFAULT ''
chamber	TEXT	NOT NULL DEFAULT 'unknown'
role	TEXT
source	TEXT
UNIQUE		(committee, member, session, chamber)

bill_committees

Column	Type	Constraint
id	INTEGER	PK
bill_id	TEXT	NOT NULL, part of UNI (FK → bills.bill_id)
committee	TEXT	NOT NULL, part of UNI (FK → committees.name_key)
session	TEXT	NOT NULL DEFAULT ''
chamber	TEXT	NOT NULL DEFAULT 'unknown'
source	TEXT
UNIQUE		(bill_id, committee, session, chamber)

schema_migrations

Column	Type	Constraint
version	INTEGER	PK
applied_at	TEXT
note	TEXT

Stamped at each migrate() run.

v_daily_file_records (view)

A CREATE VIEW that mirrors daily_file_records exactly — same column names, same column order, same row set — so no consumer must change its query from daily_file_records to v_daily_file_records.

Foreign keys

PRAGMA foreign_keys = ON is set on every connection opened by relational.connect(). The FK constraints are declarative in DDL but are only enforced when FK pragma is active (SQLite behaviour).

Never-fabricate rule

Committees with no matching roster record have an empty member set in committee_members and a WARN is logged by name. No members are invented. This is testable: test_committee_without_roster_has_no_members asserts COUNT(*) = 0 for a committee that has no roster.