A scraper for the California Legislature's Daily File publications — the Assembly and Senate pages that list bills up for hearing — plus a relational store and a local web dashboard for browsing the results.

What this is

The pipeline has three lanes:

Ingest (the scrape) — python -m scraper ingest. Downloads and caches the Assembly daily-file HTML page and the Senate daily-file PDF for a given date, plus the chambers' "upcoming events" calendars. Raw pages are cached on disk (cache/raw/), so re-running the same date hits the network zero times. A legitimate "no publication that day" (weekend / recess) is treated as a normal outcome, not an error.
Store & analyze — parse, migrate, and correlate load the cached pages into a SQLite database (db/daily_file.sqlite, schema in SCHEMA.md); analyze runs subject-matter tagging on bills (built-in keyword tagger by default; optionally a local Ollama model, see llm: in config.yaml).
Dashboard (web server) — python -m scraper dashboard. Serves an API plus a static frontend on http://127.0.0.1:8000/ so you can browse bills, committees, members, and tags without touching the database directly.

start.sh at the repo root runs the scrapes, then starts the dashboard.

How to kick it off

Prerequisites

Python 3.9+
Dependencies: pip install -r requirements.txt
Populate the database at least once so the dashboard has data to show (see below — start.sh scrapes but does not build db/daily_file.sqlite).

Just run it

./start.sh

That scrapes today's daily files (plus the events calendars), then serves the dashboard at http://127.0.0.1:8000/. Press Ctrl+C to stop.

Useful variations

# scrape a specific date, then serve
DATE=2026-09-02 ./start.sh

# force a network re-fetch even if a date is already cached
FORCE=1 ./start.sh

# serve on a different port / bind address
PORT=9000 HOST=0.0.0.0 ./start.sh

# skip the scrape and just start the dashboard
SKIP_SCRAPE=1 ./start.sh

Build the database (one time)

start.sh handles the scrape + server, but to see bills in the dashboard the SQLite database must be built from the cached pages:

python -m scraper ingest --date 2026-09-02                    # download + cache
python -m scraper ingest --date 2026-09-02 --store-events \
    --db db/daily_file.sqlite                                  # events calendar
python -m scraper parse --input cache/raw/assembly_daily_file \
    --house assembly --sqlite db/daily_file.sqlite             # bills table
python -m scraper migrate --db db/daily_file.sqlite            # relational schema
python -m scraper correlate --db db/daily_file.sqlite          # bill <-> committee
python -m scraper analyze --db db/daily_file.sqlite            # subject tags

Then ./start.sh (or python -m scraper dashboard directly) will serve it.

See INGESTION.md, DAILY_FILE_PARSE_SPEC.md, CORRELATE.md, and LLM_ANALYSIS.md for the full command reference.