81 lines
3.0 KiB
Markdown
81 lines
3.0 KiB
Markdown
# cali_scraper
|
|
|
|
A scraper for the California Legislature's Daily File publications — the
|
|
Assembly and Senate pages that list bills up for hearing — plus a
|
|
relational store and a local web dashboard for browsing the results.
|
|
|
|
## What this is
|
|
|
|
The pipeline has three lanes:
|
|
|
|
1. **Ingest (the scrape)** — `python -m scraper ingest`. Downloads and caches
|
|
the Assembly daily-file HTML page and the Senate daily-file PDF for a given
|
|
date, plus the chambers' "upcoming events" calendars. Raw pages are cached
|
|
on disk (`cache/raw/`), so re-running the same date hits the network zero
|
|
times. A legitimate "no publication that day" (weekend / recess) is treated
|
|
as a normal outcome, not an error.
|
|
2. **Store & analyze** — `parse`, `migrate`, and `correlate` load the cached
|
|
pages into a SQLite database (`db/daily_file.sqlite`, schema in
|
|
`SCHEMA.md`); `analyze` runs subject-matter tagging on bills (built-in
|
|
keyword tagger by default; optionally a local Ollama model, see `llm:` in
|
|
`config.yaml`).
|
|
3. **Dashboard (web server)** — `python -m scraper dashboard`. Serves an API
|
|
plus a static frontend on `http://127.0.0.1:8000/` so you can browse bills,
|
|
committees, members, and tags without touching the database directly.
|
|
|
|
`start.sh` at the repo root runs the scrapes, then starts the dashboard.
|
|
|
|
## How to kick it off
|
|
|
|
### Prerequisites
|
|
|
|
- Python 3.9+
|
|
- Dependencies: `pip install -r requirements.txt`
|
|
- Populate the database at least once so the dashboard has data to show (see
|
|
below — `start.sh` scrapes but does not build `db/daily_file.sqlite`).
|
|
|
|
### Just run it
|
|
|
|
```bash
|
|
./start.sh
|
|
```
|
|
|
|
That scrapes **today's** daily files (plus the events calendars), then serves
|
|
the dashboard at http://127.0.0.1:8000/. Press `Ctrl+C` to stop.
|
|
|
|
### Useful variations
|
|
|
|
```bash
|
|
# scrape a specific date, then serve
|
|
DATE=2026-09-02 ./start.sh
|
|
|
|
# force a network re-fetch even if a date is already cached
|
|
FORCE=1 ./start.sh
|
|
|
|
# serve on a different port / bind address
|
|
PORT=9000 HOST=0.0.0.0 ./start.sh
|
|
|
|
# skip the scrape and just start the dashboard
|
|
SKIP_SCRAPE=1 ./start.sh
|
|
```
|
|
|
|
### Build the database (one time)
|
|
|
|
`start.sh` handles the scrape + server, but to see bills in the dashboard the
|
|
SQLite database must be built from the cached pages:
|
|
|
|
```bash
|
|
python -m scraper ingest --date 2026-09-02 # download + cache
|
|
python -m scraper ingest --date 2026-09-02 --store-events \
|
|
--db db/daily_file.sqlite # events calendar
|
|
python -m scraper parse --input cache/raw/assembly_daily_file \
|
|
--house assembly --sqlite db/daily_file.sqlite # bills table
|
|
python -m scraper migrate --db db/daily_file.sqlite # relational schema
|
|
python -m scraper correlate --db db/daily_file.sqlite # bill <-> committee
|
|
python -m scraper analyze --db db/daily_file.sqlite # subject tags
|
|
```
|
|
|
|
Then `./start.sh` (or `python -m scraper dashboard` directly) will serve it.
|
|
|
|
See `INGESTION.md`, `DAILY_FILE_PARSE_SPEC.md`, `CORRELATE.md`, and
|
|
`LLM_ANALYSIS.md` for the full command reference. |