# cali_scraper

A scraper for the California Legislature's Daily File publications — the
Assembly and Senate pages that list bills up for hearing — plus a
relational store and a local web dashboard for browsing the results.

## What this is

The pipeline has three lanes:

1. **Ingest (the scrape)** — `python -m scraper ingest`. Downloads and caches
   the Assembly daily-file HTML page and the Senate daily-file PDF for a given
   date, plus the chambers' "upcoming events" calendars. Raw pages are cached
   on disk (`cache/raw/`), so re-running the same date hits the network zero
   times. A legitimate "no publication that day" (weekend / recess) is treated
   as a normal outcome, not an error.
2. **Store & analyze** — `parse`, `migrate`, and `correlate` load the cached
   pages into a SQLite database (`db/daily_file.sqlite`, schema in
   `SCHEMA.md`); `analyze` runs subject-matter tagging on bills (built-in
   keyword tagger by default; optionally a local Ollama model, see `llm:` in
   `config.yaml`).
3. **Dashboard (web server)** — `python -m scraper dashboard`. Serves an API
   plus a static frontend on `http://127.0.0.1:8000/` so you can browse bills,
   committees, members, and tags without touching the database directly.

`start.sh` at the repo root runs the scrapes, then starts the dashboard.

## How to kick it off

### Prerequisites

- Python 3.9+
- Dependencies: `pip install -r requirements.txt`
- Populate the database at least once so the dashboard has data to show (see
  below — `start.sh` scrapes but does not build `db/daily_file.sqlite`).

### Just run it

```bash
./start.sh
```

That scrapes **today's** daily files (plus the events calendars), then serves
the dashboard at http://127.0.0.1:8000/. Press `Ctrl+C` to stop.

### Useful variations

```bash
# scrape a specific date, then serve
DATE=2026-09-02 ./start.sh

# force a network re-fetch even if a date is already cached
FORCE=1 ./start.sh

# serve on a different port / bind address
PORT=9000 HOST=0.0.0.0 ./start.sh

# skip the scrape and just start the dashboard
SKIP_SCRAPE=1 ./start.sh
```

### Build the database (one time)

`start.sh` handles the scrape + server, but to see bills in the dashboard the
SQLite database must be built from the cached pages:

```bash
python -m scraper ingest --date 2026-09-02                    # download + cache
python -m scraper ingest --date 2026-09-02 --store-events \
    --db db/daily_file.sqlite                                  # events calendar
python -m scraper parse --input cache/raw/assembly_daily_file \
    --house assembly --sqlite db/daily_file.sqlite             # bills table
python -m scraper migrate --db db/daily_file.sqlite            # relational schema
python -m scraper correlate --db db/daily_file.sqlite          # bill <-> committee
python -m scraper analyze --db db/daily_file.sqlite            # subject tags
```

Then `./start.sh` (or `python -m scraper dashboard` directly) will serve it.

See `INGESTION.md`, `DAILY_FILE_PARSE_SPEC.md`, `CORRELATE.md`, and
`LLM_ANALYSIS.md` for the full command reference.