# Ingestion Layer (t_f595542f)

HTTP ingestion for the CA Legislature Daily File. Built on the verified spec in
`daily_file_sources.yaml` / `DAILY_FILE_PARSE_SPEC.md` (task t_a171f4d1).

## Layout

```
scraper/
  __main__.py      # python -m scraper -> cli.main()
  cli.py           # argparse entry: ingest / list / check
  config.py        # loads config.yaml, validates sources, resolves paths
  http_client.py   # retry + exponential back-off; typed 404/403/5xx errors
  cache.py         # raw-byte cache (sha256 sidecar) -- re-parse = no network
  request_log.py   # append-only JSONL audit log (one line per request attempt)
  alerts.py        # pluggable alert hooks (log / webhook); fired on 404/5xx/403
  sources.py       # date -> URL resolution (Senate PDF filename decode)
  ingest.py        # Ingestor() orchestrator + IngestResult
config.yaml        # single source of config: URLs, retry policy, alert sink
scripts/
  run_ingest.sh    # cron wrapper (nightly 02:00)
  test_ingest_offline.py   # 20 deterministic checks, no real network
cache/raw/         # <source>/<YYYY-MM-DD>.{html|pdf} + .meta.json (gitignored)
cache/request_log.jsonl
```

## Usage

```bash
# one date, both chambers (downloads + caches):
python -m scraper ingest --date 2026-09-02

# one chamber only, JSON output:
python -m scraper ingest --date 2026-09-02 --source senate_daily_file --json

# force a network refresh even if cached:
python -m scraper ingest --date 2026-09-02 --force

# cache status without touching the network:
python -m scraper check --date 2026-09-02

# list configured sources + URLs:
python -m scraper list
```

## Cron (nightly 02:00 PT)

```crontab
0 2 * * *    /Users/kyle/Documents/cali_scraper/scripts/run_ingest.sh
```

`run_ingest.sh` ingests *today* (override with `CALI_SCRAPER_DATE=YYYY-MM-DD`,
force a refresh with `CALI_SCRAPER_FORCE=1`). Exit code semantics:

| exit | meaning                                                            |
|------|--------------------------------------------------------------------|
|  0   | ok / cached / "no publication that day" (expected 404) — all fine |
|  1   | real failure: antibot 403 / transient 5xx after retries / unexpected 404 |
|  2   | config or usage error (e.g. invalid --date)                        |
|  3   | `check` found a chamber missing from the cache                     |

A non-legislative day (weekend / recess) returns a clean expected-404 with **exit
0** — it is NOT a cron failure. Real outages (5xx / antibot) exit 1 so cron mail
fires only for problems worth paging on. Every request attempt is appended to
`cache/request_log.jsonl` as the audit trail that pairs with the alert hooks.

## Key design decisions (read before downstream work)

- **Neither page is date-addressable by URL.** The Assembly page is always the
  *current live* file (date is a data value read from the hearing table). The
  Senate "Daily File" lives only as a date-encoded PDF
  (`SEN_{type}_{year}_R_{YYYYMMDD}.PDF`). So `--date` is the **cache key and the
  output label**, not a query parameter. The Senate PDF URL is *predicted* from
  the date and confirmed from the live index page.
- **Assembly = server-rendered HTML** (safe plain GET). **Senate = PDF behind an
  antibot 403.** The layer records the 403 as an `antibot` status + alert rather
  than retry-storming; mitigation is the `blocked-page-recovery` skill or a
  headless browser session pre-warm. (On a good connection the PDF *can* be
  served — verified: SEN_COM_2025_R_20260902.PDF, 269,550 bytes, 200.)
- **404 is not an error on the Senate path.** `missing_is_expected_404` in
  config.yaml turns a "no publication that day" 404 into a clean, expected,
  exit-0 outcome. An *unexpected* 404 (e.g. on Assembly) alerts as a warning.
- **Cache = the network guard.** On a cache hit the Ingestor serves raw bytes
  from disk and makes zero HTTP calls — this is how "re-parses don't re-hit the
  network" is guaranteed (proven in test #2).
- **Downstream handoffs:** t_fb8081e7 (parser) reads `cache/raw/<source>/<date>.html`
  (Assembly) and `.pdf` (Senate); t_8e5a3821 (QA) can drive `test_ingest_offline.py`
  and the live `--json` CLI. The parser may add a `--source`-aware fixture loader;
  the raw bytes are the contract.
```