Initial commit

INGESTION.md

@@ -0,0 +1,90 @@
+# Ingestion Layer (t_f595542f)
+
+HTTP ingestion for the CA Legislature Daily File. Built on the verified spec in
+`daily_file_sources.yaml` / `DAILY_FILE_PARSE_SPEC.md` (task t_a171f4d1).
+
+## Layout
+
+```
+scraper/
+  __main__.py      # python -m scraper -> cli.main()
+  cli.py           # argparse entry: ingest / list / check
+  config.py        # loads config.yaml, validates sources, resolves paths
+  http_client.py   # retry + exponential back-off; typed 404/403/5xx errors
+  cache.py         # raw-byte cache (sha256 sidecar) -- re-parse = no network
+  request_log.py   # append-only JSONL audit log (one line per request attempt)
+  alerts.py        # pluggable alert hooks (log / webhook); fired on 404/5xx/403
+  sources.py       # date -> URL resolution (Senate PDF filename decode)
+  ingest.py        # Ingestor() orchestrator + IngestResult
+config.yaml        # single source of config: URLs, retry policy, alert sink
+scripts/
+  run_ingest.sh    # cron wrapper (nightly 02:00)
+  test_ingest_offline.py   # 20 deterministic checks, no real network
+cache/raw/         # <source>/<YYYY-MM-DD>.{html|pdf} + .meta.json (gitignored)
+cache/request_log.jsonl
+```
+
+## Usage
+
+```bash
+# one date, both chambers (downloads + caches):
+python -m scraper ingest --date 2026-09-02
+
+# one chamber only, JSON output:
+python -m scraper ingest --date 2026-09-02 --source senate_daily_file --json
+
+# force a network refresh even if cached:
+python -m scraper ingest --date 2026-09-02 --force
+
+# cache status without touching the network:
+python -m scraper check --date 2026-09-02
+
+# list configured sources + URLs:
+python -m scraper list
+```
+
+## Cron (nightly 02:00 PT)
+
+```crontab
+0 2 * * *    /Users/kyle/Documents/cali_scraper/scripts/run_ingest.sh
+```
+
+`run_ingest.sh` ingests *today* (override with `CALI_SCRAPER_DATE=YYYY-MM-DD`,
+force a refresh with `CALI_SCRAPER_FORCE=1`). Exit code semantics:
+
+| exit | meaning                                                            |
+|------|--------------------------------------------------------------------|
+|  0   | ok / cached / "no publication that day" (expected 404) — all fine |
+|  1   | real failure: antibot 403 / transient 5xx after retries / unexpected 404 |
+|  2   | config or usage error (e.g. invalid --date)                        |
+|  3   | `check` found a chamber missing from the cache                     |
+
+A non-legislative day (weekend / recess) returns a clean expected-404 with **exit
+0** — it is NOT a cron failure. Real outages (5xx / antibot) exit 1 so cron mail
+fires only for problems worth paging on. Every request attempt is appended to
+`cache/request_log.jsonl` as the audit trail that pairs with the alert hooks.
+
+## Key design decisions (read before downstream work)
+
+- **Neither page is date-addressable by URL.** The Assembly page is always the
+  *current live* file (date is a data value read from the hearing table). The
+  Senate "Daily File" lives only as a date-encoded PDF
+  (`SEN_{type}_{year}_R_{YYYYMMDD}.PDF`). So `--date` is the **cache key and the
+  output label**, not a query parameter. The Senate PDF URL is *predicted* from
+  the date and confirmed from the live index page.
+- **Assembly = server-rendered HTML** (safe plain GET). **Senate = PDF behind an
+  antibot 403.** The layer records the 403 as an `antibot` status + alert rather
+  than retry-storming; mitigation is the `blocked-page-recovery` skill or a
+  headless browser session pre-warm. (On a good connection the PDF *can* be
+  served — verified: SEN_COM_2025_R_20260902.PDF, 269,550 bytes, 200.)
+- **404 is not an error on the Senate path.** `missing_is_expected_404` in
+  config.yaml turns a "no publication that day" 404 into a clean, expected,
+  exit-0 outcome. An *unexpected* 404 (e.g. on Assembly) alerts as a warning.
+- **Cache = the network guard.** On a cache hit the Ingestor serves raw bytes
+  from disk and makes zero HTTP calls — this is how "re-parses don't re-hit the
+  network" is guaranteed (proven in test #2).
+- **Downstream handoffs:** t_fb8081e7 (parser) reads `cache/raw/<source>/<date>.html`
+  (Assembly) and `.pdf` (Senate); t_8e5a3821 (QA) can drive `test_ingest_offline.py`
+  and the live `--json` CLI. The parser may add a `--source`-aware fixture loader;
+  the raw bytes are the contract.
+```