cali_scraper/INGESTION.md

Ingestion Layer (t_f595542f)

HTTP ingestion for the CA Legislature Daily File. Built on the verified spec in daily_file_sources.yaml / DAILY_FILE_PARSE_SPEC.md (task t_a171f4d1).

Layout

scraper/
  __main__.py      # python -m scraper -> cli.main()
  cli.py           # argparse entry: ingest / list / check
  config.py        # loads config.yaml, validates sources, resolves paths
  http_client.py   # retry + exponential back-off; typed 404/403/5xx errors
  cache.py         # raw-byte cache (sha256 sidecar) -- re-parse = no network
  request_log.py   # append-only JSONL audit log (one line per request attempt)
  alerts.py        # pluggable alert hooks (log / webhook); fired on 404/5xx/403
  sources.py       # date -> URL resolution (Senate PDF filename decode)
  ingest.py        # Ingestor() orchestrator + IngestResult
config.yaml        # single source of config: URLs, retry policy, alert sink
scripts/
  run_ingest.sh    # cron wrapper (nightly 02:00)
  test_ingest_offline.py   # 20 deterministic checks, no real network
cache/raw/         # <source>/<YYYY-MM-DD>.{html|pdf} + .meta.json (gitignored)
cache/request_log.jsonl

Usage

# one date, both chambers (downloads + caches):
python -m scraper ingest --date 2026-09-02

# one chamber only, JSON output:
python -m scraper ingest --date 2026-09-02 --source senate_daily_file --json

# force a network refresh even if cached:
python -m scraper ingest --date 2026-09-02 --force

# cache status without touching the network:
python -m scraper check --date 2026-09-02

# list configured sources + URLs:
python -m scraper list

Cron (nightly 02:00 PT)

0 2 * * *    /Users/kyle/Documents/cali_scraper/scripts/run_ingest.sh

run_ingest.sh ingests today (override with CALI_SCRAPER_DATE=YYYY-MM-DD, force a refresh with CALI_SCRAPER_FORCE=1). Exit code semantics:

exit	meaning
0	ok / cached / "no publication that day" (expected 404) — all fine
1	real failure: antibot 403 / transient 5xx after retries / unexpected 404
2	config or usage error (e.g. invalid --date)
3	check found a chamber missing from the cache

A non-legislative day (weekend / recess) returns a clean expected-404 with exit 0 — it is NOT a cron failure. Real outages (5xx / antibot) exit 1 so cron mail fires only for problems worth paging on. Every request attempt is appended to cache/request_log.jsonl as the audit trail that pairs with the alert hooks.

Key design decisions (read before downstream work)

• Neither page is date-addressable by URL. The Assembly page is always the current live file (date is a data value read from the hearing table). The Senate "Daily File" lives only as a date-encoded PDF (SEN_{type}_{year}_R_{YYYYMMDD}.PDF). So --date is the cache key and the output label, not a query parameter. The Senate PDF URL is predicted from the date and confirmed from the live index page.
• Assembly = server-rendered HTML (safe plain GET). Senate = PDF behind an antibot 403. The layer records the 403 as an antibot status + alert rather than retry-storming; mitigation is the blocked-page-recovery skill or a headless browser session pre-warm. (On a good connection the PDF can be served — verified: SEN_COM_2025_R_20260902.PDF, 269,550 bytes, 200.)
• 404 is not an error on the Senate path. missing_is_expected_404 in config.yaml turns a "no publication that day" 404 into a clean, expected, exit-0 outcome. An unexpected 404 (e.g. on Assembly) alerts as a warning.
• Cache = the network guard. On a cache hit the Ingestor serves raw bytes from disk and makes zero HTTP calls — this is how "re-parses don't re-hit the network" is guaranteed (proven in test #2).
• Downstream handoffs: t_fb8081e7 (parser) reads cache/raw/<source>/<date>.html (Assembly) and .pdf (Senate); t_8e5a3821 (QA) can drive test_ingest_offline.py and the live --json CLI. The parser may add a --source-aware fixture loader; the raw bytes are the contract.