scraper/CLAUDE.md

CLAUDE.md — Guardian Daily Newscast Pipeline Guide

Project Overview

A Python pipeline that turns The Guardian's daily US news into MP3 newscast audio files.

Three scripts run in sequence:

1. main.py — Scrapes Guardian articles (outputs data/output.json)
2. reader.py — Generates teleprompter scripts via local LLM (outputs data/scripts.json)
3. tts_generator.py — Converts scripts to MP3 audio (outputs data/audio/*.mp3)

Coding Standards

• Language: Python 3.x
• Style: PEP 8 compliant.
• Async: Use asyncio and httpx.AsyncClient. Always ensure clients are closed properly (e.g., via await client.close()).
• Logging: Use loguru for all logging.
• Typing: Use Python type hints throughout.
• Data Validation: Use Pydantic models for any structured data.

Architecture & Key Patterns

Pipeline Overview

[Stage 1] main.py (Guardian Scraper)  --> data/output.json
[Stage 2] reader.py (LLM Script Gen)  --> data/scripts.json
[Stage 3] tts_generator.py (TTS)      --> data/audio/*.mp3

Each stage reads the previous stage's output file. Stages are independent and can be re-run in any order (but logically must follow 1 → 2 → 3).

Stage 1: Guardian Scraper (main.py + scraper/)

Client-Parser-Model Pattern:

• GuardianClient (scraper/client.py): Handles HTTP networking, rate limits, and concurrency via asyncio.Semaphore. Implements exponential backoff for 429/5xx responses.
• GuardianParser (scraper/parser.py): Pure functions for HTML extraction. Body fallback logic: (1) div[data-gu-context="body"], (2) <article>, (3) <main>.
• Article / ScrapeResult (scraper/models.py): Pydantic models for data consistency.
• utils.py: Disk I/O (save_results_to_json).

Stage 2: LLM Script Generator (reader.py)

• Sends each article body to Ollama (http://127.0.0.1:11434/api/generate, model gemma4:e2b).
• System prompt enforces clean teleprompter-ready text (no stage directions, no speaker labels).
• Retry logic: up to 3 attempts with exponential backoff (10s, 20s). 2s delay between requests.
• Incremental save to data/scripts.json during processing.

Stage 3: TTS Audio Generator (tts_generator.py)

• KokoroTTS: Wraps Kokoro KPipeline with voice af_heart, outputs WAV at 24kHz.
• convert_wav_to_mp3: ffmpeg subprocess (-q:a 2).
• ScriptItem: Pydantic model validating input from data/scripts.json.
• Concurrent processing with asyncio.Semaphore(CONCURRENCY_LIMIT=1).
• Cleans up intermediate WAV files after conversion.

DevContainer (.devcontainer/)

A VS Code Dev Container enables reproducible development. Two files define it:

• Dockerfile (based on python:3.11-slim):

  • Installs system deps: git, curl, ffmpeg.
  • Copies and installs requirements.txt (project root) into the container at /app/.
  • Sets default env vars: OLLAMA_URL, MODEL_NAME, PORT, POLL_INTERVAL, DATA_DIR, TTS_VOICE.
  • Creates /app/data/audio/ for output.

• devcontainer.json:

  • Points VS Code to the Dockerfile.
  • Forwards port 8080 (newscast API).
  • postCreateCommand: runs pip install -r requirements.txt in the VS Code Python interpreter for IDE features (linting, completions).
  • remoteEnv: overrides Dockerfile defaults with workspace-relative values (e.g., DATA_DIR points to ${workspaceFolder}/data).
  • Extensions: ms-python.python, ms-python.vscode-pylance.

Key env vars (Dockerfile defaults, overridable via devcontainer.json remoteEnv):

Variable	Default	Description
OLLAMA_URL	http://host.docker.internal:11434/api/generate	Ollama API endpoint (use host.docker.internal to reach host from container)
MODEL_NAME	gemma4:e2b	Ollama model identifier
PORT	8080	Newscast API listen port
POLL_INTERVAL	21600	API polling interval in seconds (6 hours)
DATA_DIR	/app/data	Base directory for all pipeline output files
TTS_VOICE	af_heart	Kokoro TTS voice

Common Tasks

Add a New Pipeline Stage

1. Create a new script (e.g., stage_name.py) in the project root.
2. Read the previous stage's output file.
3. Write results to a new output file.
4. Add it to the && chain in README.md Quick Start.
5. Update the pipeline diagram in this document.
6. Add the output file to the Output Files table.

Change Target URL (Stage 1)

Update the url variable in main.py (line 16). The URL is built dynamically from today's date:

url = f"https://www.theguardian.com/us-news/{year}/{month}/{day}/all"

Modify Concurrency (Stage 1)

Change the concurrency_limit passed to GuardianClient in main.py (line 19):

client = GuardianClient(concurrency_limit=5)

Adjust TTS Voice or Quality

• Voice: Change VOICE = "af_heart" in tts_generator.py (line 16).
• Audio quality: Change -q:a 2 in the convert_wav_to_mp3 function (line 62).
• Sample rate: Change 24000 in sf.write() (line 51).

Change LLM Model (Stage 2)

Update MODEL_NAME = "gemma4:e2b" in reader.py (line 8). Ensure the model is pulled in Ollama (ollama pull <model>).

Add Extraction Fields (Stage 1)

This is a three-step process:

1. Update the Article model in scraper/models.py.
2. Update extract_article_body (or add a new method) in scraper/parser.py.
3. Update the Article instantiation in main.py's scrape_article function.

Adjust Rate Limiting (Stage 1)

Modify the wait_time calculation in scraper/client.py (lines 36, 42).

Modify Retry Logic (Stage 2)

Change the retry count (line 40: if attempt < 3) and the backoff delay (line 41: wait = attempt * 10) in reader.py.

Update Data Flow Between Stages

When an output file format changes:

1. Update the output format in the producing script (e.g., main.py stage 1).
2. Update the input format in the consuming script (e.g., reader.py stage 2).
3. Update type hints, Pydantic models, and JSON parsing in both files.
4. Update this document's pipeline diagram and output tables.

Critical Files

• main.py: Stage 1 orchestration (Guardian scraper).
• reader.py: Stage 2 orchestration (LLM script generation).
• tts_generator.py: Stage 3 orchestration (TTS audio generation).
• scraper/client.py: HTTP networking + concurrency control.
• scraper/parser.py: HTML extraction (Guardian page structure).
• scraper/models.py: Pydantic data schemas (Article, ScrapeResult, ScriptItem).
• scraper/utils.py: Disk I/O helpers.
• requirements.txt: Python dependencies.