# CLAUDE.md — Guardian Daily Newscast Pipeline Guide

## Project Overview

A Python pipeline that turns The Guardian's daily US news into MP3 newscast audio files.

Three scripts run in sequence:
1. **main.py** — Scrapes Guardian articles (outputs `data/output.json`)
2. **reader.py** — Generates teleprompter scripts via local LLM (outputs `data/scripts.json`)
3. **tts_generator.py** — Converts scripts to MP3 audio (outputs `data/audio/*.mp3`)

## Coding Standards

- **Language**: Python 3.x
- **Style**: PEP 8 compliant.
- **Async**: Use `asyncio` and `httpx.AsyncClient`. Always ensure clients are closed properly (e.g., via `await client.close()`).
- **Logging**: Use `loguru` for all logging.
- **Typing**: Use Python type hints throughout.
- **Data Validation**: Use Pydantic models for any structured data.

## Architecture & Key Patterns

### Pipeline Overview

```
[Stage 1] main.py (Guardian Scraper)  --> data/output.json
[Stage 2] reader.py (LLM Script Gen)  --> data/scripts.json
[Stage 3] tts_generator.py (TTS)      --> data/audio/*.mp3
```

Each stage reads the previous stage's output file. Stages are independent and can be re-run in any order (but logically must follow 1 → 2 → 3).

### Stage 1: Guardian Scraper (`main.py` + `scraper/`)

**Client-Parser-Model Pattern:**

- `GuardianClient` (`scraper/client.py`): Handles HTTP networking, rate limits, and concurrency via `asyncio.Semaphore`. Implements exponential backoff for 429/5xx responses.
- `GuardianParser` (`scraper/parser.py`): Pure functions for HTML extraction. Body fallback logic: (1) `div[data-gu-context="body"]`, (2) `<article>`, (3) `<main>`.
- `Article` / `ScrapeResult` (`scraper/models.py`): Pydantic models for data consistency.
- `utils.py`: Disk I/O (`save_results_to_json`).

### Stage 2: LLM Script Generator (`reader.py`)

- Sends each article body to Ollama (`http://127.0.0.1:11434/api/generate`, model `gemma4:e2b`).
- System prompt enforces clean teleprompter-ready text (no stage directions, no speaker labels).
- Retry logic: up to 3 attempts with exponential backoff (10s, 20s). 2s delay between requests.
- Incremental save to `data/scripts.json` during processing.

### Stage 3: TTS Audio Generator (`tts_generator.py`)

- `KokoroTTS`: Wraps Kokoro `KPipeline` with voice `af_heart`, outputs WAV at 24kHz.
- `convert_wav_to_mp3`: ffmpeg subprocess (`-q:a 2`).
- `ScriptItem`: Pydantic model validating input from `data/scripts.json`.
- Concurrent processing with `asyncio.Semaphore(CONCURRENCY_LIMIT=1)`.
- Cleans up intermediate WAV files after conversion.

## Common Tasks

### Add a New Pipeline Stage

1. Create a new script (e.g., `stage_name.py`) in the project root.
2. Read the previous stage's output file.
3. Write results to a new output file.
4. Add it to the `&&` chain in README.md Quick Start.
5. Update the pipeline diagram in this document.
6. Add the output file to the Output Files table.

### Change Target URL (Stage 1)

Update the `url` variable in `main.py` (line 16). The URL is built dynamically from today's date:

```python
url = f"https://www.theguardian.com/us-news/{year}/{month}/{day}/all"
```

### Modify Concurrency (Stage 1)

Change the `concurrency_limit` passed to `GuardianClient` in `main.py` (line 19):

```python
client = GuardianClient(concurrency_limit=5)
```

### Adjust TTS Voice or Quality

- Voice: Change `VOICE = "af_heart"` in `tts_generator.py` (line 16).
- Audio quality: Change `-q:a 2` in the `convert_wav_to_mp3` function (line 62).
- Sample rate: Change `24000` in `sf.write()` (line 51).

### Change LLM Model (Stage 2)

Update `MODEL_NAME = "gemma4:e2b"` in `reader.py` (line 8). Ensure the model is pulled in Ollama (`ollama pull <model>`).

### Add Extraction Fields (Stage 1)

This is a three-step process:
1. Update the `Article` model in `scraper/models.py`.
2. Update `extract_article_body` (or add a new method) in `scraper/parser.py`.
3. Update the `Article` instantiation in `main.py`'s `scrape_article` function.

### Adjust Rate Limiting (Stage 1)

Modify the `wait_time` calculation in `scraper/client.py` (lines 36, 42).

### Modify Retry Logic (Stage 2)

Change the retry count (line 40: `if attempt < 3`) and the backoff delay (line 41: `wait = attempt * 10`) in `reader.py`.

### Update Data Flow Between Stages

When an output file format changes:
1. Update the output format in the producing script (e.g., `main.py` stage 1).
2. Update the input format in the consuming script (e.g., `reader.py` stage 2).
3. Update type hints, Pydantic models, and JSON parsing in both files.
4. Update this document's pipeline diagram and output tables.

## Critical Files

- `main.py`: Stage 1 orchestration (Guardian scraper).
- `reader.py`: Stage 2 orchestration (LLM script generation).
- `tts_generator.py`: Stage 3 orchestration (TTS audio generation).
- `scraper/client.py`: HTTP networking + concurrency control.
- `scraper/parser.py`: HTML extraction (Guardian page structure).
- `scraper/models.py`: Pydantic data schemas (Article, ScrapeResult, ScriptItem).
- `scraper/utils.py`: Disk I/O helpers.
- `requirements.txt`: Python dependencies.