155 lines
6.5 KiB
Markdown
155 lines
6.5 KiB
Markdown
# CLAUDE.md — Guardian Daily Newscast Pipeline Guide
|
|
|
|
## Project Overview
|
|
|
|
A Python pipeline that turns The Guardian's daily US news into MP3 newscast audio files.
|
|
|
|
Three scripts run in sequence:
|
|
1. **main.py** — Scrapes Guardian articles (outputs `data/output.json`)
|
|
2. **reader.py** — Generates teleprompter scripts via local LLM (outputs `data/scripts.json`)
|
|
3. **tts_generator.py** — Converts scripts to MP3 audio (outputs `data/audio/*.mp3`)
|
|
|
|
## Coding Standards
|
|
|
|
- **Language**: Python 3.x
|
|
- **Style**: PEP 8 compliant.
|
|
- **Async**: Use `asyncio` and `httpx.AsyncClient`. Always ensure clients are closed properly (e.g., via `await client.close()`).
|
|
- **Logging**: Use `loguru` for all logging.
|
|
- **Typing**: Use Python type hints throughout.
|
|
- **Data Validation**: Use Pydantic models for any structured data.
|
|
|
|
## Architecture & Key Patterns
|
|
|
|
### Pipeline Overview
|
|
|
|
```
|
|
[Stage 1] main.py (Guardian Scraper) --> data/output.json
|
|
[Stage 2] reader.py (LLM Script Gen) --> data/scripts.json
|
|
[Stage 3] tts_generator.py (TTS) --> data/audio/*.mp3
|
|
```
|
|
|
|
Each stage reads the previous stage's output file. Stages are independent and can be re-run in any order (but logically must follow 1 → 2 → 3).
|
|
|
|
### Stage 1: Guardian Scraper (`main.py` + `scraper/`)
|
|
|
|
**Client-Parser-Model Pattern:**
|
|
|
|
- `GuardianClient` (`scraper/client.py`): Handles HTTP networking, rate limits, and concurrency via `asyncio.Semaphore`. Implements exponential backoff for 429/5xx responses.
|
|
- `GuardianParser` (`scraper/parser.py`): Pure functions for HTML extraction. Body fallback logic: (1) `div[data-gu-context="body"]`, (2) `<article>`, (3) `<main>`.
|
|
- `Article` / `ScrapeResult` (`scraper/models.py`): Pydantic models for data consistency.
|
|
- `utils.py`: Disk I/O (`save_results_to_json`).
|
|
|
|
### Stage 2: LLM Script Generator (`reader.py`)
|
|
|
|
- Sends each article body to Ollama (`http://127.0.0.1:11434/api/generate`, model `gemma4:e2b`).
|
|
- System prompt enforces clean teleprompter-ready text (no stage directions, no speaker labels).
|
|
- Retry logic: up to 3 attempts with exponential backoff (10s, 20s). 2s delay between requests.
|
|
- Incremental save to `data/scripts.json` during processing.
|
|
|
|
### Stage 3: TTS Audio Generator (`tts_generator.py`)
|
|
|
|
- `KokoroTTS`: Wraps Kokoro `KPipeline` with voice `af_heart`, outputs WAV at 24kHz.
|
|
- `convert_wav_to_mp3`: ffmpeg subprocess (`-q:a 2`).
|
|
- `ScriptItem`: Pydantic model validating input from `data/scripts.json`.
|
|
- Concurrent processing with `asyncio.Semaphore(CONCURRENCY_LIMIT=1)`.
|
|
- Cleans up intermediate WAV files after conversion.
|
|
|
|
### DevContainer (`.devcontainer/`)
|
|
|
|
A VS Code Dev Container enables reproducible development. Two files define it:
|
|
|
|
- **`Dockerfile`** (based on `python:3.11-slim`):
|
|
- Installs system deps: `git`, `curl`, `ffmpeg`.
|
|
- Copies and installs `requirements.txt` (project root) into the container at `/app/`.
|
|
- Sets default env vars: `OLLAMA_URL`, `MODEL_NAME`, `PORT`, `POLL_INTERVAL`, `DATA_DIR`, `TTS_VOICE`.
|
|
- Creates `/app/data/audio/` for output.
|
|
|
|
- **`devcontainer.json`**:
|
|
- Points VS Code to the Dockerfile.
|
|
- Forwards port `8080` (newscast API).
|
|
- `postCreateCommand`: runs `pip install -r requirements.txt` in the VS Code Python interpreter for IDE features (linting, completions).
|
|
- `remoteEnv`: overrides Dockerfile defaults with workspace-relative values (e.g., `DATA_DIR` points to `${workspaceFolder}/data`).
|
|
- Extensions: `ms-python.python`, `ms-python.vscode-pylance`.
|
|
|
|
**Key env vars** (Dockerfile defaults, overridable via `devcontainer.json` `remoteEnv`):
|
|
|
|
| Variable | Default | Description |
|
|
|---|---|---|
|
|
| `OLLAMA_URL` | `http://host.docker.internal:11434/api/generate` | Ollama API endpoint (use `host.docker.internal` to reach host from container) |
|
|
| `MODEL_NAME` | `gemma4:e2b` | Ollama model identifier |
|
|
| `PORT` | `8080` | Newscast API listen port |
|
|
| `POLL_INTERVAL` | `21600` | API polling interval in seconds (6 hours) |
|
|
| `DATA_DIR` | `/app/data` | Base directory for all pipeline output files |
|
|
| `TTS_VOICE` | `af_heart` | Kokoro TTS voice |
|
|
|
|
## Common Tasks
|
|
|
|
### Add a New Pipeline Stage
|
|
|
|
1. Create a new script (e.g., `stage_name.py`) in the project root.
|
|
2. Read the previous stage's output file.
|
|
3. Write results to a new output file.
|
|
4. Add it to the `&&` chain in README.md Quick Start.
|
|
5. Update the pipeline diagram in this document.
|
|
6. Add the output file to the Output Files table.
|
|
|
|
### Change Target URL (Stage 1)
|
|
|
|
Update the `url` variable in `main.py` (line 16). The URL is built dynamically from today's date:
|
|
|
|
```python
|
|
url = f"https://www.theguardian.com/us-news/{year}/{month}/{day}/all"
|
|
```
|
|
|
|
### Modify Concurrency (Stage 1)
|
|
|
|
Change the `concurrency_limit` passed to `GuardianClient` in `main.py` (line 19):
|
|
|
|
```python
|
|
client = GuardianClient(concurrency_limit=5)
|
|
```
|
|
|
|
### Adjust TTS Voice or Quality
|
|
|
|
- Voice: Change `VOICE = "af_heart"` in `tts_generator.py` (line 16).
|
|
- Audio quality: Change `-q:a 2` in the `convert_wav_to_mp3` function (line 62).
|
|
- Sample rate: Change `24000` in `sf.write()` (line 51).
|
|
|
|
### Change LLM Model (Stage 2)
|
|
|
|
Update `MODEL_NAME = "gemma4:e2b"` in `reader.py` (line 8). Ensure the model is pulled in Ollama (`ollama pull <model>`).
|
|
|
|
### Add Extraction Fields (Stage 1)
|
|
|
|
This is a three-step process:
|
|
1. Update the `Article` model in `scraper/models.py`.
|
|
2. Update `extract_article_body` (or add a new method) in `scraper/parser.py`.
|
|
3. Update the `Article` instantiation in `main.py`'s `scrape_article` function.
|
|
|
|
### Adjust Rate Limiting (Stage 1)
|
|
|
|
Modify the `wait_time` calculation in `scraper/client.py` (lines 36, 42).
|
|
|
|
### Modify Retry Logic (Stage 2)
|
|
|
|
Change the retry count (line 40: `if attempt < 3`) and the backoff delay (line 41: `wait = attempt * 10`) in `reader.py`.
|
|
|
|
### Update Data Flow Between Stages
|
|
|
|
When an output file format changes:
|
|
1. Update the output format in the producing script (e.g., `main.py` stage 1).
|
|
2. Update the input format in the consuming script (e.g., `reader.py` stage 2).
|
|
3. Update type hints, Pydantic models, and JSON parsing in both files.
|
|
4. Update this document's pipeline diagram and output tables.
|
|
|
|
## Critical Files
|
|
|
|
- `main.py`: Stage 1 orchestration (Guardian scraper).
|
|
- `reader.py`: Stage 2 orchestration (LLM script generation).
|
|
- `tts_generator.py`: Stage 3 orchestration (TTS audio generation).
|
|
- `scraper/client.py`: HTTP networking + concurrency control.
|
|
- `scraper/parser.py`: HTML extraction (Guardian page structure).
|
|
- `scraper/models.py`: Pydantic data schemas (Article, ScrapeResult, ScriptItem).
|
|
- `scraper/utils.py`: Disk I/O helpers.
|
|
- `requirements.txt`: Python dependencies.
|