153 lines
4.9 KiB
Markdown
153 lines
4.9 KiB
Markdown
# Guardian Daily Newscast Pipeline
|
|
|
|
Turns The Guardian's daily US news headlines into a spoken MP3 newscast.
|
|
|
|
Three scripts run in sequence: **scrape** the articles, **generate** teleprompter scripts via a local LLM, **convert** scripts to audio.
|
|
|
|
## Prerequisites
|
|
|
|
External tools you must have installed before running:
|
|
|
|
- **Ollama** — with the `gemma4:e2b` model pulled (`ollama pull gemma4:e2b`). Used in stage 2 to generate newscast scripts.
|
|
- **ffmpeg** — used in stage 3 to convert WAV to MP3 (`brew install ffmpeg` on macOS, `apt install ffmpeg` on Ubuntu).
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
# 1. Install Python dependencies
|
|
pip install -r requirements.txt
|
|
|
|
# 2. Run the full pipeline (stages 1, 2, 3 in order)
|
|
python main.py && python reader.py && python tts_generator.py
|
|
|
|
# 3. Find your newscast files in data/audio/*.mp3
|
|
ls data/audio/
|
|
```
|
|
|
|
## Installation
|
|
|
|
```bash
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
Dependencies: `httpx`, `beautifulsoup4`, `pydantic`, `loguru`, `kokoro`, `soundfile`, `torch`.
|
|
|
|
## The Pipeline
|
|
|
|
The project has three distinct stages that **must be run in order**:
|
|
|
|
```
|
|
[Stage 1] main.py (Guardian Scraper)
|
|
|
|
|
v
|
|
data/output.json
|
|
|
|
|
v
|
|
[Stage 2] reader.py (LLM Script Generator)
|
|
|
|
|
v
|
|
data/scripts.json
|
|
|
|
|
v
|
|
[Stage 3] tts_generator.py (TTS Audio Generator)
|
|
|
|
|
v
|
|
data/audio/*.mp3
|
|
```
|
|
|
|
### Stage 1: Guardian Scraper (`main.py`)
|
|
|
|
- Fetches today's Guardian US news home page (URL built from current date).
|
|
- Extracts all article links, then fetches each article's body text concurrently (semaphore-limited to 5).
|
|
- **Output:** `data/output.json` — a JSON object with an `articles` array (each entry: `headline`, `url`, `body`, `scraped_at`).
|
|
|
|
### Stage 2: LLM Script Generator (`reader.py`)
|
|
|
|
- Reads `data/output.json`.
|
|
- Sends each article body to a local Ollama instance (`http://127.0.0.1:11434/api/generate`, model `gemma4:e2b`).
|
|
- System prompt instructs the model to produce clean teleprompter-ready spoken text (no stage directions, no speaker labels).
|
|
- Retries failed requests up to 3 times (exponential backoff: 10s, 20s). 2s delay between each request to avoid overloading Ollama.
|
|
- **Output:** `data/scripts.json` — a JSON array of objects (each: `headline`, `script`).
|
|
|
|
### Stage 3: TTS Audio Generator (`tts_generator.py`)
|
|
|
|
- Reads `data/scripts.json`.
|
|
- Uses Kokoro TTS (local, voice `af_heart`, 24kHz) to generate WAV audio.
|
|
- Converts WAV to MP3 via ffmpeg (`-q:a 2`).
|
|
- Cleans up intermediate WAV files after conversion.
|
|
- **Output:** `data/audio/<sanitized_headline>.mp3` files — one MP3 per article.
|
|
|
|
## Usage
|
|
|
|
Run all three stages in order:
|
|
|
|
```bash
|
|
python main.py
|
|
python reader.py
|
|
python tts_generator.py
|
|
```
|
|
|
|
Or chain them with `&&`:
|
|
|
|
```bash
|
|
python main.py && python reader.py && python tts_generator.py
|
|
```
|
|
|
|
Each stage is independent and can be re-run without re-running earlier stages (e.g., re-run stage 3 if audio quality needs to change, without re-scraping).
|
|
|
|
## Project Structure
|
|
|
|
```
|
|
.
|
|
├── main.py # Stage 1: Guardian scraper
|
|
├── reader.py # Stage 2: LLM script generation
|
|
├── tts_generator.py # Stage 3: TTS audio generation
|
|
├── requirements.txt # Python dependencies
|
|
├── README.md # This file
|
|
├── CLAUDE.md # Developer guide
|
|
├── scraper/
|
|
│ ├── __init__.py
|
|
│ ├── client.py # GuardianClient (HTTP + concurrency)
|
|
│ ├── parser.py # GuardianParser (HTML extraction)
|
|
│ ├── models.py # Pydantic models (Article, ScrapeResult)
|
|
│ └── utils.py # Disk I/O helpers
|
|
├── data/
|
|
│ ├── output.json # Stage 1 output (scraped articles)
|
|
│ ├── scripts.json # Stage 2 output (generated scripts)
|
|
│ └── audio/ # Stage 3 output (MP3 files)
|
|
│ ├── article_1.mp3
|
|
│ ├── article_2.mp3
|
|
│ └── ...
|
|
└── .claude/
|
|
└── settings.local.json
|
|
```
|
|
|
|
## Dependencies
|
|
|
|
### Python (via `pip install -r requirements.txt`)
|
|
|
|
| Package | Purpose |
|
|
|---|---|
|
|
| httpx | Async HTTP client for scraping |
|
|
| beautifulsoup4 | HTML parsing |
|
|
| pydantic | Data validation models |
|
|
| loguru | Logging |
|
|
| kokoro | Local TTS engine |
|
|
| soundfile | WAV audio I/O |
|
|
| torch | Required by kokoro |
|
|
|
|
### External (not installed via pip)
|
|
|
|
| Tool | Requirement |
|
|
|---|---|
|
|
| Ollama | Running locally, with `gemma4:e2b` |
|
|
| ffmpeg | Available on PATH |
|
|
|
|
## Output Files
|
|
|
|
| File | Description |
|
|
|---|---|
|
|
| `data/output.json` | JSON object containing scraped articles with `headline`, `url`, `body`, and `scraped_at` fields. |
|
|
| `data/scripts.json` | JSON array of objects with `headline` and `script` fields. |
|
|
| `data/audio/*.mp3` | One MP3 file per article, named from the sanitized headline. |
|