scraper/README.md

Guardian Daily Newscast Pipeline

Turns The Guardian's daily US news headlines into a spoken MP3 newscast.

Three scripts run in sequence: scrape the articles, generate teleprompter scripts via a local LLM, convert scripts to audio.

Prerequisites

External tools you must have installed before running:

• Ollama — with the gemma4:e2b model pulled (ollama pull gemma4:e2b). Used in stage 2 to generate newscast scripts.
• ffmpeg — used in stage 3 to convert WAV to MP3 (brew install ffmpeg on macOS, apt install ffmpeg on Ubuntu).

Quick Start

# 1. Install Python dependencies
pip install -r requirements.txt

# 2. Run the full pipeline (stages 1, 2, 3 in order)
python main.py && python reader.py && python tts_generator.py

# 3. Find your newscast files in data/audio/*.mp3
ls data/audio/

Installation

pip install -r requirements.txt

Dependencies: httpx, beautifulsoup4, pydantic, loguru, kokoro, soundfile, torch.

The Pipeline

The project has three distinct stages that must be run in order:

[Stage 1] main.py (Guardian Scraper)
              |
              v
       data/output.json
              |
              v
[Stage 2] reader.py (LLM Script Generator)
              |
              v
       data/scripts.json
              |
              v
[Stage 3] tts_generator.py (TTS Audio Generator)
              |
              v
       data/audio/*.mp3

Stage 1: Guardian Scraper (main.py)

• Fetches today's Guardian US news home page (URL built from current date).
• Extracts all article links, then fetches each article's body text concurrently (semaphore-limited to 5).
• Output: data/output.json — a JSON object with an articles array (each entry: headline, url, body, scraped_at).

Stage 2: LLM Script Generator (reader.py)

• Reads data/output.json.
• Sends each article body to a local Ollama instance (http://127.0.0.1:11434/api/generate, model gemma4:e2b).
• System prompt instructs the model to produce clean teleprompter-ready spoken text (no stage directions, no speaker labels).
• Retries failed requests up to 3 times (exponential backoff: 10s, 20s). 2s delay between each request to avoid overloading Ollama.
• Output: data/scripts.json — a JSON array of objects (each: headline, script).

Stage 3: TTS Audio Generator (tts_generator.py)

• Reads data/scripts.json.
• Uses Kokoro TTS (local, voice af_heart, 24kHz) to generate WAV audio.
• Converts WAV to MP3 via ffmpeg (-q:a 2).
• Cleans up intermediate WAV files after conversion.
• Output: data/audio/<sanitized_headline>.mp3 files — one MP3 per article.

Usage

Run all three stages in order:

python main.py
python reader.py
python tts_generator.py

Or chain them with &&:

python main.py && python reader.py && python tts_generator.py

Each stage is independent and can be re-run without re-running earlier stages (e.g., re-run stage 3 if audio quality needs to change, without re-scraping).

Project Structure

.
├── main.py                    # Stage 1: Guardian scraper
├── reader.py                  # Stage 2: LLM script generation
├── tts_generator.py           # Stage 3: TTS audio generation
├── requirements.txt           # Python dependencies
├── README.md                  # This file
├── CLAUDE.md                  # Developer guide
├── scraper/
│   ├── __init__.py
│   ├── client.py              # GuardianClient (HTTP + concurrency)
│   ├── parser.py              # GuardianParser (HTML extraction)
│   ├── models.py              # Pydantic models (Article, ScrapeResult)
│   └── utils.py               # Disk I/O helpers
├── data/
│   ├── output.json            # Stage 1 output (scraped articles)
│   ├── scripts.json           # Stage 2 output (generated scripts)
│   └── audio/                 # Stage 3 output (MP3 files)
│       ├── article_1.mp3
│       ├── article_2.mp3
│       └── ...
└── .claude/
    └── settings.local.json

Dependencies

Python (via pip install -r requirements.txt)

Package	Purpose
httpx	Async HTTP client for scraping
beautifulsoup4	HTML parsing
pydantic	Data validation models
loguru	Logging
kokoro	Local TTS engine
soundfile	WAV audio I/O
torch	Required by kokoro

External (not installed via pip)

Tool	Requirement
Ollama	Running locally, with gemma4:e2b
ffmpeg	Available on PATH

Output Files

File	Description
data/output.json	JSON object containing scraped articles with headline, url, body, and scraped_at fields.
data/scripts.json	JSON array of objects with headline and script fields.
data/audio/*.mp3	One MP3 file per article, named from the sanitized headline.