initial commit
This commit is contained in:
@@ -0,0 +1,152 @@
|
||||
# Guardian Daily Newscast Pipeline
|
||||
|
||||
Turns The Guardian's daily US news headlines into a spoken MP3 newscast.
|
||||
|
||||
Three scripts run in sequence: **scrape** the articles, **generate** teleprompter scripts via a local LLM, **convert** scripts to audio.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
External tools you must have installed before running:
|
||||
|
||||
- **Ollama** — with the `gemma4:e2b` model pulled (`ollama pull gemma4:e2b`). Used in stage 2 to generate newscast scripts.
|
||||
- **ffmpeg** — used in stage 3 to convert WAV to MP3 (`brew install ffmpeg` on macOS, `apt install ffmpeg` on Ubuntu).
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# 1. Install Python dependencies
|
||||
pip install -r requirements.txt
|
||||
|
||||
# 2. Run the full pipeline (stages 1, 2, 3 in order)
|
||||
python main.py && python reader.py && python tts_generator.py
|
||||
|
||||
# 3. Find your newscast files in data/audio/*.mp3
|
||||
ls data/audio/
|
||||
```
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
Dependencies: `httpx`, `beautifulsoup4`, `pydantic`, `loguru`, `kokoro`, `soundfile`, `torch`.
|
||||
|
||||
## The Pipeline
|
||||
|
||||
The project has three distinct stages that **must be run in order**:
|
||||
|
||||
```
|
||||
[Stage 1] main.py (Guardian Scraper)
|
||||
|
|
||||
v
|
||||
data/output.json
|
||||
|
|
||||
v
|
||||
[Stage 2] reader.py (LLM Script Generator)
|
||||
|
|
||||
v
|
||||
data/scripts.json
|
||||
|
|
||||
v
|
||||
[Stage 3] tts_generator.py (TTS Audio Generator)
|
||||
|
|
||||
v
|
||||
data/audio/*.mp3
|
||||
```
|
||||
|
||||
### Stage 1: Guardian Scraper (`main.py`)
|
||||
|
||||
- Fetches today's Guardian US news home page (URL built from current date).
|
||||
- Extracts all article links, then fetches each article's body text concurrently (semaphore-limited to 5).
|
||||
- **Output:** `data/output.json` — a JSON object with an `articles` array (each entry: `headline`, `url`, `body`, `scraped_at`).
|
||||
|
||||
### Stage 2: LLM Script Generator (`reader.py`)
|
||||
|
||||
- Reads `data/output.json`.
|
||||
- Sends each article body to a local Ollama instance (`http://127.0.0.1:11434/api/generate`, model `gemma4:e2b`).
|
||||
- System prompt instructs the model to produce clean teleprompter-ready spoken text (no stage directions, no speaker labels).
|
||||
- Retries failed requests up to 3 times (exponential backoff: 10s, 20s). 2s delay between each request to avoid overloading Ollama.
|
||||
- **Output:** `data/scripts.json` — a JSON array of objects (each: `headline`, `script`).
|
||||
|
||||
### Stage 3: TTS Audio Generator (`tts_generator.py`)
|
||||
|
||||
- Reads `data/scripts.json`.
|
||||
- Uses Kokoro TTS (local, voice `af_heart`, 24kHz) to generate WAV audio.
|
||||
- Converts WAV to MP3 via ffmpeg (`-q:a 2`).
|
||||
- Cleans up intermediate WAV files after conversion.
|
||||
- **Output:** `data/audio/<sanitized_headline>.mp3` files — one MP3 per article.
|
||||
|
||||
## Usage
|
||||
|
||||
Run all three stages in order:
|
||||
|
||||
```bash
|
||||
python main.py
|
||||
python reader.py
|
||||
python tts_generator.py
|
||||
```
|
||||
|
||||
Or chain them with `&&`:
|
||||
|
||||
```bash
|
||||
python main.py && python reader.py && python tts_generator.py
|
||||
```
|
||||
|
||||
Each stage is independent and can be re-run without re-running earlier stages (e.g., re-run stage 3 if audio quality needs to change, without re-scraping).
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
.
|
||||
├── main.py # Stage 1: Guardian scraper
|
||||
├── reader.py # Stage 2: LLM script generation
|
||||
├── tts_generator.py # Stage 3: TTS audio generation
|
||||
├── requirements.txt # Python dependencies
|
||||
├── README.md # This file
|
||||
├── CLAUDE.md # Developer guide
|
||||
├── scraper/
|
||||
│ ├── __init__.py
|
||||
│ ├── client.py # GuardianClient (HTTP + concurrency)
|
||||
│ ├── parser.py # GuardianParser (HTML extraction)
|
||||
│ ├── models.py # Pydantic models (Article, ScrapeResult)
|
||||
│ └── utils.py # Disk I/O helpers
|
||||
├── data/
|
||||
│ ├── output.json # Stage 1 output (scraped articles)
|
||||
│ ├── scripts.json # Stage 2 output (generated scripts)
|
||||
│ └── audio/ # Stage 3 output (MP3 files)
|
||||
│ ├── article_1.mp3
|
||||
│ ├── article_2.mp3
|
||||
│ └── ...
|
||||
└── .claude/
|
||||
└── settings.local.json
|
||||
```
|
||||
|
||||
## Dependencies
|
||||
|
||||
### Python (via `pip install -r requirements.txt`)
|
||||
|
||||
| Package | Purpose |
|
||||
|---|---|
|
||||
| httpx | Async HTTP client for scraping |
|
||||
| beautifulsoup4 | HTML parsing |
|
||||
| pydantic | Data validation models |
|
||||
| loguru | Logging |
|
||||
| kokoro | Local TTS engine |
|
||||
| soundfile | WAV audio I/O |
|
||||
| torch | Required by kokoro |
|
||||
|
||||
### External (not installed via pip)
|
||||
|
||||
| Tool | Requirement |
|
||||
|---|---|
|
||||
| Ollama | Running locally, with `gemma4:e2b` |
|
||||
| ffmpeg | Available on PATH |
|
||||
|
||||
## Output Files
|
||||
|
||||
| File | Description |
|
||||
|---|---|
|
||||
| `data/output.json` | JSON object containing scraped articles with `headline`, `url`, `body`, and `scraped_at` fields. |
|
||||
| `data/scripts.json` | JSON array of objects with `headline` and `script` fields. |
|
||||
| `data/audio/*.mp3` | One MP3 file per article, named from the sanitized headline. |
|
||||
Reference in New Issue
Block a user