Adds visitor dashboard and related features

2026-06-28 06:41:23 +00:00
parent a7cccbf170
commit 77d1c04d86
35 changed files with 1938 additions and 8 deletions
@@ -95,6 +95,8 @@ Open [http://localhost:4200](http://localhost:4200) and ensure the backend is ru
    ├── Dockerfile              # Python 3.12-slim + gunicorn/uvicorn
    ├── requirements.txt
    ├── seed.py                 # Initial data for all models
+    ├── inject_test_logs.py     # Test data injector for visitor stats pipeline
+    ├── geo/                    # MaxMind GeoLite2 City database
    └── app/
        ├── main.py             # FastAPI app, CORS, router registration
        ├── config.py           # ENV-based settings (KMTN_* prefix)
@@ -104,7 +106,8 @@ Open [http://localhost:4200](http://localhost:4200) and ensure the backend is ru
        └── api/
            ├── programs.py     # CRUD /api/programs
            ├── events.py       # CRUD /api/events
-            └── tiers.py        # CRUD /api/tiers
+            ├── tiers.py        # CRUD /api/tiers
+            └── stats.py        # Visitor stats endpoints (summary, timeseries, geo, parse)
 ```

 ## Local Development
@@ -197,6 +200,72 @@ ng test              # Angular + Vitest

 The backend API is self-documented at [http://localhost:8000/docs](http://localhost:8000/docs) (Swagger UI). The database auto-seeds on first startup — visit that URL to interactively test endpoints.

+## Admin Dashboard — Visitor Stats
+
+A visitor analytics dashboard built on a **log parsing pipeline**: Nginx writes access logs to a shared volume that the FastAPI backend reads and processes. No extra services required (no Logstash, Prometheus, or message queues).
+
+### How it works
+
+1. **Nginx** writes structured JSON access logs to **daily log files** on a shared mount: `/logs/access-YYYY-MM-DD.log`. A `map` directive extracts the date from the ISO timestamp, so Nginx rotates to a new file at midnight each day.
+2. **FastAPI** (via [log_parser.py](backend/app/log_parser.py)) scans `/logs/` for daily files that haven't been parsed yet, enriches each entry with IP geolocation (MaxMind GeoLite2 City), and stores aggregates in PostgreSQL.
+3. **PostgreSQL** tables (`visitor_stats_daily`, `visitor_stats_geo`, `log_parse_state`) hold the aggregated data and a "last parsed date" marker.
+4. **Angular admin dashboard** reads from the stats API endpoints to render charts and a world map.
+
+### Why daily files
+
+The currently-being-written file is never read — no partial lines, no race conditions, no file locking needed. Stats have a minimum one-day lag.
+
+### Stats API endpoints
+
+| Method | Endpoint | Description |
+|---|---|---|
+| `GET` | `/api/stats/summary` | Total unique visitors for a date range |
+| `GET` | `/api/stats/timeseries` | Daily unique visitor counts (chart data) |
+| `GET` | `/api/stats/geo` | Geographic breakdown by country (map data) |
+| `GET` | `/api/stats/parse` | Trigger log parsing (SSE progress stream) |
+
+All endpoints require admin authentication (JWT token). Date range is optional; defaults to the last 30 days.
+
+### GeoLite2 setup
+
+The log parser requires a MaxMind GeoLite2 City database for IP geolocation:
+
+1. Download from [https://dev.maxmind.com/geoip/geolite2-free-geolocation-data](https://dev.maxmind.com/geoip/geolite2-free-geolocation-data)
+2. Place the `.mmdb` file at `backend/geo/GeoLite2-City.mmdb`
+3. The Dockerfile copies it into the container at `/app/geo/GeoLite2-City.mmdb`
+
+### Injecting test data
+
+To verify the pipeline works (e.g., after deploying or before a demo), use the test log injector script at [backend/inject_test_logs.py](backend/inject_test_logs.py). It writes realistic traffic from 25 well-known public IPs across multiple past days — data is only aggregated a day in arrears, so all dates are strictly before today.
+
+```bash
+# Run inside the api container (writes to shared /logs volume)
+docker compose exec api python /app/inject_test_logs.py /logs 14
+
+# Or run locally (dev mode)
+cd backend
+python inject_test_logs.py /logs 14
+```
+
+The second argument controls how many days of data to generate (default: 14). Each day gets a deterministic mix of visitors from the IP pool, so re-running produces the same output (idempotent — files are appended, so check first).
+
+After injecting, trigger parsing via the admin dashboard or:
+
+```bash
+curl -X GET "http://localhost:8000/api/parse?token=YOUR_JWT"
+```
+
+### Shared volume setup
+
+For docker-compose, the `logs` named volume is mounted in both the `frontend` (Nginx) and `api` containers at `/logs`:
+
+```yaml
+volumes:
+  logs:/logs
+```
+
+For Kubernetes (minikube), use a `hostPath` PersistentVolume with `ReadWriteMany` access mode, claimed by a PersistentVolumeClaim and mounted in both pods.
+
 ## API Reference

 The backend exposes CRUD endpoints for three models: