This page is a high-level tour of the system. The full source code, migration plan, and decision log live in the GitHub repository.
What it is
Social Sentiment Dashboard is a sentiment monitor for the EV maker Rivian. It continuously watches what owners and prospective buyers are saying across four very different channels: Reddit posts and comments, Apple App Store reviews, X/Twitter posts, and official NHTSA safety complaints. It scores each item for sentiment, surfaces recurring issues, and pushes anything safety-related to a review queue with Slack alerts. Everything lands in a single FastAPI and Next.js dashboard instead of four scattered feeds.
Why it matters
For a hardware company, customer feedback is critical but hopelessly fragmented. The same unhappy owner might vent on Reddit, leave a one-star App Store review, tag the brand on X, and file an NHTSA complaint, so no single team ever sees the whole picture. Safety and service problems are the easiest to miss and the most expensive to catch late. This project pulls every source into one place, tracks whether sentiment is rising or falling, and makes sure a critical safety complaint reaches a human in Slack within minutes.
How it works
It runs as an event-driven Kafka pipeline rather than a cron job or a batch script, so the sources, the sentiment model, the safety flagging, and the alerting all scale and fail independently. The pipeline stages (ingestion, HuggingFace-based sentiment analysis, safety flagging, and Slack notification) each run as independent consumers, and Postgres backs the FastAPI API and the Next.js dashboard, including the safety-triage queue for reviewing flagged posts.
Architecture
Pollers feed raw topics; five stage consumers move items through analysis, flagging, and notification; the API publishes commands back into the pipeline.
Everything runs under Docker Compose: a KRaft-mode Kafka broker, Postgres 16, a poller process (fetch loops plus backfill and command consumers), a worker process (five stage consumers), the FastAPI backend, and the Next.js frontend, with Kafka UI for topic and consumer-lag inspection.
The Kafka pipeline
Seven topics (three partitions each) carry JSON payloads validated by Pydantic v2 schemas. Raw items land in social.items.raw and nhtsa.complaints.raw; an ingest writer dedupes and persists them, then emits items.new. An analyzer consumer scores sentiment with a RoBERTa model and emits items.analyzed; a flagger promotes safety- and service-related posts to flags.created; a notifier delivers severity-filtered alerts to Slack. A pipeline.commands topic lets the API trigger polls and refreshes on demand.
Delivery is at-least-once: consumers commit offsets only after the database transaction commits, and every write is idempotent under redelivery via unique constraints, so a crash between commit and offset advance never duplicates data.
Data sources
Sources are pluggable behind a small Source ABC, so adding one means implementing a fetch method and registering it. Current sources: Reddit (public JSON with RSS fallback), NHTSA complaints, Apple App Store reviews (RSS), and X/Twitter via an Apify scraper with owner-voice queries and a promo filter (kept inert unless a token is configured, so no accidental spend).
Dashboard & API
The FastAPI backend exposes stats (sentiment time series, per-entity and per-source breakdowns, weekly changes, executive summary), filtered post search, clustered issue cards, NHTSA complaint analytics, and a flag queue where resolving requires a note. The Next.js dashboard covers Overview, Charts, Entities, Issues, Posts, Flagged, and NHTSA views, plus a manual “fetch now” trigger that publishes a command into the pipeline. The backend ships with 47 tests.