NewsBlur

NewsBlur is a feature‑rich, self‑hosted feed reader that exposes a robust set of developer hooks while maintaining an approachable API surface. At its core, the application ingests RSS/Atom feeds, normalizes the content, and surfaces it through a web UI that mimics the original publisher layout. The system is designed for real‑time delivery: a background worker pipeline fetches feeds, parses them with BeautifulSoup or lxml, and pushes new stories into a Redis queue that the Django front‑end consumes. This architecture allows developers to scale horizontally by adding more Celery workers or Redis shards without changing the core code.

Architecture & Technical Stack

• Back‑end: Python 3.7+ with Django 2.x/3.x, using a dual‑database approach: PostgreSQL for relational data (users, subscriptions) and MongoDB for unstructured story blobs. Celery + RabbitMQ orchestrate asynchronous fetching, while Redis stores transient state (story IDs, feed schedules) and minimal caching. Optional Elasticsearch integration provides full‑text search across all stories.
• Front‑end: A classic MVC stack built on Backbone.js, Underscore.js, and jQuery. The UI is rendered server‑side for SEO and progressive enhancement but also consumes REST endpoints via JSON, enabling third‑party clients.
• Containerization: The repo ships a docker-compose.yml that provisions all services (PostgreSQL, MongoDB, RabbitMQ, Redis, Elasticsearch, Nginx, HAProxy). The make nb target builds a multi‑container stack that can be deployed on any Docker host or orchestrator (K8s, ECS).

Core Capabilities & APIs

• RESTful Endpoints: /api/v1/feeds, /api/v1/stories, and /api/v1/search expose CRUD operations for feeds, stories, and saved searches. Authentication is token‑based (Django Rest Framework) and can be extended via OAuth2 or custom middleware.
• Webhooks: Developers can subscribe to story_added and feed_updated events. The webhook payload contains the story ID, feed metadata, and a direct link to the original article.
• Feed Import/Export: Supports OPML import/export, making migration from other readers trivial. The import logic normalizes duplicate feeds and merges subscriptions automatically.
• Tagging & Filtering: Tag‑based routing is exposed through API calls (/api/v1/tags). The backend uses a simple many‑to‑many relation to filter stories on the fly, which is useful for building custom dashboards or analytics.

Deployment & Infrastructure

NewsBlur’s Docker stack can run on a single VM or be spread across multiple nodes. The front‑end is stateless; all session data lives in Redis, enabling horizontal scaling behind a load balancer. The Celery workers can be replicated independently to handle bursty feed traffic, and RabbitMQ’s clustering guarantees message durability. For high‑availability PostgreSQL and MongoDB setups, the repo includes Helm charts that can be adapted to managed services (RDS, Atlas).

Integration & Extensibility

The codebase is intentionally modular. The feed‑parsing logic lives in a separate Python package (newsblur.feedparser), which can be swapped out for a custom parser. Plugin hooks are exposed via Django signals (feed_fetched, story_parsed), allowing developers to inject analytics, profanity filters, or custom content transformations. The web UI exposes a public API for third‑party clients such as Reeder, ReadKit, and Unread, which means you can build native or CLI tools that interact with your NewsBlur instance.

Developer Experience

Documentation is comprehensive: the README covers setup, architecture diagrams, and API references. The community around NewsBlur is active on GitHub Issues and a dedicated Slack channel, making it easy to get help with custom deployments. The code is well‑structured with clear separation of concerns, making it approachable for contributors who want to add new features or fix bugs. Licensing is MIT, so you can use and modify the code in commercial products without restrictions.

Use Cases

• Personal Knowledge Management: A developer can host NewsBlur to aggregate industry blogs, then build a custom analytics layer that surfaces trending topics across their feed set.
• Enterprise Aggregation: Companies can deploy NewsBlur behind a corporate VPN, ingest internal RSS feeds (e.g., HR updates), and expose a unified portal for employees.
• IoT & Edge Devices: With its lightweight front‑end and Docker support, NewsBlur can run on Raspberry Pi to provide a local news kiosk that updates in real time.

Advantages

• Performance: The dual‑database design keeps story blobs fast to read/write while keeping relational data lightweight. Redis caching reduces latency for river assembly.
• Flexibility: Developers can swap out any component (e.g., replace RabbitMQ with Kafka) without breaking the API contract.
• Open Source & Extensible: MIT license and a plugin architecture lower barriers to custom extensions, unlike proprietary readers that lock users into closed ecosystems.
• Real‑Time Delivery: The push‑style architecture ensures stories appear within seconds of publication, a feature not commonly found in other self‑hosted readers.

In summary, NewsBlur offers a mature, modular stack that balances developer ergonomics with production‑ready scalability. Its well‑documented APIs and extensible hooks make it an excellent foundation for building custom news aggregation services, analytics dashboards, or even fully‑customized reader apps.