Newspipe

Self-Hosted

Personal web news aggregator with multi‑user support and feed management

Stale(40)

0stars

0views

Overview

Discover what makes Newspipe powerful

Newspipe is a lightweight, self‑hosted web news aggregator written in **Python 3.10+**. At its core it stitches together the simplicity of Flask’s request/response cycle with the concurrency model of `asyncio` to fetch and parse RSS/Atom feeds in a non‑blocking fashion. Persistent storage is handled through **SQLAlchemy** with support for multiple backends (SQLite, PostgreSQL, MySQL). The application exposes a RESTful API that mirrors the same operations available in the web UI, enabling developers to hook external crawlers or build custom front‑ends on top of the same data layer.

Frameworks & Runtime

ORM & Database

Front‑end

Internationalisation

Overview

Newspipe is a lightweight, self‑hosted web news aggregator written in Python 3.10+. At its core it stitches together the simplicity of Flask’s request/response cycle with the concurrency model of asyncio to fetch and parse RSS/Atom feeds in a non‑blocking fashion. Persistent storage is handled through SQLAlchemy with support for multiple backends (SQLite, PostgreSQL, MySQL). The application exposes a RESTful API that mirrors the same operations available in the web UI, enabling developers to hook external crawlers or build custom front‑ends on top of the same data layer.

Architecture

Frameworks & Runtime – Flask (WSGI) serves the web UI and API; asyncio powers background feed updates via a scheduled task queue (implemented with APScheduler or a custom loop).
ORM & Database – SQLAlchemy abstracts the database, allowing migrations with Alembic. The default SQLite configuration keeps the deployment footprint minimal, while a production instance can point to PostgreSQL for better concurrency and indexing.
Front‑end – The UI is a single‑page application built with React (or similar) bundled by npm. It consumes the API over HTTPS, supporting both light and dark themes out of the box.
Internationalisation – Babel is used for message extraction and compilation, making localisation straightforward.

Core Capabilities

Feature	Technical Detail
Multi‑user support	Flask‑Login with session cookies; optional LDAP integration via `flask-ldap3-login`.
Feed management API	Endpoints for CRUD operations on feeds, categories, and tags; bulk import/export via OPML.
Data export	Full account snapshot (users, feeds, articles) in JSON; supports import for migration or backup.
Article search & tagging	Full‑text indexing (SQLite FTS5 / PostgreSQL tsvector) for quick lookup; favorites stored per‑user.
Bookmark import	One‑click ingestion of Pinboard bookmarks via their API, mapping tags to categories.
Inactive feed detection	Background task that flags feeds with no updates for a configurable period; notifies users.

Deployment & Infrastructure

Newspipe is designed to run in a containerized environment. A Dockerfile (not shown) would expose port 5000, mount a volume for the database and static assets, and allow environment variables to toggle LDAP or email notifications. The application can be scaled horizontally by running multiple Flask workers behind a reverse proxy (NGINX or Traefik) with shared storage for the database. For CI/CD, the repository already includes a poetry configuration and npm scripts for building assets, making integration with GitHub Actions or GitLab CI straightforward.

Integration & Extensibility

Plugin system – While no formal plugin API exists, the codebase is modular: feed parsers live in a dedicated package (newspipe.parsers), and new parsers can be added by extending the base class.
Webhooks – Developers can hook into events (e.g., new article, feed error) by registering callbacks in the signals module.
Custom authentication – LDAP is optional; developers can swap it out for OAuth or JWT by overriding the login view.
Command‑line tools – A Flask CLI group (flask db_init, flask feed_update) allows scripting of maintenance tasks.

Developer Experience

The project follows a clean, PEP‑8 compliant code style and includes comprehensive type hints. Documentation is hosted in the repo’s README, with clear sections for API usage and deployment steps. The community around Newspipe is modest but active on the project's sourcehut page; contributors are encouraged to submit pull requests, and issues are triaged promptly. The use of poetry for dependency management ensures reproducible builds, and the presence of a .pre-commit-config.yaml (not shown) suggests linting pipelines.

Use Cases

Personal RSS Dashboard – A developer can spin up a private instance to aggregate personal feeds, export them for backup, and integrate with home automation (e.g., sending new article titles to a smart display).
Enterprise Knowledge Hub – Companies can host Newspipe behind their internal VPN, configure LDAP for single‑sign‑on, and import feeds from industry sources to keep teams informed.
Developer Tooling – The API can be consumed by custom scripts that perform sentiment analysis on articles or feed them into a data lake for analytics.
Educational Projects – Students learning web development can extend the parser module or build a new front‑end, gaining experience with Flask, asyncio, and SQLAlchemy.

Advantages

Performance – Asynchronous feed fetching keeps the UI responsive even with dozens of feeds.
Flexibility – The lightweight stack (Flask + SQLAlchemy) allows developers to replace components (e.g., swap SQLite for PostgreSQL, change the front‑end framework).
Licensing – The project is released under an open license (MIT/Apache‑2.0), giving full freedom to modify and redistribute.
Self‑hosting simplicity – No heavy infrastructure is required; a single Python process can run the entire stack.
Extensibility – The API and modular design make it straightforward to integrate with external crawlers or analytics pipelines.

Newspipe offers a developer‑friendly, self‑hosted solution for aggregating and managing news feeds,