Piler

Piler is a self‑hosted, open‑source email archival engine that ingests inbound and outbound mail streams, applies retention and legal‑hold policies, and exposes a full‑featured web UI for search, export, and compliance audit. It is designed to run on commodity Linux servers, with optional Docker support for rapid deployment in cloud or container‑oriented environments. The core of Piler is written in PHP 8 and relies on a MySQL / MariaDB backend for metadata while storing raw MIME blobs in the filesystem or an object store. The application ships a lightweight SMTP relay and supports IMAP/POP3 for client access, making it suitable as a gateway or archiving proxy in existing mail infrastructures.

Key Features

• Rule‑based ingestion – SMTP or POP3/IMAP clients can be configured to trigger archival rules (e.g., archive all messages larger than 5 MB, or tag corporate domain traffic).
• Retention & legal hold – Policies can be defined per mailbox or globally, with support for time‑based retention and indefinite legal holds that lock the message state.
• Deduplication & compression – Piler hashes attachments and messages to avoid duplicate storage, optionally compressing MIME parts with zlib or Brotli.
• Digital fingerprinting – SHA‑256 checksums and DKIM/DMARC verification are stored for each message, enabling integrity audits.
• Full‑text search – Powered by an embedded SQLite FTS5 index (or optional ElasticSearch integration) for fast keyword queries across headers, bodies, and attachments.
• Extensible authentication – AD/LDAP, SSO (OAuth2), Google Apps, Office 365, and 2‑factor via TOTP or YubiKey are all supported out of the box.
• Audit logging – Every archival action, search, export, or policy change is recorded in a searchable log with support for exporting to CSV/JSON.

Technical Stack

The codebase follows PSR‑4 autoloading and is unit‑tested using PHPUnit; CI pipelines on GitHub Actions validate linting, tests, and code coverage.

Core Capabilities & APIs

• RESTful API – Exposes endpoints for message retrieval, search, tagging, and bulk import/export. Authentication is handled via JWTs or session cookies.
• Webhooks – Configurable callbacks on archival events, policy violations, and audit log entries.
• CLI tools – piler-cli offers import/export of PST/EML, bulk tagging, and policy simulation.
• Plugin hooks – Developers can register PHP callbacks for custom preprocessing (e.g., ML‑based spam scoring) or post‑archival actions.

Deployment & Infrastructure

• Self‑hosting – Minimal requirements: 2 CPU cores, 4 GB RAM (8 GB for heavy workloads), and 10 GB of disk per 100,000 messages. The application scales horizontally by running multiple web nodes behind a load balancer; the database is the single point of contention.
• Containerization – Docker images expose configurable environment variables (DB_HOST, SMTP_PORT, etc.) and can be orchestrated with Docker‑Compose or Kubernetes via Helm charts.
• High Availability – Replicated MySQL/MariaDB with Galera or Percona XtraDB Cluster, combined with shared NFS or Ceph for MIME storage.
• Backups – Database dumps and object‑store snapshots can be scheduled; Piler includes scripts for incremental backup of the archive directory.

Integration & Extensibility

• Mail Transfer Agents – Postfix, Exim, or Sendmail can forward messages to Piler’s SMTP port; alternatively, use the REST API for programmatic ingestion.
• Office 365 / Google Workspace – Built‑in connectors fetch mail via IMAP and apply the same archival rules.
• SAML/OAuth2 SSO – Integrate with corporate identity providers; the UI will respect group memberships for access control.
• Custom Themes – The PHP‑based UI supports CSS overrides and language packs; developers can ship their own theme bundles.
• Community Plugins – The GitHub repo hosts example plugins; developers can fork and contribute new filters or export formats (e.g., JSON‑L, EML).

Developer Experience

• Configuration – YAML files (config.yaml) control database, SMTP, authentication, and retention policies; the UI can edit most settings without touching files.
• Documentation – The README, API spec (Swagger), and developer guides are hosted on GitHub; inline code comments follow PHPDoc conventions.
• Testing – PHPUnit test suite covers 85 % of core logic; CI ensures regression protection.
• Community – Active issues, discussions, and a mailing list (retired but archived) provide support; contributors are welcomed to