Bencher

Self-Hosted

Continuous benchmarking to detect performance regressions before they hit production

Active(86)

758stars

0views

Updated 5 days ago

Overview

Discover what makes Bencher powerful

Bencher is a self‑hostable continuous benchmarking platform that plugs into existing CI pipelines to surface performance regressions before they reach production. At its core, Bencher is a thin wrapper around any benchmark harness (e.g., Go `testing.B`, Rust `criterion`, Python `pytest-benchmark`) that captures raw results, normalizes them into a common JSON schema, and persists the data in a time‑series database. The platform exposes a REST API for ingestion, querying, and alerting, while the web console visualizes trends per branch, testbed, or custom metric. Developers can treat Bencher like any other CI artifact: run `bencher run` locally, push to a GitHub Action, and let the server surface alerts via webhooks or Slack.

Backend

CLI

Web UI

Data Pipeline

Overview

Bencher is a self‑hostable continuous benchmarking platform that plugs into existing CI pipelines to surface performance regressions before they reach production. At its core, Bencher is a thin wrapper around any benchmark harness (e.g., Go testing.B, Rust criterion, Python pytest-benchmark) that captures raw results, normalizes them into a common JSON schema, and persists the data in a time‑series database. The platform exposes a REST API for ingestion, querying, and alerting, while the web console visualizes trends per branch, testbed, or custom metric. Developers can treat Bencher like any other CI artifact: run bencher run locally, push to a GitHub Action, and let the server surface alerts via webhooks or Slack.

Architecture

Backend: A Go‑based HTTP server (bencher-server) implements the public REST API. It uses PostgreSQL for metadata (projects, benchmarks, thresholds) and TimescaleDB as a time‑series store for raw measurement points. The server is container‑friendly and can be deployed behind a reverse proxy (NGINX, Traefik) with TLS termination.
CLI: The bencher binary is written in Go and communicates with the API over HTTPS. It accepts arbitrary benchmark adapters (adapter::json, adapter::magic) and serializes results into a canonical JSON payload. The CLI can be invoked in CI or locally, making it agnostic to the underlying language of the test harness.
Web UI: A single‑page React application (TSX) served by the same Go server. It consumes the REST API, renders interactive charts with Recharts, and allows threshold configuration through a declarative JSON schema. The UI is bundled via Vite, enabling hot reloading during self‑hosting development.
Data Pipeline: Benchmarks are ingested as POST requests to /api/v1/results. The server validates the payload, upserts metadata, and writes measurements to TimescaleDB. An internal scheduler (cron‑style) runs nightly jobs that compute rolling percentiles and compare against configured thresholds, emitting alerts via webhook or email.

Core Capabilities

Feature	Description
Benchmark Ingestion	Wrap any benchmark harness; results are stored in a normalized format.
Time‑Series Storage	TimescaleDB allows efficient queries over large spans (months/years).
Trend Analysis	Graphs per branch/testbed; auto‑computes Δ% relative to baseline.
Threshold & Alerting	Customizable upper/lower bounds per benchmark; alerts via webhooks, Slack, or email.
REST API	Full CRUD for projects, benchmarks, thresholds; supports pagination and filtering.
CLI Plugins	Adapters can be extended via Go modules; community adapters exist for Rust, Python, etc.
CI Integration	GitHub Actions, GitLab CI/CD, and self‑hosted GitHub App provide out‑of‑the‑box workflows.

Deployment & Infrastructure

Containerization: Docker images are available for the CLI, server, and console. The docker-compose.yml in the repo demonstrates a minimal self‑hosted stack (PostgreSQL, TimescaleDB, API, UI). Kubernetes manifests are also provided for production workloads.
Scalability: The stateless API can be horizontally scaled behind a load balancer. TimescaleDB handles high write throughput; sharding is possible for extremely large projects.
Self‑Hosting Requirements: A PostgreSQL 13+ instance, TimescaleDB extension, TLS certificates, and at least 2 GB RAM for the server. The CLI has no runtime dependencies beyond Go.
Observability: Prometheus metrics are exposed (/metrics) for latency, request count, and database health. The UI also logs events to a central log aggregator.

Integration & Extensibility

Plugins: The adapter system allows developers to write custom parsers for any benchmark output. New adapters can be published as Go modules and referenced in the CLI config.
Webhooks: Benchmarks that exceed thresholds trigger a JSON payload to any URL, enabling integration with CI dashboards, Slack, or custom monitoring tools.
REST API: Full programmatic access means you can build your own dashboards, merge Bencher data into existing analytics pipelines, or automate threshold adjustments.
CLI Configuration: YAML/JSON config files let you map benchmark names to human‑readable labels, set per‑benchmark thresholds, and define testbed metadata.

Developer Experience

Documentation: Comprehensive docs cover every command, API endpoint, and architectural diagram. The docs/reference/architecture.md file contains a flow‑chart of data ingestion.
Community: The project is open source on GitHub with an active issue tracker and a Slack channel for quick support. Contributions to adapters or UI components are welcomed.
Configuration: The CLI uses a simple bencher.yaml file; the server reads environment variables for database URLs, TLS certs, and webhook endpoints. No code changes are required to add a new benchmark or change thresholds.
Testing: Unit tests for the CLI and server are written in Go, ensuring rapid feedback on changes.

Use Cases

CI Performance Gates – Run bencher run in a GitHub Action; if any benchmark exceeds its threshold, the job fails and blocks PR merge.
Long‑Term Regression Tracking – Store nightly benchmark results for a web app; visualize