Kibitzr

Kibitzr is a lightweight, self‑hosted automation engine that turns arbitrary web pages and command‑line tasks into event‑driven notifications. Written in pure Python, it can run on any platform that supports Python 2 or 3—Windows, Linux, macOS—and is designed to fit comfortably on the smallest virtual machines offered by cloud providers. The core idea is simple: poll a target resource, evaluate a condition expressed in XPath/CSS or shell output, and push the result to a messaging channel such as Slack, Telegram, or Mailgun. The application is intentionally minimalistic; it has no heavy dependencies and exposes a single YAML configuration file that defines tasks, triggers, and notifiers.

Technical Stack & Architecture

• Language: Python 2/3 (compatibility through six and future)
• Core Engine: Event loop built on asyncio‑like primitives, enabling concurrent polling of multiple URLs or command streams without blocking.
• Web Interaction: Uses selenium with headless browsers (Chrome/Firefox) for full‑browser scenarios, and requests + BeautifulSoup / lxml for lightweight HTTP+HTML parsing.
• Data Store: Stateless; all state is persisted in a local SQLite database (or optional Redis for high‑throughput deployments).
• Notification Services: Integrates natively with Slack (Incoming Webhooks), Telegram Bot API, and Mailgun’s SMTP gateway.
• Extensibility: Supports custom plugins written in Python or Bash, loaded at runtime. Plugins can expose new notification backends, custom parsers, or even entire task types.

The architecture is deliberately modular: a Poller fetches resources, a Parser evaluates conditions, and a Notifier dispatches alerts. This separation allows developers to drop in new components without touching the core loop.

Core Capabilities

• Human‑friendly YAML: Tasks are defined in a single file, e.g.

  tasks:
    - name: check-slack-release
      url: https://slack.com/changelog
      parser:
        type: xpath
        expression: //div[@class='release']/h2/text()
      notifier:
        type: slack
        channel: '#devops'
  

• Browser‑level Automation: Selenium scripts can log in, navigate through multi‑step flows, and capture dynamic content.
• Command Execution: Run arbitrary shell commands or SSH sessions; capture stdout/stderr and trigger on exit codes.
• Conditional Logic: Use regular expressions, comparison operators, or custom Python functions to decide when a notification should fire.
• Rate‑Limiting & Scheduling: Built‑in cron‑style scheduling (interval, daily, weekly) and per‑task rate limits to avoid spamming.

Deployment & Infrastructure

• Containerization: A Docker image is available (kibitzr/kibitzr) that bundles the Python runtime, Selenium drivers, and a minimal headless browser. The image is only ~150 MB.
• Scalability: Because the engine is event‑driven and stateless, multiple instances can run behind a load balancer. For high‑volume polling, the SQLite store can be swapped for Redis or PostgreSQL.
• Self‑Hosting: No external dependencies beyond the notification services. All credentials are stored locally and can be encrypted with a master key.
• Resource Footprint: Benchmarks show ~30 MB RAM and 1 CPU core for a single polling task; ideal for AWS t3.micro or GCP f1-micro instances.

Integration & Extensibility

• Plugin API: Exposes a simple registration hook (kibitzr.register_plugin) where developers can add new parsers, notifiers, or task types. Plugins are discovered via entry points (kibitzr.plugins).
• Webhooks: Kibitzr can expose an HTTP endpoint that accepts JSON payloads to trigger tasks on demand.
• SDK: A lightweight Python client (kibitzr-sdk) lets developers programmatically create, update, or delete tasks without editing YAML.

Developer Experience

• Configuration: A single YAML file keeps everything in one place. Validation is performed at startup, providing clear error messages for malformed tasks.
• Documentation: The README and inline comments cover every API surface. A live‑coding tutorial (Zapier integration) demonstrates how to expose Kibitzr as a Zapier app.
• Community: Active GitHub issues and a dedicated chat channel mean quick support for edge cases. The license is permissive (MIT), encouraging fork‑and‑customize.

Use Cases

• Continuous Integration: Notify on long‑running TeamCity builds or GitLab pipeline completions via Slack.
• Release Monitoring: Poll npm, PyPI, or Docker Hub for new package versions and alert teams.
• Security Alerts: Scrape CVE databases or vendor advisories, then push to a dedicated Telegram channel.
• Operational Dashboards: Periodically run SSH commands on remote servers, parse output, and email summaries to sysadmins.
• Compliance Checks: Verify that critical documents (e.g., passport readiness) are up‑to‑date by scraping government portals.

Advantages Over Alternatives