ChangeDetection competitor-intelligence monitor

A bounded discovery and intelligence layer for ChangeDetection.io. It discovers newly published competitor pages, keeps a curated representative watch set, classifies real content changes, and sends immediate or digest notifications to Discord.

The current example profile uses the reserved .example namespace; replace it with your own private config.json and ignore.json before discovery. Source code and templates are separate from your private targets, credentials, and monitoring history. A generic example at the current revision is not a guarantee that older Git history or test fixtures are generic; see Publishing safely.

What this repository separates

These are independent mechanisms and clocks:

Discovery — sitemaps, RSS/Atom, configured Certificate Transparency sources, and bounded same-domain links find URLs.
Watch reconciliation — useful representatives are imported, tagged, paused, or reactivated through the ChangeDetection API.
Page fetching — ChangeDetection checks existing watches on its own schedule.
Event collection — this monitor polls retained ChangeDetection history, classifies changes, and delivers notifications.
Snapshot retention — ChangeDetection history records distinct content versions, not every successful unchanged check.

A gap between two snapshot timestamps is therefore not a check interval or detection delay. Timing claims require raw successful-fetch telemetry identifying the last unchanged check and first changed check.

Main behavior

recursive sitemap and feed discovery;
bounded same-domain crawl with canonicalization and hard page/depth/size ceilings;
optional Certificate Transparency discovery with bounded retries and silent first baseline;
representative active selection, normally no more than two pages per host and semantic family;
durable exact/regex exclusions;
pause-not-delete reconciliation so historical snapshots survive deselection, including pages paused after their removal is confirmed by the configured number of complete inventories;
exclusion of individual company/person/domain/IP records, routine public-record churn, aggregate lists/statistics, archives, tests, discovery documents, duplicate languages/aliases, and static assets;
anti-bot challenge detection represented as a coverage gap, never as “no changes”;
semantic realignment of positional text diffs, so one inserted or removed repeated card is reported as one change rather than a cascade of false replacements, with bounded excerpts that always retain evidence from both old and new sides;
classification of a broad Estonian-to-English localization with unchanged numeric/commercial values as minor localization rather than as a pricing or policy alert; actual value changes remain important;
suppression of pure nested navigation-list changes, navigation moves, rendered icon labels, and identical minor fragments repeated across three or more pages;
source and Certificate Transparency health retained in status/state rather than mixed into competitor-intelligence digests;
durable immediate and digest outboxes with Discord rate-limit retry;
global command locking and systemd failure alerts;
explicit timestamp semantics in event records, plus the last successful unchanged check and first successful changed check captured from consecutive collector observations whenever available.

The supplied generic template deliberately sets respect_robots to false. Crawling remains limited by same-domain allowlists, one-second pacing, response timeouts, and hard sitemap/page/depth ceilings. Change that policy in your own deployment if required.

Repository layout

monitor.py                         discovery, reconciliation, collection, digest
notify_failure.py                  throttled systemd failure notification
test_monitor.py                    45 deterministic engine regressions
config.example.json                fictional `.example` scope template
ignore.example.json                durable exclusion template
compose.yaml                       digest-pinned ChangeDetection base stack
compose.browser.yaml               optional digest-pinned JS browser override
systemd/                           portable unit templates and timers
scripts/setup.sh                   initialize a checkout and optionally start Compose
scripts/doctor.py                  repository/config/live readiness checks
scripts/render_systemd.py          render host paths and runtime account into units
scripts/install-systemd.sh         install and verify timers
scripts/uninstall-systemd.sh       remove units but preserve data
verification/                      live E2E, scope, persistence, and optional UI checks
deploy/caddy/                      optional public reverse-proxy example

Runtime configuration, credentials, queues, discovery state, snapshots, and generated evidence are intentionally excluded from Git.

Requirements

Linux with Bash and Python 3.11 or newer (monitor runtime uses only the standard library); systemd is needed only for the supplied recurring timers;
Git, CA certificates, Docker Engine with Compose v2, and permission to use that Docker daemon;
Linux amd64 or arm64 for the pinned ChangeDetection image and optional browser image;
a Discord bot that can view and send messages in your own destination channel/thread;
optional: Caddy for public HTTPS;
optional UI verifier: a Chrome/Chromium CDP session and websockets from requirements-verification.txt.

Linux / WSL on a work computer

Use a machine and targets approved by your organization's policy. This is a new local deployment, not access to anyone else's existing monitoring installation. A clone is not a runtime backup: you must supply your own private configuration and credentials. Keep the default UI loopback-only; public HTTPS is optional, not a prerequisite.

For Ubuntu 24.04+ or a comparable Debian-family distribution with Python 3.11+:

sudo apt update
sudo apt install -y git python3 python3-venv ca-certificates

Install Docker Engine and its Compose plugin using the official Linux instructions. On Windows, use WSL 2 with an Ubuntu distribution, run these commands in its Bash terminal, and either enable that distribution in Docker Desktop's WSL integration or install a Docker Engine inside the distribution. Do not configure two competing daemons. Native PowerShell and Git Bash are not supported runtimes for these scripts. Store the checkout in the WSL Linux filesystem, such as ~/src, not a Windows-mounted directory, so Linux permissions work as expected.

Check prerequisites before cloning or running setup:

python3 --version                 # must be 3.11 or newer
bash --version
git --version
docker compose version           # Compose v2, not docker-compose v1
docker info >/dev/null           # must succeed as your normal runtime user

The scripts call docker without sudo. If daemon access fails, resolve it with your administrator; do not run the entire setup as root. Docker-group membership grants root-equivalent host access and should not be added casually on a managed computer. No runtime pip install is required.

For recurring timers, check systemctl is-system-running and confirm systemd is actually running (a degraded result needs investigation; merely having systemctl installed is insufficient). For WSL, follow Microsoft's systemd instructions if needed; otherwise skip timer installation and run commands manually. Monitoring does not continue while the computer is asleep, shut down, or WSL/Docker is stopped. Use an always-on approved Linux host if continuous monitoring is required.

The base installation does not run Sockpuppet or add elevated container capabilities. The optional browser override adds SYS_ADMIN, which its Chromium sandbox needs. It is not published on a host port, but this remains an elevated capability: review it against the destination host's security policy before enabling the override.

Fresh installation

1. Clone and initialize

Use the repository's advertised Git clone URL, not merely the web UI's page URL. Replace the prompt value with the URL supplied by the publisher; no particular hostname or remote hosting service is assumed.

mkdir -p "$HOME/src"
cd "$HOME/src"
read -r -p 'Git clone URL: ' REPOSITORY_URL
git clone -- "$REPOSITORY_URL" changedetection-competitor-monitor
cd changedetection-competitor-monitor
git remote get-url origin
./scripts/test.sh
umask 077
./scripts/setup.sh --no-start

The local test gate should end with All repository gates passed. It does not contact competitor sites or deliver Discord messages. setup.sh --no-start requires the Docker CLI/Compose plugin for model validation but does not start containers. Use one deployment per host unless you deliberately isolate both the Compose project and systemd unit names: the defaults use fixed names and a second checkout is not an independent installation.

setup.sh creates, without overwriting existing files:

.env from .env.example;
config.json from config.example.json;
ignore.json from ignore.example.json;
secrets.env from secrets.env.example with mode 0600;
local state/ and data/changedetection/ directories.

These local copies are ignored by Git. Edit the copies, not the tracked *.example templates. .env configures Compose; secrets.env supplies the monitor's API/bot credentials; config.json and ignore.json contain your private monitoring scope. Do not reuse a production configuration unless you intentionally want that exact private scope and have authorization.

Review .env in your editor before starting. Retain CHANGEDETECTION_BIND=127.0.0.1 and PUBLIC_BASE_URL=http://localhost:5000 for local use. If port 5000 is occupied, choose a free CHANGEDETECTION_PORT, then update both PUBLIC_BASE_URL and config.json's changedetection_base to match. A custom CHANGEDETECTION_DATA_DIR must point to private persistent storage.

2. Start and configure ChangeDetection

docker compose up -d
docker compose ps

Open http://localhost:5000 in your browser (or your configured PUBLIC_BASE_URL). In WSL with normal localhost forwarding, use the same URL from the Windows browser. If it fails, first check docker compose ps and docker compose logs --tail=100 changedetection; do not expose the port publicly as a workaround. In ChangeDetection:

set its native application password;
open Settings → API;
create/copy an API token.

Compose's startup ordering does not itself prove API readiness; the live doctor in step 4 is the readiness gate before timers are installed. The datastore contains watches, retained snapshots, and native-login/API material; it is private even if its container is stopped.

The pinned ChangeDetection 0.60.4 release uses a native password-only login; there is no application username. The optional Caddy example intentionally adds no proxy Basic Auth.

3. Configure monitor targets and secrets

Edit config.json:

replace REPLACE_WITH_DISCORD_CHANNEL_OR_THREAD_ID;
replace REPLACE_WITH_DISCORD_USER_ID;
review competitor domains, seeds, exclusions, crawl ceilings, and robots policy;
change the user-agent contact URL.

The shipped target is intentionally fictional and will not resolve. Add at least one real private target before running discovery.

Edit secrets.env:

CD_API_TOKEN=REPLACE_WITH_CHANGEDETECTION_API_TOKEN
DISCORD_BOT_TOKEN=REPLACE_WITH_DISCORD_BOT_TOKEN

Protect all four private configuration files (the secrets file must remain mode 0600):

chmod 600 .env config.json ignore.json secrets.env
python3 scripts/doctor.py --mode configured

Do not paste tokens into tracked examples, shell commands, screenshots, or issue reports. Check git status --short before committing source changes; ignored files are not encrypted or automatically backed up.

4. Prove live prerequisites

python3 scripts/doctor.py --mode live
python3 scripts/doctor.py --mode live --verify-discord-send
python3 monitor.py status
python3 monitor.py discover --no-send
python3 monitor.py collect --no-send

The normal live doctor checks the exact ChangeDetection API token, Discord bot identity, and destination channel/thread without sending a message. The second command explicitly creates, reads back, deletes, and confirms the absence of one probe message to prove the bot can actually deliver there without leaving test noise. The systemd installer repeats that probe as the selected runtime account before installing any units. The first discovery inventory is a silent baseline.

--no-send suppresses Discord-client creation: discovery and collection still need the ChangeDetection API token because they read or reconcile live watches, while digest --no-send needs neither API nor Discord credentials. It is not a dry run—discovery can still create, pause, or reactivate watches.

5. Install recurring timers

sudo ./scripts/install-systemd.sh --user "$USER"

Installed schedule:

discovery/reconciliation: every 6 hours;
event collection: every 10 minutes;
daily digest: 08:30 UTC.

These timer files are deployment-policy defaults, not engine requirements. Edit the three files under systemd/ before installation if another cadence or timezone is appropriate.

Verify:

systemctl list-timers 'competitor-monitor*'
systemctl status competitor-monitor-discover.timer competitor-monitor-collect.timer competitor-monitor-digest.timer

Remove the units without deleting the repository, state, or ChangeDetection datastore:

sudo ./scripts/uninstall-systemd.sh

Optional browser rendering with Sockpuppet

Sockpuppet is not required for the default html_requests fetch backend. Enable it only for watches that require JavaScript rendering, Chromium screenshots, or browser-based selectors:

docker compose -f compose.yaml -f compose.browser.yaml up -d

Then use "fetch_backend": "html_webdriver" for only the relevant competitor profile or watch. To return to HTTP-only fetching:

docker compose -f compose.yaml -f compose.browser.yaml down --remove-orphans
docker compose up -d

Optional public HTTPS with Caddy

Copy deploy/caddy/changedetection.caddy.example into your Caddy site directory, replace changedetection.example.com, validate, and reload Caddy. Keep the Compose port bound to loopback.

sudo caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile
sudo systemctl reload caddy

Use ChangeDetection’s native login. Do not add a second Basic Auth layer unless you intentionally want double authentication.

Day-to-day commands

python3 monitor.py discover           # discover, reconcile, and notify
python3 monitor.py discover --no-send # reconcile without Discord delivery
python3 monitor.py collect            # classify retained history and notify
python3 monitor.py digest             # deliver queued minor topics/gaps
python3 monitor.py status             # current watch and run summary

All paths are overrideable:

python3 monitor.py status \
  --config /path/config.json \
  --ignore /path/ignore.json \
  --state-dir /var/lib/competitor-monitor \
  --env-file /run/secrets/competitor-monitor.env

Verification

Run the local, secret-free gate:

./scripts/test.sh

It runs the engine suite, repository/secret-hygiene tests, Python compilation, shell syntax checks, Compose validation, portable systemd rendering, and systemd-analyze verify when available.

Live acceptance tools intentionally require a configured local deployment:

python3 verification/e2e_verify.py
python3 verification/verify_representative_scope.py
python3 verification/audit_watch_scope.py
python3 verification/verify_persistence.py before
docker compose restart
python3 verification/verify_persistence.py after

e2e_verify.py creates reversible test watches, proves before/after diffs plus immediate/digest/silent notification paths, reads deliveries back, and removes its fixtures. It writes generated reports under ignored verification/ paths.

The optional UI check requires a separately started disposable Chrome CDP session:

python3 -m venv .venv
.venv/bin/pip install -r requirements-verification.txt
CD_UI_PASSWORD='your native password' \
  .venv/bin/python verification/verify_ui.py \
  --url https://changedetection.example.com/ \
  --cdp-url http://127.0.0.1:9222

Updating an existing clone

Run from the same checkout used for installation, as its normal owner. git pull updates tracked source only; it does not import someone else's targets/history or replace your ignored private files. It also does not refresh an existing local configuration when example schemas change.

1. Inspect and pause this installation

cd "$HOME/src/changedetection-competitor-monitor"
git remote get-url origin
git status --short
git branch --show-current

Expect branch main and no tracked edits. Preserve/review any local edits first; do not use git reset --hard or git clean to force an update. If origin is missing, set it to the publisher's advertised Git clone URL, not a credentials-bearing URL.

If you installed this checkout's systemd timers, pause both the timers and any in-flight jobs before updating. Skip this block for manual-only installations. The fixed unit names below must belong to this installation; do not stop units belonging to another checkout.

sudo systemctl stop competitor-monitor-discover.timer competitor-monitor-collect.timer competitor-monitor-digest.timer
sudo systemctl stop competitor-monitor-discover.service competitor-monitor-collect.service competitor-monitor-digest.service

Stop containers using the same file set used to start them (choose one):

# Default HTTP-only deployment:
docker compose down --remove-orphans
# If the optional browser override was enabled, use this instead:
# docker compose -f compose.yaml -f compose.browser.yaml down --remove-orphans

down preserves the bind-mounted datastore. Before pulling, make a private backup of .env, config.json, ignore.json, secrets.env, state/, and the actual CHANGEDETECTION_DATA_DIR while writers are stopped. Keep it outside the checkout with restricted permissions; include custom/external datastore paths and preserve ownership/modes. Git is not that backup. Do not use down -v or delete any data directory as an update step.

2. Pull and validate

git pull --ff-only origin main
./scripts/test.sh
./scripts/setup.sh --no-start

If Git reports divergence, conflicts, or a failed pull, stop and resolve it without discarding local work. If tests fail, leave timers paused and inspect the failure. setup.sh --no-start is safe to repeat: it validates configuration/Compose and creates missing files but does not overwrite existing configuration. Review the updated README and git log -5 --oneline; compare changed example settings with your private files and apply any necessary schema changes manually.

3. Restart and verify

# Default deployment (use the browser file set instead if previously enabled):
docker compose up -d
python3 scripts/doctor.py --mode live
python3 monitor.py status

Require the doctor to return "ok": true and status to match your own intended watch inventory. For a timed installation, reinstall the updated units at this same path, then verify all three timers are enabled/active:

sudo ./scripts/install-systemd.sh --user "$USER"
systemctl list-timers 'competitor-monitor*'
systemctl is-active competitor-monitor-discover.timer competitor-monitor-collect.timer competitor-monitor-digest.timer

The installer sends, reads back, and deletes one Discord probe. Manual-only users skip it; the next deliberate monitor command is their next run. If recovery requires older application/data versions, restore the matching private backup as well as source—checking out an older commit alone is not a datastore rollback.

Stop or remove this installation

Pause monitoring only: use the two systemctl stop commands above if timers were installed. These stops are temporary; enabled timers can return after a reboot. For a persistent pause, disable all three timers with sudo systemctl disable --now competitor-monitor-discover.timer competitor-monitor-collect.timer competitor-monitor-digest.timer, then stop the three services. ChangeDetection keeps fetching existing watches until its Compose stack is also stopped.

Stop page fetching/UI too: run docker compose down --remove-orphans from this checkout, or the equivalent browser-override down command above. This removes this Compose project's containers/network, not the bind-mounted datastore, private configuration, monitor ledger, or historical snapshots. Do not stop the entire Docker daemon.

Remove scheduled integration: first stop the timers and services as above, then run:

sudo ./scripts/uninstall-systemd.sh
systemctl list-timers 'competitor-monitor*'

The remover disables/removes the fixed systemd unit files and reloads systemd. It does not stop an already-running monitor service for you, remove Compose containers/images, revoke tokens, remove an optional Caddy site, or erase source/config/state/datastore. Use the explicit service-stop and Compose commands before deleting or moving the checkout. If Caddy was separately configured, remove only that deployment's site and validate/reload Caddy separately.

Erase private data: this is a separate, irreversible operator decision, not part of uninstall. After stopping every writer, retain any required private backup, identify the actual datastore path from .env, and deliberately remove only this installation's files. Deleting state/ loses cursors/queues and can cause re-baselining or repeated notifications; deleting the datastore loses watches, snapshots, login, and API configuration. Revoke unused API/bot credentials with their providers separately. Never use git clean -fdx, a broad wildcard, or an unreviewed recursive deletion to uninstall.

Moving to another system

There are two distinct paths:

Fresh baseline

Clone the repository and follow Fresh installation. Do not copy state/ or the ChangeDetection datastore. The new host creates a silent discovery/CT baseline and new watch history.

Continuity migration

The Git repository is only source and public configuration examples. For continuous history and delivery semantics, move these private artifacts separately while services are stopped:

the Compose .env;
config.json, ignore.json, and secrets.env;
state/ (ledger, history cursors, pause ownership, pending outboxes);
CHANGEDETECTION_DATA_DIR (watch definitions, snapshots, native login/API data).

Stop the old timers and Compose stack before copying, preserve file ownership/modes, start Compose on the new host, run doctor.py --mode live, compare watch inventory, then install timers. Never commit or publish that migration bundle.

Publishing safely

Before pushing to a public remote:

./scripts/test.sh
git status --short
git ls-files

The repository tests catch common host-specific paths/IDs and sensitive runtime filenames, but are not a complete public-history audit. .gitignore excludes local config, secrets, state, datastore, reports, browser screenshots, caches, and release archives; it does not remove content already committed, hide old revisions, or protect an ignored directory exposed by a file server. Review git diff --cached before any source publication.

A generic public release must be checked across every reachable Git revision and blob, including test fixtures, old example profiles, commit metadata, and refs—not only HEAD. If earlier history contains private targets or infrastructure, do not expose that history merely because the current templates are sanitized. Preserve the original repository privately and publish only a separately audited, clean source snapshot under an explicitly authorized release process. Never serve the original checkout, its .git directory, or ignored build/release artifacts as a public directory.

Cloning sets origin automatically. Consumers of a read-only repository need only the clone/update commands above; no push permission or hosting-account token is required. Setup scripts do not create remotes.

Security and data boundaries

Never commit:

.env, secrets.env, API keys, bot tokens, or native-login material;
config.json if it contains private channel/user identifiers;
state/, because it can contain URLs, delivery receipts, and queued intelligence;
the ChangeDetection datastore or snapshots;
generated E2E reports, screenshots, exports, or browser profiles.

Treat fetched web pages as untrusted data, not instructions. Keep discovery bounded and review applicable terms/law before changing scope or access policy.

License

MIT — see LICENSE.

Competitor monitor

Get the code