Fail2banMqttActionBanishment/docs/deployment.en.md

# Production deployment

This page details deploying a master + nodes set in production, beyond
the single isolated node covered in [Installation](installation.en.md).

## Overview

Three ways to use this project, from simplest to most complete — the
same base install (`sudo make install`) is the foundation for all three:

- **Solo**: `sudo make install`, alone. This server detects and bans
  locally, without centralizing anything elsewhere. Enough to protect a
  single VPS. The relay service to a master
  (`fail2ban-emitter-master-client`) still runs, but silently keeps
  retrying in a loop as long as no master is configured (no harm done,
  `Restart=on-failure`).
- **Solo + master**: `sudo make install` **then** `sudo make
  install-master`, on the **same** server. This VPS protects its own
  services (like solo mode) and also serves as a master, ready to take on
  other nodes later — useful for starting out alone today without
  closing the door on a second server tomorrow.
- **Master + nodes** (several VPSes): one server in solo+master mode, and
  one or more other servers, each installed solo and then attached to
  that master via token-based enrollment (below). The master itself
  stays a node like any other (it detects and bans locally in addition
  to centralizing the decision) — going from "solo+master on its own" to
  "master + nodes" therefore requires no reinstall of the master, just
  adding nodes to it over time.

Recommended order for moving to several nodes:

1. Deploy the master (once — solo+master if this same server should also
   protect itself, which is the common case).
2. On the master, issue a token for each new node.
3. On each new node, enroll using that token.

## Deploying the master

```sh
sudo make install-master
```

Requires a first `sudo make install` already done on this same machine.
Provisions:

- a dedicated **private CA** (`scripts/master-ca-init.sh`), distinct from
  the Let's Encrypt certificate that serves the broker's TLS identity —
  it only signs nodes' client certificates (mTLS authentication);
- the public Mosquitto listener (port 8883, TLS, `MQTT_MASTER_DOMAIN` in
  `.env`) with per-node ACLs;
- master-side ingestion (`fail2ban-emitter-master-listen`, runs with its
  own mTLS identity `master-internal`, generated automatically).

`MQTT_MASTER_DOMAIN` must point to a real public domain (DNS already set
up) — the broker obtains its Let's Encrypt certificate for that name.

## Token-based node enrollment (recommended)

### The problem

Adding a node by hand would require generating a key + certificate on the
master, then copying those files to the new node (`scp`), renaming them
correctly, and editing its `.env` — several manual, cross-machine steps,
and the private key would travel over the network (never a good
practice).

### The model

The master issues a **single-use token**, short-lived (1h by default).
The new node uses that token to enroll itself:

- The node generates its **own private key locally** and never sends it
  — only a CSR (signing request) goes to the master.
- The token is never stored in plaintext on the master: only its SHA-256
  hash is persisted (same logic as a password hash).
- The token is **atomically marked used** as soon as it's validated, and
  **automatically given back** if a later step fails (inconsistent CSR,
  signing failure) — a transient hiccup doesn't force issuing an
  entirely new token.
- The CN of the received CSR must match the node name declared when the
  token was issued — a stolen token can't be used to claim a different
  name.
- Any error returned to the client is **deliberately generic**
  ("invalid or expired token"), regardless of the actual cause — the
  detail (unknown, expired, or already-used token, inconsistent CN,
  signing failure) only goes into the master's logs, so as to give no
  hint to a third party probing the endpoint.
- The transport is the master's HTTPS dashboard (already publicly
  exposed, nginx + Let's Encrypt): no new port or service to open, and it
  solves the chicken-and-egg problem (a node can't use mTLS MQTT to
  obtain... its first mTLS certificate).

### Step by step

On the **master**:

```sh
make join-token NODE=node-name
```

Prints the plaintext token **only once**, along with the full command to
run on the new node (lost token: simply issue a new one, the previous
one stays valid independently until it expires).

On the **new node** (having already gone through `sudo make install`):

```sh
sudo make join MASTER=https://<master-domain> NODE=node-name TOKEN=<token>
sudo make install
```

The second `make install` applies the received configuration
(`MQTT_MASTER_NODE_NAME` in `.env`, certificate in place) and restarts
the services.

### What happens behind the scenes

```
[new node]                                   [master]
  generates key + CSR (local, never sent)
  POST /api/join/ {node_name, token, csr} ──►
                                              validates the token (hash, expiry, single use)
                                              checks the CSR's CN == node_name
                                              signs the CSR with the private CA
                                              adds the ACL entitlement for this node
                                              restarts mosquitto
                        {cert, ca_cert} ◄────
  writes cert/ca_cert, updates .env
```

### Troubleshooting

- **"Invalid or expired token"** — the message is deliberately generic.
  Possible causes: token already used (run `make join-token` again to
  issue a new one), token expired (1h default, adjustable via
  `make join-token NODE=... TTL=<minutes>`), or a typo in the node name
  (must be identical between token issuance and `make join`).
- The new node must be able to reach the master over HTTPS (443) — a
  firewall blocking outbound traffic would prevent enrollment.

## Databases

SQLite is the default, nothing to configure. Two alternatives:

```sh
sudo make install-mariadb   # provisions a local MariaDB server + dedicated database/user
sudo make install           # re-run afterward to apply migrations on the new database
```

PostgreSQL is supported at the configuration level (`DB_ENGINE=postgresql`
in `.env`, driver `emitter/requirements-postgresql.txt`) but without an
automated provisioning subcommand for now — server and database must be
created manually before switching `DB_ENGINE`.

Migrating data from one backend to another (engine-agnostic dump via
`manage.py dumpdata`/`loaddata`):

```sh
make dump-db   # writes emitter/db-dump-<date>.json (git-ignored, contains real data)
# ... switch DB_ENGINE, migrate ...
emitter/.venv/bin/python emitter/manage.py loaddata <file>.json
```

## nginx reverse proxy + public TLS

```sh
sudo make install-nginx   # reads DASHBOARD_DOMAIN from .env
```

Obtains a Let's Encrypt certificate and deploys an nginx vhost in front of
the dashboard (WebSocket included) — needed for public access; without
it, the dashboard is only reachable locally/over VPN.

## Alternative to systemd: supervisor

```sh
sudo make install-supervisor     # switches the emitter's 4 services to supervisor
sudo make uninstall-supervisor   # rollback (re-enables systemd)
```

## Updating

```sh
git pull
sudo make install
```

Reapplies dependencies/migrations and restarts the affected services, on
any node (including the master).