denis/voice-assistant
1
0
Fork 0
mirror of https://github.com/deunix-educ/voice-assistant.git synced 2026-10-13 00:11:57 +02:00
Code Issues Packages Projects Releases Wiki Activity

Assistant vocal local ESP32 : étapes 1 à 15, et 15b écrite

Browse Source 
État du projet au premier commit :
- firmware ESP32 : micro I2S, lecture anti-gigue, MQTT (état retenu, testament),
  mains libres, domotique de démonstration, chien de garde, présence du serveur ;
- serveur Python : sessions, VAD, faster-whisper, pyannote, identification,
  droits par profil, domotique, mot de réveil, Piper ;
- broker Mosquitto : configuration, comptes et ACL (non activés en développement) ;
- étape 15b écrite, en attente d'essai : agent MQTT pour machines Linux.

Étapes 1 à 15 validées. 203 tests, pyright sans erreur.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Denis Defolie
2026-09-30 17:40:55 +02:00
commit d20976fea2
108 changed files with 15022 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README-EN.md
+461
View File
@@ -0,0 +1,461 @@
# Local voice assistant — ESP32 + Raspberry Pi
A **fully local** voice assistant, with no cloud service: an ESP32 captures
speech, streams raw PCM over MQTT to a Python server that transcribes,
interprets and answers with speech synthesis; the ESP32 plays the answer back
and executes the command.
*Version française : [README.md](README.md).*
---
## 1. Architecture
```
INMP441 -> ESP32 -> MQTT -> Python server (Raspberry Pi / PC)
|
+---------+---------+
| |
VAD Diarization /
| identification
+---------+---------+
|
faster-whisper
|
Assistant
|
Piper
|
MQTT audio/out
|
ESP32 -> MAX98357A -> speaker
```
Four often-confused stages, never to be mixed up:
| Stage | Question it answers |
|---|---|
| VAD | Is someone speaking? |
| Diarization | Who speaks when? (SPEAKER_00, SPEAKER_01…) |
| Identification | Who is this person? (Denis, Marie, Unknown) |
| STT | What is being said? |
> Voice identification is **probabilistic**. It is never an authentication or
> security mechanism.
---
## 2. Hardware
| Item | Detail |
|---|---|
| ESP32-WROOM-32 | two I2S peripherals: I2S0 as receiver, I2S1 as transmitter |
| INMP441 | I2S microphone, **3.3 V**, L/R pin to GND (left channel) |
| MAX98357A | mono class-D amplifier, **5 V**, SD tied to VIN, GAIN left floating (9 dB) |
| Button | push-to-talk between GPIO 4 and GND (internal pull-up) |
| LED | with a 330 Ω resistor on GPIO 2 |
| Server | Raspberry Pi 4/5 64-bit or Debian PC |
### Wiring
**Full diagram: [docs/wiring.svg](docs/wiring.svg)** — detailed pinout, power
budget and multimeter checks (in French): **[docs/BROCHAGE.md](docs/BROCHAGE.md)**.
```
ESP32 INMP441 (microphone)
3V3 ----------------------- VDD
GND ----------------------- GND, L/R
GPIO 25 (SCK) ------------- SCK
GPIO 26 (WS) ------------- WS
GPIO 33 (SD) ------------- SD
ESP32 MAX98357A (amplifier, step 2)
5V (VIN) ------------------- VIN
GND ----------------------- GND
GPIO 27 (BCLK) ------------- BCLK
GPIO 14 (LRC) ------------- LRC
GPIO 22 (DIN) ------------- DIN
ESP32 User interface
GPIO 4 ------ button ------ GND
GPIO 2 --[330 Ω]-- LED ---- GND
```
No strapping pin (GPIO 0, 2, 5, 12, 15) carries an audio signal: their level at
reset selects the boot mode. GPIO 2 drives the LED as an output only, which does
not affect booting.
---
## 3. Audio format, identical everywhere
| Parameter | Value |
|---|---|
| Sample rate | 16 000 Hz |
| Resolution | signed 16-bit, little-endian (`pcm_s16le`) |
| Channels | 1 (mono) |
| Bit rate | 32 000 bytes/s |
| Chunk | 50 ms = 800 samples = **1 600 bytes** |
No Base64: MQTT payloads carry raw PCM bytes.
---
## 4. Installation
```bash
# 1. System packages (mosquitto, ffmpeg, venv) + serial port access
make install-system
# Log out and back in to activate the "dialout" group.
# 2. Python environment and dev tooling (pyright, pytest, platformio)
make install-dev
# PlatformIO gets its own venv (.pio-venv): its dependencies clash with the
# voice server's.
# 3. Speech recognition and synthesis (step 6): faster-whisper, Piper, ~550 MB of models
make install-speech
make models
# 4. Diarization (step 8): pyannote + CPU torch (~1.3 GB), Hugging Face token required
make install-diarization
cp server/.env.example server/.env # then HF_TOKEN=... (see step 8)
make models
# 5. Secrets (used from step 3 onwards)
cp firmware/include/secrets.h.example firmware/include/secrets.h
cp server/.env.example server/.env
# 6. Help
make help
```
---
## 5. Usage
```bash
make fw-build # build the firmware
make fw-upload SERIAL_PORT=/dev/ttyUSB0
make record OUT=recordings/test.wav # record one session (step 1)
make play OUT=recordings/test.wav # play it back (picks a working player)
make inspect OUT=recordings/test.wav # analyse it without listening
make play-esp OUT=recordings/test.wav # play it on the ESP32 speaker (step 2)
make tone-esp TONE=440 # pure tone on the ESP32 speaker (step 2)
make run # assistant: answers every question (step 6)
make speech-check TEXT="Quelle heure est-il ?" # recognition and synthesis, no ESP32
make say WAKE="Hey Mycroft" TEXT="Ouvre la porte du salon." VOICE=fr_FR-tom-medium # WAV + MP3 for a phone
make transcribe OUT=server/recordings/<file>.wav
make vad OUT=server/recordings/<file>.wav # speech found by the VAD (step 7)
make vad-all # VAD over every server recording
make diarize OUT=server/recordings/<file>.wav # who speaks when, who, who said what (steps 8-10)
make diarize-demo # two voices: yours and Piper's
make enroll NAME=Denis # voice profile from the last 5 sessions (step 9)
make profiles # list profiles; make forget NAME=Denis deletes one
make mqtt-watch # text messages voice/… and home/# (no audio)
make mqtt-user NAME=esp32-01 # MQTT account for a board, or the server (step 15)
make agent-run # Linux agent of this PC, test mode (step 15b)
make mqtt-agents # agent states, commands and reports
make fw-monitor # serial monitor
make test # Python tests
make lint # pyright type check
```
---
## 6. Progress
The exact state of the project, step by step, lives in **[CONTEXT.md](CONTEXT.md)**.
Current step: **15b — Linux machines on the network**. Steps 1 to 15 are validated.
### Step 15b — Linux machines on the network
"Hey Mycroft, éteins le PC du bureau": each machine runs an **agent**
([agent/voice_agent.py](agent/voice_agent.py)), a small MQTT client that only runs
the actions in **its own** list (`/etc/voice-agent/agent.yaml`): fixed commands,
no shell. The server only sends an action name. "Allume" wakes a machine that is
off through **Wake-on-LAN** (a network packet received by the sleeping network card).
| Action | Words | Rights (config.yaml) |
|---|---|---|
| `wake` (server) | allume, démarre, réveille | standard |
| `lock` | verrouille | standard |
| `shutdown` | éteins, arrête | full + confirmation |
| `reboot` | redémarre, relance | full + confirmation |
1. Try it on this PC without shutting anything down: `make agent-run` (creates `agent/agent.yaml` in test mode).
2. `make run`, then "Hey Mycroft, éteins le PC du bureau" → confirmation → "oui":
the agent prints `essai : systemctl poweroff (non execute)`.
3. Real machine: copy `agent/` to it, `make -C agent install`, edit
`/etc/voice-agent/agent.yaml`, and name it in `server/config.yaml` (`machines.list`).
**Anonymous broker**: any device on the network can then publish on `agent/<machine>/command`.
In production, use the accounts and rights of step 15 (`mosquitto/acl` has the agent rules).
### Step 15 — reliability
| What | How |
|---|---|
| MQTT accounts and rights | no anonymous access; a board reads and writes **its own** topics only ([docs/MOSQUITTO.md](docs/MOSQUITTO.md) §2) |
| Server presence | `voice/server/status`: retained `online`, `offline` last will; server gone = board microphones off |
| Watchdog | board frozen for more than 15 s → reboot, reason `watchdog` in the `boot` event |
| Network | TCP connection to the broker within 1 s; Wi-Fi restarted after 30 s without it |
| Log | copied to `server/logs/voice-server.log` (rotating files, 4 MB at most) |
Setup, **in this order**: a broker that is still open already accepts clients that have an account.
1. `make mqtt-user NAME=voice-server`, then `make mqtt-user NAME=esp32-01` (passwords: `openssl rand -hex 16`).
2. `server/.env`: `MQTT_USERNAME=voice-server`, `MQTT_PASSWORD=...`; `secrets.h`: `MQTT_PASSWORD="..."`.
3. `make fw-upload` and `make run`: everything works as before, the server announces its presence.
4. `make mosquitto-docker`: the broker now requires the accounts.
### Step 14 — per-profile rights
| Level | Rights | Given to |
|---|---|---|
| **complet** (full) | everything; sensitive actions (open the door) after a "oui" **confirmation** | profiles listed in `access.users`, **if the voice is clearly recognised** (score ≥ 0.6) |
| **standard** | lights, shutters, closing the door | enrolled profile without a level (`access.default`) |
| **limite** (limited) | conversation only | unknown voice (`access.unknown`) |
Rules per device and action in `config.yaml` (section `access`); an action missing from the
rules requires the full level. Names in `access.users` are those shown by `make profiles`.
1. `make run` (server only, firmware unchanged).
2. "Hey Mycroft, allume la lumière" → executed (Denis, full).
3. "Hey Mycroft, ouvre la porte" → "Confirmez-vous : ouvrir la porte du salon ?" →
"Hey Mycroft, oui" → "J'ouvre la porte du salon."; "non" → "D'accord, j'annule."
4. A phone voice (standard profile): "ouvre la porte" → refused; a non-enrolled voice:
"allume la lumière" → "Je ne vous ai pas reconnu…". Ready-made sentences:
`recordings/voix-test/tom-porte.mp3`, `pierre-lumiere.mp3`; more with `make say`. The wake word is spoken by an English
voice (`WAKE`): a synthetic voice saying it the French way is not recognised.
**Reminder**: a voice is never proof of identity. These rights prevent mistakes and guests'
commands, not an attack (recording, impersonation).
### Step 13 — wake word (hands-free)
The server asks the board to stream its microphone continuously (`voice/<board>/audio/stream`)
and listens for the wake word with openWakeWord: **"Alexa"**, **"Hey Mycroft"** or **"Hey
Jarvis"**, pronounced the English way. Once heard, it keeps the command until the final
silence, then handles it like a push-to-talk session. The button still works at any time.
LED: **short flash every 2 s** = continuous listening; **on** = wake word heard, command in
progress; slow blink = ready, no continuous listening.
1. `make install-wakeword` then `make models` (5 ONNX models, 5 MB).
2. `make fw-upload SERIAL_PORT=/dev/ttyUSB0` (step 13 firmware), then `make run`.
3. "Alexa, allume la lumière" → LED on during the command, then answer and command.
4. A sentence without the wake word → nothing happens, nothing is recorded.
5. "Alexa" alone, then silence → back to standby after 4 s.
**Privacy**: before the wake word, only the last 500 ms are kept in memory, nothing is
written to disk (checked by a test). But audio flows continuously over the network: as long
as the broker accepts anonymous clients (step 15), anyone on the LAN can listen to it.
Turn listening off: `wake.enabled: false`, or
`make mqtt-control MSG='{"cmd":"listen","enabled":false}'`.
### Step 12 — anti-jitter output buffer
The board plays nothing until it holds **300 ms of audio in reserve** (priming); the
Wi-Fi may then deliver chunks late or in bursts without it being heard. If the reserve
runs dry, playback stops cleanly and resumes once the reserve is rebuilt (re-priming)
instead of stuttering. The `played` report gives the delay to the first sample, the
**minimum margin** (network delay still absorbable) and the number of re-primings.
1. `make fw-upload SERIAL_PORT=/dev/ttyUSB0` (step 12 firmware).
2. `make tone-mqtt`: expected `reamorcages 0`, minimum margin ≈ 250 ms.
3. `make tone-mqtt JITTER=200`: a 200 ms stall every 500 ms; sound still continuous,
`reamorcages 0`, minimum margin ≈ 100 ms.
4. `make tone-mqtt JITTER=400`: beyond the reserve; clean pauses (re-primings > 0),
never any crackling.
5. `make run`: assistant answers, with `reamorcages 0` at the usual signal level.
### Step 11 — home automation
A command is three words: an **action** (allumer, éteindre, ouvrir, fermer), a **device**
(lumière, volets, porte) and a **room** (listed in `config.yaml`, section `home`). Without a
room, the board's own room is used. The server publishes `home/<room>/<device>/set` for any
home automation system; when the room is the board's, the board also executes it: demo LED
on GPIO 21 for the light, simulated shutters and door (serial log).
1. Wire the LED (optional): GPIO 21 → 330 Ω → LED → GND ([docs/BROCHAGE.md](docs/BROCHAGE.md) §10).
2. `make fw-upload SERIAL_PORT=/dev/ttyUSB0` (step 11 firmware), then `make run`.
3. In another terminal: `make mqtt-watch` (voice/… as text and home/#).
4. "Allume la lumière" → "J'allume la lumière du salon.", LED on, `home/salon/light/set`
and the confirmation `{"event":"device",...,"ok":true}`.
5. "Ferme les volets de la cuisine" → `home/cuisine/shutter/set` only (another room).
6. "Allume les volets" → "Je ne sais pas allumer les volets."; "Ouvre" → "Que dois-je ouvrir ?".
**Security**: voice identification is never proof. Do not connect a door strike or an
alarm to these commands; per-profile rights come in step 14.
### Step 10 — who said what
Whisper gives the time of each word, the diarization the time of each speaker turn, on
the same audio: each word goes to whoever was speaking at that moment. The server
publishes the result on `voice/<board>/transcript` and the assistant calls the person by name.
```json
{"session": "453cce", "utterances": [
{"speaker": "Denis", "text": "Ouvrir la porte.", "start": 0.0, "end": 1.26, "score": 0.6},
{"speaker": "Siwis", "text": "Je suis la voix de synthèse...", "start": 1.5, "end": 4.46, "score": 0.6},
{"speaker": "Denis", "text": "Fermez la porte.", "start": 4.92, "end": 5.86, "score": 0.6}]}
```
1. `make diarize OUT=server/recordings/<file>.wav`: who said what, and the MQTT message, no board needed.
2. `make mqtt-watch` in one terminal, `make run` in another.
3. "Bonjour": the board answers "Bonjour Denis !"; `voice/esp32-01/transcript` shows up.
4. A two-voice session (you, a phone voice, you again): three utterances.
### Step 9 — speaker identification
Each voice found by the diarization is summarised by an **embedding** (256 numbers,
WeSpeaker model from the pyannote pipeline), compared by **cosine similarity** with the
enrolled profiles. Above the threshold (0.45, measured on this bench), the name;
otherwise "Inconnu". **It is an estimate, never a security feature**: a recording or an
impersonation can fool it.
1. `make run`, then 5 varied sentences of 1 to 4 s, alone, at a normal distance.
2. `make enroll NAME=Denis`: profile built from those 5 sessions; sessions that are too
short, contain several voices or do not match the others are dropped.
3. New questions (the server picks up the profile without restarting): the log shows
`SPEAKER_00 = Denis (0.74)`; another person or a loudspeaker voice gives
`Inconnu (0.05, proche de Denis)`.
4. `make diarize-demo`: your voice recognised, Piper's voice "Inconnu".
Profiles are **biometric data**: `server/profiles/`, never versioned, one readable JSON
file per person; `make forget NAME=...` deletes it.
### Step 8 — diarization: who speaks when
The speech found by the VAD is split into speaker turns: `SPEAKER_00` from 0.0 to 1.3 s,
`SPEAKER_01` from 1.4 to 2.9 s... Labels only hold within one session: putting a name on
a voice is step 9. Model `pyannote/speaker-diarization-community-1` (33 MB), run on CPU,
**pyannote telemetry disabled** (enabled by default in version 4).
1. Create an account on <https://huggingface.co>, then **accept the conditions** at
<https://huggingface.co/pyannote/speaker-diarization-community-1>.
2. Create a "Read" token at <https://huggingface.co/settings/tokens> and put it in
`server/.env`: `HF_TOKEN=hf_...` (`cp server/.env.example server/.env` if missing).
3. `make install-diarization` then `make models`. No Internet access afterwards.
4. `make diarize-demo`: your voice, Piper's voice, your voice → two speakers expected.
5. `make run`: one session where two people take turns; the log shows
`2 locuteur(s) : SPEAKER_00 0.00-1.30, SPEAKER_01 1.40-2.90`.
Too heavy for a Raspberry Pi 4: set `diarization.enabled: false` in `config.yaml`.
### Step 7 — voice activity detection (VAD)
The server now transcribes speech only: leading and trailing silence is removed,
sessions without speech (silent press, click) are dropped without calling Whisper.
Silero ships with faster-whisper: nothing more to install.
1. `make vad-all`: one character per 32 ms window (`#` speech, `_` silence) for
each recording, plus the number of sessions without speech.
2. `make vad OUT=server/recordings/<file>.wav` then `make play OUT=recordings/parole.wav`:
speech only, silences removed.
3. `make run` (restart it: the code changed), ask a few questions and press once
without speaking. The log shows `parole 1.36 s sur 2.00 s (segments 0.49-1.85 s)`,
and `sans parole (VAD)` for the silent press, answered immediately.
Settings in `config.yaml`, section `vad`: threshold, minimum pause, minimum speech, padding.
### Step 6 — MVP: the assistant answers
1. `make install-speech` then `make models` (faster-whisper `small`, Piper voice
`fr_FR-siwis-medium`, about 550 MB in `server/models/`, once).
2. `make speech-check`: Piper says "Quelle heure est-il ?", Whisper reads it
back, the assistant answers; no hardware needed.
3. `make transcribe OUT=server/recordings/<file>.wav`: transcribes a sentence
recorded by the ESP32 in step 4.
4. `make run`: press the button, ask "Quelle heure est-il ?", release; the board
answers. Recognised questions: time, date, hello, thanks, your name; anything
else is repeated back as understood.
The log shows the time spent in each stage: `transcription 1.0 s, synthese
0.1 s, reponse prete en 1.1 s`. `make run-echo` keeps the step 5 echo mode,
without models.
### Step 5 — the server makes the ESP32 speak, over MQTT
1. `make fw-upload SERIAL_PORT=/dev/ttyUSB0` (step 5 firmware).
2. `make tone-mqtt`: 3 s of pure tone sent over MQTT; the board report is printed
(`chunks 60/60, underruns 0`).
3. `make play-mqtt OUT=server/recordings/<file>.wav`: any WAV, converted to
16 kHz mono, peak at −3 dBFS.
4. `make run-echo`: the server plays each sentence back on the board that said it.
### Step 4 — voice to the server, over MQTT
1. `make install` (server dependencies: paho-mqtt, PyYAML, numpy...).
2. `make run`: the server connects to the broker and waits for sessions.
3. `make fw-upload SERIAL_PORT=/dev/ttyUSB0`, then press the button, speak, release.
4. The server writes `server/recordings/esp32-01_<date>_<session>.wav` and logs a
report: `complete`, received / announced chunks, peak and RMS.
5. `make play OUT=server/recordings/<file>.wav` or `make inspect OUT=...`.
If the broker is unreachable when the button is pressed, the session goes over
the serial link: `make record` remains available for troubleshooting.
### Step 3 — Wi-Fi and MQTT
1. Broker: see [docs/MOSQUITTO.md](docs/MOSQUITTO.md) (Debian service or container).
2. `cp firmware/include/secrets.h.example firmware/include/secrets.h`, then set
the Wi-Fi (2.4 GHz) and the broker IP address.
3. `make mqtt-watch MQTT_HOST=<broker ip>` in a first terminal.
4. `make fw-upload SERIAL_PORT=/dev/ttyUSB0` in a second one.
Expected in the first terminal: `voice/esp32-01/state {"status":"online",...}`
and `voice/esp32-01/event {"event":"boot",...}`. After cutting the ESP32 power:
`voice/esp32-01/state {"status":"offline"}` within 15 s.
### Step 2 — test the audio output
1. Wire the MAX98357A amplifier and the speaker (§2), including C1 and C2.
2. `make fw-upload SERIAL_PORT=/dev/ttyUSB0`
3. At boot, a **440 Hz beep** plays: it tests the output on its own.
4. `make tone-esp`: 3 s of pure tone streamed from the PC in real time. Any
dropout is heard as a click.
5. `make play-esp OUT=recordings/test.wav`: the step 1 recording played back on
the ESP32 speaker.
Expected: clean tone, no clicks or crackle; the ESP32 report shows `0 refuses`
(rejected) and `0 sous-alimentations` (underruns).
### Step 1 — test the audio input
1. Wire microphone, button and LED as in §2.
2. `make fw-upload SERIAL_PORT=/dev/ttyUSB0`
3. `make record OUT=recordings/test.wav`
4. Press the button, say a sentence, release it.
5. Listen: `make play OUT=recordings/test.wav`
6. With no audio output: `make inspect OUT=recordings/test.wav` draws the signal
envelope in ASCII and reports peak, RMS and DC offset.
Expected: the sentence is intelligible and not clipped; the script reports a peak
between 10 % and 90 % of full scale. Above 99 %, lower `MIC_GAIN` in
[firmware/include/config.h](firmware/include/config.h); below 2 %, raise it.
---
## 7. Repository layout
```
voice-assistant/
├── firmware/ ESP32 firmware (PlatformIO, Arduino)
│ ├── include/ config.h, secrets.h.example
│ └── src/ main.cpp + one .h/.cpp pair per class
├── server/ Python server (one module per class)
├── tools/ standalone test tools
├── mosquitto/ voice.conf
├── Makefile every system operation
├── CONTEXT.md progress log and frozen technical decisions
└── README.md / README-EN.md
```
---
## 8. License
MIT — see [LICENSE](LICENSE).