Build your own Lumi

uv is a tool that installs Python and everything Lumi needs, without touching the rest of your computer. Copy each block into a terminal (Terminal on macOS, PowerShell on Windows).

minilumi/ is one small Python package: listen.py (when you stop talking), hear.py, think.py, speak.py, visit.py (the conversation loop) and screen.py (the face). config.toml holds three presets: workstation, laptop and cloud.

Python 3.11+. Base dependencies are just numpy, sounddevice and anthropic; local speech is the optional local extra. HTTP to model servers is the standard library, so any OpenAI-compatible server works.

Lumi records you until you stop talking, then hands the recording to a model that turns it into words. Those words are what she "heard".

hear.py takes 16 kHz mono audio. With kind = "whisper-local" it runs faster-whisper in-process; with kind = "openai" it posts a WAV to any /v1/audio/transcriptions endpoint.

The original used NVIDIA Parakeet behind an OpenAI-style server: fast and accurate for English. Accuracy problems were almost always the microphone, not the model: a directional mic pointed at the visitor beat every software fix.

A language model is great at conversation but knows nothing about your event. So Lumi gives it a cheat sheet first: the rooms, the sessions, where lunch is. Then it answers using only that.

event.py builds the system prompt from your CSV files: persona, event facts, rooms and sessions. The visitor's name, the time and their next sessions go into their first message, not the system prompt. think.py streams the reply back word by word.

Keep the system prompt byte-for-byte identical for every visitor. Then the model server can reuse its prompt cache and skip re-reading thousands of tokens. Anything per-person in the system prompt breaks the cache for everyone. Small models sometimes "think out loud" in <think> tags; minilumi asks the server not to, and strips any that slip through.

Lumi doesn't wait for the whole answer. As soon as the first sentence is written, she starts saying it while the model writes the next one. That's why she feels quick.

visit.py regroups the streamed text into sentences; speak.py turns each into audio with Kokoro (in-process via kokoro-onnx, or any /v1/audio/speech server that returns WAV).

Names are where TTS falls apart. Keep a small pronunciation list (word → respelling) and apply it right before speaking, never to the captions. Some phrases are simply misread ("Uh oh" came out wrong; "Oh no" didn't). Test every scripted line by listening to it.

Lumi listens to how loud the room is all the time. When someone is clearly louder than the room, she starts recording. When they've been quiet for a moment, she stops and answers.

listen.py measures the room as a low percentile of the last 1.5 s of loudness. A turn starts at 4× the room level, continues while it's above 2×, and ends after 0.7 s below that. Anything shorter than 0.25 s, or never really loud, is dropped (coughs, clicks).

We recorded 75 seconds of real crowd noise and replayed it through the endpointer over and over while tuning. The room level must be re-measured continuously; one measurement at startup goes stale the moment a crowd walks in. Known limit: someone speaking softly in a loud crowd won't be heard. Lowering the bar let stray "yeah"s from passers-by in. We kept the bar high.

The face is just a web page. Python tells the page what's happening ("listening", "speaking", "here's the sentence") and the page animates the mouth to match the voice.

screen.py serves ui/ and pushes messages with Server-Sent Events. Each spoken sentence arrives with its envelope, the loudness of the audio every 20 ms, and the mouth's height follows it.

When sentences queue up, a sentence's audio starts in the future (after the ones ahead of it). The original screen guarded against clock skew with a symmetric clamp, which treated "starts in 8 seconds" as an error. Lip sync silently broke for every multi-sentence reply, and that got through eleven separate code reviews. Only a review of the whole branch caught it. Clamp asymmetrically: tolerate a little lateness, a lot of earliness.

Copy example-event/ to my-event/, open the copy in a spreadsheet or text editor, and replace the made-up "Example Con" with your own event: rooms with spoken directions, sessions, attendees, and any facts in event.txt. Then set event_dir = "my-event" in config.toml. (my-event/ is git-ignored, so your real data stays out of git.)

rooms.csv: name, floor, directions. sessions.csv: title, room, start, end, speaker. attendees.csv: badge, name, and their sessions separated by ;. A scanned badge greets that person with their next session.

Build a "pretend it's 8 AM on day one" clock for rehearsals, so you can test what Lumi says at every point of the event. In minilumi that's the now= argument of Visit. And keep the last good copy of the data: a half-saved CSV at the wrong moment shouldn't leave Lumi knowing nothing.

Change her name, tone and voice. Have her greet people by nickname. The original had a pronunciation list, playful greetings, and a few surprise "shows" for specific badges.

Personality is the PERSONA line at the top of event.py. For nicknames, add a column to attendees.csv, read it in load_event (event.py), and use it in Visit._greeting.

Scripted moments should skip the language model: match the badge before calling it, then play pre-written lines with pre-rendered audio and a scripted animation. Every word is reviewed in advance, and nothing can improvise its way into an awkward moment in front of a crowd.

Have Lumi start by herself when the computer starts, and restart herself if she ever stops. The simplest way: a terminal that opens at login and runs her in a loop.

A barcode scanner types into whichever window has focus, so Lumi's terminal must keep focus. Wrap the command in a loop so a crash restarts it in two seconds.

On Linux, the original read the scanner directly (evdev, with a udev rule for permissions), so focus didn't matter. It also had a self-check command that failed on a muted speaker, kept transcripts out of logs entirely, and capped the GPU clock after the machine powered itself off under sustained load.