Back to projects

Exploring

JARVIS

A local desktop voice assistant experiment that's meant to actually do things, not just transcribe them.

voice / assistant / experimental

JV
Pythonfaster-whisperOllamaKokoro TTS

Why I built it

JARVIS is a local desktop voice assistant inspired by the idea of having a personal computer assistant that can actually perform useful actions, not just answer questions out loud.

The interaction is built around natural voice rather than typing commands: speak, get understood, get something done.

What it does

Bram speaks, speech gets transcribed, the assistant reasons about the request, a safe tool may be selected, the action gets performed, and JARVIS responds through speech.

Key features

Voice input

  • Push-to-talk interaction
  • Natural spoken input, including Dutch
  • faster-whisper (Whisper small), CUDA/float16 where available, 16 kHz mic input

Voice output

  • Responds in English with a short assistant/butler-like style
  • Started on Windows/SAPI via pyttsx3, later moved toward Kokoro TTS
  • Multiple voice experiments along the way

Reasoning

  • Local models through Ollama during development
  • Qwen model experimentation
  • Exploration of alternative LLM providers

Tools / actions

  • Open and close applications
  • Control volume
  • Open websites, run searches
  • Report CPU, RAM and disk information
  • Date/time
  • Media-related actions, Spotify experimentation

How it works

  1. 1Bram speaks
  2. 2Speech is transcribed
  3. 3Assistant reasons about the request
  4. 4A safe tool may be selected
  5. 5The action is performed
  6. 6JARVIS responds through speech

Architecture notes

The pipeline is deliberately split into distinct stages: conversation, reasoning, tool selection, execution, and spoken feedback. JARVIS acknowledges an action naturally in speech and then executes it. The two are kept as separate steps rather than one blended guess.

That separation is what makes tool experiments (opening apps, checking system info, media control) safe to try without turning the assistant into an unrestricted shell.

Tech stack

Pythonfaster-whisperOllamaKokoro TTS

Interesting technical details

Detail

Explicit tools only

No arbitrary shell or code execution. JARVIS can only call a fixed set of explicit, named tools, nothing more general than that.

Detail

Controlled autonomy

Human approval where appropriate, sandboxed actions, and read-only internet access where possible. The goal is useful autonomy with an explicit ceiling, not maximum autonomy.

Detail

Confidence vs. real state

An assistant saying "I opened Spotify" is useless if Spotify didn't actually open. Making tool execution match what the assistant claims it did, not just sound like it did, turned out to be one of the more interesting parts of this project.

Challenges & decisions

Decision

Speech recognition in practice

Accuracy, Dutch transcription specifically, GPU/CUDA setup, and voice latency all needed real tuning. None of it worked well by default.

Decision

TTS reliability

Getting text-to-speech to sound natural and respond reliably took multiple attempts, from Windows/SAPI through to Kokoro TTS.

Decision

Local inference speed

Running reasoning locally through Ollama means inference speed is a constant, practical constraint on how the assistant can respond.

Decision

Saying it vs. doing it

The gap between conversational confidence and actual tool state (the assistant sounding sure vs. the action actually having happened) is a real engineering problem, not just a nice-to-have polish item.

Current status

Active experiment: exploring what a useful local voice assistant looks like.

  • Push-to-talk voice interaction with Dutch input and English spoken responses works
  • A working set of tool experiments exists (apps, volume, system info, media, search); not all are production-reliable
  • Local reasoning runs through Ollama with Qwen model experimentation
  • TTS has moved from Windows/SAPI toward Kokoro TTS through several voice experiments

What's next

The interesting part of this project isn't the voice interface itself. It's the boundary between what the assistant says it did and what actually happened on the machine. Getting that boundary right matters more than any individual tool.

Wake-word activationBetter conversational memory and interruption handlingHigher tool reliability and more natural spoken feedbackMore autonomy while keeping explicit safety boundaries