NVIDIA engineer builds talking Halloween garage door with Nemotron open models

A garage door that complains in pirate

According to a post on Hugging Face, the author built MonsterHouse for Halloween so that visiting children could talk to the house itself. The character is a garage door with a mouth, given a backstory as a building made from the timbers of a sunken pirate ship. It brushes visitors off at first, responds immediately to a polite "please", and gives a warm farewell after "thank you".

The author says NVIDIA Nemotron Voice Agent provides the conversational foundation. Nemotron speech recognition handles listening, Nemotron 3.5 Bolt E4B 4GB writes replies, Magpie TTS Zeroshot produces the voice, and Audio2Face-3D drives the mouth animation in Unreal Engine 5. The character mesh was built in Autodesk Maya before export to Unreal Engine.

How the pipeline is put together

Bolt E4B is described as an open small language model for conversational agents. The author chose it after testing more than a dozen models, using the 4-bit quantized release as the default, with the BF16 checkpoint available for comparison. The author says it has a smaller footprint than Qwen3 30B-A3B and Qwen 3.6 35B-A3B. A LoRA adapter trained on a dataset of over 5000 multi-turn conversations supplies the personality; the data was created and judged by a local agent and a very large LLM.

Magpie TTS Zeroshot conditions the voice on a short reference sample rather than a trained voice. The house has seven moods, and emotion tags from the LLM select delivery. Each emotion needs a 3-10 second sample, so the author added a voice-sampling block to the control interface.

The work is split across two GPUs. An NVIDIA RTX PRO 6000 Blackwell Max-Q runs the language and voice pipeline, while an NVIDIA RTX PRO 4000 Blackwell runs Unreal Engine graphics and Audio2Face-3D. The author says compute, not memory, is the constraint, and the split keeps the animation frame rate steady.

For speed, the pipeline releases speakable text before a full answer is finished, and prepared greetings and jokes use cached text and recordings. The author reports median end-of-speech-to-audio times of 1.70 seconds with Bolt Q4 and TTS, and 0.72 seconds for cached responses. Speech-end detection currently waits for 300 ms of silence, and a 100 ms setting compensates for Audio2Face-3D buffering.

Controls, vision and open items

A control desk lets the author audition cached lines, change the opening mood, tune the mouth animation and monitor performance. Live mode runs the trick-or-treat character, demo mode has the house explain itself, and microphone mode lets the author speak through the house directly, bypassing recognition, dialogue generation and speech synthesis.

An optional vision layer uses Nemotron 3 Nano Omni to describe costumes, while YOLO and ByteTrack count and track people in view. In a synthetic test video of five costumed sheep, the saved analysis identified pirate, vampire, witch, pumpkin and ghost outfits. The author notes this is not evidence of a successful Halloween night, that real groups and night lighting still need testing, and that costume identification and retorts may need post-training.

Remaining goals include better animation, performance testing with real visitors and automatic candy handout. The author has not yet reached the aim of leaving the house to run Halloween unattended.

Building a similar character

The author says the MonsterHouse code is not packaged for release, but all the NVIDIA models are available. The suggested path starts with cloning the Nemotron Voice Agent and running its Generic Assistant example, which includes Nemotron Speech ASR, an LLM, Magpie TTS, voice activity detection and end-of-utterance handling.

The Voice Agent includes Bolt E4B profiles for cloud, server and single-GPU deployments, and the model can be tried on build.nvidia.com. The author suggests conversational mode for short replies, disabling reasoning for simple exchanges, keeping state such as who has said please in application code, and training a LoRA adapter when prompting falls short. NVIDIA NeMo AutoModel has a LoRA recipe for Bolt E4B.

Magpie TTS Multilingual is the Voice Agent default, while Magpie TTS Zeroshot suits custom characters and requires requesting access to run locally. The author advises using only voices you have permission to use. For limited hardware, the author suggests generating hundreds of responses offline with Bolt E4B and Magpie, then serving them from a cache with Nemotron 3 ASR and a Python conversation router.