Major Rewrite

A rewrite incorporating developments from the last few days.

.gitignore

@@ -0,0 +1,2 @@
+*.pyc
+mOrpheus.txt

README.md

@@ -1,17 +1,25 @@
 # mOrpheus Virtual Assistant Demo
 
-This project implements the Morpheus Virtual Assistant, which integrates speech recognition, text generation, and text-to-speech (TTS) synthesis. The assistant utilizes the following components:
+This project implements the mOrpheus Virtual Assistant, which integrates speech recognition, text generation, and text-to-speech (TTS) synthesis. The assistant leverages several state-of-the-art components:
 
 - **Whisper** for speech recognition.
-- **LM Studio API** for text generation (chat) and text-to-speech.
+- **LM Studio API** for both text generation (chat) and text-to-speech synthesis.
 - **SNAC-based decoder** to convert TTS token streams into PCM audio.
 
 ## Features
 
-- **Speech Recognition:** Captures audio input from the microphone and transcribes it using the Whisper model.
-- **Text Generation:** Uses LM Studio’s chat API to generate responses based on the transcribed input.
-- **Text-to-Speech:** Synthesizes speech from text using LM Studio’s TTS API and decodes the token stream with a SNAC-based decoder.
-- **Audio Playback:** Plays the generated audio and checks its duration to warn if the audio might be truncated.
+- **Speech Recognition:**  
+  Captures audio from the microphone and transcribes it using the Whisper model.
+
+- **Text Generation:**  
+  Uses LM Studio’s chat API to generate natural language responses based on transcribed input.
+
+- **Text-to-Speech:**  
+  Converts the generated text into speech via LM Studio’s TTS API. The text is cleaned (removing newlines, markdown symbols, and emojis) before being sent to the TTS engine, and the resulting token stream is decoded using a SNAC-based decoder.
+  
+
+- **Audio Playback & Activation:**  
+  Plays the generated audio and waits for the next activation. Long responses are segmentented then recombined at playback. The assistant supports both push‑to‑talk (keypress) and hotword (“Hey Assistant”) activation, or both simultaneously, as configured.
 
 ## Requirements
 
@@ -45,61 +53,143 @@ This project implements the Morpheus Virtual Assistant, which integrates speech
 
 3. **Configure the application:**
 
-   - Create a `config.yaml` file in the project root.
-   - Populate it with your configuration details for Whisper, LM Studio API, audio settings, etc.
+   - Create a `settings.yml` file in the project root.
+   - Populate it with your configuration details for Whisper, LM Studio API, audio settings, interaction mode, etc.
 
-   Example `config.yaml`:
+   Example `settings.yml`:
 
    ```yaml
+   # -------------------------------
+   # Configuration for Whisper STT
+   # -------------------------------
    whisper:
-     model_name: "base"
+     model: "small.en"
      sample_rate: 16000
 
-   lm_studio_api:
-     api_url: "http://your-api-url.com"
+   # -------------------------------
+   # Configuration for LM Studio (Chat & TTS)
+   # -------------------------------
+   lm:
+     api_url: "http://127.0.0.1:1234/v1"
+     
      chat:
-       endpoint: "/v1/chat"
-       model: "gemma"
+       endpoint: "/chat/completions"
+       model: "gemma-3-12b-it"
        system_prompt: "You are a helpful assistant."
-       max_tokens: 150
+       max_tokens: 256
        temperature: 0.7
        top_p: 0.9
-       repetition_penalty: 1.0
-     tts:
-       endpoint: "/v1/tts"
-       model: "orpheus"
-       default_voice: "default"
-       max_tokens: 200
-       temperature: 0.8
+       repetition_penalty: 1.1
+       max_response_time: 10.0
 
+     tts:
+       endpoint: "/completions"
+       model: "orpheus-3b-ft.gguf@q2_k"
+       default_voice: "tara"
+       max_tokens: 4096
+       temperature: 0.6
+       top_p: 0.9
+       repetition_penalty: 1.0
+       speed: 1.0
+       max_segment_duration: 20
+
+   # -------------------------------
+   # TTS Audio Output Configuration
+   # -------------------------------
    tts:
      sample_rate: 24000
 
+   # -------------------------------
+   # Audio Device Configuration
+   # -------------------------------
    audio:
      input_device: null
      output_device: null
+     hotword_sample_rate: 16000
 
-   desired_tts_duration: 20
+   # -------------------------------
+   # Voice Activity Detection (VAD) Configuration
+   # -------------------------------
+   vad:
+     mode: 2
+     frame_duration_ms: 30
+     silence_threshold_ms: 1000
+     min_record_time_ms: 2000
+
+   # -------------------------------
+   # Hotword Detection Configuration
+   # -------------------------------
+   hotword:
+     enabled: true
+     phrase: "Hey Assistant"
+     sensitivity: 0.7
+     timeout_sec: 5
+     retries: 3
+
+   # -------------------------------
+   # Segmentation Configuration for TTS
+   # -------------------------------
+   segmentation:
+     max_words: 60
+
+   # -------------------------------
+   # Speech Quality Configuration
+   # -------------------------------
+   speech:
+     normalize_audio: false
+     default_pitch: 0
+     min_speech_confidence: 0.5
+     max_retries: 3
+
+   # -------------------------------
+   # Interaction Configuration
+   # -------------------------------
+   interaction:
+     mode: "both"  # Options: "push_to_talk", "hotword", or "both"
+     post_audio_delay: 0.5
    ```
 
-4. **Run the Assistant:**
+4. **Run LM Studio**
+Before activating the assistant you need to have LM Studio running both the LLM and Orpheus model as defined in the settings.yml in API mode. This is only accessibly in Power User or Developer Mode respectively.
+
+5. **Run the Assistant:**
 
    ```bash
-   python morpheus_demo.py
+   python morpheus.py
    ```
-Make sure you have your choosen LLM model and the Orpheus 4-bit GGUF loaded inside LM Studio and that you are in API mode. 
+
+
+## Suggested Models
+
+These are the models I tested with, your milage may vary with additional models.
+
+**LLM Models:**
+- gemma-3-12b-it-GGUF/gemma-3-12b-it-Q3_K_L.gguf
+
+**Orpheus Models:**
+- lex-au/Orpheus-3b-FT-Q2_K.gguf - Fastest inference (~50% faster tokens/sec than Q8_0).
+- lex-au/Orpheus-3b-FT-Q4_K_M.gguf - Balanced quality/speed.
+- lex-au/Orpheus-3b-FT-Q8_0.gguf - Original high-quality model.
 
 ## Usage
 
-- The assistant will begin by listening for your voice input.
-- After transcribing your speech, it will generate a text response using the LM Studio API.
-- The response is then converted to speech using the TTS API, decoded via SNAC, and played back.
-- If the generated audio duration is below the configured threshold, a warning is printed.
+- **Activation:**  
+  The assistant listens for activation either via a hotword ("Hey Assistant") or a push-to-talk keypress (or both, depending on your settings).  
+- **Speech Processing:**  
+  It records your speech, transcribes it using Whisper, and generates a text response via LM Studio’s chat API.
+- **TTS Synthesis:**  
+  The response is cleaned to remove unwanted characters (e.g., emojis, newlines, markdown formatting) and then sent to LM Studio’s TTS API. The SNAC-based decoder converts the TTS token stream into PCM audio.
+- **Audio Playback:**  
+  The generated audio is played back, and a brief delay is applied before the assistant waits for the next activation.
 
 ## Contributing
 
-Feel free to open issues or submit pull requests with improvements or bug fixes.
+Feel free to open issues or submit pull requests with improvements or bug fixes. I'm fairly new at this and feel this could see a lot of improvements.
 
 ## License
 
 This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.
+
+## Notice
+
+See the [NOTICE] (NOTICE) file for details.

config.yaml

@@ -1,33 +0,0 @@
-# Configuration for Morpheus Virtual Assistant
-
-whisper:
-  model_name: "small.en"          # Name of the Whisper model used for speech recognition.
-  sample_rate: 16000              # Audio sample rate (Hz) for recording.
-
-lm_studio_api:
-  api_url: "http://127.0.0.1:1234"  # Base URL for the LM Studio API.
-  chat:
-    endpoint: "/v1/chat/completions"         # API endpoint for chat-based text generation.
-    model: "gemma-3-1b-it"                     # Model ID for text generation (Gemma).
-    system_prompt: "You are a smart assistant with a knack for humor."
-      # System prompt to guide Gemma's responses.
-    max_tokens: 2500                         # Maximum tokens to generate for chat responses.
-    temperature: 0.7                         # Sampling temperature (controls randomness).
-    # Optional parameters; remove or ignore if your model does not support them.
-    top_p: 0.9                               # Top-p (nucleus) sampling parameter.
-    repetition_penalty: 1.1                  # Penalty to reduce repetitive text.
-  tts:
-    endpoint: "/v1/completions"              # API endpoint for text-to-speech synthesis.
-    model: "orpheus-3b-0.1-ft"               # Model ID for TTS (Orpheus).
-    default_voice: "tara"                    # Default TTS voice.
-    max_tokens: 2500                         # Maximum tokens to generate for TTS output.
-    temperature: 0.6                         # Sampling temperature for TTS.
-    top_p: 0.9                               # Top-p sampling parameter for TTS.
-    repetition_penalty: 1.0                  # Repetition penalty for TTS generation.
-
-tts:
-  sample_rate: 24000            # Audio sample rate (Hz) for TTS output.
-
-audio:
-  input_device: 15              # Audio input device ID (e.g., NVidia Broadcast via Windows DirectSound).
-  output_device: 21             # Audio output device ID (e.g., Corsair Void Elite via Windows DirectSound).

mOrpheus.html

@@ -0,0 +1,376 @@
+<!DOCTYPE html>
+<html>
+<head>
+  <meta charset='utf-8'>
+  <title>File Summary</title>
+  <style>
+    body { font-family: Arial, sans-serif; margin: 20px; }
+    pre  { background-color: #f4f4f4; padding: 10px; white-space: pre-wrap; }
+  </style>
+</head>
+<body>
+  <h1>File Summary</h1>
+  <h2>Directory Structure</h2><pre>├── NOTICE├── config.yaml├── modules/│   ├── lmstudio_client.py│   ├── snac_decoder.py│   ├── virtual_assistant.py│   └── whisper_recognizer.py├── morpheus_demo.py</pre><h2>Files Content</h2><h3>NOTICE</h3><pre>This project includes code derived from "orpheus-tts-local" by Isaiah Bjork, available at:
+https://github.com/isaiahbjork/orpheus-tts-local
+
+Copyright 2025 Isaiah Bjork
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at:
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed 
+under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR 
+CONDITIONS OF ANY KIND, either express or implied. See the License for the 
+specific language governing permissions and limitations under the License.</pre><h3>config.yaml</h3><pre># Configuration for Morpheus Virtual Assistant
+
+whisper:
+  model_name: "small.en"          # Name of the Whisper model used for speech recognition.
+  sample_rate: 16000              # Audio sample rate (Hz) for recording.
+
+lm_studio_api:
+  api_url: "http://127.0.0.1:1234"  # Base URL for the LM Studio API.
+  chat:
+    endpoint: "/v1/chat/completions"         # API endpoint for chat-based text generation.
+    model: "gemma-3-1b-it"                     # Model ID for text generation (Gemma).
+    system_prompt: "You are a smart assistant with a knack for humor."
+      # System prompt to guide Gemma's responses.
+    max_tokens: 2500                         # Maximum tokens to generate for chat responses.
+    temperature: 0.7                         # Sampling temperature (controls randomness).
+    # Optional parameters; remove or ignore if your model does not support them.
+    top_p: 0.9                               # Top-p (nucleus) sampling parameter.
+    repetition_penalty: 1.1                  # Penalty to reduce repetitive text.
+  tts:
+    endpoint: "/v1/completions"              # API endpoint for text-to-speech synthesis.
+    model: "orpheus-3b-0.1-ft"               # Model ID for TTS (Orpheus).
+    default_voice: "tara"                    # Default TTS voice.
+    max_tokens: 2500                         # Maximum tokens to generate for TTS output.
+    temperature: 0.6                         # Sampling temperature for TTS.
+    top_p: 0.9                               # Top-p sampling parameter for TTS.
+    repetition_penalty: 1.0                  # Repetition penalty for TTS generation.
+
+tts:
+  sample_rate: 24000            # Audio sample rate (Hz) for TTS output.
+
+audio:
+  input_device: 15              # Audio input device ID (e.g., NVidia Broadcast via Windows DirectSound).
+  output_device: 21             # Audio output device ID (e.g., Corsair Void Elite via Windows DirectSound).</pre><h3>morpheus_demo.py</h3><pre>import yaml
+from modules.virtual_assistant import VirtualAssistant
+
+CONFIG_PATH = "config.yaml"
+with open(CONFIG_PATH, "r") as f:
+    config = yaml.safe_load(f)
+
+if __name__ == "__main__":
+    assistant = VirtualAssistant(config)
+    assistant.run()</pre><h3>modules\lmstudio_client.py</h3><pre>import os
+import time
+import json
+import wave
+import requests
+import threading
+import asyncio
+from .snac_decoder import tokens_decoder_sync
+
+class LMStudioClient:
+    """
+    Interfaces with the LM Studio API for text generation (chat) and text-to-speech.
+    """
+    def __init__(self, config_lm_api, tts_sample_rate):
+        self.api_url = config_lm_api["api_url"]
+        self.text_endpoint = config_lm_api["chat"]["endpoint"]
+        self.tts_endpoint = config_lm_api["tts"]["endpoint"]
+        self.default_model = config_lm_api["chat"]["model"]
+        self.tts_model = config_lm_api["tts"]["model"]
+        self.system_prompt = config_lm_api["chat"]["system_prompt"]
+        self.default_voice = config_lm_api["tts"]["default_voice"]
+        self.max_tokens = config_lm_api["chat"]["max_tokens"]
+        self.temperature = config_lm_api["chat"]["temperature"]
+        self.top_p = config_lm_api["chat"]["top_p"]
+        self.repetition_penalty = config_lm_api["chat"]["repetition_penalty"]
+        self.tts_max_tokens = config_lm_api["tts"]["max_tokens"]
+        self.tts_temperature = config_lm_api["tts"]["temperature"]
+        self.headers = {"Content-Type": "application/json"}
+        self.tts_sample_rate = tts_sample_rate
+
+    def generate_text(self, user_input):
+        messages = [
+            {"role": "system", "content": self.system_prompt},
+            {"role": "user", "content": user_input}
+        ]
+        payload = {
+            "model": self.default_model,
+            "messages": messages,
+            "max_tokens": self.max_tokens,
+            "temperature": self.temperature,
+            "top_p": self.top_p,
+            "repeat_penalty": self.repetition_penalty,
+            "stream": False
+        }
+        url = self.api_url + self.text_endpoint
+        print(f"Generating text for messages: {messages}")
+        response = requests.post(url, headers=self.headers, json=payload)
+        if response.status_code != 200:
+            raise RuntimeError(f"Text generation failed: {response.status_code} {response.text}")
+        data = response.json()
+        generated_text = data.get("choices", [{}])[0].get("message", {}).get("content", "").strip()
+        print(f"Generated text: {generated_text}")
+        return generated_text
+
+    def synthesize_speech(self, text, voice=None, output_file=None, desired_tts_duration=20):
+        voice = voice if voice else self.default_voice
+        prompt = f"<|audio|>{voice}: {text}<|eot_id|>"
+        payload = {
+            "model": self.tts_model,
+            "prompt": prompt,
+            "max_tokens": self.tts_max_tokens,
+            "temperature": self.tts_temperature,
+            "top_p": self.top_p,
+            "repeat_penalty": self.repetition_penalty,
+            "stream": True
+        }
+        url = self.api_url + self.tts_endpoint
+        print(f"Generating speech for prompt: {prompt}")
+        response = requests.post(url, headers=self.headers, json=payload, stream=True)
+        if response.status_code != 200:
+            raise RuntimeError(f"TTS request failed: {response.status_code} {response.text}")
+
+        def token_generator():
+            for line in response.iter_lines():
+                if line:
+                    decoded_line = line.decode("utf-8")
+                    if decoded_line.startswith("data: "):
+                        data_str = decoded_line[6:]
+                        if data_str.strip() == "[DONE]":
+                            break
+                        try:
+                            data = json.loads(data_str)
+                            token_text = data.get("choices", [{}])[0].get("text", "")
+                            yield token_text
+                        except json.JSONDecodeError as e:
+                            print(f"Error decoding JSON: {e}")
+
+        # Decode tokens into audio bytes using our SNAC-based decoder.
+        audio_bytes = tokens_decoder_sync(token_generator())
+        if not output_file:
+            output_file = f"outputs/{voice}_{int(time.time())}.wav"
+        os.makedirs(os.path.dirname(output_file), exist_ok=True)
+        with wave.open(output_file, "wb") as wf:
+            wf.setnchannels(1)
+            wf.setsampwidth(2)
+            wf.setframerate(self.tts_sample_rate)
+            wf.writeframes(audio_bytes)
+        print(f"Audio saved to {output_file}")
+        # Check the duration of the generated WAV file.
+        with wave.open(output_file, "rb") as wf:
+            frames = wf.getnframes()
+            rate = wf.getframerate()
+            duration = frames / float(rate)
+        if duration < desired_tts_duration:
+            print(f"Warning: Generated audio is only {duration:.2f} seconds long. Consider increasing tts_max_tokens in your configuration.")
+        return output_file</pre><h3>modules\snac_decoder.py</h3><pre>import torch
+import numpy as np
+import asyncio
+import threading
+import queue
+from snac import SNAC
+
+# Monkey-Patch torch.load to use weights_only=True by default
+original_torch_load = torch.load
+def patched_torch_load(*args, **kwargs):
+    kwargs.setdefault("weights_only", True)
+    return original_torch_load(*args, **kwargs)
+torch.load = patched_torch_load
+
+# Load the SNAC model used for decoding LM Studio TTS tokens into PCM audio.
+snac_model = SNAC.from_pretrained("hubertsiuzdak/snac_24khz").eval()
+snac_device = "cuda" if torch.cuda.is_available() else "cpu"
+print(f"Using SNAC on device: {snac_device}")
+snac_model = snac_model.to(snac_device)
+
+def convert_to_audio(multiframe, count):
+    if len(multiframe) < 7:
+        return
+    codes_0 = torch.tensor([], device=snac_device, dtype=torch.int32)
+    codes_1 = torch.tensor([], device=snac_device, dtype=torch.int32)
+    codes_2 = torch.tensor([], device=snac_device, dtype=torch.int32)
+    num_frames = len(multiframe) // 7
+    frame = multiframe[:num_frames*7]
+    for j in range(num_frames):
+        i = 7 * j
+        if codes_0.shape[0] == 0:
+            codes_0 = torch.tensor([frame[i]], device=snac_device, dtype=torch.int32)
+        else:
+            codes_0 = torch.cat([codes_0, torch.tensor([frame[i]], device=snac_device, dtype=torch.int32)])
+        if codes_1.shape[0] == 0:
+            codes_1 = torch.tensor([frame[i+1]], device=snac_device, dtype=torch.int32)
+            codes_1 = torch.cat([codes_1, torch.tensor([frame[i+4]], device=snac_device, dtype=torch.int32)])
+        else:
+            codes_1 = torch.cat([codes_1, torch.tensor([frame[i+1]], device=snac_device, dtype=torch.int32)])
+            codes_1 = torch.cat([codes_1, torch.tensor([frame[i+4]], device=snac_device, dtype=torch.int32)])
+        if codes_2.shape[0] == 0:
+            codes_2 = torch.tensor([frame[i+2]], device=snac_device, dtype=torch.int32)
+            codes_2 = torch.cat([codes_2, torch.tensor([frame[i+3]], device=snac_device, dtype=torch.int32)])
+            codes_2 = torch.cat([codes_2, torch.tensor([frame[i+5]], device=snac_device, dtype=torch.int32)])
+            codes_2 = torch.cat([codes_2, torch.tensor([frame[i+6]], device=snac_device, dtype=torch.int32)])
+        else:
+            codes_2 = torch.cat([codes_2, torch.tensor([frame[i+2]], device=snac_device, dtype=torch.int32)])
+            codes_2 = torch.cat([codes_2, torch.tensor([frame[i+3]], device=snac_device, dtype=torch.int32)])
+            codes_2 = torch.cat([codes_2, torch.tensor([frame[i+5]], device=snac_device, dtype=torch.int32)])
+            codes_2 = torch.cat([codes_2, torch.tensor([frame[i+6]], device=snac_device, dtype=torch.int32)])
+    codes = [codes_0.unsqueeze(0), codes_1.unsqueeze(0), codes_2.unsqueeze(0)]
+    if torch.any(codes[0] < 0) or torch.any(codes[0] > 4096) or \
+       torch.any(codes[1] < 0) or torch.any(codes[1] > 4096) or \
+       torch.any(codes[2] < 0) or torch.any(codes[2] > 4096):
+        return
+    with torch.inference_mode():
+        audio_hat = snac_model.decode(codes)
+    audio_slice = audio_hat[:, :, 2048:4096]
+    detached_audio = audio_slice.detach().cpu()
+    audio_np = detached_audio.numpy()
+    audio_int16 = (audio_np * 32767).astype(np.int16)
+    audio_bytes = audio_int16.tobytes()
+    return audio_bytes
+
+def turn_token_into_id(token_string, index):
+    token_string = token_string.strip()
+    last_token_start = token_string.rfind("<custom_token_")
+    if last_token_start == -1:
+        print("No token found in the string")
+        return None
+    last_token = token_string[last_token_start:]
+    if last_token.startswith("<custom_token_") and last_token.endswith(">"):
+        try:
+            number_str = last_token[14:-1]
+            return int(number_str) - 10 - ((index % 7) * 4096)
+        except ValueError:
+            return None
+    else:
+        return None
+
+async def tokens_decoder(token_gen):
+    buffer = []
+    count = 0
+    async for token_text in token_gen:
+        token = turn_token_into_id(token_text, count)
+        if token is None:
+            continue
+        if token > 0:
+            buffer.append(token)
+            count += 1
+            if count % 7 == 0 and count > 27:
+                buffer_to_proc = buffer[-28:]
+                audio_samples = convert_to_audio(buffer_to_proc, count)
+                if audio_samples is not None:
+                    yield audio_samples
+
+def tokens_decoder_sync(syn_token_gen):
+    audio_queue = queue.Queue()
+
+    async def async_token_gen():
+        for token in syn_token_gen:
+            yield token
+
+    async def async_producer():
+        async for audio_chunk in tokens_decoder(async_token_gen()):
+            audio_queue.put(audio_chunk)
+        audio_queue.put(None)  # Sentinel
+
+    def run_async():
+        asyncio.run(async_producer())
+
+    thread = threading.Thread(target=run_async)
+    thread.start()
+    audio_segments = []
+    while True:
+        audio = audio_queue.get()
+        if audio is None:
+            break
+        audio_segments.append(audio)
+    thread.join()
+    return b"".join(audio_segments)</pre><h3>modules\virtual_assistant.py</h3><pre>import wave
+import time
+import numpy as np
+import sounddevice as sd
+from .whisper_recognizer import WhisperRecognizer
+from .lmstudio_client import LMStudioClient
+
+class VirtualAssistant:
+    """
+    The main virtual assistant class that integrates Whisper, LM Studio API for chat and TTS,
+    and decodes TTS tokens into audio using the SNAC-based decoder.
+    """
+    def __init__(self, config):
+        self.recognizer = WhisperRecognizer(
+            model_name=config["whisper"]["model_name"],
+            sample_rate=config["whisper"]["sample_rate"]
+        )
+        self.lm_client = LMStudioClient(
+            config_lm_api=config["lm_studio_api"],
+            tts_sample_rate=config["tts"]["sample_rate"]
+        )
+        self.input_device = config.get("audio", {}).get("input_device", None)
+        self.output_device = config.get("audio", {}).get("output_device", None)
+        self.desired_tts_duration = config.get("desired_tts_duration", 20)
+
+    def play_audio(self, filename):
+        print("▶️ Playing audio...")
+        with wave.open(filename, "rb") as wf:
+            sample_rate = wf.getframerate()
+            audio_data = wf.readframes(wf.getnframes())
+            audio_array = np.frombuffer(audio_data, dtype=np.int16).astype(np.float32) / 32767.0
+            sd.play(audio_array, sample_rate, device=self.output_device)
+            sd.wait()
+
+    def get_wav_duration(self, filename):
+        with wave.open(filename, "rb") as wf:
+            frames = wf.getnframes()
+            rate = wf.getframerate()
+            return frames / float(rate)
+
+    def run(self):
+        print("\n🔄 Starting the virtual assistant. Press Ctrl+C to exit.\n")
+        try:
+            while True:
+                user_text = self.recognizer.transcribe(duration=5, device=self.input_device)
+                if not user_text.strip():
+                    print("⚠️ No speech detected. Please try again.")
+                    continue
+                response_text = self.lm_client.generate_text(user_text)
+                audio_file = self.lm_client.synthesize_speech(
+                    response_text,
+                    desired_tts_duration=self.desired_tts_duration
+                )
+                duration = self.get_wav_duration(audio_file)
+                print(f"Audio duration: {duration:.2f} seconds.")
+                self.play_audio(audio_file)
+                print("Waiting extra 1 second after playback to ensure full audio is played.")
+                time.sleep(duration + 1.0)
+        except KeyboardInterrupt:
+            print("\n👋 Exiting gracefully. Goodbye!")</pre><h3>modules\whisper_recognizer.py</h3><pre>import whisper
+import sounddevice as sd
+import scipy.io.wavfile as wav
+
+class WhisperRecognizer:
+    """
+    Uses the Whisper model to record and transcribe audio from the microphone.
+    """
+    def __init__(self, model_name, sample_rate):
+        print("🔊 Loading Whisper model...")
+        self.model = whisper.load_model(model_name)
+        self.sample_rate = sample_rate
+
+    def transcribe(self, duration=5, device=None):
+        print("\n🎙️ Listening...")
+        audio = sd.rec(int(duration * self.sample_rate), samplerate=self.sample_rate, channels=1, device=device)
+        sd.wait()
+        wav.write("input.wav", self.sample_rate, audio)
+        print("📝 Transcribing...")
+        result = self.model.transcribe("input.wav")
+        text = result["text"].strip()
+        print(f"👤 You said: {text}")
+        return text</pre>
+</body>
+</html>

modules/audio.py

@@ -0,0 +1,88 @@
+# modules/audio.py
+import os
+import time
+import wave
+import numpy as np
+import sounddevice as sd
+import webrtcvad
+from scipy.io.wavfile import write as wav_write
+from modules.logging import logger
+
+# Default parameters (can be overridden via configuration)
+VAD_MODE = 2
+FRAME_DURATION_MS = 30
+SILENCE_THRESHOLD_MS = 1000
+MIN_RECORD_TIME_MS = 2000
+DEFAULT_MAX_WORDS_PER_SEGMENT = 60
+
+def record_until_silence(sample_rate, device=None):
+    """
+    Records audio until a period of silence is detected.
+    """
+    try:
+        vad_inst = webrtcvad.Vad(VAD_MODE)
+        frame_length = int(sample_rate * FRAME_DURATION_MS / 1000)
+        silence_frames = int(SILENCE_THRESHOLD_MS / FRAME_DURATION_MS)
+        logger.info("Recording until %d ms of silence is detected...", SILENCE_THRESHOLD_MS)
+        recorded_frames = []
+        consecutive_silence = 0
+        start_time = time.time()
+
+        with sd.InputStream(samplerate=sample_rate, channels=1, device=device, blocksize=frame_length) as stream:
+            while True:
+                frame, _ = stream.read(frame_length)
+                frame_int16 = (np.squeeze(frame) * 32767).astype(np.int16).tobytes()
+                is_speech = vad_inst.is_speech(frame_int16, sample_rate)
+                recorded_frames.append(frame)
+                if not is_speech:
+                    consecutive_silence += 1
+                else:
+                    consecutive_silence = 0
+                elapsed_ms = (time.time() - start_time) * 1000
+                if consecutive_silence >= silence_frames and elapsed_ms > MIN_RECORD_TIME_MS:
+                    logger.info("Silence detected; stopping recording (elapsed %.0f ms)", elapsed_ms)
+                    break
+
+        audio = np.concatenate(recorded_frames, axis=0)
+        return audio
+    except Exception as e:
+        logger.error("Error during audio recording: %s", str(e))
+        raise
+
+def segment_text(text, max_words=DEFAULT_MAX_WORDS_PER_SEGMENT):
+    """
+    Splits text into segments of at most max_words, attempting to split at sentence boundaries.
+    """
+    words = text.split()
+    segments = []
+    while len(words) > max_words:
+        segment = " ".join(words[:max_words])
+        cutoff = max_words
+        for i, word in enumerate(segment.split()):
+            if word.endswith((".", "!", "?")):
+                cutoff = i + 1
+        segments.append(" ".join(words[:cutoff]).strip())
+        words = words[cutoff:]
+    if words:
+        segments.append(" ".join(words).strip())
+    return segments
+
+def combine_audio_files(file_list, output_file):
+    """
+    Combines multiple WAV files into one.
+    """
+    if not file_list:
+        return None
+    try:
+        with wave.open(file_list[0], "rb") as wf:
+            params = wf.getparams()
+        with wave.open(output_file, "wb") as out_wf:
+            out_wf.setparams(params)
+            for f in file_list:
+                with wave.open(f, "rb") as wf:
+                    out_wf.writeframes(wf.readframes(wf.getnframes()))
+        logger.info("Combined audio written to %s", output_file)
+        return output_file
+    except Exception as e:
+        logger.error("Error combining audio files: %s", str(e))
+        raise

modules/config.py

@@ -0,0 +1,20 @@
+# modules/config.py
+import os
+import yaml
+from modules.logging import logger
+
+class ConfigError(Exception):
+    pass
+
+_config_cache = None
+
+def load_config(config_file="settings.yml"):
+    global _config_cache
+    if _config_cache is None:
+        if not os.path.exists(config_file):
+            logger.error("Configuration file %s not found.", config_file)
+            raise ConfigError(f"Configuration file {config_file} not found.")
+        with open(config_file, "r", encoding="utf-8") as f:
+            _config_cache = yaml.safe_load(f)
+        logger.info("Configuration loaded from %s", config_file)
+    return _config_cache

modules/hotword_detector.py

@@ -0,0 +1,82 @@
+# modules/hotword_detector.py
+
+from typing import Optional
+import os
+import numpy as np
+import sounddevice as sd
+import time
+import tempfile
+import whisper
+from scipy.io.wavfile import write as wav_write
+from modules.logging import logger
+from modules.config import load_config
+
+class HotwordDetector:
+    def __init__(self, config=None):
+        self.config = config if config is not None else load_config()  # Use passed config if available
+        hotword_config = self.config["hotword"]
+        audio_config = self.config["audio"]
+
+        self.enabled = hotword_config["enabled"]
+        self.phrase = hotword_config["phrase"].lower()
+        self.sensitivity = hotword_config["sensitivity"]
+        self.timeout = hotword_config["timeout_sec"]
+        self.retries = hotword_config["retries"]
+        self.sample_rate = audio_config["hotword_sample_rate"]
+
+        logger.info("Loading Whisper model for hotword detection...")
+        self.whisper_model = whisper.load_model("tiny.en")
+        logger.info("Hotword detector ready (listening for '%s')", self.phrase)
+
+    def listen_for_hotword(self) -> bool:
+        if not self.enabled:
+            return True  # Bypass if disabled
+        logger.info("Listening for hotword: '%s'...", self.phrase)
+        for attempt in range(self.retries):
+            try:
+                # Use the dedicated check_for_hotword with a 1 second recording
+                if self.check_for_hotword(timeout=1.0):
+                    logger.info("Hotword detected!")
+                    return True
+                else:
+                    logger.debug("Attempt %d: Hotword not detected", attempt + 1)
+            except Exception as e:
+                logger.error("Hotword detection attempt %d failed: %s", attempt + 1, str(e))
+        logger.warning("Hotword not detected")
+        return False
+
+    def check_for_hotword(self, timeout: float = 1.0) -> bool:
+        """
+        Record for the full timeout duration and transcribe.
+        Returns True if the hotword is found in the transcription.
+        """
+        device = self.config["audio"]["input_device"]
+        try:
+            # Record audio for 'timeout' seconds
+            num_samples = int(self.sample_rate * timeout)
+            audio = sd.rec(num_samples, samplerate=self.sample_rate, channels=1, dtype='float32', device=device)
+            sd.wait()
+            # Convert audio to int16 as expected by the transcriber
+            audio_int16 = (audio.flatten() * 32767).astype('int16')
+            text = self._transcribe_audio(audio_int16)
+            logger.debug("Hotword check transcription: '%s'", text)
+            return self.phrase in text.lower()
+        except Exception as e:
+            logger.error("Hotword check failed: %s", str(e))
+            return False
+
+    def _transcribe_audio(self, audio):
+        if audio.size == 0:
+            return ""
+        # Use a temporary file for Whisper transcription
+        with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as tmp:
+            temp_file = tmp.name
+            wav_write(temp_file, self.sample_rate, audio)
+        try:
+            result = self.whisper_model.transcribe(temp_file)
+            return result.get("text", "").strip()
+        finally:
+            try:
+                os.remove(temp_file)
+            except Exception as e:
+                logger.warning("Could not delete temporary file %s: %s", temp_file, str(e))

modules/lm_client.py

@@ -0,0 +1,218 @@
+# modules/lm_client.py
+import os
+import time
+import json
+import wave
+import requests
+import re
+from typing import Optional
+from modules.logging import logger
+from modules.audio import segment_text, combine_audio_files
+from modules.snac_decoder import tokens_decoder_sync
+from modules.config import load_config
+
+def clean_text_for_tts(text: str) -> str:
+    """
+    Clean the text to be sent to the TTS engine by:
+      • Removing newline characters and excessive whitespace.
+      • Removing markdown symbols (e.g., asterisks).
+      • Removing non-ASCII characters (e.g., emojis).
+    """
+    # Remove newline characters
+    text = text.replace('\n', ' ')
+    # Remove markdown formatting
+    text = re.sub(r'\*+', '', text)
+    # Remove non-ASCII characters (e.g., emojis)
+    text = re.sub(r'[^\x00-\x7F]+', '', text)
+    # Normalize whitespace
+    text = re.sub(r'\s+', ' ', text)
+    return text.strip()
+
+class LMStudioClient:
+    def __init__(self, config):
+        self.config = config
+        lm_config = config["lm"]
+
+        self.api_url = lm_config["api_url"]
+        self.chat_endpoint = lm_config["chat"]["endpoint"]
+        self.tts_endpoint = lm_config["tts"]["endpoint"]
+
+        # Chat parameters
+        chat_config = lm_config["chat"]
+        self.chat_model = chat_config["model"]
+        self.system_prompt = chat_config["system_prompt"]
+        self.chat_max_tokens = chat_config["max_tokens"]
+        self.chat_temperature = chat_config["temperature"]
+        self.chat_top_p = chat_config["top_p"]
+        self.chat_repetition_penalty = chat_config["repetition_penalty"]
+        self.max_response_time = chat_config["max_response_time"]
+
+        # TTS parameters
+        tts_config = lm_config["tts"]
+        self.tts_model = tts_config["model"]
+        self.default_voice = tts_config["default_voice"]
+        self.tts_max_tokens = tts_config["max_tokens"]
+        self.tts_temperature = tts_config["temperature"]
+        self.tts_top_p = tts_config["top_p"]
+        self.tts_repetition_penalty = tts_config["repetition_penalty"]
+        self.speed = tts_config["speed"]
+        self.max_segment_duration = tts_config["max_segment_duration"]
+
+        self.headers = {"Content-Type": "application/json"}
+        self.retries = config["speech"]["max_retries"]
+        self.tts_sample_rate = config["tts"]["sample_rate"]
+
+        # Use a session for connection pooling
+        self.session = requests.Session()
+
+    def chat(self, user_input: str) -> str:
+        messages = [
+            {"role": "system", "content": self.system_prompt},
+            {"role": "user", "content": user_input}
+        ]
+        payload = {
+            "model": self.chat_model,
+            "messages": messages,
+            "max_tokens": self.chat_max_tokens,
+            "temperature": self.chat_temperature,
+            "top_p": self.chat_top_p,
+            "repeat_penalty": self.chat_repetition_penalty,
+            "stream": False
+        }
+        url = self.api_url + self.chat_endpoint
+        logger.debug("Chat request payload: %s", payload)
+        for attempt in range(self.retries):
+            try:
+                start_time = time.time()
+                response = self.session.post(
+                    url,
+                    headers=self.headers,
+                    json=payload,
+                    timeout=self.max_response_time
+                )
+                elapsed = time.time() - start_time
+                logger.info("Chat response received in %.2f seconds", elapsed)
+                logger.debug("LM Studio full response: %s", response.text)
+                if response.status_code != 200:
+                    logger.error("Chat API error: %s %s", response.status_code, response.text)
+                    if attempt < self.retries - 1:
+                        delay = 2 ** attempt
+                        logger.warning("Chat API error, retrying in %d seconds...", delay)
+                        time.sleep(delay)
+                        continue
+                    raise RuntimeError(f"Chat API error: {response.status_code} {response.text}")
+                data = response.json()
+                generated_text = data.get("choices", [{}])[0].get("message", {}).get("content", "").strip()
+                token_count = len(generated_text.split()) * 1.33
+                logger.info("Generated %d tokens: %s", int(token_count), generated_text[:50])
+                return generated_text
+            except requests.exceptions.Timeout:
+                delay = 2 ** attempt
+                logger.warning("Chat API timeout (attempt %d), retrying in %d seconds", attempt + 1, delay)
+                time.sleep(delay)
+                if attempt == self.retries - 1:
+                    return "I need more time to think about that. Could you ask again?"
+            except Exception as e:
+                delay = 2 ** attempt
+                logger.error("Chat API failed (attempt %d): %s", attempt + 1, str(e))
+                time.sleep(delay)
+                if attempt == self.retries - 1:
+                    return "I'm having trouble responding right now. Please try again later."
+
+    def synthesize_speech(self, text: str, voice: Optional[str] = None, output_file: Optional[str] = None) -> str:
+        voice = voice if voice else self.default_voice
+        if not voice.isalpha() or len(voice) > 20:
+            logger.warning("Invalid voice name '%s', using default", voice)
+            voice = self.default_voice
+
+        # Clean the text for TTS
+        cleaned_text = clean_text_for_tts(text)
+        prompt = f"<|audio|>{voice}: {cleaned_text}<|eot_id|>"
+        payload = {
+            "model": self.tts_model,
+            "prompt": prompt,
+            "max_tokens": self.tts_max_tokens,
+            "temperature": self.tts_temperature,
+            "top_p": self.tts_top_p,
+            "repeat_penalty": self.tts_repetition_penalty,
+            "speed": self.speed,
+            "stream": True
+        }
+        url = self.api_url + self.tts_endpoint
+        logger.debug("TTS request payload: %s", payload)
+        for attempt in range(self.retries):
+            try:
+                response = self.session.post(
+                    url,
+                    headers=self.headers,
+                    json=payload,
+                    stream=True,
+                    timeout=self.max_segment_duration + 5
+                )
+                if response.status_code != 200:
+                    logger.error("TTS API error: %s %s", response.status_code, response.text)
+                    if attempt < self.retries - 1:
+                        delay = 2 ** attempt
+                        logger.warning("TTS API error, retrying in %d seconds...", delay)
+                        time.sleep(delay)
+                        continue
+                    raise RuntimeError(f"TTS API error: {response.status_code} {response.text}")
+
+                def token_generator():
+                    for line in response.iter_lines():
+                        if line:
+                            decoded_line = line.decode("utf-8")
+                            if decoded_line.startswith("data: "):
+                                data_str = decoded_line[6:]
+                                if data_str.strip() == "[DONE]":
+                                    break
+                                try:
+                                    data = json.loads(data_str)
+                                    token_text = data.get("choices", [{}])[0].get("text", "")
+                                    yield token_text
+                                except json.JSONDecodeError as e:
+                                    logger.error("JSON decode error: %s", e)
+
+                audio_bytes = tokens_decoder_sync(token_generator())
+                if not output_file:
+                    timestamp = int(time.time())
+                    output_file = f"outputs/{voice}_{timestamp}.wav"
+                os.makedirs("outputs", exist_ok=True)
+                with wave.open(output_file, "wb") as wf:
+                    wf.setnchannels(1)
+                    wf.setsampwidth(2)
+                    wf.setframerate(self.tts_sample_rate)
+                    wf.writeframes(audio_bytes)
+                logger.info("Audio saved to %s", output_file)
+                return output_file
+            except Exception as e:
+                delay = 2 ** attempt
+                logger.error("TTS synthesis failed (attempt %d): %s", attempt + 1, str(e))
+                time.sleep(delay)
+                if attempt == self.retries - 1:
+                    raise
+
+    def synthesize_long_text(self, text: str, voice: Optional[str] = None) -> str:
+        voice = voice if voice else self.default_voice
+        segments = segment_text(text, max_words=self.config["segmentation"]["max_words"])
+        logger.info("Text segmented into %d parts", len(segments))
+        file_list = []
+        for i, seg in enumerate(segments):
+            seg_filename = f"outputs/{voice}_{int(time.time())}_{i}.wav"
+            try:
+                self.synthesize_speech(seg, voice=voice, output_file=seg_filename)
+                file_list.append(seg_filename)
+                time.sleep(0.2)  # Small delay between segments
+            except Exception as e:
+                logger.error("Failed to synthesize segment %d: %s", i, str(e))
+                if file_list:
+                    break
+                raise
+        combined_filename = f"outputs/{voice}_{int(time.time())}_combined.wav"
+        combine_audio_files(file_list, combined_filename)
+        for f in file_list:
+            try:
+                os.remove(f)
+            except Exception as e:
+                logger.warning("Could not remove temporary file %s: %s", f, str(e))
+        return combined_filename

modules/logging.py

@@ -0,0 +1,35 @@
+# modules/logging.py
+import os
+import sys
+import logging
+from logging.handlers import RotatingFileHandler
+from datetime import datetime
+
+LOG_DIR = "log"
+os.makedirs(LOG_DIR, exist_ok=True)
+log_filename = f"{LOG_DIR}/assistant_{datetime.now().strftime('%Y%m%d_%H%M%S')}.log"
+error_log_filename = f"{LOG_DIR}/errors_{datetime.now().strftime('%Y%m%d_%H%M%S')}.log"
+
+logger = logging.getLogger("AssistantLogger")
+logger.setLevel(logging.DEBUG)
+
+# Console handler
+ch = logging.StreamHandler(sys.stdout)
+ch.setLevel(logging.DEBUG)
+
+# Rotating file handler for all logs (max 5MB per file, up to 3 backups)
+fh = RotatingFileHandler(log_filename, maxBytes=5 * 1024 * 1024, backupCount=3, encoding="utf-8")
+fh.setLevel(logging.DEBUG)
+
+# Rotating file handler for errors only
+eh = RotatingFileHandler(error_log_filename, maxBytes=5 * 1024 * 1024, backupCount=3, encoding="utf-8")
+eh.setLevel(logging.ERROR)
+
+formatter = logging.Formatter('%(asctime)s [%(levelname)s] %(message)s', datefmt='%H:%M:%S')
+ch.setFormatter(formatter)
+fh.setFormatter(formatter)
+eh.setFormatter(formatter)
+
+logger.addHandler(ch)
+logger.addHandler(fh)
+logger.addHandler(eh)

modules/performance.py

@@ -0,0 +1,34 @@
+# modules/performance.py
+import time
+from modules.logging import logger
+
+class PerformanceMonitor:
+    def __init__(self, report_interval=60):
+        self.start_time = time.time()
+        self.token_count = 0
+        self.audio_chunks = 0
+        self.api_calls = 0
+        self.errors = 0
+        self.last_report_time = time.time()
+        self.report_interval = report_interval
+
+    def add_tokens(self, count=1):
+        self.token_count += count
+
+    def add_audio_chunk(self):
+        self.audio_chunks += 1
+
+    def add_api_call(self):
+        self.api_calls += 1
+
+    def add_error(self):
+        self.errors += 1
+
+    def report(self, force=False):
+        now = time.time()
+        if force or (now - self.last_report_time) >= self.report_interval:
+            elapsed = now - self.start_time
+            tokens_per_sec = self.token_count / elapsed if elapsed > 0 else 0
+            logger.info("Performance: %.1f tokens/sec | %d API calls | %d errors | %d audio chunks",
+                        tokens_per_sec, self.api_calls, self.errors, self.audio_chunks)
+            self.last_report_time = now

modules/snac_decoder.py

@@ -1,121 +1,92 @@
+# modules/snac_decoder.py
+
+import time
 import torch
 import numpy as np
-import asyncio
-import threading
-import queue
-from snac import SNAC
+from modules.logging import logger
+from snac import SNAC  # Ensure that the snac module is installed
 
-# Monkey-Patch torch.load to use weights_only=True by default
-original_torch_load = torch.load
-def patched_torch_load(*args, **kwargs):
-    kwargs.setdefault("weights_only", True)
-    return original_torch_load(*args, **kwargs)
-torch.load = patched_torch_load
-
-# Load the SNAC model used for decoding LM Studio TTS tokens into PCM audio.
+# Load SNAC model
 snac_model = SNAC.from_pretrained("hubertsiuzdak/snac_24khz").eval()
 snac_device = "cuda" if torch.cuda.is_available() else "cpu"
-print(f"Using SNAC on device: {snac_device}")
+logger.info("Using SNAC on device: %s", snac_device)
 snac_model = snac_model.to(snac_device)
+cuda_stream = torch.cuda.Stream() if snac_device == "cuda" else None
 
 def convert_to_audio(multiframe, count):
     if len(multiframe) < 7:
-        return
-    codes_0 = torch.tensor([], device=snac_device, dtype=torch.int32)
-    codes_1 = torch.tensor([], device=snac_device, dtype=torch.int32)
-    codes_2 = torch.tensor([], device=snac_device, dtype=torch.int32)
+        return None
     num_frames = len(multiframe) // 7
-    frame = multiframe[:num_frames*7]
+    frame = multiframe[:num_frames * 7]
+    codes_0 = torch.zeros(num_frames, dtype=torch.int32, device=snac_device)
+    codes_1 = torch.zeros(num_frames * 2, dtype=torch.int32, device=snac_device)
+    codes_2 = torch.zeros(num_frames * 4, dtype=torch.int32, device=snac_device)
+    frame_tensor = torch.tensor(frame, dtype=torch.int32, device=snac_device)
     for j in range(num_frames):
-        i = 7 * j
-        if codes_0.shape[0] == 0:
-            codes_0 = torch.tensor([frame[i]], device=snac_device, dtype=torch.int32)
-        else:
-            codes_0 = torch.cat([codes_0, torch.tensor([frame[i]], device=snac_device, dtype=torch.int32)])
-        if codes_1.shape[0] == 0:
-            codes_1 = torch.tensor([frame[i+1]], device=snac_device, dtype=torch.int32)
-            codes_1 = torch.cat([codes_1, torch.tensor([frame[i+4]], device=snac_device, dtype=torch.int32)])
-        else:
-            codes_1 = torch.cat([codes_1, torch.tensor([frame[i+1]], device=snac_device, dtype=torch.int32)])
-            codes_1 = torch.cat([codes_1, torch.tensor([frame[i+4]], device=snac_device, dtype=torch.int32)])
-        if codes_2.shape[0] == 0:
-            codes_2 = torch.tensor([frame[i+2]], device=snac_device, dtype=torch.int32)
-            codes_2 = torch.cat([codes_2, torch.tensor([frame[i+3]], device=snac_device, dtype=torch.int32)])
-            codes_2 = torch.cat([codes_2, torch.tensor([frame[i+5]], device=snac_device, dtype=torch.int32)])
-            codes_2 = torch.cat([codes_2, torch.tensor([frame[i+6]], device=snac_device, dtype=torch.int32)])
-        else:
-            codes_2 = torch.cat([codes_2, torch.tensor([frame[i+2]], device=snac_device, dtype=torch.int32)])
-            codes_2 = torch.cat([codes_2, torch.tensor([frame[i+3]], device=snac_device, dtype=torch.int32)])
-            codes_2 = torch.cat([codes_2, torch.tensor([frame[i+5]], device=snac_device, dtype=torch.int32)])
-            codes_2 = torch.cat([codes_2, torch.tensor([frame[i+6]], device=snac_device, dtype=torch.int32)])
+        idx = j * 7
+        codes_0[j] = frame_tensor[idx]
+        codes_1[j * 2] = frame_tensor[idx + 1]
+        codes_1[j * 2 + 1] = frame_tensor[idx + 4]
+        codes_2[j * 4] = frame_tensor[idx + 2]
+        codes_2[j * 4 + 1] = frame_tensor[idx + 3]
+        codes_2[j * 4 + 2] = frame_tensor[idx + 5]
+        codes_2[j * 4 + 3] = frame_tensor[idx + 6]
     codes = [codes_0.unsqueeze(0), codes_1.unsqueeze(0), codes_2.unsqueeze(0)]
-    if torch.any(codes[0] < 0) or torch.any(codes[0] > 4096) or \
-       torch.any(codes[1] < 0) or torch.any(codes[1] > 4096) or \
-       torch.any(codes[2] < 0) or torch.any(codes[2] > 4096):
-        return
-    with torch.inference_mode():
+    if (torch.any(codes[0] < 0) or torch.any(codes[0] > 4096) or
+        torch.any(codes[1] < 0) or torch.any(codes[1] > 4096) or
+        torch.any(codes[2] < 0) or torch.any(codes[2] > 4096)):
+        return None
+    stream_ctx = torch.cuda.stream(cuda_stream) if cuda_stream is not None else torch.no_grad()
+    with stream_ctx, torch.inference_mode():
         audio_hat = snac_model.decode(codes)
-    audio_slice = audio_hat[:, :, 2048:4096]
-    detached_audio = audio_slice.detach().cpu()
-    audio_np = detached_audio.numpy()
-    audio_int16 = (audio_np * 32767).astype(np.int16)
-    audio_bytes = audio_int16.tobytes()
+        audio_slice = audio_hat[:, :, 2048:4096]
+        if snac_device == "cuda":
+            audio_int16_tensor = (audio_slice * 32767).to(torch.int16)
+            audio_bytes = audio_int16_tensor.cpu().numpy().tobytes()
+        else:
+            audio_np = audio_slice.detach().cpu().numpy()
+            audio_int16 = (audio_np * 32767).astype(np.int16)
+            audio_bytes = audio_int16.tobytes()
     return audio_bytes
 
 def turn_token_into_id(token_string, index):
     token_string = token_string.strip()
-    last_token_start = token_string.rfind("<custom_token_")
-    if last_token_start == -1:
-        print("No token found in the string")
+    if "<custom_token_" not in token_string:
         return None
-    last_token = token_string[last_token_start:]
-    if last_token.startswith("<custom_token_") and last_token.endswith(">"):
-        try:
-            number_str = last_token[14:-1]
-            return int(number_str) - 10 - ((index % 7) * 4096)
-        except ValueError:
-            return None
-    else:
+    last_token_start = token_string.rfind("<custom_token_")
+    if last_token_start == -1 or not token_string.endswith(">"):
+        return None
+    try:
+        number_str = token_string[last_token_start + 14:-1]
+        return int(number_str) - 10 - ((index % 7) * 4096)
+    except (ValueError, IndexError):
         return None
 
-async def tokens_decoder(token_gen):
+token_cache = {}
+MAX_CACHE_SIZE = 1000
+
+def tokens_decoder(token_gen):
     buffer = []
     count = 0
-    async for token_text in token_gen:
-        token = turn_token_into_id(token_text, count)
-        if token is None:
-            continue
-        if token > 0:
+    min_frames_required = 28
+    process_every = 7
+    for token_text in token_gen:
+        cache_key = (token_text, count % 7)
+        if cache_key in token_cache:
+            token = token_cache[cache_key]
+        else:
+            token = turn_token_into_id(token_text, count)
+            if token is not None and len(token_cache) < MAX_CACHE_SIZE:
+                token_cache[cache_key] = token
+        if token is not None and token > 0:
             buffer.append(token)
             count += 1
-            if count % 7 == 0 and count > 27:
-                buffer_to_proc = buffer[-28:]
+            if count % process_every == 0 and count >= min_frames_required:
+                buffer_to_proc = buffer[-min_frames_required:]
                 audio_samples = convert_to_audio(buffer_to_proc, count)
                 if audio_samples is not None:
                     yield audio_samples
 
 def tokens_decoder_sync(syn_token_gen):
-    audio_queue = queue.Queue()
-
-    async def async_token_gen():
-        for token in syn_token_gen:
-            yield token
-
-    async def async_producer():
-        async for audio_chunk in tokens_decoder(async_token_gen()):
-            audio_queue.put(audio_chunk)
-        audio_queue.put(None)  # Sentinel
-
-    def run_async():
-        asyncio.run(async_producer())
-
-    thread = threading.Thread(target=run_async)
-    thread.start()
-    audio_segments = []
-    while True:
-        audio = audio_queue.get()
-        if audio is None:
-            break
-        audio_segments.append(audio)
-    thread.join()
+    audio_segments = list(tokens_decoder(syn_token_gen))
     return b"".join(audio_segments)

modules/virtual_assistant.py

@@ -1,60 +1,138 @@
-import wave
+# modules/virtual_assistant.py
+
+import os
 import time
+import wave
 import numpy as np
 import sounddevice as sd
-from .whisper_recognizer import WhisperRecognizer
-from .lmstudio_client import LMStudioClient
+from typing import Optional
+from modules.logging import logger
+from modules.whisper_recognizer import WhisperRecognizer
+from modules.lm_client import LMStudioClient
+from modules.hotword_detector import HotwordDetector
+from modules.performance import PerformanceMonitor
+from modules.config import load_config
 
 class VirtualAssistant:
-    """
-    The main virtual assistant class that integrates Whisper, LM Studio API for chat and TTS,
-    and decodes TTS tokens into audio using the SNAC-based decoder.
-    """
-    def __init__(self, config):
+    def __init__(self, config_path: str = "settings.yml"):
+        self.config = load_config(config_path)
         self.recognizer = WhisperRecognizer(
-            model_name=config["whisper"]["model_name"],
-            sample_rate=config["whisper"]["sample_rate"]
+            model_name=self.config["whisper"]["model"],
+            sample_rate=self.config["whisper"]["sample_rate"],
+            config=self.config
         )
-        self.lm_client = LMStudioClient(
-            config_lm_api=config["lm_studio_api"],
-            tts_sample_rate=config["tts"]["sample_rate"]
-        )
-        self.input_device = config.get("audio", {}).get("input_device", None)
-        self.output_device = config.get("audio", {}).get("output_device", None)
-        self.desired_tts_duration = config.get("desired_tts_duration", 20)
-
-    def play_audio(self, filename):
-        print("▶️ Playing audio...")
-        with wave.open(filename, "rb") as wf:
-            sample_rate = wf.getframerate()
-            audio_data = wf.readframes(wf.getnframes())
-            audio_array = np.frombuffer(audio_data, dtype=np.int16).astype(np.float32) / 32767.0
-            sd.play(audio_array, sample_rate, device=self.output_device)
-            sd.wait()
-
-    def get_wav_duration(self, filename):
-        with wave.open(filename, "rb") as wf:
-            frames = wf.getnframes()
-            rate = wf.getframerate()
-            return frames / float(rate)
+        self.lm_client = LMStudioClient(self.config)
+        # Always initialize hotword detector if enabled in config
+        self.hotword_detector = HotwordDetector(config=self.config) if self.config["hotword"]["enabled"] else None
+        self.performance = PerformanceMonitor()
+        self._running = False
 
     def run(self):
-        print("\n🔄 Starting the virtual assistant. Press Ctrl+C to exit.\n")
+        self._running = True
+        logger.info("Assistant started. Press ENTER or say '%s' to interact.", self.config["hotword"]["phrase"])
+        
         try:
-            while True:
-                user_text = self.recognizer.transcribe(duration=5, device=self.input_device)
-                if not user_text.strip():
-                    print("⚠️ No speech detected. Please try again.")
-                    continue
-                response_text = self.lm_client.generate_text(user_text)
-                audio_file = self.lm_client.synthesize_speech(
-                    response_text,
-                    desired_tts_duration=self.desired_tts_duration
-                )
-                duration = self.get_wav_duration(audio_file)
-                print(f"Audio duration: {duration:.2f} seconds.")
-                self.play_audio(audio_file)
-                print("Waiting extra 1 second after playback to ensure full audio is played.")
-                time.sleep(duration + 1.0)
+            while self._running:
+                try:
+                    if not self._wait_for_activation():
+                        continue
+                    # Record and process user input
+                    user_text = self.recognizer.transcribe()
+                    if not user_text:
+                        logger.warning("No speech detected.")
+                        continue
+                    # Get and process response
+                    response_text = self.lm_client.chat(user_text)
+                    self.performance.add_tokens(len(response_text.split()))
+                    
+                    # Synthesize speech
+                    word_count = len(response_text.split())
+                    if word_count > self.config["segmentation"]["max_words"]:
+                        logger.info("Response is long (%d words). Segmenting...", word_count)
+                        output_file = self.lm_client.synthesize_long_text(response_text)
+                    else:
+                        output_file = self.lm_client.synthesize_speech(response_text)
+                    
+                    # Play audio and wait for the activation signal before next loop
+                    self.play_audio(output_file)
+                    self._wait_for_activation()  # Wait again after audio playback
+                except Exception as e:
+                    logger.error("Error in main loop: %s", str(e))
+                    time.sleep(1)
+                    
         except KeyboardInterrupt:
-            print("\n👋 Exiting gracefully. Goodbye!")
+            logger.info("Exiting assistant. Goodbye!")
+        finally:
+            try:
+                self.performance.report(force=True)
+            except Exception as e:
+                logger.error("Error in performance report: %s", str(e))
+            self._running = False
+
+    def stop(self):
+        self._running = False
+
+    def _flush_stdin(self):
+        """Flush any lingering input from stdin."""
+        try:
+            import sys
+            import termios
+            termios.tcflush(sys.stdin, termios.TCIFLUSH)
+        except Exception:
+            try:
+                import msvcrt
+                while msvcrt.kbhit():
+                    msvcrt.getch()
+            except Exception:
+                pass
+
+    def _wait_for_activation(self) -> bool:
+        """
+        Wait for activation either by detecting a keypress (push-to-talk)
+        or by detecting the hotword, whichever comes first.
+        """
+        logger.info("Waiting for activation: press ENTER or say the hotword...")
+        hotword_timeout = self.config["hotword"]["timeout_sec"] if self.hotword_detector else 0
+        elapsed = 0.0
+        check_interval = 0.5
+        while elapsed < hotword_timeout:
+            if self._check_for_keypress():
+                self._flush_stdin()
+                return True
+            if self.hotword_detector and self.hotword_detector.check_for_hotword(timeout=check_interval):
+                return True
+            time.sleep(check_interval)
+            elapsed += check_interval
+        # Fallback to blocking push-to-talk input if neither hotword nor keypress detected within the timeout
+        input("Press ENTER to speak...")
+        return True
+
+    def _check_for_keypress(self) -> bool:
+        """Non-blocking keypress check."""
+        try:
+            import msvcrt  # Windows
+            return msvcrt.kbhit()
+        except ImportError:
+            import sys
+            import select  # Unix
+            return sys.stdin in select.select([sys.stdin], [], [], 0)[0]
+
+    def play_audio(self, filename: str):
+        """Play audio with normalization and error handling."""
+        if not os.path.exists(filename):
+            logger.error("Audio file not found: %s", filename)
+            return
+        try:
+            with wave.open(filename, "rb") as wf:
+                sample_rate = wf.getframerate()
+                audio_data = wf.readframes(wf.getnframes())
+            audio_array = np.frombuffer(audio_data, dtype=np.int16).astype(np.float32) / 32767.0
+            if self.config["speech"]["normalize_audio"]:
+                max_val = np.max(np.abs(audio_array))
+                if max_val > 0:
+                    audio_array = audio_array / max_val
+            sd.play(audio_array, samplerate=sample_rate)
+            sd.wait()
+        except Exception as e:
+            logger.error("Audio playback error: %s", str(e))
+            raise

modules/whisper_recognizer.py

@@ -1,23 +1,54 @@
-import whisper
+# modules/whisper_recognizer.py
+from typing import Optional
+import os
+import time
+import torch
+import numpy as np
 import sounddevice as sd
-import scipy.io.wavfile as wav
+from scipy.io.wavfile import write as wav_write
+import tempfile
+import whisper
+from modules.logging import logger
+from modules.audio import record_until_silence
+from modules.config import load_config
 
 class WhisperRecognizer:
-    """
-    Uses the Whisper model to record and transcribe audio from the microphone.
-    """
-    def __init__(self, model_name, sample_rate):
-        print("🔊 Loading Whisper model...")
-        self.model = whisper.load_model(model_name)
+    def __init__(self, model_name: str = "base", sample_rate: int = 16000, config=None):
+        self.config = config if config is not None else load_config()
+        logger.info("Loading Whisper model (%s)...", model_name)
+        device = "cuda" if torch.cuda.is_available() else "cpu"
+        self.model = whisper.load_model(model_name, device=device)
         self.sample_rate = sample_rate
+        logger.info("Whisper model loaded on %s", device)
 
-    def transcribe(self, duration=5, device=None):
-        print("\n🎙️ Listening...")
-        audio = sd.rec(int(duration * self.sample_rate), samplerate=self.sample_rate, channels=1, device=device)
-        sd.wait()
-        wav.write("input.wav", self.sample_rate, audio)
-        print("📝 Transcribing...")
-        result = self.model.transcribe("input.wav")
-        text = result["text"].strip()
-        print(f"👤 You said: {text}")
-        return text
+    def transcribe(self, device: Optional[int] = None) -> str:
+        """Record and transcribe audio with error handling"""
+        try:
+            logger.info("Recording...")
+            audio = record_until_silence(
+                self.sample_rate,
+                device=device or self.config["audio"]["input_device"]
+            )
+            if audio.size == 0:
+                logger.warning("No audio recorded")
+                return ""
+            # Use a temporary file for audio
+            with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as tmp:
+                temp_file = tmp.name
+                wav_write(temp_file, self.sample_rate, audio)
+            logger.info("Transcribing...")
+            start_time = time.time()
+            result = self.model.transcribe(temp_file)
+            elapsed = time.time() - start_time
+            logger.debug("Transcription took %.2f seconds", elapsed)
+            text = result.get("text", "").strip()
+            if not text:
+                logger.warning("No speech detected in audio")
+            try:
+                os.remove(temp_file)
+            except Exception as e:
+                logger.warning("Could not delete temporary file %s: %s", temp_file, str(e))
+            return text
+        except Exception as e:
+            logger.error("Transcription failed: %s", str(e))
+            return ""

morpheus.py

@@ -0,0 +1,30 @@
+#!/usr/bin/env python
+import argparse
+import sys
+from modules.virtual_assistant import VirtualAssistant
+from modules.logging import logger
+
+def main():
+    parser = argparse.ArgumentParser(description="Start the mOrpheus Virtual Assistant.")
+    parser.add_argument(
+        "-c", "--config",
+        type=str,
+        default="settings.yml",
+        help="Path to configuration YAML file."
+    )
+    args = parser.parse_args()
+
+    try:
+        assistant = VirtualAssistant(config_path=args.config)
+        logger.info("Starting mOrpheus virtual assistant...")
+        assistant.run()
+    except KeyboardInterrupt:
+        logger.info("Assistant interrupted by user. Shutting down.")
+    except Exception as e:
+        logger.critical(f"Fatal error: {str(e)}", exc_info=True)
+        sys.exit(1)
+
+    logger.info("mOrpheus shutdown complete")
+
+if __name__ == "__main__":
+    main()

requirements.txt

@@ -1,9 +1,12 @@
-torch==2.5.1 --index-url https://download.pytorch.org/whl/cu121
+# Install PyTorch with CUDA support as needed.
+# For example, if using CUDA 12.6, run:
+# pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126
+
 openai-whisper
-transformers
-sounddevice
-numpy
-scipy
-pyyaml
-requests
+sounddevice>=0.4.6
+numpy>=1.24
+scipy>=1.10.0
+webrtcvad
+requests>=2.31.0
+pyyaml>=6.0.1
 snac

settings.yml

@@ -0,0 +1,88 @@
+# -------------------------------
+# Configuration for Whisper STT
+# -------------------------------
+whisper:
+  model: "small.en"        # Name of the Whisper model to use for speech-to-text.
+  sample_rate: 16000       # Audio sample rate (in Hz) for recording and transcription.
+
+# -------------------------------
+# Configuration for LM Studio (Chat & TTS)
+# -------------------------------
+lm:
+  api_url: "http://127.0.0.1:1234/v1"  # Base URL for the LM Studio inference server.
+  
+  chat:
+    endpoint: "/chat/completions"       # API endpoint for chat-based text generation.
+    model: "gemma-3-12b-it"             # Model identifier for chat generation.
+    system_prompt: "You are a helpful assistant."  # System prompt to set the context.
+    max_tokens: 256                     # Increased for more coherent responses.
+    temperature: 0.7                    # Sampling temperature; controls randomness.
+    top_p: 0.9                          # Top-p (nucleus sampling) value.
+    repetition_penalty: 1.1             # Penalty factor to reduce repetitive outputs.
+    max_response_time: 10.0             # Increased timeout (in seconds) for slower responses.
+
+  tts:
+    endpoint: "/completions"            # API endpoint for text-to-speech synthesis.
+    model: "orpheus-3b-ft.gguf@q2_k"    # Model identifier for TTS synthesis.
+    default_voice: "tara"               # Default voice for TTS output.
+    max_tokens: 4096                    # Optimal for Orpheus.
+    temperature: 0.6                    # Sampling temperature for TTS.
+    top_p: 0.9                          # Top-p sampling value for TTS generation.
+    repetition_penalty: 1.0             # Penalty to prevent repetitive TTS output.
+    speed: 1.0                          # More natural speed.
+    max_segment_duration: 20            # Increased maximum seconds per TTS segment.
+
+# -------------------------------
+# TTS Audio Output Configuration
+# -------------------------------
+tts:
+  sample_rate: 24000          # Sample rate (in Hz) for the generated TTS audio.
+
+# -------------------------------
+# Audio Device Configuration
+# -------------------------------
+audio:
+  input_device: null          # Specify input device ID, or leave null to use the system default.
+  output_device: null         # Specify output device ID, or leave null to use the system default.
+  hotword_sample_rate: 16000  # Sample rate for hotword detection.
+
+# -------------------------------
+# Voice Activity Detection (VAD) Configuration
+# -------------------------------
+vad:
+  mode: 2                     # VAD aggressiveness (0 is least, 3 is most aggressive).
+  frame_duration_ms: 30       # Duration of each audio frame (in ms) for VAD analysis.
+  silence_threshold_ms: 1000  # Duration (in ms) of consecutive silence to stop recording.
+  min_record_time_ms: 2000    # Minimum recording duration (in ms).
+
+# -------------------------------
+# Hotword Detection Configuration
+# -------------------------------
+hotword:
+  enabled: true               # Whether hotword detection is enabled.
+  phrase: "Hey Cassie"        # Hotword phrase (case insensitive).
+  sensitivity: 0.7            # Sensitivity (0-1); higher is more strict.
+  timeout_sec: 5              # Time (in seconds) to listen for hotword before timeout.
+  retries: 3                  # Maximum attempts to detect the hotword.
+
+# -------------------------------
+# Segmentation Configuration for TTS
+# -------------------------------
+segmentation:
+  max_words: 60               # Maximum number of words per segment when splitting long TTS responses.
+
+# -------------------------------
+# Speech Quality Configuration
+# -------------------------------
+speech:
+  normalize_audio: false      # Whether to normalize audio volume.
+  default_pitch: 0            # Pitch adjustment (-20 to +20).
+  min_speech_confidence: 0.5  # Minimum confidence for accepting speech.
+  max_retries: 3              # Maximum retries for API calls.
+
+# -------------------------------
+# Interaction Configuration
+# -------------------------------
+interaction:
+  mode: "both"              # Options: "push_to_talk", "hotword", or "both" to enable both simultaneously.
+  post_audio_delay: 0.5     # Delay (in seconds) after audio playback before next activation.