Python · Voice · prototype · 2025

Cosmo

Windows voice assistant on LiveKit Agents and Google's realtime voice model. Its tools open apps, move the mouse, type, search the web and check the weather.

Problem
Controlling a Windows PC hands-free is still clumsy.
What I built
A voice assistant on LiveKit Agents and Google's realtime voice model, with tools that open apps, move the mouse, type and search.
Outcome
Working prototype.
Role
Solo project

Tech: Python, LiveKit Agents, Google realtime voice

Repository

What it is

Cosmo is a voice assistant for a Windows PC. You talk to it, it answers out loud, and it can act on the machine: open and close apps, move the mouse, type, press keys, change the volume, search the web, check the weather and play files.

Controlling a computer hands-free is still clumsy on Windows. I wanted to see how far a realtime voice model could get if it was given direct control of the desktop.

It's a working prototype. It isn't packaged or hardened for anyone else to install casually.

How it works

Cosmo runs on LiveKit Agents with Google's realtime voice model. The model hears you, decides what to do, and calls a set of Python tools that do the work on the desktop. Some examples:

  • google_search runs a web search and reads back the top few results.
  • get_weather looks up the weather for a city, or guesses the city if you don't give one.
  • open and close find and focus or close application windows.
  • Play_file searches your folders for a media file by name and plays it.
  • move_cursor_tool, type_text_tool and press_hotkey_tool drive the mouse and keyboard.
  • control_volume_tool changes the system volume.

Its behaviour comes from a set of prompts written in a mix of Hindi and English, so it can be spoken to in either. Saying "shutdown", "quit", "exit" or "close cosmo" stops it.

The tools use pywin32, pyautogui and pynput for window and input control. That is also why it only runs on Windows.

What I worked on

I wrote the agent setup and the desktop tools and wired them into the LiveKit session. The persona prompts credit Arham Khan as the assistant's designer.

Limits

  • Windows only.
  • It controls the real mouse and keyboard. Whatever it's told to type or click happens on your actual desktop, so run it somewhere that's safe.
  • It needs an internet connection and API keys for LiveKit, Google and OpenWeather, which may be rate-limited or paid.
  • There are no automated tests. The tools were tried by hand.

Links