TONWERK-AI

Setup

Tonwerk-AI has two parts: the server on a Mac, which runs the models and stores the songs, and the app on iPhone, iPad or Mac, which operates it. The server is set up once and then updates itself. If you just want to look first, use the demo mode.

What you need

1. Install the server

On the Mac that is to compute, in Terminal:

xcode-select --install
git clone https://github.com/Marcel-B/Tonwerk-AI.git ~/repos/Tonwerk-AI
~/repos/Tonwerk-AI/deploy/setup-mac.sh

The first line installs Apple's Command Line Tools (git, compiler) if they are missing. The script then checks the Mac and installs what is missing: Homebrew, uv, ffmpeg, Node.js and the .NET 10 SDK. After that it fetches YuE2 as a checkout of YuE Studio in the tested version, builds the bridge to the Neural Engine, downloads the models (about 7 GB) and starts the server as a service that runs while the user is logged in. At the end it computes once as a test.

The script may run as often as you like; it skips what is already there. If the app YuE Studio is already installed, the server shares its models and song folder (~/Music/YuE Studio), and both see the same library.

Options

Further tools come with options, on the first run or later:

--transcription
SheetSage2 for transcription (about 2 GB)
--voices
stem separation and Seed-VC for other voices
--speech
the speech lab with mlx-audio
--images
image models for painted covers: FLUX.2 Klein, Z-Image Turbo, Krea 2 Turbo and FLUX 3 Klein (the last two only on a Mac with 32 GB)
--midi
Basic Pitch for audio to MIDI
--whisper
Whisper for lyrics that follow along
--montage
music videos from your own photos and clips
--ace
ACE-Step as a second song model
--singing
SoulX-Singer for the Singing page (about 3 GB)
--all
everything above

To add an option later without losing the earlier ones:

~/repos/Tonwerk-AI/deploy/setup-mac.sh $(cat "$HOME/Library/Application Support/YuE UI/setup-options") --images

Models download on first use; until then, the page in the app shows that it is loading.

Updates

The script turns on automatic updates: every five minutes the Mac looks for new versions, waits until nothing is computing, backs up the database and settings to ~/Backups/YueUI and installs the new version. Waiting jobs survive the restart. deploy/update.sh status shows the state, deploy/update.sh off turns it off.

2. Make the server reachable

The server listens only on the Mac itself (127.0.0.1:5090) and has no login of its own. It becomes reachable through Tailscale, only for the devices of your tailnet and with an HTTPS certificate:

tailscale serve --bg --https=8443 5090

After that it is called https://<name-of-the-mac>.<tailnet>.ts.net:8443; tailscale serve status shows the exact name. In a browser this address with /ui/ appended opens the web interface; the app needs the same address. Never put the server on the open internet with tailscale funnel: anyone could then make and delete songs.

3. Install the app

4. Connect the app

  1. Turn on Tailscale on the device.
  2. Open Tonwerk-AI. On first launch the app asks for the server's address: the one from step 2, for example mac-mini.tailnet.ts.net:8443. It adds https:// itself; the web interface's address with /ui/… works too.
  3. Connect. The app loads the library, playlists and queue and stays up to date live through an event stream.

On the server Mac itself, http://127.0.0.1:5090 works too, without Tailscale. You choose another server in the app's settings under Use another server. The address is the only thing the app remembers about the server.

On first use the app asks for the permissions a page needs: Local Network for a server in your home network, Microphone for voices in the speech lab and for singing, Bluetooth for MIDI instruments over Bluetooth. Without them, the rest keeps working.

Try it without a server

On the start screen, Try without a server leads into a built-in sample library with three short songs synthesised for it, playlists, stars, notes and a score. It lives in the app, nothing goes onto the network, and changes last only until the app quits. What needs the models (new songs, voices, most tools) says there that it needs your own server. You get back in the settings with Connect your own server.

What the server can use in addition

LM Studio
For Draft lyrics the Mac needs LM Studio with a language model, by default google/gemma-4-26b-a4b-qat (around 15 GB; with 16 GB of memory better google/gemma-4-e4b). LM Studio need not be open: the server starts it and loads the model itself, and unloads it after every draft.
Claude
The Sketch needs, and the lyrics and the translation of cover descriptions can use, an API key of your own from Anthropic (console.anthropic.com). It goes into ~/.config/tonwerk/anthropic.key (one line, chmod 600). What then goes to Anthropic is described under Privacy; Anthropic bills you for the costs.
Nextcloud
A nightly backup of database and songs to your own Nextcloud (or any WebDAV storage): WebDAV address, user and app password in ~/.config/tonwerk/nextcloud.env as NEXTCLOUD_WEBDAV_URL, NEXTCLOUD_USERNAME and NEXTCLOUD_APP_PASSWORD.
External drive
deploy/move-to-volume.sh shows how large which parts (models, songs, stems, …) are; deploy/move-to-volume.sh /Volumes/<drive> yue2 songs moves them to an external SSD and leaves a link behind, --back brings a part back. For the service to read the drive, its dotnet needs Full Disk Access in System Settings.

Problems

No connection
Is Tailscale turned on on the device and in the same tailnet? Does tailscale serve status on the Mac show port 8443? Does the address open the web interface in the device's browser? Is the Mac awake and a user logged in (the service runs in their session)?
A tool says something is missing
Then the matching option is not set up yet. Run setup-mac.sh again with the saved options plus the new one.
A job waits for a long time
The Mac computes one large model at a time; the queue shows at the top which one holds the memory and what each job is waiting for. If the app YuE Studio is open, a second model runs there and needs memory: better close it.
Something goes wrong
Tools → Log shows the server's messages and copies a report without settings, keys and addresses that you can send me.