Local LLM (llama.cpp)

About

Local LLM (llama.cpp) runs workflows and prompts on your Mac with llama.cpp and Metal. Cleanup, rewrites, summaries and translations work without sending text to the cloud. Besides the recommended Gemma 4 models, you can use any GGUF model from Hugging Face that comes as a single file, for example a small 0.5B to 4B model that cleans up dictations in about a second, or import a .gguf file you already have.

Features

• Fully local prompt processing with llama.cpp and Metal on Apple silicon
• Gemma 4, Qwen3.5 and LFM2.5 models, the same files as in TypeWhisper for Windows
• Single-file GGUF models from Hugging Face: enter the repository, pick a file, done
• Import of .gguf files from your Mac
• Downloads pinned to the repository’s current commit and checked against their SHA-256
• Thinking turned off for text cleanup, so answers come right away
• Generation temperature slider for more precise or more creative output
• Optional Hugging Face token for higher download rate limits and private or gated models
• No API key required

Recommended Models

Model	Download Size	Recommended RAM	Status
Gemma 4 E2B (Q4_K_M)	~3.1 GB	8 GB+	Recommended
Gemma 4 E4B (Q4_K_M)	~5 GB	16 GB+	Recommended
Gemma 4 E4B (Q8_0)	~8.2 GB	16 GB+	Supported
Gemma 4 26B-A4B (Q4_K_M, MoE)	~17 GB	32 GB+	Experimental
Qwen3.5 2B (Q4_K_M)	~1.3 GB	8 GB+	Experimental
LFM2.5 2.6B (Q4_K_M)	~1.7 GB	8 GB+	Experimental

The plugin recommends Gemma 4 E2B or E4B based on your Mac’s memory.

Your Own Models

• From Hugging Face - Enter a repository such as unsloth/Qwen3.5-2B-GGUF, add a quantization like unsloth/Qwen3.5-2B-GGUF:Q4_K_M, or paste a huggingface.co link. Click Find GGUF Files, pick a file and click Add & Download.
• From a file - Click Import GGUF File… and choose a .gguf file. The plugin copies it into its own folder and your original file stays where it is. When the file is on the same volume as TypeWhisper’s data, usually the Mac’s startup disk, the copy takes no extra space; from another volume it needs space for a full copy.

Models with a chat template in their GGUF file work best; most instruct and chat models have one. Models split into several files and projector files (mmproj-*.gguf) are not offered.

Setup

1. In TypeWhisper, open Settings > Discover plugins and install Local LLM (llama.cpp). The add-on then appears in the Settings sidebar, and its page opens.
2. Pick a recommended model and click Download & Load, or add your own model under Your Models. Experimental models use Try anyway.
3. After the model is loaded, select Local LLM (llama.cpp) as the LLM provider in your workflow or prompt.

Notes

• The first model also downloads llama.cpp 0.7.0 (about 12 MB) from GitHub. It runs as a local server that only your Mac can reach, with a new random key on every start.
• After Auto-unload or a crash of the model server, the next prompt loads the model again.
• The plugin needs a Mac with Apple silicon and TypeWhisper 1.8.0 or later. It is not available in the App Store edition, because it runs the downloaded llama.cpp program.
• LFM2.5 uses the LFM Open License, which allows free commercial use only below USD 10 million in annual revenue. Models you add keep the license of their repository.