Skip to content

Local LLM Setup on M3 Max (llama.cpp + Hugging Face)

Updated #tooling#llm#llama-cpp#mlx#apple-silicon#local-inference#agentic-coding

LM Studio

  • open and run server
  • apply
    • update .codex config
    • add to opencode config

Local LLM Setup on M3 Max (llama.cpp + Hugging Face)

Overview

  • Run open-source LLMs locally on M3 Max 36GB
  • Stack: llama.cpp for inference + hf CLI for model downloads
  • Exposes OpenAI-compatible API → drops into Codex, OpenCode, Aider, Continue
  • Claude Code support via LiteLLM proxy (Anthropic ↔ OpenAI translation)
  • MLX is a faster alternative for raw throughput — see Alternative Runtimes
  • Analogy: llama-server is to local LLMs what Postgres is to local DBs — a daemon you point clients at

Default Model: Qwen3.6-27B

  • Top pick for 36GB as of Apr 2026
    • 27B dense, ~16GB at Q4_K_M
    • Beats prior-gen Qwen3.5-397B-A17B on coding benchmarks
    • 262k native context (extensible to 1M)
    • Apache 2.0
    • GGUF community quants already published

Setup Steps

1. Install both tools

brew install llama.cpp hf
  • llama.cpp ships with Metal enabled by default on macOS
  • hf is the renamed HuggingFace CLI (was huggingface-cli)

2. Authenticate with Hugging Face

hf auth login

3. Download a model (GGUF format)

  • Coding default — Qwen3.6-27B (~16GB):

    hf download unsloth/Qwen3.6-27B-GGUF \
  Qwen3.6-27B-Q4_K_M.gguf \
  --local-dir ~/models

  • General — Gemma 3 27B (~16GB):

    hf download bartowski/gemma-3-27b-it-GGUF \
  gemma-3-27b-it-Q4_K_M.gguf \
  --local-dir ~/models

  • Quant cheat sheet for 36GB:

    • Q4_K_M → default, best size/quality tradeoff
    • Q5_K_M → +25% size, slight quality bump
    • IQ4_XS → smaller, near Q4_K_M quality (uses imatrix)
    • Q8_0 → near-lossless, only viable for ≤14B models

4. Run the server (OpenAI-compatible API)

llama-server \
  -m ~/models/Qwen3.6-27B-Q4_K_M.gguf \
  -ngl 99 \
  -c 16384 \
  --flash-attn on \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --port 8080
  • Endpoint live at http://localhost:8080/v1
  • Critical flags:
    • -ngl 99 → all layers on GPU (Metal). Always set this
    • --flash-attn on → ~30% speedup, lower memory (use on|off|auto, no longer a bare toggle)
    • --cache-type-k/v q8_0 → halves KV cache memory
    • -c 16384 → 16k context (KV cache scales linearly, only bump if needed)
    • --mlock → optional, pins model in RAM, prevents macOS swapping

5. Wire into agentic coding tools

OpenCode (recommended — native OpenAI-compatible, no proxy)

  • Install: 
    npm install -g opencode-ai
# or: brew install sst/tap/opencode
  • Configure ~/.config/opencode/opencode.json: 
    {
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "local": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Local llama.cpp",
      "options": {
        "baseURL": "http://localhost:8080/v1",
        "apiKey": "dummy"
      },
      "models": {
        "qwen3.6-27b": {
          "name": "Qwen3.6 27B (local)"
        }
      }
    }
  }
}
  • Run: opencode → use /models to pick local/qwen3.6-27b
  • Quick env-var alternative (no config file): 
    export OPENAI_BASE_URL=http://localhost:8080/v1
export OPENAI_API_KEY=dummy
opencode

Codex / Aider (also OpenAI-compatible)

export OPENAI_BASE_URL=http://localhost:8080/v1
export OPENAI_API_KEY=dummy

Claude Code (needs Anthropic ↔ OpenAI proxy)

pip install 'litellm[proxy]' --break-system-packages
litellm --model openai/qwen \
  --api_base http://localhost:8080/v1 \
  --api_key dummy --port 4000

export ANTHROPIC_BASE_URL=http://localhost:4000
export ANTHROPIC_AUTH_TOKEN=dummy
export ANTHROPIC_MODEL=qwen
export ANTHROPIC_SMALL_FAST_MODEL=qwen
  • Alternative: claude-code-router — purpose-built proxy for Claude Code with per-task model routing

Tool-call expectations

  • OpenCode: best of the bunch for local — provider-aware, handles smaller-model quirks
  • Claude Code: works but rough — Claude Code expects frontier-model quality tool calls; expect malformed JSON occasionally
  • All tools: bump -c to 64k+ for usable agentic work (tools eat context fast)
  • If tool calls fail repeatedly → add --chat-template chatml to llama-server

6. Verify performance

llama-bench -m ~/models/Qwen3.6-27B-Q4_K_M.gguf -ngl 99 -fa 1
  • Expected on M3 Max for 27B Q4_K_M:
    • Prompt eval: ~200–400 tok/s
    • Generation: ~18–22 tok/s
  • If gen <10 tok/s → swapping. Close Chrome/Docker, check sudo memory_pressure

Optimization Knobs (matter a lot)

  • Quant level: 4-bit is the sweet spot. Below 4-bit = noticeable quality loss
  • Context length: KV cache scales linearly. Drop to 8k–16k unless you need long context
  • Flash attention: always enable on M-series (--flash-attn on)
  • KV cache quantization: q8_0 halves memory with negligible quality hit
  • Speculative decoding: pair big model with tiny draft model → 1.5–2x speedup
    • Example: 27B target + 1.5B draft, both same family
    • Best gains on predictable text (code, structured output)
  • Memory pressure: keep ~6GB free for macOS or you'll swap and tank tok/s
  • Unified memory caveat: Chrome/Docker/etc fight for the same pool. Close before long sessions

Speculative Decoding Example

llama-server \
  -m ~/models/Qwen3.6-27B-Q4_K_M.gguf \
  -md ~/models/Qwen3-1.5B-Q4_K_M.gguf \
  -ngl 99 -ngld 99 \
  --draft-max 16 --draft-min 4 \
  --flash-attn on -c 8192
  • Draft model generates N tokens, big model verifies in one pass
  • -md = draft model path, -ngld = draft model GPU layers

Models That Won't Fit on 36GB

These are the current frontier open models — all need data-center GPUs:

  • Kimi K2.6 (Moonshot) — 1T total / 32B active MoE
    • Smallest viable quant: 350GB (2-bit dynamic GGUF)
    • Needs 8x H100 for production
  • DeepSeek-V4-Pro — 1.6T total / 49B active MoE
    • 1M-token context with hybrid CSA/HCA attention
    • Even V4-Flash (284B/13B active) needs ~140GB at 4-bit
  • GLM-5.1 (zai-org) — 754B total / 40B active MoE
    • Smallest quant (Unsloth dynamic 1-bit): 200GB
    • Strong on SWE-Bench Pro and Terminal-Bench 2.0

warning: MoE saves compute, not memory. All experts must stay resident. Active param count (the smaller number) only tells you inference cost, not RAM requirement.

For these models: use APIs (OpenRouter, DeepInfra, provider direct) instead of local hosting.

Other Models That Fit on 36GB

  • Gemma 3 27B (~16GB at 4-bit) — strong general reasoning, multimodal
  • Qwen2.5-Coder 32B (~18GB at 4-bit) — previous default, still solid
  • Llama 3.3 70B at IQ2_XXS (~22GB) — pushes limits, slower, quality dips noticeably
  • Qwen3 14B / Gemma 3 12B at 4-bit — fast, leaves headroom for big context

Common Gotchas

  • Forgot -ngl 99 → runs on CPU, 10x slower
  • Context too large → OOM crash. Watch sudo memory_pressure while running
  • Wrong chat template → garbage output. Pass --chat-template chatml|llama3|gemma if auto-detect fails
  • Gated model 401 → accept license in browser first, even with valid token
  • Partial GGUF download → cryptic errors. Verify ls -lh matches HF size
  • Flag drift: llama.cpp renames flags often. Recent breakage:
    • --flash-attn → now requires value: --flash-attn on|off|auto
    • Run llama-server -h | grep <flag> if a flag errors

Quick Alias

Add to ~/.zshrc:

alias llm-serve='llama-server -m ~/models/Qwen3.6-27B-Q4_K_M.gguf -ngl 99 -c 16384 --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 --port 8080'

Alternative Runtimes

  • MLX / mlx-lm — Apple's native framework, ~10–30% faster on Apple Silicon
    • Separate runtime, separate model format (MLX folder, not GGUF — no interop with llama.cpp)
    • Use when raw throughput matters more than ecosystem/quant variety
    • Install: pip install mlx-lm --break-system-packages
    • Download: hf download unsloth/Qwen3.6-27B-UD-MLX-4bit --local-dir ~/models/qwen3.6-mlx
    • Run: mlx_lm.server --model ~/models/qwen3.6-mlx --port 8080
    • Tradeoffs: smaller model selection, fewer quant variants, no spec decoding yet
  • Ollama — wraps llama.cpp, easier ergonomics, model registry
    • Use as always-on daemon: brew install ollama && ollama serve
    • Endpoint at http://localhost:11434/v1
  • LM Studio — GUI for browsing/comparing quants
    • Bundles both MLX and llama.cpp runtimes (pick per model)

Resources

  • GPU Kernels ML Optimization
  • temp Claude Code