A founder of a small software business shares a detailed, experience-based assessment of running local Qwen 27B models on an RTX 6000 Pro (96GB VRAM) for real business workflows. The post argues that local models are not a replacement for frontier models like Claude Opus but serve distinct use cases: privacy-sensitive customer support analysis, telemetry-based revenue recovery, and bounded maintenance tasks. Key findings include persistent looping/hallucination issues on long-horizon agentic tasks, the importance of quantization settings and model card tuning parameters, and the operational complexity of running local inference (identity, metering, power monitoring). The setup uses llama.cpp with speculative decoding (MTP) achieving 130-200 tok/s, and fine-tunes like Qwopus. The author concludes local models excel at specific, supervised tasks but cannot replace cloud SOTA for unsupervised agentic coding work, especially in Go.
