A research paper (ICLR 2026) from Together.ai demonstrates that smaller LLMs like Llama-3-70B and Qwen-72B can match or outperform GPT-4o on long-context tasks using a 'Divide & Conquer' framework. The approach splits long documents into parallel chunks processed by worker models, with a manager model aggregating results. Three noise sources are identified: model noise (superlinear confusion growth with length), task noise (cross-chunk dependencies), and aggregator noise (poor result stitching). The framework addresses aggregator noise through carefully planned worker prompts. Engineering benefits include lower cost (cheaper worker models), faster processing (parallel execution), and easy tuning (only 5 samples needed to find optimal chunk size). The approach works best for QA, retrieval, and summarization tasks, but fails when tasks require tracking context across the entire document.
