LongCat 2.0: The Beginning of the End of NVIDIA MOAT?
This title could be clearer and more informative.Try out Clickbait Shieldfor free (5 uses left this month).
LongCat 2.0 is a 1.6 trillion parameter open-weights LLM from Meituan-backed AI lab LongCat, trained entirely on Chinese AI hardware (ASIC-based AI6 superpods) without NVIDIA GPUs. The model matches near state-of-the-art performance and introduces three key architectural innovations: (1) LongCat Sparse Attention with streaming-aware indexing, cross-layer indexing with distillation, and hierarchical coarse-to-fine retrieval; (2) N-gram embeddings allocating 135B parameters to learn multi-token phrase representations up to 5-grams, particularly beneficial for code; and (3) use of the Muon optimizer and multi-tier on-policy distillation (MOPD) for post-training. The model supports a 1 million token context window and achieves 59.5% on SWEBench Pro, positioning it as a strong coding and agentic model. Its successful training on Chinese hardware with no loss spikes signals a meaningful step toward NVIDIA independence for frontier AI training.
•13m watch time
Questions this post answers
What architectural innovations does LongCat 2.0 introduce compared to DeepSeek V3?
LongCat 2.0 introduces three main innovations over DeepSeek's baseline: LongCat Sparse Attention (streaming-aware indexing, cross-layer index sharing across 2 layers with distillation, and hierarchical coarse-to-fine retrieval), N-gram embeddings allocating 135B parameters to learn token-sequence representations up to 5-grams, and use of the Muon optimizer. The n-gram embedding approach is more conservatively integrated than DeepSeek's equivalent, attaching directly to the embedding module rather than as a full transformer block. Engineers tracking open-weights LLM architecture decisions find the latest technical comparisons on daily.dev.
How does cross-layer indexing in sparse attention work and why does it matter for long context models?
Cross-layer indexing reuses a single sparse attention index across multiple transformer layers instead of recomputing it per layer. LongCat shares one index across two layers; GLM 5.2 shares across four. LongCat also applies cross-layer distillation during training so the shared index learns what tokens are useful for both layers simultaneously. This matters especially with multi-token prediction (MTP), where naive setups would pay the indexer cost once per draft step across a million-token context. Developers building on long-context LLMs keep up with sparse attention research through daily.dev.
What Chinese AI hardware did LongCat use to train LongCat 2.0 and how does it compare to NVIDIA GPUs?
LongCat 2.0 was trained on over 50,000 AI6 superpods, where AI6 refers to application-specific integrated circuits for AI — not NVIDIA GPUs. The LongCat blog explicitly compared their AI6 superpod software ecosystem to NVIDIA's, noting it is less mature. Despite this, training completed with no rollbacks or unrecoverable loss spikes, demonstrating that a frontier-scale 1.6 trillion parameter model can be trained end-to-end on Chinese hardware. Teams watching the Chinese AI hardware ecosystem for supply chain and infrastructure decisions follow developments on daily.dev.
119.8K Impressions3 Comments