Rock Matrix, working with the Hong Kong University of Science and Technology and Carnegie Mellon University, has open-sourced EdgeRock-4B, a small language model whose memory use stays flat no matter how long the context gets. The base model is Alibaba's Qwen3.5-4B, according to QbitAI, the project's GitHub page and its arXiv paper.
EdgeRock was not pretrained from scratch. The team converted Qwen3.5-4B in stages. Its 24 Gated DeltaNet layers, which already carry a fixed-size state, were kept. Its 8 full-attention layers, which normally store keys and values for every past token, were turned into sliding-window attention over the last 1,024 tokens. To stop the model forgetting what falls out of that window, the team added Rock Memory Anchors: for each group of KV heads, every 256 candidate tokens the model keeps the top 1,024 most useful ones in a small anchor bank.
The result is a KV cache fixed at 72 MiB at 256K context, against 8 GiB for Qwen3.5-4B, a 99.1% cut. The full per-sequence state is 97.76 MiB. On a single 80 GB Hopper GPU, EdgeRock served 512 concurrent streams at 12,124 tokens per second, 6.42 times Qwen3.5-4B's 1,889 tokens per second at its batch limit of 64, with peak memory of 61.26 GiB. With 128K tokens in and 128K out, it held a steady 233.7 tokens per second, while Qwen slowed from about 169 to 151.3. GPU time fell to 24.9% of Qwen's at 4K input and 15.6% at 32K.
The trade-off on accuracy is small, by the team's numbers. EdgeRock keeps 96.3% of Qwen3.5-4B's AIME score and 97.5% on MATH-500, beats it on 8 benchmarks, and keeps 95.3% of its vision scores. Its mean pass@1 across AIME 2024 to 2026 is 82.9%, against 18.8% for the strongest other bounded-state model tested, RWKV-7 G1j 13.3B, and it ranks first among bounded-state models on 24 of 29 text benchmarks. In an ablation, a retrieval task scored 94.1% with the memory anchors and 0.4% without them.
The weights accept text and image input and come with quantized builds. They are released under the EdgeRock model license, which the team says allows commercial use. Larger sizes are planned.
SEE ALSO: MemTensor Open-Sources MemT 1.0, Memory-Native Models That Keep Long Histories in a Fixed State · Baidu Unveils Unlimited-OCR: Constant KV Cache Delivers SOTA Performance on Long Documents