There’s extra. To make its use of reinforcement studying as environment friendly as attainable, DeepSeek has additionally developed a brand new algorithm referred to as Group Relative Coverage Optimization (GRPO). It first used GRPO a 12 months in the past, to construct a mannequin referred to as DeepSeekMath.
We’ll skip the particulars—you simply must know that reinforcement studying includes calculating a rating to find out whether or not a possible transfer is sweet or dangerous. Many present reinforcement-learning strategies require an entire separate mannequin to make this calculation. Within the case of huge language fashions, which means a second mannequin that could possibly be as costly to construct and run as the primary. As a substitute of utilizing a second mannequin to foretell a rating, GRPO simply makes an informed guess. It’s low-cost, however nonetheless correct sufficient to work.
A standard method
DeepSeek’s use of reinforcement studying is the principle innovation that the corporate describes in its R1 paper. However DeepSeek just isn’t the one agency experimenting with this system. Two weeks earlier than R1 dropped, a staff at Microsoft Asia introduced a mannequin referred to as rStar-Math, which was educated in an analogous means. “It has equally enormous leaps in efficiency,” says Matt Zeiler, founder and CEO of the AI agency Clarifai.
AI2’s Tulu was additionally constructed utilizing environment friendly reinforcement-learning strategies (however on high of, not as a substitute of, human-led steps like supervised fine-tuning and RLHF). And the US agency Hugging Face is racing to duplicate R1 with OpenR1, a clone of DeepSeek’s mannequin that Hugging Face hopes will expose much more of the elements in R1’s particular sauce.
What’s extra, it’s an open secret that high corporations like OpenAI, Google DeepMind, and Anthropic could already be utilizing their very own variations of DeepSeek’s method to coach their new era of fashions. “I’m positive they’re doing virtually the very same factor, however they’ll have their very own taste of it,” says Zeiler.
However DeepSeek has a couple of trick up its sleeve. It educated its base mannequin V3 to do one thing referred to as multi-token prediction, the place the mannequin learns to foretell a string of phrases directly as a substitute of one by one. This coaching is cheaper and seems to spice up accuracy as properly. “If you concentrate on the way you converse, while you’re midway by way of a sentence, you understand what the remainder of the sentence goes to be,” says Zeiler. “These fashions needs to be able to that too.”
It has additionally discovered cheaper methods to create massive knowledge units. To coach final 12 months’s mannequin, DeepSeekMath, it took a free knowledge set referred to as Widespread Crawl—an enormous variety of paperwork scraped from the web—and used an automatic course of to extract simply the paperwork that included math issues. This was far cheaper than constructing a brand new knowledge set of math issues by hand. It was additionally more practical: Widespread Crawl consists of much more math than every other specialist math knowledge set that’s accessible.
And on the {hardware} facet, DeepSeek has discovered new methods to juice previous chips, permitting it to coach top-tier fashions with out coughing up for the newest {hardware} available on the market. Half their innovation comes from straight engineering, says Zeiler: “They positively have some actually, actually good GPU engineers on that staff.”
