According to The Verge, the report published Thursday says benchmark scores suggest "that Anthropic has made roughly two years of historical progress over the last year." METR's researchers wrote that they "are confident" the acceleration is not simply due to more training compute, and that the majority of it, as of October, "is due to AI," with that contribution growing.
Employees using the company's most capable internal model self-reported being about 12 times as productive on average, roughly double what they reported a few months earlier. Most of the acceleration still comes from AI assisting researchers rather than working independently, but "there are some signs that fully autonomous contributions will play a larger role soon." The researchers found no clear signs of diminishing returns or bottlenecks.
METR also reports that over the past six months Anthropic finished training three models it has not released, in July, August-September and September-October. The company shared non-public information with the researchers, but under the agreement it was allowed to redact parts of the final report, and according to The Verge it did so liberally. The report comes as Anthropic prepares to go public.