METR’s $2,500 benchmark proves AI is still worse than humans at coding
metr introduced the 'expenditure horizon' metric showing humans improve nanoGPT at ~$2,500/percent, while only top models like gpt-5.5 and opus-4.8 offer viable automation, signaling agents remain inefficient for complex r&d.
metr released a framework to quantify automation value by defining the 'expenditure horizon'—the budget point where ai costs equal human costs for a specific task. applied to the nanogpt speedrun, the analysis estimated humans spend roughly $2,500 per one-percent performance gain, derived from 16 hours of work at a $150 hourly rate. in head-to-head tests with a $10,000 budget, gpt-5 and opus-4.1 failed to contribute verifiable improvements, whereas gpt-5.5 and opus-4.8 achieved 1 to 1.5 percent gains respectively. the results suggest that while cutting-edge models are beginning to cross the cost-effectiveness threshold for complex engineering, the majority of ai agents remain economically unviable for high-stakes r&d tasks.