DeepSeek V4 Flash for 8 sets of Harness: Pi Agent has the highest success rate and saves money, Claude Code is the fastest but most expensive

Comparative news, according to monitoring, AI Agent infrastructure company Composio connected DeepSeek V4 Flash to 8 Harness sets and tested with the same batch of 30 Agent tasks. Harness is an execution framework outside of the model, responsible for context, tool calls, and task execution. The task involves real apps such as Gmail, GitHub, Slack, Calendar, Notion, etc. The passing conditions for the 8 sets of harnesses are as follows: 1. Pi Agent: 20/302. Oh My Pi: 17/303. Claude Code: 16/304. Codex: 16/305. Deep Agents: 16/306. Hermes Agent: 15/307. Prime Agent: 15/248. OpenCode: 14/30 By simply changing Harness, DeepSeek V4 Flash can go from 14 tasks to 20. The cost and speed also vary greatly. Pi Agent only costs around $0.028 per successful mission, and Claude Code is $0.195, which is nearly 7x. Claude Code is the fastest, with a median time of 122.7 seconds; Oh My Pi is the slowest at 272.4 seconds. However, Pi's test configuration is different from the uniform conditions: it uses high instead of maximum inference strength, and 24 of the 30 tasks use the official DeepSeek API instead of OpenRouter. So this set of gaps isn't necessarily entirely attributable to Harness. Composio previously used Kimi K3 for another round of Harness reviews. At the time, Oh My Pi ranked first with 22/25, and the original Pi Agent was 18/25. OMP itself is an enhanced coding version forked from Pi, adding capabilities such as sub-agents. After switching to DeepSeek V4 Flash, the original Pi came first.