JetBrains’ Junie evaluation pipeline analyzed how two frontier-Lab LLMs perform when plugged into the same coding agent. In a private benchmark, Claude Opus 4.7 and Gemini 3.5 Flash solved the same number of tasks, illustrating that similar success rates can conceal distinct execution profiles.
The study emphasizes that a single metric like task resolution does not convey how an agent arrived at a solution or how much effort and risk were involved in getting there.
A FOUR-PERSPECTIVE EVALUATION FRAMEWORK
For each benchmark task, the pipeline combines context from the issue and repository, the generated patch, test results, and an execution trace. It evaluates outcomes across four dimensions: functional outcome, execution efficiency, patch quality, and process quality. The goal is to supplement binary success with a richer understanding of how the solution was produced. TechStaged has also covered New arXiv preprint introduces FedEHR-Agents for Federated EHR Modeling.
Costs, latency, and tool usage are tracked to distinguish productive work from unnecessary exploration.
KEY RESULTS FROM 523 TASKS ACROSS FOUR DATASETS
Across 523 tasks, Opus resolved 267 tasks (51.1%), while Gemini resolved 254 (48.6%). Both models produced the same outcome on 430 tasks (214 solved, 216 failed), with 93 tasks separating them.
A direct example cited shows the same task can be reached via very different paths: one model completed the patch quickly with a focused search, while the other explored a broader module before implementing changes. The overall score can mask substantial differences in how objectives are met.
TRAJECTORIES, PATCH QUALITY, AND THE COSTS OF DIFFERENT PATHS
Opus tended to identify the underlying cause more often, but its runs frequently lacked executable validation, with 123 runs not performing executable checks (68 of which still resolved the task). This creates a risk that a patch’s correctness is never confirmed.
Gemini more often performed an executable check, but its trajectories showed higher risk of unnecessary expansion and reliance on unverified APIs or test fixtures. In a subset of runs, Gemini exhibited moderate to severe hallucination (195 runs) and notable redundancy (80 runs).
The study notes that trajectory analysis can reveal strengths and weaknesses not visible from the final patch alone, underscoring the value of model-specific profiling for model selection and deployment decisions.
WHAT “MODEL PROFILES” ADD TO BENCHMARKING
The researchers describe model-specific profiles that summarize strengths and weaknesses. Opus demonstrates stronger diagnostic capabilities and a tendency to switch to implementation and testing phases, while Gemini shows strengths in using executable feedback to refine solutions.
The pairwise comparisons illustrate how identical task outcomes can arise from different reasoning and execution patterns, making a purely score-based comparison insufficient for model selection in agentic coding.
CONTEXT AND IMPLICATIONS
The work cautions that benchmark scores should be interpreted with care, as performance can vary by task type and repository structure. It also highlights the importance of considering trajectory data and patch quality when evaluating coding agents.
Overall, the study advocates for more granular evaluation to inform model choice and to guide future improvements in agentic-coding benchmarks.
RELATED COVERAGE
SOURCES
- The JetBrains Blog: From Leaderboards to Model Profiles: A Deep Dive Evaluation of LLMs for Agentic Coding Published · Primary source
