ABCDEFGHIJKLMNOPQRSTUVWXYZ
1
Model% ResolvedOrgDateLogsTrajsSite
2
o371.7OpenAI2024-12-20
3
🆕 🥇 🤠 CodeStory Midwit Agent + swe-search
62.2-2024-12-21✓✓🔗
4
🆕 🥈 devlo58.22024-12-13✓✓🔗
5
🆕 🥉 Emergent E1 (v2024-12-23)
57.22024-12-23✓✓🔗
6
🆕 Gru(2024-12-08)
572024-12-08✓✓🔗
7
🆕 EPAM AI/Run Developer Agent v20241212 + Anthopic Claude 3.5 Sonnet
55.42024-12-12✓✓🔗
8
🆕 🥇 Amazon Q Developer Agent (v20241202-dev)
552024-12-02✓✓🔗
9
🆕 🥈 devlo54.22024-11-08✓✓🔗
10
🥉 🤠 ✅ OpenHands + CodeAct v2.1 (claude-3-5-sonnet-20241022)
532024-10-29✓✓🔗
11
🆕 Engine Labs (2024-11-25)
51.82024-11-25✓✓🔗
12
🆕 🤠 Agentless-1.5 + Claude-3.5 Sonnet (20241022)
50.82024-12-02✓✓🔗
13
Solver (2024-10-28)
502024-10-28✓✓🔗
14
🆕 Bytedance MarsCode Agent
502024-11-25✓✓🔗
15
🆕 nFactorial (2024-11-05)
49.22024-11-05✓✓🔗
16
Tools + Claude 3.5 Sonnet (2024-10-22)
492024-10-22✓✓🔗
17
🤠 ✅ Composio SWE-Kit (2024-10-25)
48.62024-10-25✓✓🔗
18
🆕 🤠 ✅ AppMap Navie v2
47.22024-11-06✓✓🔗
19
Emergent E1 (v2024-10-12)
46.62024-10-23✓✓🔗
20
🆕 🤠 AutoCodeRover-v2.0 (Claude-3.5-Sonnet-20241022)
46.22024-11-08✓✓🔗
21
Solver (2024-09-12)
45.42024-09-24✓✓🔗
22
Gru(2024-08-24)
45.22024-08-24✓✓🔗
23
Solver (2024-09-12)
43.62024-09-20✓✓🔗
24
nFactorial (2024-10-30)
41.62024-10-30✓✓🔗
25
🆕 Nebius AI Qwen 2.5 72B Generator + LLama 3.1 70B Critic
40.62024-11-13✓✓🔗
26
Tools + Claude 3.5 Haiku
40.62024-10-22✓✓🔗
27
Honeycomb40.62024-08-20✓✓🔗
28
🤠 Composio SWEkit + Claude 3.5 Sonnet (2024-10-16)
40.62024-10-16✓✓🔗
29
EPAM AI/Run Developer Agent v20241029 + Anthopic Claude 3.5 Sonnet
39.62024-10-29✓✓🔗
30
Amazon Q Developer Agent (v20240719-dev)
38.82024-07-21✓✓🔗
31
🤠 Agentless-1.5 + GPT 4o (2024-05-13)
38.82024-10-28✓✓🔗
32
AutoCodeRover (v20240620) + GPT 4o (2024-05-13)
38.42024-06-28✓-🔗
33
🤠 ✅ SWE-agent + Claude 3.5 Sonnet
33.62024-06-20✓✓-
34
🆕 Artemis Agent v1 (2024-11-20)
322024-11-20✓✓🔗
35
nFactorial (2024-10-07)
31.62024-10-07✓✓🔗
36
🤠 Lingma Agent + Lingma SWE-GPT 72b (v0925)
28.82024-10-02✓✓🔗
37
EPAM AI/Run Developer Agent + GPT4o
272024-10-16✓✓🔗
38
🤠 ✅ AppMap Navie + GPT 4o (2024-05-13)
26.22024-06-15✓-🔗
39
nFactorial (2024-10-01)
25.82024-10-01✓✓🔗
40
Amazon Q Developer Agent (v20240430-dev)
25.62024-05-09✓-🔗
41
🤠 Lingma Agent + Lingma SWE-GPT 72b (v0918)
252024-09-18✓✓🔗
42
EPAM AI/Run Developer Agent + GPT4o
242024-08-20✓✓🔗
43
🤠 ✅ SWE-agent + GPT 4o (2024-05-13)
23.22024-07-28✓✓🔗
44
🤠 ✅ SWE-agent + GPT 4 (1106)
22.42024-04-02✓✓🔗
45
🤠 ✅ SWE-agent + Claude 3 Opus
18.22024-04-02✓✓-
46
🤠 Lingma Agent + Lingma SWE-GPT 7b (v0925)
18.22024-10-02✓✓🔗
47
🤠 Lingma Agent + Lingma SWE-GPT 7b (v0918)
10.22024-09-18✓✓🔗
48
🤠 ✅ RAG + Claude 3 Opus
72024-04-02✓-🔗
49
🤠 ✅ RAG + Claude 2
4.42023-10-10✓--
50
🤠 ✅ RAG + GPT 4 (1106)
2.82024-04-02✓--
51
🤠 ✅ RAG + SWE-Llama 7B
1.42023-10-10✓--
52
🤠 ✅ RAG + SWE-Llama 13B
1.22023-10-10✓--
53
🤠 ✅ RAG + ChatGPT 3.5
0.42023-10-10✓--
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100