SWE MERA Challenge

Existing benchmarks for this task face a fundamental dilemma: they are either compromised by data contamination, or exposed to agents that exploit internet access to copy ready-made commits directly from GitHub. To resolve this, we introduce a two-tier leaderboard guarded by a network allow-list that grants access to standard software repositories, like PyPI, while blocking leakage of solutions. The same design also accommodates self-hosted open-source models, which participants can run on organizer-provided GPUs without any external network access. The public tier is open for continuous, virtually unlimited debugging, with its allow-list extended dynamically by organizers as participants request legitimate tools for their agents. The private tier, by contrast, permits only three scoring attempts against a hidden dataset --- striking a deliberate balance that rewards genuine model generalization and protects the academic integrity of the final evaluation. To ensure a smooth participant journey, each team receives a Starter Kit containing a local network-gateway simulator, a baseline ReAct agent, and documentation, so that agents can be developed and validated entirely offline before any submission. The public Stage I leaderboard supports virtually unlimited iteration and provides continuous feedback.

Competition Timeline