SWE MERA Challenge

All details on how to participate in the competition are provided here: SWE-MERA-CHALLENGE

Existing benchmarks for this task face a fundamental dilemma: they are either compromised by data contamination, or exposed to agents that exploit internet access to copy ready-made commits directly from GitHub.

To resolve this, we introduce a two-tier leaderboard guarded by a network allow-list that grants access to standard software repositories, like PyPI, while blocking leakage of solutions. The same design also accommodates self-hosted open-source models, which participants can run on organizer-provided GPUs without any external network access.

Public (Stage I)

The public tier is open for continuous, virtually unlimited debugging, with its allow-list extended dynamically by organizers as participants request legitimate tools for their agents.

Private (Stage II, Stage III)

The private tier, by contrast, permits only three scoring attempts against a hidden dataset --- striking a deliberate balance that rewards genuine model generalization and protects the academic integrity of the final evaluation.

We prepare baseline how to submit the solution described bellow. You could change model or harness inside those baseline

Competition Timeline

Requirements

uv, Docker, Git, and 60 GB of free disk space.

Baseline

All details are here: SWE-MERA-CHALLENGE

Install

uv pins Python 3.12, creates the virtual environment, downloads the interpreter when necessary, and installs the pinned dependencies:

git clone https://github.com/MERA-Evaluation/SWE-MERA-CHALLENGE.git
cd SWE-MERA-CHALLENGE

uv python pin 3.12
uv venv --python 3.12 .venv
uv pip install --python .venv/bin/python -r requirements.txt
uv run --python .venv/bin/python -- harbor --version

Model configuration

Store provider credentials in .env, then select the model and agent using the harbor run arguments below.

cp configs/.env.example .env
vim .env

Run the full benchmark

uv run --python .venv/bin/python -- harbor run \
  --path tasks \
  --agent mini-swe-agent \
  --model openrouter/qwen/qwen3.7-flash \
  --env-file .env \
  --jobs-dir jobs/qwen3.7-flash-run-001 \
  --n-attempts 1 \
  --n-concurrent 2 \
  --yes

Estimated cost: Approximately 1.5 USD for a full run with this model.

Submit results

Point the packaging script at the timestamped job directory:

scripts/create-submission.sh jobs/run-001/2026-08-30__05-45-50

This creates sample_submission.zip. Attach that file to the submission form on the competition website.

Rules

• Give each task exactly one attempt.
• Submit only patches produced by the agent during the run.
• Do not use gold patches, evaluation tests, or test-specific workarounds.
• Retain the Harbor job directory and trajectories for verification.