UPDATE: Stage 2 has started!

Existing benchmarks for this task face a fundamental dilemma: they are either compromised by data contamination, or exposed to agents that exploit internet access to copy ready-made commits directly from GitHub.

To resolve this, we introduce a two-tier leaderboard guarded by a network allow-list that grants access to standard software repositories, like PyPI, while blocking leakage of solutions. The same design also accommodates self-hosted open-source models, which participants can run on organizer-provided GPUs without any external network access.

Public (Stage I)

The public tier is open for continuous, virtually unlimited debugging, with no restrictions on either the models or the harnesses used.

Private (Stage II, Stage III)

The private tier, by contrast, permits only three scoring attempts per day against a hidden dataset — striking a deliberate balance that rewards genuine model generalization and protects the academic integrity of the final evaluation. Stage II still allows debugging: for example, the allow-list is extended dynamically by organizers as participants request legitimate tools for their agents.

We provide a baseline that shows how to run and submit a solution. You can replace either the model or the harness inside this baseline.

Competition Timeline

Period	Phase	Languages
Jul 13 -- Oct 12, 2026	Stage I (Public)	Go, Python, PHP
Oct 23 -- Nov 20, 2026	Stage II (Private)	Go, Python, PHP, Language 1
Nov 21 -- Nov 23, 2026	Stage III (Private)	Go, Python, PHP, Language 1, Language 2
Nov 2026	Final private scoring	---

Data (Stage 2)

53 tasks in 4 languages and different complexities.

Metric

The leaderboard metric is pass@1: the fraction of tasks solved on a single attempt.

• Each trial is scored by the verifier with a binary reward: 1 if the agent's patch makes the hidden evaluation tests pass, 0 otherwise (written to <trial>/verifier/reward.txt).
• Since the rules allow exactly one attempt per task (--n-attempts 1), pass@1 is simply the mean reward across tasks: pass@1 = (number of trials with reward 1) / (number of tasks).
• This is the same value Harbor reports as Mean in the run summary table and in the run-level result.json (stats.evals.<eval>.metrics[0].mean), so you can see your provisional score locally before submitting. The submission checker recomputes it independently from the per-trial verifier/reward.txt files.

Example: a run over 3 tasks with rewards 1, 1, 0 has pass@1 = 2/3 ≈ 0.667.

Requirements

• uv, Docker, Git, and 60 GB of free disk space (each task ships its own Docker image of 1–2 GB; a full run pulls all of them).
• On Apple Silicon: task images are linux/amd64. Docker (or Colima) runs them via emulation — make sure Rosetta/qemu is available. Pre-pulling images with --platform linux/amd64 (see Troubleshooting) avoids build-time surprises.

Evaluation Environment

All the tasks will run on NVIDIA PRO 6000, a task will be given 2 CPU, 8 Gb RAM and 96 GB VRAM. Your submission will have up to 4 hours to download a model and up to 30 minutes per each task.

Baseline

More details can be found here: SWE-MERA-CHALLENGE

Install

uv pins Python 3.12, creates the virtual environment, downloads the interpreter when necessary, and installs the pinned dependencies:

git clone https://github.com/MERA-Evaluation/SWE-MERA-CHALLENGE.git
cd SWE-MERA-CHALLENGE

uv python pin 3.12
uv venv --python 3.12 .venv
uv pip install --python .venv/bin/python -r requirements.txt
uv run --python .venv/bin/python -- harbor --version

Model configuration

Store provider credentials in .env, then select the model and agent using the harbor run arguments below.

cp configs/.env.example .env
vim .env

Run a single task (recommended first step for Stage 1)

Before a full run, validate your setup end-to-end on one task. This builds one image, runs the agent, and produces a scored trial in a few minutes:

uv run --python .venv/bin/python -- harbor run \
  --path tasks \
  --include-task-name py-009 \
  --agent mini-swe-agent \
  --model openrouter/qwen/qwen3.7-flash \
  --env-file .env \
  --jobs-dir jobs/debug \
  --n-attempts 1 \
  --n-concurrent 1 \
  --agent-setup-timeout-multiplier 10 \
  --yes

The reward for each trial is written to <trial>/verifier/reward.txt (1 = solved, 0 = not). Inspect a finished run with harbor view jobs/debug.

Run the full benchmark

uv run --python .venv/bin/python -- harbor run \
  --path tasks \
  --agent mini-swe-agent \
  --model openrouter/qwen/qwen3.7-flash \
  --env-file .env \
  --jobs-dir jobs/qwen3.7-flash-run-001 \
  --n-attempts 1 \
  --n-concurrent 2 \
  --agent-setup-timeout-multiplier 10 \
  --yes

Estimated cost: Approximately 1.5 USD for a full run with this model.

Submit results (Stage 1)

Point the packaging script at the timestamped job directory (the run itself, not its parent):

scripts/create-submission.sh jobs/run-001/2026-08-30__05-45-50

This creates sample_submission.zip. Upload this file using the "Submit solution" button at the top of this page.

What the checker expects. The ZIP must contain exactly one Harbor run: the contents of a single timestamped job directory, with the run metadata files at the archive root and one directory per trial:

sample_submission.zip
├── config.json          # run config
├── lock.json            # environment/task lock
├── result.json          # run-level results and reward stats
├── job.log
└── <task>__<id>/        # one directory per trial
    ├── result.json      # trial results
    ├── agent/           # agent trajectories
    ├── artifacts/       # includes the produced model_patch.diff
    └── verifier/
        ├── reward.txt   # 1 or 0 — the trial score
        └── report.json, parse_result.json, ...

Common mistakes that lead to rejection:

• Zipping the parent jobs/ directory or several runs at once — the checker requires exactly one run (harbor_runs_found: 0 or more than 1).
• Packaging hand-picked files (e.g. a CSV of patch files) instead of a genuine Harbor job directory — the checker reads the run's result.json and per-trial verifier/reward.txt, nothing else.
• Re-packaging a run after manually editing results — keep the original job directory intact; organizers may request it for verification (see Rules).

Stage 2 Submission

Participants provide:

• submission.yaml with only agent and model;
• the inference build context in inference/;
• custom_agent/ when agent: custom_agent is selected.

run.sh and inference/test.sh are trusted organizer files. The official evaluator always uses its own copies of these scripts and ignores copies from the uploaded ZIP.

run.sh validates the submission, builds and starts the inference image, checks its API, and launches Harbor. It removes this run's container and image before starting and on exit, including failure. The runner preserves the exit code of the failed stage. Do not run concurrent submissions with the same username.

The inference docker build has a fixed six-hour limit (BUILD_TIMEOUT_SEC=21600). GNU timeout (or gtimeout on macOS) sends SIGTERM when that limit expires and SIGKILL after a further 60 seconds if the build command has not exited. A timeout stops the run before inference or Harbor starts, prints an error, and triggers cleanup. The runner returns 124 for a normal timeout or 137 for forced killing. This limit is independent of model startup and task timeouts in task.toml.

From the repository root, run the public tasks locally with:

./example_stage2/run.sh --username user_id

--username is mandatory and names Docker resources and the default output directory. It is not an authentication credential or a field in the uploaded YAML. Use 1-64 lowercase letters, digits, ., _ or -, starting with a letter or digit. The ODS evaluator supplies a name derived from the submission ID automatically. An old username field in YAML is rejected.

To override paths or concurrency:

SWE_MERA_TASKS_DIR="$PWD/tasks" \
SWE_MERA_JOBS_DIR="$PWD/jobs/stage2-local" \
SWE_MERA_N_CONCURRENT=2 \
./example_stage2/run.sh --username user_id

Without overrides, tasks come from <project>/tasks and results go to <project>/jobs/stage2/<username>. Private evaluation must supply its task directory. The runner requires the project's .venv/bin/python and .venv/bin/harbor and calls Harbor directly.

The inference API is published on 127.0.0.1:8000 by default. The organizer exposes three GPUs and runs Harbor with 10 concurrent tasks by default. The bundled Qwen/Qwen3.5-9B example serves qwen35-9b-chat using three data-parallel replicas and --max-num-seqs 512. The agent timeout multiplier is 1; a task with a 3600-second agent limit therefore gets 3600 seconds. The verifier timeout is unchanged.

custom_agent is imported by the Harbor runner; only its tool commands execute inside the no-internet task container. Production evaluation must therefore run participant agents on an isolated worker without ODS credentials.

Troubleshooting

Problems we hit ourselves while validating the baseline:

• Agent setup times out (default 360 s). Installing agent dependencies inside the container can be slow on cold caches or emulated CPUs. Pass --agent-setup-timeout-multiplier 10.
• Image pulls fail intermittently during the run. The registry (auth.docker.io) sometimes times out mid-build. Pre-pull the task images before starting:

  docker pull --platform linux/amd64 rewive/swe-mera-challenge:pallets-click-3208

  Image tags follow the pattern rewive/swe-mera-challenge:<owner>-<repo>-<pr-number> and are listed in each task's YAML file under tasks/.
• Apple Silicon (arm64). All task images are linux/amd64; use Docker Desktop or Colima with emulation. Everything (build, agent run, verifier) works under emulation, just slower — budget roughly 2–3× the wall-clock time.
• Checking where a run went wrong. Look at <trial>/trial.log (agent/verifier lifecycle), <trial>/verifier/command_test.log (test output), and <trial>/verifier/reward.txt (final score). harbor view <jobs-dir> gives a summary table.

Rules

• Give each task exactly one attempt (--n-attempts 1).
• Submit only patches produced by the agent during the run.
• Do not use gold patches, evaluation tests, or test-specific workarounds.
• Retain the Harbor job directory and trajectories for verification.

Support

If you want to ask any questions, there is an official support channel in Mattermost: https://mm.ods.ai/ods/channels/swe-mera-challenge. To access Mattermost use your ODS.ai account (choose authorize via ODS.ai on the Mattermost welcome page).