**UPDATE**: Stage 2 has started!

Existing benchmarks for this task face a fundamental dilemma: they are either **compromised by data contamination**, or exposed to agents that **exploit internet access to copy ready-made commits directly from GitHub**.

To resolve this, we introduce a **two-tier leaderboard** guarded by a network allow-list that grants access to standard software repositories, like PyPI, while blocking leakage of solutions. The same design also accommodates **self-hosted open-source models**, which participants can run on organizer-provided GPUs without any external network access.

### Public (Stage I)

The public tier is open for continuous, virtually unlimited debugging, with no restrictions on either the models or the harnesses used.

### Private (Stage II, Stage III)

The private tier, by contrast, permits only **three scoring attempts per day** against a hidden dataset — striking a deliberate balance that rewards genuine model generalization and protects the academic integrity of the final evaluation. Stage II still allows debugging: for example, the allow-list is extended dynamically by organizers as participants request legitimate tools for their agents.

We provide a baseline that shows how to run and submit a solution. You can replace either the model or the harness inside this baseline.

**Competition Timeline**

| Period | Phase | Languages |
| --- | --- | --- |
| Jul 13 -- Oct 12, 2026 | Stage I (Public) | Go, Python, PHP |
| Oct 23 -- Nov 20, 2026 | Stage II (Private) | Go, Python, PHP, Language 1 |
| Nov 21 -- Nov 23, 2026 | Stage III (Private) | Go, Python, PHP, Language 1, Language 2 |
| Nov 2026 | Final private scoring | --- |

## Data (Stage 2)

48 tasks in 4 languages and different complexities. 

## Metric

The leaderboard metric is **pass@1**: the fraction of tasks solved on a single attempt.

- Each trial is scored by the verifier with a binary reward: `1` if the agent's patch makes the hidden evaluation tests pass, `0` otherwise (written to `<trial>/verifier/reward.txt`).
- Since the rules allow exactly one attempt per task (`--n-attempts 1`), pass@1 is simply the mean reward across tasks: `pass@1 = (number of trials with reward 1) / (number of tasks)`.
- This is the same value Harbor reports as **Mean** in the run summary table and in the run-level `result.json` (`stats.evals.<eval>.metrics[0].mean`), so you can see your provisional score locally before submitting. The submission checker recomputes it independently from the per-trial `verifier/reward.txt` files.

Example: a run over 3 tasks with rewards `1, 1, 0` has pass@1 = 2/3 ≈ 0.667.

## Requirements

- uv, Docker, Git, and **60 GB of free disk space** (each task ships its own Docker image of 1–2 GB; a full run pulls all of them).
- On Apple Silicon: task images are `linux/amd64`. Docker (or Colima) runs them via emulation — make sure Rosetta/qemu is available. Pre-pulling images with `--platform linux/amd64` (see Troubleshooting) avoids build-time surprises.

## Evaluation Environment
All the tasks will run on **NVIDIA PRO 6000**, a task will be given 2 CPU, 8 Gb RAM and 48 GB VRAM. Your submission will have up to 4 hours to download a model and up to 30 minutes per each task.

## Baseline

More details can be found here: [SWE-MERA-CHALLENGE](https://github.com/MERA-Evaluation/SWE-MERA-CHALLENGE)

### Install

uv pins Python 3.12, creates the virtual environment, downloads the interpreter when necessary, and installs the pinned dependencies:

```bash
git clone https://github.com/MERA-Evaluation/SWE-MERA-CHALLENGE.git
cd SWE-MERA-CHALLENGE

uv python pin 3.12
uv venv --python 3.12 .venv
uv pip install --python .venv/bin/python -r requirements.txt
uv run --python .venv/bin/python -- harbor --version
```

### Model configuration

Store provider credentials in `.env`, then select the model and agent using the `harbor run` arguments below.

```bash
cp configs/.env.example .env
vim .env
```

### Run a single task (recommended first step for Stage 1)

Before a full run, validate your setup end-to-end on one task. This builds one image, runs the agent, and produces a scored trial in a few minutes:

```bash
uv run --python .venv/bin/python -- harbor run \
 --path tasks \
 --include-task-name py-009 \
 --agent mini-swe-agent \
 --model openrouter/qwen/qwen3.7-flash \
 --env-file .env \
 --jobs-dir jobs/debug \
 --n-attempts 1 \
 --n-concurrent 1 \
 --agent-setup-timeout-multiplier 10 \
 --yes
```

The reward for each trial is written to `<trial>/verifier/reward.txt` (`1` = solved, `0` = not). Inspect a finished run with `harbor view jobs/debug`.

### Run the full benchmark

```bash
uv run --python .venv/bin/python -- harbor run \
 --path tasks \
 --agent mini-swe-agent \
 --model openrouter/qwen/qwen3.7-flash \
 --env-file .env \
 --jobs-dir jobs/qwen3.7-flash-run-001 \
 --n-attempts 1 \
 --n-concurrent 2 \
 --agent-setup-timeout-multiplier 10 \
 --yes
```

> **Estimated cost:** Approximately 1.5 USD for a full run with this model.

### Submit results (Stage 1)

Point the packaging script at the **timestamped job directory** (the run itself, not its parent):

```bash
scripts/create-submission.sh jobs/run-001/2026-08-30__05-45-50
```

This creates `sample_submission.zip`. Upload this file using the "Submit solution" button at the top of this page.

**What the checker expects.** The ZIP must contain **exactly one Harbor run**: the contents of a single timestamped job directory, with the run metadata files at the archive root and one directory per trial:

```
sample_submission.zip
├── config.json # run config
├── lock.json # environment/task lock
├── result.json # run-level results and reward stats
├── job.log
└── <task>__<id>/ # one directory per trial
 ├── result.json # trial results
 ├── agent/ # agent trajectories
 ├── artifacts/ # includes the produced model_patch.diff
 └── verifier/
 ├── reward.txt # 1 or 0 — the trial score
 └── report.json, parse_result.json, ...
```

Common mistakes that lead to rejection:

- Zipping the **parent** `jobs/` directory or several runs at once — the checker requires exactly one run (`harbor_runs_found: 0` or more than 1).
- Packaging hand-picked files (e.g. a CSV of patch files) instead of a genuine Harbor job directory — the checker reads the run's `result.json` and per-trial `verifier/reward.txt`, nothing else.
- Re-packaging a run after manually editing results — keep the original job directory intact; organizers may request it for verification (see Rules).
 
# Stage 2 Submission

Participants provide:

- `submission.yaml` with only `agent` and `model`;
- the inference build context in `inference/`;
- `custom_agent/` when `agent: custom_agent` is selected.

`run.sh` and `inference/test.sh` are trusted organizer files.
The official evaluator always uses its own copies of these scripts and ignores
copies from the uploaded ZIP.

`run.sh` validates the submission, builds and starts the inference image, checks
its API, and launches Harbor. It removes this run's container and image before
starting and on exit, including failure. The runner preserves the exit code of
the failed stage. Do not run concurrent submissions with the same username.

The inference `docker build` has a fixed six-hour limit (`BUILD_TIMEOUT_SEC=21600`).
GNU `timeout` (or `gtimeout` on macOS) sends SIGTERM when that limit expires and
SIGKILL after a further 60 seconds if the build command has not exited. A timeout
stops the run before inference or Harbor starts, prints an error, and triggers
cleanup. The runner returns 124 for a normal timeout or 137 for forced killing.
This limit is independent of model startup and task timeouts in `task.toml`.

From the repository root, run the public tasks locally with:

```bash
./example_stage2/run.sh --username user_id
```

`--username` is mandatory and names Docker resources and the default output
directory. It is not an authentication credential or a field in the uploaded
YAML. Use 1-64 lowercase letters, digits, `.`, `_` or `-`, starting with a
letter or digit. The ODS evaluator supplies a name derived from the submission
ID automatically. An old `username` field in YAML is rejected.

To override paths or concurrency:

```bash
SWE_MERA_TASKS_DIR="$PWD/tasks" \
SWE_MERA_JOBS_DIR="$PWD/jobs/stage2-local" \
SWE_MERA_N_CONCURRENT=2 \
./example_stage2/run.sh --username user_id
```

Without overrides, tasks come from `<project>/tasks` and results go to
`<project>/jobs/stage2/<username>`. Private evaluation must supply its task
directory. The runner requires the project's `.venv/bin/python` and
`.venv/bin/harbor` and calls Harbor directly.

The inference API is published on `127.0.0.1:8000` by default. The organizer
exposes three GPUs and runs Harbor with 10 concurrent tasks by default. The
bundled `Qwen/Qwen3.5-9B` example serves `qwen35-9b-chat` using three data-parallel
replicas and `--max-num-seqs 512`. The agent timeout multiplier is 1; a task with
a 3600-second agent limit therefore gets 3600 seconds. The verifier timeout is
unchanged.

`custom_agent` is imported by the Harbor runner; only its tool commands execute
inside the no-internet task container. Production evaluation must therefore run
participant agents on an isolated worker without ODS credentials.


## Troubleshooting

Problems we hit ourselves while validating the baseline:

- **Agent setup times out (default 360 s).** Installing agent dependencies inside the container can be slow on cold caches or emulated CPUs. Pass `--agent-setup-timeout-multiplier 10`.
- **Image pulls fail intermittently during the run.** The registry (auth.docker.io) sometimes times out mid-build. Pre-pull the task images before starting:
 ```bash
 docker pull --platform linux/amd64 rewive/swe-mera-challenge:pallets-click-3208
 ```
 Image tags follow the pattern `rewive/swe-mera-challenge:<owner>-<repo>-<pr-number>` and are listed in each task's YAML file under `tasks/`.
- **Apple Silicon (arm64).** All task images are `linux/amd64`; use Docker Desktop or Colima with emulation. Everything (build, agent run, verifier) works under emulation, just slower — budget roughly 2–3× the wall-clock time.
- **Checking where a run went wrong.** Look at `<trial>/trial.log` (agent/verifier lifecycle), `<trial>/verifier/command_test.log` (test output), and `<trial>/verifier/reward.txt` (final score). `harbor view <jobs-dir>` gives a summary table.

## Rules

- Give each task exactly one attempt (`--n-attempts 1`).
- Submit only patches produced by the agent during the run.
- Do not use gold patches, evaluation tests, or test-specific workarounds.
- Retain the Harbor job directory and trajectories for verification.

## Support

If you want to ask any questions, there is an official support channel in Mattermost: [https://mm.ods.ai/ods/channels/swe-mera-challenge](https://mm.ods.ai/ods/channels/swe-mera-challenge). To access Mattermost use your ODS.ai account (choose authorize via ODS.ai on the Mattermost welcome page).

Fresh, uncontaminated real-world software engineering tasks across Go, Python, PHP, and more.

Public Stage2

Новости

Agreement

Я согласен(-на) с <a href="https://storage.yandexcloud.net/ds-ods/files/data/docs/odsaifooter/20250312_Service_agreement_ODSai.pdf">Соглашением об услугах</a>. Обработка персональных данных Участников Активностей производится на основании Соглашения об услугах.

Благодарим за интерес к соревнованию!

Заполните, пожалуйста, форму ниже.

Согласия, отмеченные&nbsp; (*) - обязательные.

#	Teams (3)	pass@1	Last	Entries
1	c8842bdaf637d	0.1698	10h	3
2	Pavel Adamenko	0.1509	27h	10
3	mb59c52789515	0.0000000001	10h	3