<h3>Task description</h3>

<p>The goal of this benchmark is to provide an open and interactive interface for AutoML system evaluation on a wide range of tasks and datasets. We design our benchmark for both Academic datasets and real-world industrial datasets in order to get a better understanding of the current state of AutoML system performance. This benchmark is extensible and will have additional dataset groups with their updated versions coming with respect to the benchmark roadmap.</p>

<p>Benchmark solutions are end-to-end AutoML systems suited for both automatically building ML models on a given dataset as well as using their best-fitted model for inference on test data for the given dataset. Solutions are sent to the automatic testing system and evaluated on groups of datasets (see Dataset section).</p>

<h3>Evaluation procedure</h3>

<p>Solution evaluation consists of 3 phases:</p>

<ol>
	<li>Check. The solution is evaluated on a single small dataset. This step is required to check if the solution is correct. Check phase evaluation provides a detailed log of errors for participants.</li>
	<li>Live Test. Solutions are evaluated on a small yet representative subset of datasets from each dataset group (OpenML, Finance, ODS). These datasets are evaluated at any given time and provide live feedback on the leaderboard. Participants are provided with scores and time consumption and each of these datasets as well as their total score on this Live Test on the leaderboard.</li>
	<li>Large Test. Solutions are evaluated on a complete set of datasets from each group on a monthly basis. This evaluation is resource-intensive and carried out with a monthly schedule. Participants are provided with scores and time consumption for each dataset as well as their total score on each dataset group on the Large Test leaderboard. Detailed scores on each dataset of each participant are also provided for further analysis by the community after every run of the Large Test.<br />
	Important: Participants must choose up to 2 of their submissions eligible for evaluation on the Large Test (same mechanic as choosing your final submissions on regular competitions).</li>
</ol>

<h3>Benchmark Roadmap</h3>

<ul>
	<li>October-November: making participants familiar with the submission system, evaluation on the open group of datasets (OpenML CC 18). Crowdsourcing ODS datasets.</li>
	<li>November: providing participants with the Finance group of datasets. First run of the Large Test. End of datasets crowdsourcing in 2021.</li>
	<li>December: complete runs on all groups of datasets, integration of benchmark into open AutoML course.</li>
</ul>

<h3>Metrics and scores</h3>

<p>The complete scoring process of each AutoML solution consists of the following 3 steps:<br />
<br />
Step 1. For every dataset group on each dataset evaluate respective metric_value on test data predictions:</p>

<ul>
	<li>Binary classification: ROC-AUC</li>
	<li>Multiclass classification: ROC-AUC (one-vs-all)</li>
	<li>Regression: RMSE</li>
</ul>

<p>Step 2. For each metric value on each dataset calculate its relative dataset_score compared to the metric value of linear baseline:<br />
<code>dataset_score = metric_value / metric_baseline</code></p>

<p>Step 3. For each dataset group calculate its group_score as the average dataset_score within this group. <code>total_score</code> is the average <code>dataset_score</code> across all datasets in the current benchmark.</p>

<h3>Resource constraints</h3>

<ul>
	<li>12Gb memory</li>
	<li>4 vCPU</li>
	<li>50Mb solution archive size</li>
	<li>5 minutes for each dataset for Live Test and 1 hour runtime per dataset for Large Test.</li>
</ul>

<p>Official support channel: <a href="https://opendatascience.slack.com/archives/C02M54WP9EU"><strong>#automl_benchmark</strong></a> in ODS.ai slack. If you are not registered, please<a href="https://ods.ai/join-community" rel="noopener noreferrer" target="_blank"> join the community</a>.</p>

Open AutoML benchmark in the form of a container-based competition on both Academic (OpenML CC18) and industrial (Finance and ODS) datasets.

Public

Private

Check

Baseline

Dataset contribution

User Agreement

I accept <a target="_blank" rel="noopener noreferrer" href="https://storage.yandexcloud.net/datasouls-ods/CourseFest_1/Docs/UserAgreement.pdf">Terms of User Agreement and Privacy Policy</a>

Privacy Policy

I give my <a target="_blank" rel="noopener noreferrer" href="https://storage.yandexcloud.net/datasouls-ods/CourseFest_1/Docs/Consent.pdf">consent</a> to «Sorevnovaniya Analiza Dannykh» LLC to process and transfer my personal data to the partners of the event for information interaction

Consent

I <a target="_blank" rel="noopener noreferrer" href="https://storage.yandexcloud.net/datasouls-ods/CourseFest_1/Docs/Email.pdf">agree</a> to receive newsletters from «Sorevnovaniya Analiza Dannykh» LLC

#	Teams (3)	OpenML Benchmark	Last	Entries
1	OOC Narek Maloyan	6.5086548608	3m	37
LightAutoML time-utilizing		1.0881029257
AutoGluon benchmark mode		1.0867399491
H2O v. 3.34.0.3 default		1.081668093
LightAutoML fast run		1.080346944
2	Stebenkov Andrey	1.0789208913	8m	1
MLJar v 0.11.1 default		1.076993503
TPOT v. 0.11.7 default		1.0698691134
3	Nikolay Nikitin	-10.9641074168	2m	7