Advanced Tutorial

Here is an advanced tutorial. If you are new to CodaBench, please refer to get started tutorial first. In this article, you'll learn how to use more advanced features and how to create benchmarks using either the editor or bundles. Before proceeding to our tutorial, make sure you have registered for an account on the Codabench website.

The image below is an overview of the benchmark creation process

Creating a Benchmark by Editor¶

In this chapter, I'll take you step by step through the Editor's approach to creating benchmark, including algorithm type and dataset type.

Step 1: Click on Management in the top right corner of Codabench's home page under Competitions.¶

When you click on it, you will see the screen as shown in the screenshot below.

Step 2: Click on the Create button in the top right corner of Competition Management.¶

Step 3: Fill in the Details tab content.¶

Step 4: Fill in the Participant Tab.¶

Step 5: Fill in the Pages Tab.¶

Step 6: Fill in the Phases Tab.¶

When you click on the Manage Tasks/Datasets button, you will see the screenshot shown below

Click the Add Dataset button in the diagram to upload the resource files needed to create the competition.

Creating a phase will require a bundle of the following types (.zip format), I'll give you more details on how to write these bundle later.

Now you just need to have this concept in your mind

Here are the screenshots of the 5 types of bundles after they were uploaded

Step 7: Fill in the Leaderboard Tab.¶

Step 8: Save and Publish the Benchmark¶

Creating a Benchmark by Bundle¶

Creating a benchmark through bundles is a much more efficient way than using editors.

Simple Version Example: CLASSIFY WHEAT SEEDS¶

Step 1: Download bundle¶

https://github.com/codalab/competition-examples/blob/master/codabench/wheat_seeds/code_submission_bundle.zip

Click on the link above to download the bundle in the screenshot.

Step 2: Go to the benchmark upload page¶

Step 3: Upload the bundle¶

After the bundle has been uploaded, you will see the screenshot shown below.

Step 4: View your new benchmark¶

Benchmark Examples¶

Example bundles for code & dataset competition can be found here:

https://github.com/codalab/competition-examples/tree/master/codabench

Iris¶

Iris Codabench Bundle is a simple benchmark involving two phases, code submission and results submission.

AutoWSL¶

Two versions of the Automated Weakly Supervised Learning Benchmark: - Code submission benchmark - Dataset submission benchmark

Mini-AutoML¶

Mini-AutoML Bundle is a benchmark template for Codabench, featuring code submission to multiple datasets (tasks).

How do I set up submission comments for multiple submissions?¶

Steps¶

Step 1: Click the edit button¶

Step 2: Enable multiple submissions on leaderboard¶

Step 3: Set up submission comment¶

Step 4: Save all changes¶

Step 5: Leave a comment before making submission¶

Step 6: Check out the leaderboard¶