AmuEval

A key aspect of building machine learning systems is comparing the performance of different methods designed to solve the same task. To achieve this, data is divided into training and test sets. The training data is used to develop new methods, while the test data is used to evaluate their performance.

The quality of a method is measured using a selected metric, which compares the method’s results with the expected outcomes. Depending on the task’s specifics, a metric is chosen that best aligns with the goal of the artificial intelligence system.

One effective way to select the best method is organizing an open competition, where participants develop solutions for a given task. The most popular platform for hosting such competitions is Kaggle, known for its flexibility in supporting various competition types. However, its downside is the time-consuming nature of setting up competitions.

The AmuEval platform was created to simplify competition organization. It allows organizers to:

* Provide training data,

* Select a metric,

* Upload expected answers for the test set (which remain hidden).

Participants submit their method’s results for the test set. The performance metrics of all solutions can be compared on a leaderboard, which updates after each submission.

To learn more about AMUEval, visit the repository.