mirdata: Software for Reproducible Usage of Datasets

Abstract

There are a number of efforts in the MIR community towards increased reproducibility, such as creating more open datasets, publishing code, and the use of common software libraries, e.g. for evaluation. However, when it comes to datasets, there is usually little guarantee that researchers are using the exact same data in the same way, which among other issues, makes comparisons of different methods on the “same” datasets problematic. In this paper, we first show how (often unknown) differences in datasets can lead to significantly different experimental results. We propose a solution to these problems in the form of an open source library, mirdata, which handles datasets in their current distribution modes, but controls for possible variability. In particular, it contains tools which: (1) validate if the user’s data (e.g. audio, annotations) is consistent with a canonical version of the dataset; (2) load annotations in a consistent manner; (3) download or give instructions for obtaining data; and (4) make it easy to perform track metadata-specific analysis.

Related

July 2020 | IJCAI - International Joint Conference on Artificial Intelligence

Seq-U-Net: A One-Dimensional Causal U-Net for Efficient Sequence Modelling

Daniel Stoller, Mi Tian, Sebastian Ewert, and Simon Dixon

July 2020 | WCCI/IJCNN - IEEE World Congress on Computational Intelligence / International Joint Conference on Neural Networks

Using a Neural Network Codec Approximation Loss to Improve Source Separation Performance in Limited Capacity Networks

Ishwarya Ananthabhotla, Sebastian Ewert, Joseph A. Paradiso