Bird Audio Detection Challenge – results

We designed the Bird Audio Detection challenge to help solve a widespread practical problem in automatic animal sound surveying. If you have acoustic monitoring data, and you’re looking for a system that will help you detect bird sounds in your sound recordings, you’ll find some tools and software out there, but often you’ll find the results aren’t as good as you’d hoped: too many missed-detections and false-positives, and it takes a lot of manual effort to tweak the settings usefully, or to postprocess the results. As we argue in our MLSP 2016 paper, the state of the art needs improving in robustness and generality.

So we were pleased to see 30 different submissions to the challenge, from teams all around the world. Would the best method come from a team of audio experts? Bird experts? Deep-learning experts? Students with a new idea?

The teams could send in multiple sets of results during the submission period, and see the “preview scores” live. Quite a few got a preview score of almost 90%, which is a strong score for this difficult challenge:

Graph of ongoing scores

We’re happy to announce the final results:

and to announce the prize winners:

(1) Highest scoring system (£500):
Thomas Grill (“bulbul”)
– This uses deep learning (a convolutional neural network), with signal-processing techniques (silence/noise trimming, data augmentation) to improve quality, and pseudo-labelling which helps the classifier to adapt to new conditions.
(2=) Judges’ award (€250):
Emre Cakir, Sharath Adavanne, Giambattista Parascandolo, Konstantinos Drossos, Tuomas Virtanen (“cakir”)
– This achieves a result close to the highest-scoring system, using deep learning, yet is much less computationally intensive. The strongest submission from this method did not use data augmentation, domain adaptation or ensembling (all of which would increase the computation needed). The method is based on a hybrid of convolutional and recurrent neural networks.
(2=) Judges’ award (€250):
Anshul Thakur, Jyothi Jain, Padmanabhan Rajan, A.D. Dileep (“IIT_Mandi”)
– The committee selected this method – one of the few that did not use deep learning – because it has a low computational complexity (it can run efficiently) and their GMM/SVM method was felt to be well-suited to the binary decision-making task in the challenge, with some modifications (feature warping, probabilistic sequence kernel) designed to encourage generalisable detection.

For further details, including descriptions/code for many systems, please see the final results table.

After the deadline for submissions had passed, there was useful discussion on the bird detection mailing list, with teams comparing their approaches against each other. There is source code available for many of the submissions. The methods will be discussed formally as part of the EUSIPCO 2017 special session on Bird Audio Signal Processing.

Our thanks to all the teams who joined our challenge.