Call for PhD applications at the Machine Listening Lab

The Machine Listening Lab is welcoming PhD applications for September 2022 entry. Applicants from all nationalities can apply across different funding schemes and PhD programmes. Current PhD funding opportunities for September 2022 entry include:

Applicants are encouraged to contact prospective supervisors before submitting their application – please send an email to your selected supervisors with your CV and draft research proposal.

Suggested PhD topics offered by Machine Listening Lab academics include:

Computational analysis of chick vocalisations: from categorisation to live feedback
(for PhD in Computer Science funded by a QMUL Principal’s studentship)

Supervisors: Dr. Emmanouil Benetos and Dr. Elisabetta Versace

The assessment of animals’ emotional state and welfare is a central issue for behavioural neuroscience and ethical farming. Animal vocalisations provide a rich set of information on the inner state of animals and can be used to influence animal behaviour in industrial settings, such as chicken farms. However, there is limited research on using vocalisations for monitoring animal welfare, and on the use of audio technologies for automatic welfare assessment of poultry in industrial settings.

In this project, we will develop computational methods to automatically categorise vocalisations of domestic chickens, infer their emotional state, provide live feedback and identify stimuli that can improve animal welfare. The PhD project will build upon pilot work led by the supervisors [i] on automatic recognition of chick calls using machine learning and signal processing methods. This project will lead to computationally efficient and robust machine learning methods and systems for automatically monitoring poultry welfare from audio, as well as will investigate research questions related to poultry development, behaviour, and well-being in industrial settings. Prospective candidates should be curious, self-motivated and have experience in one or more of the following: Bioacoustics, Cognitive Science, Artificial Intelligence/Machine Learning, Digital Signal Processing.

[i] C. Wang, E. Benetos, S. Wang, and E. Versace, “Joint Scattering for Automatic Chick Call Recognition”, 2022 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), submitted.

Discourse structure recognition for broadcast summarisation
(for PhD in Computer Science funded by a China Scholarship Council studentship)

Supervisors: Prof. Matthew Purver, Dr Huy Phan

This project will investigate new methods for inferring and detecting dialogue and narrative structure in broadcast data, primarily TV programmes, to enable more effective and meaningful semantic search and linking, and/or generation of concise meaningful summaries. Current methods for search and summarization are based around models of word and phrase meaning: these are effective when the key information is expressed though verbal content, and our recent work has shown that e.g. news broadcast segmentation can be improved by incorporating recent advances in NLP methods (Ghinassi, DataTV 2021). However, in many domains this is not the case: political debates need to be understood as networks of linked but contrasting opinions; dramas as often non-linear plotlines with key events and changes. Effective models of these formats must therefore include information about these elements of narrative/discourse structure, and learn to detect these in real datasets.

Methods for recognising and inferring narrative structure are available from related work on text understanding and summarization (including work from our lab, Droog-Hayes et al, ICSC 2019). This project will extend them to be suitable for spoken broadcast data and to use state-of-the-art neural network NLP methods, following our recent work in topic segmentation (Ghinassi, 2021) and related recent work in text segmentation (Angelov, 2020; Lukasik et al., 2021), while integrating information from multiple modalities including text, audio and video, again following our recent advances in multimodal modelling (Ghinassi et al., in prep.; Rohanian et al., Interspeech 2020). It will then build on this to develop models for jointly learning to generate compressed summaries while detecting key events and points in the discourse (discussion points, key characters, plot changes etc.), by adapting recent neural methods for dialogue act detection and summarization (Goo & Chen, 2018; Goo et al., 2018), and combining with suitable graph structures

Multitask modelling for overlapping sound sources
(for PhD in Artificial Intelligence and Music)

Supervisor: Dr Huy Phan

Overlapping sound sources are the main error source in a modelling system. Examples are polyphonic audio events in audio event detection and polyphonic music in multi-instrument transcription. In deep-learning context, the most common approach to deal with event overlaps is to treat the modelling task as a multi-label classification problem. By doing this, we inherently consider multiple one-vs.-rest classification problems, which are jointly solved by a single (i.e. shared) network. This project investigates to frame the task as a multi-class classification problem by considering each possible label combination as one class. To circumvent the large number of arising classes due to combinatorial explosion, decomposition of the label space will be explored to form multiple groups of category labels and yield a multi-task problem in a divide-and-conquer fashion, where each of the tasks is a multi-class classification problem. Network architectures will be then devised for multi-task modelling. The approach will be validated on databases with high overlapping degree of sound sources for polyphonic audio event detection and polyphonic music transcription.

Personalized scientific-grading sleep monitoring in home environments
(for PhD in Computer Science funded by a China Scholarship Council studentship)

Supervisor: Dr Huy Phan

Good sleep is crucial in maintaining one’s mental and physical health while sleep disorders are linked with plethora of different ailments, such as cardiovascular diseases, dementia, and depression to name a few. Accurate and cost-effective monitoring of sleep not only has great medical value but also allows individuals to self-assess and self-manage their sleep. Its importance gives rise to increasing demand in bringing sleep monitoring from labs to home environments for longitudinal monitoring. Existing commercial devices has limited value for scientific purposes, such as sleep disorders assessment or studying sleep under conditions like dementia. Novel wearable in-ear EEG devices, whose sensors are fitted neatly inside an ear canal to measure brain activities, hold a great potential for home-based use. Furthermore, they have been shown to be comparable to the lab-based polysomnography for sleep scoring.

However, there are two data-related challenges with these devices. First, manually labelling a large amount of their data is difficult and expensive as it requires to have polysomnography data recorded in parallel. Second, it exhibits strong “trait-like characteristics” specific to individuals and significant night-to-night variation in an individual’s sleep. This project aims to develop personalized deep learning methods to overcome these challenges. Domain adaptation methods will be explored to transfer knowledge from a large polysomnography database to ear-EEG. Regularization techniques will be developed given a potentially small amount of labelled ear-EEG data. Semi-supervised/unsupervised domain adaptation will be also explored to leverage unlabelled data. Furthermore, we will also investigate continual learning methods to keep up a personalized model with the possible “concept drift” caused by the long-term changes in a person’s sleep patterns. While these methods are required to learn from sequential, and potentially small, data, they also need to overcome catastrophic forgetting.

Resource-efficient models for music understanding
(for PhD in Artificial Intelligence and Music)

Supervisors: Dr. Emmanouil Benetos and Prof. Phillip Stanley-Marbell

State-of-the-art models for music understanding and music information research are often very hard to run on small and embedded devices such as mobile phones, single-board computers, and other microprocessors. At the same time, the computational cost, footprint, and environmental impact for building and deploying deep learning models for music understanding is constantly increasing. This PhD project will investigate methods for creating resource-efficient models for music understanding, applied to various tasks in music information research that involve music audio data, such as automatic music transcription, audio fingerprinting, or music tagging. Methods to be investigated can include but are not limited to sparse training, network pruning, binary neural networks, post-training inference, and knowledge distillation.

The successful candidate will investigate, propose and develop novel machine learning methods and software tools for resource-efficient music understanding, and will apply them to address tasks of their choice within the wider field of music information research. This will result in models that can be deployed on small or embedded devices, or on offline models where learning and inference times and computational resources are drastically reduced.

Self-supervision in machine listening
(for PhD in Artificial Intelligence and Music)

Supervisor: Dr. Emmanouil Benetos
in collaboration with Bytedance

Self-supervised learning methods aim to provide an alternative to supervised representation learning, eliminating the need for large annotated datasets. Self-supervision has advanced rapidly in recent years with applications across several modalities, and can be ideally used in machine listening and music understanding tasks which have been historically data-deprived compared to other domains. This PhD project will investigate methods for self-supervised learning applied to various tasks in applied to various tasks in music information research that involve music audio data, such as automatic music transcription, audio fingerprinting, or music tagging. Methods to be investigated can include but are not limited to contrastive self-supervised learning, formulation of appropriate pretext tasks, transferability to downstream tasks, and links between self-supervised and semi-supervised learning for music understanding.

The successful candidate will investigate, propose and develop novel self-supervised representation learning methods and software tools for music understanding, and will apply them to address tasks of their choice within the wider field of music information research. This will result in models that can learn from unlabelled data while performing comparably or surpassing supervised learning methods.

Sound source separation and localisation
(for PhD in Computer Science funded by a China Scholarship Council studentship)

Supervisors: Dr. Emmanouil Benetos and Prof. Mark Sandler

Audio and music source separation has been an active area of research, with applications in sound recording & production, broadcasting, and audio consumption. In recent years, deep learning techniques have dominated the topic, along with the use of priors for informed source separation. While the research community has mostly focused on single-channel source separation, real-world, professional studio applications make use of several microphones which are used in consort and their outputs are combined (or mixed) to create a composite signal.

This PhD project will investigate and propose machine learning methods for sound source separation and localisation (SSSL), which will separate a mixture recording into its constituent sources, in the context of a studio setup involving multiple microphones including spot and main feeds, while at the same time using information on the spatial location of sound sources to improve separation performance. The proposed research will lead to a new paradigm for informed sound source separation, where prior information will be provided both by spot mics related to constituent sources, as well as by providing or automatically inferring the spatial location of sound sources in the scene. The proposed project will draw knowledge from and will contribute knowledge to the fields of machine learning, digital signal processing, and acoustics, and will advance the broader fields of audio & music technology and signal separation. Upon completion, the project will provide new high fidelity and computationally efficient algorithms and models for separating and enhancing sources in studio practices.

Using Signal-informed Source Separation (SISS) principals to improve instrument separation from legacy recordings
(for PhD in Artificial Intelligence and Music)

Supervisors: Prof. Mark Sandler and Dr. Emmanouil Benetos

The recently proposed Signal-Informed Source Separation (SISS) paradigm from c4dm belongs to the broader category of Informed Source Separation (ISS), with the unique and specific attribute of using one audio signal to inform the separation of another. The informing source is a close approximation of a coherent component in the mixture. A current AIM PhD is examining this paradigm for live ensemble recordings when the spot mic signal informs the separation of the main mix. An alternative viewpoint is that the informing signal as a caricature of its corresponding component in the main feed. This suggests that we should investigate other caricatures for musical instrument separation and modification, especially for re-mixing and up-mixing of legacy commercial recordings. For example, a session musician could play the guitar line in the Beatles’ “She Loves You” to separate Harrison’s part, or it could be rendered from a MIDI transcription. Preliminary, confirmatory evidence for this approach appears in [1 & 2] which explore crude implementations with good outcomes but do not develop the approach further.

This PhD will develop skills in deep learning, especially architectures employing conditioning, as well as novel cost functions, perhaps incorporating physical models of the instruments to be separated. Applicants would benefit from a background in Machine Learning and DSP, coupled with knowledge of modern music recording, processing and mixing techniques.

[1] P. Smaragdis & G. J. Mysore, ‘Separation by “humming”: User-guided sound extraction from monophonic mixtures’, in IEEE WASPAA, 2009.
[2] Y. Li et al, ‘Learning to Denoise Historical Music’, in ISMIR, 2020.

Posted in Uncategorized

CfP: EURASIP JASMP special issue on Recent Advances in Computational Sound Scene Analysis

EURASIP Journal on Audio, Speech and Music Processing

Special issue on Recent Advances in Computational Sound Scene Analysis
Deadline: 1st April 2022

Topics of interest include but are not limited to:

  • Methodology: signal processing, machine learning, auditory perception, taxonomies, and ontologies related to sound scenes and events
  • Tasks and applications: acoustic scene classification, sound event detection and localization, sound source separation, audio tagging, audio captioning, detection of rare sound events, anomaly audio event detection, computational bioacoustic scene analysis, urban soundscape analysis, and cross-modal analysis (e.g. audio recognition/analysis with information from video, texts, image, language, etc.)
  • Machine learning methodologies for sound scene analysis: self-supervised learning, few-shot learning, meta-learning, generative models, explainable machine learning, continual learning, curriculum learning, active learning, multi-task learning, and attention mechanisms
  • Human-centered sound scene analysis: human-computer interaction and interfaces, user-centered evaluation, visualization of audio events and scenes, and user annotation
  • Evaluation, datasets, software tools, and reproducibility in computational sound scene and event analysis
  • Ethics and policy: legal and societal aspects of computational sound scene analysis; ethical and privacy issues related to designing, implementing and deploying sound scene analysis systems; privacy-preserving sound scene analysis; federated learning for sound scene analysis
  • Performance metrics: studies for developing effective evaluation metrics and tools for related tasks in audio scene analysis, event detection, and audio tagging

The EURASIP Journal on Audio, Speech, and Music Processing recognizes novel contributions of the following types within its area:

  • Empirical Research: Data-driven research, new experimental results, and new data sets
  • Methodology: New theory and methods for the processing of speech, audio, and music signals
  • Software: New software implementations and toolboxes for speech, audio, and music processing
  • Review: Timely and comprehensive overview and tutorial material covering recent developments within the field

Submission instructions:

Guest Editors:
Jakob Abeßer, Fraunhofer IDMT, Germany
Emmanouil Benetos, Queen Mary University of London, UK
Annamaria Mesaros, Tampere University, Finland
Wenwu Wang, University of Surrey, UK

Posted in Uncategorized

MLLab at Interspeech 2021

The Machine Listening Lab will be participating to the 22nd Annual Conference of the International Speech Communication Association (INTERSPEECH 2021), taking place on 30 August – 3 September both online and in Brno, Czech Republic. The following papers will be presented by MLLab members:

Posted in Uncategorized

MLLab students and staff to join the Alan Turing Institute

Two MLLab PhD students and three MLLab academics will join the Alan Turing Institute, the UK’s national institute in artificial intelligence and data science, in Autumn 2021.

The following MLLab PhD students will join the Turing as Enrichment students in 2021/22:

  • Lele Liu – Enrichment project: Cross-domain automatic music audio-to-score transcription
  • Ilaria Manco – Enrichment project: Multimodal deep learning for music information retrieval

The Turing’s Enrichment scheme offers students enrolled on a doctoral programme at a UK university an opportunity to boost their research project with a placement at the Turing for up to 12 months.

The following MLLab academics have been appointed Turing Fellows in 2021/22:

Turing Fellows are scholars with proven research excellence in data science, artificial intelligence or a related field whose research would be significantly enhanced through active involvement with the Turing network of universities and partners.

Posted in Uncategorized

MLLab at IJCNN 2021

On 18-22 July 2021, MLLab researchers will participate virtually at the IEEE International Joint Conference on Neural Networks (IJCNN 2021), the flagship conference of the IEEE Computational Intelligence Society and the International Neural Network Society.

The following papers authored/co-authored by MLLab members will be presented at IJCNN 2021:

  • MusCaps: Generating Captions for Music Audio
    Ilaria Manco, Emmanouil Benetos, Elio Quinton and Gyorgy Fazekas
  • Revisiting the Onsets and Frames Model with Additive Attention
    Kin Wai Cheuk, Yin-Jyun Luo, Emmanouil Benetos and Dorien Herremans
Posted in Uncategorized

MLLab papers at ICASSP 2021

On 6-11 June 2021, several MLLab researchers will participate virtually at the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2021). ICASSP is the flagship conference of the IEEE’s Signal Processing Society and is the leading conference in the field of signal processing.

As in previous years, the Machine Listening Lab will have a strong presence at the conference, both in terms of numbers and overall impact. The following papers authored/co-authored by MLLab members will be presented (all papers are publicly available until 14th June 2021):

See you all at ICASSP!

Posted in Uncategorized

New data challenge: “Few-shot Bioacoustic Event Detection” at DCASE 2021

We’re pleased to announce a new data challenge: “Few-shot Bioacoustic Event Detection“, a new task within the “DCASE 2021” data challenge event.

We challenge YOU to create a system to detect the calls of birds, hyenas, meerkats and more.

This is a “few shot” task, meaning we only ever have a small number of examples of the sound to be detected. This is a great challenge for machine-learning students and researchers: it is not yet solved, and it is great practical utility for scientists and conservationists monitoring animals in the wild.

We are able to launch this task thanks to a great collaboration of people who contributed data from their own projects. These newly-curated datasets are contributed from projects recorded in Germany, USA, Kenya and Poland.

The training and validation datasets are available now to download. You can use them to develop new recognition systems. In June, the test sets will be made available, and participants will submit the results from their systems for official scoring.

Much more information on the Few-shot Bioacoustic Event Detection DCASE 2021 page.

Posted in dcase

Call for PhD applications at the Machine Listening Lab

The Machine Listening Lab is welcoming PhD applications for September 2021 entry. Applicants from all nationalities can apply across different funding schemes and PhD programmes. Current PhD funding opportunities for September 2021 entry include:

Applicants are encouraged to contact prospective supervisors before submitting their application – please send an email to your selected supervisors with your CV and draft research proposal.

Suggested PhD topics offered by Machine Listening Lab academics as part of the AIM PhD Programme include:

Music interestingness in the brain

Supervisor: Dr Huy Phan
in collaboration with Aarhus University

Measuring interestingness of a song when one is listening to the song will not only shed some light on individual music perception, allowing personalized music recommendation, but also open possibility of using music songs as a brain stimulus. This project aims to automatically measure interestingness of a music songs in the brain using Ear-EEG.

An Ear-EEG device will be used to measure the brain signal (EEG) in the ear canals when one is listening to a song, which is then assessed by machine learning algorithms (potentially deep neural networks) to map the recorded EEG signal into an interestingness measure. Data collection will be carried out and a cohort of young and healthy subjects will be recruited for this purpose. This data will allow exploring different machine learning algorithms and techniques for interestingness modelling. Personalisation and multi-modal modelling, that combines music information (either raw signals or high-level musical features, e.g. melody, music genre, etc.) and the EEG, will also be investigated. This is a joint project with the Centre for Ear-EEG, Aarhus University, and the candidate is expected to work with academics in both C4DM and the Centre for Ear-EEG.

Meta-learning for music data

Supervisor: Dr Emmanouil Benetos

Meta-learning, or “learning to learn”, is an emerging area in the broader field of machine learning. Contrary to conventional machine learning approaches where a particular task is solved using a fixed learning algorithm, the main aim of meta-learning is to learn and improve the learning algorithm itself, so that it can absorb information from one task and generalise across unseen tasks. Meta-learning has various uses in machine learning applications, for example in cases where large datasets are unavailable or when we would like to rapidly learn something about a new task without training our model from scratch. It is also closely related to other emerging machine learning concepts, such as multi-task learning, transfer learning, few-shot learning, and self-supervised learning amongst others. While meta-learning has seen a dramatic rise in research interest in recent years, its principles have seen limited adoption in the intersection of music and AI research. PhD project will investigate methods for meta-learning applied to music data, such as audio recordings or music scores. The successful candidate will investigate, propose and develop novel machine learning methods and software tools for meta-learning, and will apply them to address tasks related to music and audio data analysis. This will result in methods that can rapidly learn from limited music data, or on methods that can learn from one task and generalise to other unseen tasks related to music and audio data analysis.

Suggested PhD topics for studentships in Computer Science or Electronic Engineering programmes include:

Scalable audio event detection and localisation for domestic acoustic monitoring

Supervisor: Dr Huy Phan

Audio event detection and localisation, which is a highly active research topic, entangles the “what” and “where” questions about occurring sound events. It would enable a wide range of novel applications, particularly domestic acoustic monitoring for healthcare. In this application, it is the case that the target acoustic environments are often different from house to house, causing reverberation mismatch particularly when a system is deployed in a totally new environment. This aspect remains uncharted in the current methods proposed for audio event detection and localisation, hindering scalable deployment and robustness of the system. This project aims to evaluate the robustness of the state-of-the-art methods and propose new machine-learning (potentially deep-learning) and inference methods to address this limitation. Furthermore, apart from being robust against environmental mismatch, such a scalable system should be self-adaptive to resources available of a target device (e.g. IoT devices and mobile devices), able to detect event of interest as early as possible.

Voice and language analysis for personality disorder detection

Supervisor: Dr Huy Phan

Bipolar Disorder (BD) and Borderline Personality Disorder (BPD) are two major mental health disorders that can seriously affect the life of patients. An early and correct diagnosis of these diseases is of paramount importance for an early intervention and treatment. This project aims to develop new methods to recognise these mental health disorders. We have interviewed and collected a voice database from a significant number of participants (both healthy and with disease status), the interviews were also transcripted. This database allows to explore different machine learning, particularly deep learning, methods to analyse the bimodal data (voice and language) for disease recognition. Another important aspect of this project is that we are not only interested in recognition but also in identifying acoustic and linguistic markers that are relevant to, and hopefully underpin, the diseases.

Multi-task learning for music information retrieval

Supervisor: Dr Emmanouil Benetos

Music signals and music representations incorporate and express several concepts: pitches, onsets/offsets, chords, beats, instrument identities, sound sources, and key to name but a few. In the field of music information retrieval, methods for automatically extracting information from audio focus only on isolated concepts and tasks, thus ignoring the interdependencies and connections between musical concepts. Recent advances in machine and deep learning have showed the potential of multi-task learning (MTL), where multiple learning tasks are solved at the same time, while exploiting commonalities and differences across tasks. This research project will investigate methods for multi-task learning for music information retrieval. The successful candidate will investigate, propose and develop novel machine learning methods and software tools for jointly estimating multiple musical concepts from complex audio signals. This will result in improved learning efficiency and prediction accuracy when compared to task-specific models, and will help gain a deeper understanding on the connections between musical concepts.

Sound recognition in everyday environments

Supervisor: Dr Emmanouil Benetos

The emerging field of sound scene analysis refers to the development of software systems for automatically recognising everyday sounds and the environment/context of a recording. Applications of sound scene analysis include smart homes, urban planning, audio-based security/surveillance, indexing of sound archives, and acoustic ecology. This project will focus on recognizing sounds from everyday environments. You will carry out research and develop computational methods suitable for detecting overlapping sound events from noisy and complex audio, recorded in urban environments. In this project you will be based in the Machine Listening Lab in the Centre for Digital Music, developing new methods and software tools based on signal processing and machine learning theory.

Posted in Uncategorized

Honorary Lectureship for Helen Bear

We’re pleased to announce that Helen Bear (Yogi), who has been working with us since 2018, has been awarded a QMUL Honorary Lectureship for 3 years.

Yogi says:

“I’m delighted to continue being a part of the team in C4DM. Complementary to my work in industry, at QMUL I am excited to continue my work in applied AI for visual and audio domains. Most recently I have been working in environmental sound scene analysis for multiple tasks, such as audio geotagging. But additionally I have been creating partnerships across QM including clinicians at the St Barts NHS trust to use AI to support healthcare and patients. “

To learn more about Yogi and her work, you can read this recent interview in Wonk Magazine.

Posted in People

Welcoming new Lecturer: Huy Phan

We’re pleased to welcome new Lecturer Huy Phan to our group!

Huy is a Lecturer in AI, and joined the C4DM in April this year. His interests are a great match to the Machine Listening Lab, and we look forward to working together (remotely and in person!). Huy says:

Photo: Huy Phan

“I am a Lecturer in AI at C4DM. Before joining QMUL, I was a postdoctoral research assistant at the University of Oxford and a lecturer at the University of Kent. I received PhD degree from the University of Lübeck, Germany. I am interested in applying machine learning to temporal signal analysis and processing (e.g. audio, EEG).

“At C4DM, I hope to join force with colleagues and students to make contribution to multi-view, multi-task, privacy-preserving, and non-iid generalisation perspectives of machine learning algorithms. I will focus on applications like audio event detection and localisation, audio scene classification, speech enhancement, and healthcare.”

Posted in People