ORU Logo Human meets AI in music CREA Logo
A project of the CREA network

[ CREA Home ]

This project develops concrete case study in collaborative human-machine music performance. This is a special case of the more general problem of anticipation and creative processes in mixed human-robot, or anthrobotic, systems.


The concept explored in this project is illustrated on the right. A human musician plays freely, and an AI system controls the parameters of a robotic performer. At the same time, the human musician hears what the robot does and may adjust to it.

We use the term "robot" here in a broad sense to mean any agent that generates physical actions: this could be a dancing robot, a virtual drummer, or a sound agent that spatializes music in the concert hall. In our first experiments, reported below, the musician is a jazz pianist and the robot performer is a commercial virtual drummer (Strike2).


Short technical paper (SAIS 2019)

Title: "Anticipation in collaborative music performance using fuzzy systems: a case study"

Authors: Oscar Thörn, Peter Fogel, Peter Knudsen, Luis de Miranda, Alessandro Saffiotti

Abstract: In order to collaborate and co-create with humans, an AI system must be capable of both reactive and anticipatory behavior. We present a case study in jazz improvisation, in which a human pianist is accompained by a synthetic drummer controlled by an AI system.

Presented at: SAIS-2019: 31th workshop of the Swedish AI Society. Umeå, Sweden, June 18-19, 2019.

Full text: Link


First public concert (Örebro University, May 28, 2019)

Synopsis: A hybrid duo of a human pianist and an AI drummer perform jazz music live. This was the first public concert that demonstrated our concept of Human-AI harmonious collaboration. It was given in the occasion of a visit of more than 50 ambassadors to Örebro University, Sweden, on May 28, 2019. The video includes an introduction and comments by the authors.

Artists: Peter Knudsen (human pianist), Oscar Thörn (AI programmer), Alessandro Saffiotti (AI expert), Peter Fogel (musicologist)


Second public concert (Örebro University, June 12, 2019)

Synopsis: A duo of a human pianist and an AI drummer perform jazz tunes. This performance was part of the "public evening on AI" organized in the framework of the First International CREA Symposium on "Anticipation and Anticipatory Systems: Humans Meet Artificial Intelligence". The experiment was aimed at showing how humans and AI can understand each other and collaborate in a harmonious way. It was performed on June 12, 2019, and was attended by more than 100 people.

Introduction: Luis de Miranda

Artists: Peter Knudsen (human pianist), Oscar Thörn (AI programmer), Alessandro Saffiotti (AI expert), Peter Fogel (musicologist)


A first foray into dance (Music Tech Fest 2019)

Synopsis: In October 2019, our CREA team participated in the Music Tech Fest (MTF) hosted by Örebro University. We joined forces with the "Accents in Motion" team, who had previously researched the use of body movements to control sound. We modified our system to take input from a Vicon traking system instead of a piano, so that our AI drummer follows the movements of two dancers. A simple performance was recorded on October 19, 2019, and shown at the MTF closing night. The clip on the right is a short excerpt.

Artists (CREA): Oscar Thorn (AI programming), MD Furkanul Islam (ROS programming), Alessandro Saffiotti (supervision)

Artists (Accents in Motion): Andreas Bergsland (music technology), Joseph Wilk (music programmer), Kirsis Mustalahti (dancer), Lilian Jap (dancer)


Robot dance at the "Feast Concert" (Örebro University, January 31, 2020)

Synopsis: A human pianist, a virtual drummer and a dancing robot, all coordinated by AI. This unusual team entertained the audience of the "Festkonsert", an official yearly celebration of Örebro University with the participation of about 250 people. The performance was meant to show the use of AI to orchestrate the collaboration among human and artificial agents in a creative process. The music performed by the pianist was improvised, and an AI system decided the moods, patterns and motions to be performed by the virtual drummer and by the dancing robot.
The clip on the right is a recording of the public event. You can also watch an excerpt from one of the laboratory tests here.

Artists: Peter Knudsen (human pianist), Pepper (the robot), Oscar Thörn (AI programmer), Alessandro Saffiotti (AI expert)


The CREA music team

Left to right in the photo:

Peter Fogel (musicologist)
Alessandro Saffiotti (AI expert)
Peter Knudsen (human pianist)
Oscar Thörn (AI programmer)
Luis de Miranda (philosopher)


[ CREA Home ]

This project was supported by Örebro University through its 20-year jubileum program, and by AI4EU, the EC H2020 landmark project to develop a European AI platform and ecosystem.

CREA is part of Örebro University's larger effort to promote multidisciplinary research in AI.

Website hosted by AASS
Last updated on February 7, 2020 by A. Saffiotti