A new study explored if music generated by artificial intelligence can be as emotionally impactful as human-composed music when paired with videos. Researchers found that while artificial intelligence music could evoke similar emotional feelings in viewers, it also prompted different physiological responses linked to attention and arousal. The research was published in the journal PLOS One.
The rapid advancement of generative artificial intelligence has prompted discussions about its role in creative fields like music and visual arts. While these tools are increasingly used in media production, the general public often expresses skepticism, viewing creativity as a uniquely human quality.
In the area of music generation, Nikolaj Fišer and his colleagues noted that the ability of artificial intelligence to convey emotion has been largely overlooked in both its development and in academic research. They designed a study to investigate whether music created by new artificial intelligence tools could produce the same emotional effect as human-created music within an audiovisual setting.
To conduct their investigation, the researchers recruited 88 participants and randomly assigned them to one of three groups. Each group watched the same 14 short video clips, which were gathered from an online platform and had their original audio removed. The videos were diverse in content, showing everything from nature documentaries and suspenseful scenes to abstract animations and advertisements. The key difference between the groups was the type of musical soundtrack that accompanied the videos.
The first group listened to human-created music. These soundtracks were selected from a database of emotional movie scores composed by people. To ensure the music was a good fit for each video, the researchers first conducted a preliminary test with a small group of people who watched the silent videos and rated their emotional content. Using these ratings, an algorithm identified the best-matching human-composed track from the database for each video.
The second group heard music generated by an artificial intelligence system using sophisticated and detailed prompts. These prompts were based on keywords that people in the preliminary test used to describe the videos, such as “horror,” “suspenseful,” or “romantic.” The researchers used an artificial intelligence music generator called Stable Audio to create these tracks, instructing it with detailed descriptions of genre, mood, and instruments to match the video’s content.
The third group listened to music also created by the same artificial intelligence system, but this time using simpler and less detailed prompts. Instead of descriptive keywords, these prompts were based on the numerical emotional ratings from the preliminary test, such as the level of positivity or negativity (valence) and the level of excitement or calmness (arousal). This method provided the artificial intelligence with emotional data but less specific creative direction.
During the experiment, participants sat in a laboratory wearing headphones while their biological signals were monitored. An eye-tracking camera measured pupil dilation and blink rate, and a separate device recorded their galvanic skin response, which relates to skin sweat levels. After viewing each video with its accompanying soundtrack, participants answered questions about their emotional state. They rated how positive or negative they felt, how aroused or calm they felt, how well the music fit the video, and how familiar the music sounded.
The results from the biological measurements revealed several differences between the conditions. When participants listened to either of the artificial intelligence-generated soundtracks, their pupils dilated more widely compared to when they listened to human-created music. This suggests that the music from the artificial intelligence may have required more mental effort to process or was perceived as more arousing.
The participants’ blink rates also showed differences. The group that heard artificial intelligence music generated from detailed keyword prompts had a higher blink rate than the group that heard artificial intelligence music from simpler emotional prompts. Blink rate can be an indicator of cognitive load or attention. The galvanic skin response, a measure of arousal, also showed a distinction. The music from detailed artificial intelligence prompts was associated with a higher skin impedance level, a state linked to lower arousal, compared to both the human-created music and the simpler artificial intelligence music.
When analyzing the participants’ self-reported feelings, the researchers found that the emotional valence, or the positive or negative quality of the emotion felt, was consistent across all three groups. This indicates that the source of the music did not change the fundamental emotional character of the experience. However, participants reported feeling significantly more aroused by both types of artificial intelligence-generated music than by the human-created scores.
The study also examined how well the music fit the visuals. Participants rated the music created by artificial intelligence from detailed keyword prompts as being the most congruent with the videos. In contrast, when asked about familiarity, participants found the human-created music to be significantly more familiar than either of the artificial intelligence-generated soundtracks. This may be because human composers often follow established musical conventions in film scoring that listeners recognize, while the artificial intelligence-generated tracks may have sounded more novel or unconventional.
The study has some limitations that the researchers acknowledge. The preliminary test used to generate the musical prompts involved a small sample of only ten people, which may not represent a wider audience. The experiment also relied on a single artificial intelligence music generator, and since the technology is evolving quickly, the results might not apply to newer or different systems. Because each participant was exposed to only one type of music, the study could not directly compare individual preferences between human and artificial intelligence compositions.
For future research, the scientists suggest exploring these questions with more advanced methods, such as using electroencephalography to get a more detailed picture of brain activity. They also propose comparing the responses of professional musicians with those of non-musicians. A next step in their work involves collaborating with professional composers to create original human-made soundtracks for a more direct comparison against music generated by artificial intelligence, rather than relying on a database of existing film scores.
The study, “Emotional impact of AI-generated vs. human-composed music in audiovisual media: A biometric and self-report study,” was authored by Nikolaj Fišer, Miguel Ángel Martín-Pascual, and Celia Andreu-Sánchez.
