You may have seen some discussion about Spectrum on our social channels. We thought we’d take some time to chronicle its developments here and share it with you. The following series of posts will chronicle the developments that get us up to date.
The Spectrum project was started by Mike Legary in an effort to produce real time music videos. The mechanism driving the visuals uses a series of modelled analog circuits that perform real time analysis of audio input which mixes stock video and effects to produce our live visualization.
We joined up with Mike in the fall of 2022. At that time Spectrum was producing visuals based on audio input but the availability of visual source material to mix was limited. Additionally the visuals produced lacked a sense of cohesion or compelling narrative. It was our task to provide feedback, perspective, and insight to work with Mike to produce visuals that are engaging and compelling. Ideally, the visuals would tell a story that is a visual representation of the music.
It should be noted that at this time, Spectrum was using piano octaves to split the spectral activity of audio input to map to visual representations. The activity of a spectrum slice would load specific videos from a bank of streamed visual source material.
One of the key issues outlined in the earliest meetings was the lack of source material. A proposed solution, which will be discussed later, was to generate imagery and combine still images to produce the visual components with machine learning tools.
Beyond that, the lack of narrative was apparent. The videos featured a theme which had good continuity—all of the videos being mixed shared a setting/location—but it was devoid of meaning. And so we went spelunking to uncover why this might be. After all, the system behind the visuals was intelligently designed and effectively streaming 4k video, but something just didn’t match up.
Taking a step back from the problem we first thought about what happens when we listen to music. What information are we taking in that we are then pairing with the visuals. A piece of music is a conversation. There are multiple instruments with their own voices, conversing. Orchestration places the spotlight on individual voices, and our attention shifts to follow interesting timbres, melodies, or rhythms. Curiously though, we the listener often only pay attention to one aspect at a time. Our attention may shift but it tends to be singularly focused. This was a key moment when we realized that we were flattening the dynamic conversation of music producing monotonous visuals. The solution was to change our mental model to think in terms of narrative and dialogue.
The spectral slices (piano octaves) used in spectrum were grouped into what we called Actors. We assigned the lowest two octaves to the Environment Actor, because the bass frequencies set the foundation for melodies to sit over as harmony or dissonance. The next two sets of two octaves became our two main Character Actors. The two octave above those became our Conflict Actors, as they’re the most volatile and highest energy.
Mike engineered prompts to populate a matrix of folders with imagery which Spectrum could draw from. As the song progresses it would shift across three Acts. The tone and level of an actor select which folder to draw images from for a given Actor.
The first half of the video showcases the diagnostic views of Spectrum. The second half showcases the alien landing club theme. We were getting close to something but there were still a lot of aspects that didn’t feel right. The imagery wasn’t compelling. Generated “photos” needed to be switched frequently to hide their poor quality. The quick shifting inhibited the sense of continuity that is required to setup the gestalt relationships that can produce narrative.
As an experiment we decided to throw out colour and detail in favour of shape and shared colours to see how we could increase continuity and resolve some image quality issues.
We were getting somewhere. More to come. 🙂
