Using neural networks to create new music

I think creative sampling as we know it is going to change. Here is a sample I generated based on two Radiohead albums using a neural network. It is based on a Tensorflow implementation of the WaveNet algorithm described by Deepmind.

Deepmind is a Google-owned company focusing on artificial intelligence. They’re trying to create neural networks that are “intelligent”: can play video games, collaborate with clinicians, or solve how to use vastly less energy in data centers, for example. Last year, they published a paper about WaveNet, a deep neural network for speech and audio synthesis. While most neural network-related experiments in the field of sound and music are about the descriptors of sound or just data-sets describing audio (MIDI, for example), WaveNet actually looks at the tiniest grains of digital sound possible: samples.

screenshot from a gif in the paper “WaveNet: A Generative Model for Raw Audio”

As you probably know, digital audio consists of measurements. A CD contains 44100 data points per second, that makes up the waveform which is, when played back, converted back into a continuous signal by the DAC (Digital to Analog-converter), which will then be converted back into sound by your speakers that make the air vibrate. The WaveNet model actually uses these samples as input, trying to learn what will come after the current sample. It creates a possibility space of where one can . If it has learnt this, it can then start generating samples, based on the model the neural network has created (As I’m just starting out in the field of neural nets, this might be totally wrong, please feel free to correct me!).

screenshot from a gif in the paper “WaveNet: A Generative Model for Raw Audio”

The paper on WaveNet describes how it was originally created to do better Text-to-Speech. The researchers found, however, that when they didn’t tell the model what to say, it still generated sound. But without meaning. The samples in the paper were super interesting to me. Half a year later, the all-round musician Espen Sommer Eide wrote a (great!) article on “Deep Learning Dead Languages”. Some audio from Espen’s article:

Audio examples from Espen Sommer Eide’s article

Moving to a new studio, and finally having the time to set up computers with a proper video card to work these algorithms, I wondered how a quite diverse database of musical material would work as a training set, and decided to train a neural net on two Radiohead albums for about a few days last week. What do you think of the results? While I don’t think it’s particularily musical, it has a very organic quality that could be very useful for sampling. To make this into a musical piece, we might need yet another algorithm to structure the creations from this WaveNet algorithm in (macro)time.

Since the Deepmind paper became public, various people have made several implementations available on Github. This is the one that I used.

Splendour Lender

Jelle Mastenbroek is a Dutch artist interested in the “true meaning of objects”. With his design studio, he creates unusual and playful products reflecting on society, money and the human condition.

Quite a few of his works, such as the here pictured Splendour Lender, Money Back Guarantee 1 & 2 amongst others, take status and money as the concept.

How important is status in times of declining moral standards? In times where financial markets collapse, economies are shrinking and money tend to become a goal in itself? With this project I want to express the real nature of money and tell the true story of status; happiness. The splendour lender brings porcelain to life and shows the original function of money as a medium to exchange goods and services. By putting a euro coin you’ll experience a joyful moment. After usage the coin will be returned so the circle can go round.

I love the playfulness and delicacy of the work, and the fact that it is very transparent and can appeal to a big audience!

Sonify... your package's journey

There’s different ways of sonifying data, and it’s often hard to strike a balance between making it musical or “listenable”, and translating the data into sound in a correct manner. We’ve seen different ways of sonifying data before in this “Sonify…”-series. With FedEx’s SoundTrack, they very clearly went for music. Which seems logical considering SoundTrack is a marketing gimmick.

FedEx Soundtrack takes a tracking number, or asks you to create a fictional package, and then sonifies this. Everything from the route, delivery type, package’s weight and dimension will influence the composition. A heavier package will for example create a track with a more intense bass track. For a multinational courier company, it is quite a fun feature that will certainly get them some attention.

Max Motor Dreams

It is common knowledge that babies and small children fall asleep in the car quite easily. This could be a few things, the muffled engine noise, the slight vibration of the car, a regulated temperature. Furthermore it could be a conditioned response: when kids are put in a seat and strapped in, they can’t really move around much and are kind of forced to relax. They have probably slept in the car before so are conditioned to do it again.

Using this knowledge, Ford is promoting it’s new vehicle range with “Max Motor Dreams”, a baby crib that reproduces the sounds, movement and light of a parent’s car. Parents are even able to use an app to collect data from routes and replay it in the crib at home.

It seems super novel, and for a minute I thought it was an April Fools’ prank, but on second thought it is not such a crazy idea. We know that this combination of sound, light and movement works to put kids to sleep. So why not take it out of it’s original context and try to recreate it. It is an interesting experiment. What other environments could we recreate in another context to evoke a certain physiological response?

Vertical Studies at Sonic Acts

The theme for this year’s biannual Sonic Acts festival (23-26 February, Amsterdam) is “The Noise of Being”. The festival generally focuses on contemporary and historical developments within art, music, science and technology. Over the last decade, the festival’s focus has shifted more and more from the universe to planet earth. This edition, it hits a little closer to home, speculating on the strange and anxious state of being human.

One of the commissioned works this year is the collaboration between Espen Sommer Eide and Signe Lidén. The two have worked together in the past during Sonic Acts’ Dark Ecology programme, recreating the historic soundscapes of the Russian settlement Nikel in the work “Altitude and History”. Through working with the local communities, they brought back the sound of wood being chopped, the lighting of the morning stoves, the sounds of animals and birds in the forest, which has since been burnt away. Vertical Studies builds upon this work.

For Vertical Studies, they reimagined the beautiful water tower in the Dutch settlement Sint Jansklooster into a vertical field-lab, where Eide and Lidén are introducing their ongoing investigations into connections between sound, history, wind and weather. For this, they use several specially constructed instruments, created for the recording and playback of sounds in the vertical dimension. By recording sounds using instruments with strings resonating with certain wind speeds, they created an archive of sounds of wind at various heights.

By using this material in the 46m high water tower, they are able to let the audience experience the effect of sounds at different altitudes in different ways, creating a kind of concert hall for vertically recorded sound. It’s quite the trip from Amsterdam (a 1.5 hour bus drive), but the building alone is already worth it, and to hear the ongoing research of Eide and Lidén in this unique space is quite the experience. It can take quite some time to get used to the space- I kept slowly going up and down the stairs, trying to find the right position to be able to fully take in all of the interventions in the space, which is logical with a piece in a new context like this. The artists use various ways of making the space resonate beautifully: by using glass resonators and very physical low resonating tones, they create a very diegetic work consisting of their archive of sounds. While the piece is about using the sounds recorded at different heights, the piece is also very much about the beautiful architecture of the water tower itself.

“Vertical Studies” can be visited during the upcoming Sonic Acts festival, which is from the 23rd until the 26th of February in Amsterdam. The excursion leaves on the 23rd from Amsterdam, at 12PM and 02PM. The installation in the water tower can also be visited on 4, 11, 18 and 25 February. Have a look at the website for more information.

Chijikinkutsu

Since ancient times, Japanese people have been sensitive to perceive nature as it is, from the sound of the wind through pine trees or the singing of insects in each season. ‘Chijikinkutsu’ was developed with this kind of delicate sense. - Nelo Akamatsu

Nelo Akamatsu is a Japanese multidisciplinary artist. “Chijikinkutsu”, the title of his latest work, is a contraction of the words “Chijiki” and “Suikinkutsu”. The first meaning “geomagnetism” and the latter is the word for a sound installation ornament for traditional Japanese gardens, invented in the 16th century.

In “Chijikinkutsu”, sewing needles are floating on water in glass tumblers which are magnetized, so they are affected by geomagnetism and turn like a compass. When electricity is applied to the coil which is on the outside of the glass tumbler, it creates a temporary magnetic field drawing the needle to the coil. The needle hits the glass creating a very delicate sound.

“A round surface of water in the glass with a floating magnetized needle, reminds me somewhat of a tiny earth with geomagnetism. The smaller the sounds of the glasses will be, the more keenly viewers’ sensibility will be sharpened. In the meantime, they realize that the sounds don’t come from outside of their bodies, but already exist inside of their mind.” - Nelo Akamatsu

I love the minimalist approach of only using a few elements and letting the complexity emerge from having multiple instances. Also, where most sound art is often very “visual” in simply showing the elements that the installation experience exists of (the speakers, the wire, the sensors), this work does not show the regular elements but maybe an even more stripped version of it. A speaker is also a coil being pushed and pulled, it also works with electromagnetism. In that sense, Akamatsu’s work is a very, very rudimentary work of sound art.

Spectrogram: Mislata

Edu Comelles is a Valencia-based sound designer, musician and curator. His works often make you aware of certain sociocultural elements, or your everyday surroundings.

For his latest project, Spectrogram: Mislata, Edu recorded the city of Mislata for a whole month. The resulting archive consisting of local urban, rural, social and cultural soundscapes was analyzed and turned into a spectrogram: a visualization of the spectral information in an audio-file.

By doing this, he makes the intangible heritage of Mislata visible without being very intrusive. The murals combine various spectrograms from different recordings. The image is full of details, complexities and graphic nuances. The murals are composed of parts of spectrograms of sounds from birds, cars, sirens, buses, church bells and all kinds of hums, hisses and roars that Edu found in the city of Mislata. Those textures, put together, form a portrait of the city and it’s inhabitants.

Sound art often takes time to immerse yourself in. Pieces in public space, even if they do tell a story about the neighborhood they’re in, can be an annoyance to the locals. By visualizing the soundscape, Spectrogram: Mislata blends in with the surroundings. The viewer can wonder about the sounds visualized on the mural at their own pace, when they pass it every day.

Looking Forward: Sound during the Dutch Design Week

It’s already October, and that means the Dutch Design Week is almost upon us. During this nine-day event, the best of Dutch design is exhibited, and there are lots of workshops, seminars and parties to attend.

For a sound artist, the DDW used to be quite uninteresting. Last year, experience designers Arvid Jense and Marie Caye thought up Bizarre Sound Creatures: an exhibition space for sound art creations in the heart of the DDW.

This year, Bizarre Sound Creatures is back with another edition. Amongst the installations exhibited are the Dato Duo, the Pigstrument, but also other exciting interactive installations. See an impression of last year’s edition below:

Additionally, I am involved in organizing a small symposium on sound design for the everyday life, together with other institutions, which is being organized at the Bizarre Sound Creatures venue as well. We hope this will grow, and will become a larger symposium next year.

Next to Bizarre Sound Creatures, artist duo Strijbos & van Rijswijk is also curating a selection of interactive sound works that can be experienced at the DDW Music Lab. It’s safe to say that sound in design is becoming a hot topic. Now let’s just hope the boundaries between disciplines will blur, and different designers will work together towards a world where things not only look and feel better, but also sound more fitting.

Pigstrument

Marie Caye is a French design researcher, experience designer and artist who’s currently studying at the Design Academy Eindhoven in the Netherlands. For one of her latest projects, the Pigstrument, she’s researching what the possible musical culture of pigs could be.

To know what sort of music culture and instrument would fit the pig way of life, she spent a lot of time finding out what it is like to be a pig. She made a Pig Empathy Suit to be able to live between pigs for a day. From their farmer, Marie found out that the pigs seemed to enjoy spitting stones on the metal fences because of the sound it created. To get the most out of the instrument she worked together with experience designer Arvid Jense, who sound designed the tubular bells on the instrument.

The instrument Marie created seems a logical choice: it can tumble and fall over and will still function, is suited for multiple “players”, and doesn’t rely on anything electronic. On her blog she writes, early on in the project: “One big question is, how to make the instrument affordances suited to pig behavior? I have seen that they will push relentlessly any object. I was thinking of trying to prevent the object to be turned around by pigs when my teacher Wieteke pointed out, it’s better to use this attitude of the pig in the way of playing the instrument.” I think the fact Marie documented her process so well adds to the viability of the project.

I like Marie’s idea to actually try and live like you’re in the skin of another species, to experience what might be an ideal music instrument. Her research blog got me interested in music for cats, “virtual reality” free range chickens and more. Very interesting and mostly untapped design areas. We might never know what the pigs actually think of it, although looking at the footage, it seems like they really enjoy it.

Pietre Sonore

I’ve often wondered why there aren’t more sculptors making use of sound in their work. Even though Wikipedia has quite a list, it’s not something you run into very often. Also, most of the artists on the list have only made use of sound one, or only quite a few times in their career.

Pinuccio Sciola, a quite renowned Italian sculptor who sadly passed away two months ago, was one of those artists who used sound in his sculptures, and even performed them. Often called “the man who makes the rocks sing”, his most famous works are the Pietre Sonore, or the Sounding Stones: sculptures, often made of limestone, that he used to play with his hands or with small rocks.

His ideas on these sounding stones were quite spiritual: he believed that stones have a voice that has always been there, but he only helps to release it. His limestone sculptures for example make a sound of a “liquid quality” when played, because geologically seen limestone is just fossillized water. As if the memory of water was imprisoned inside the stone during glaciation. An interesting way of looking at the world which might be very abstract, but helps one to think about long timespans.

To the rational minds that are sceptical of his ideas, Sciola only had only one thing to say: “I want to remind those who are sceptical that all the information circulated today by computers and digital networks in the end is stored and elaborated using infinitesimal silicon crystals.”