Minimum Viable Podcast #14: User Experience in Sound Design

The 14th episode of the Minimum Viable Podcast introduces a special guest – Johannes Helberger, a sound artist from Berlin who used his artistic experience to become one of few pioneer experts in the uncharted field of UX in sound design.

Johannes told our Managing director and the show host Michal Blažej about his work at the Sound Innovation Lab, about the past, presence and the future of sonic UX and they also touched on cultural differences in this field. They discussed that even the biggest players on the market haven't figured out yet how sound should be used for interaction design.

If the topic of UX in sound design grabbs your attention, Johannes recommends learning about the concept of Calm Technology. The book Sonic interaction design by Franinović and Serafin and the blogposts by Iain McGregor are also worth reading. Or watch Johannes's talk at IFA Berlin.

The Minimum Viable Podcast is brought to you by the CX design studio Lighting Beetle* and is usually recorded in the Slovak language. For more content in English, including case studies and handouts, visit our Journal.

You can find the Minimum Viable Podcast on:

I work in the UX and most of the interactions and experiences we build are about vision – what the user sees. Why should I even consider sound to be relevant in an interaction with a machine?

When you talk about experience, it implies that sound can help in certain situations or cases. We know that sound triggers emotions for example. When you design an experience, you maybe want people to get into a certain mood, so that people can enjoy it much more. Sound helps alter how people experience whatever it is that you design.

The strength of sound in experience is that it might help you to get into emotion, get you into a certain mood. When talking about the functional aspect of sound, it is in reference to machines or computerized objects that interact with us. We have more and more embedded systems without screens, we have to change the whole concept of interaction. You can think of voice assistance or also the rise of smart speakers which have a complete audio interface. Thus, we can observe a trend that some fields are shifting away from visual to audio experience. However, the visual part still remains a big part of design as we aim towards creating holistic experiences or a big buzzword these days – immersive experiences. Here the aim is to immerse the user, the visitor into the experience you design. With this aim in mind, you need to consider all senses. Or at least it would be beneficial for your design if you did so.

What are the real world applications of sound in human-computer interaction? You already mentioned some examples, but what are the general categories, where sound design is useful for building great user experience.

There were scientific studies and all kinds of indications that by using the sound in the right way, people feel safer or stay longer in some places, like shops for example. However, they also might leave sooner because of sound. It can also raise or decrease the perceived value of products in stores. There also was a nice study about wine tasting. It showed that it tasted better with the right music and it also caused people to buy more wine. There was also another study conducted in hospitals, which indicates that music can have a positive effect on our health and on the other hand, noise has a severe impact on our health in a bad way. It can help us to concentrate while working or can help to let go, when we meditate, and drive us when we run. It can also alter our moral evaluation and induction of empathy in films.

We also recently made a project with Volkswagen. It shows that it can create trust in autonomous vehicles for example. Frederik Robinson, a colleague of mine, did the same thing with social robots and he proved that people would trust robots more with the right sound. Some of the real life examples, which I find very interesting. On my way to give a talk about this particular topic, I was a bit late, therefore I took a taxi and I discussed the topic with the driver. You need to know that it was in Berlin – the driver sometimes picks up people under the influence of various substances from parties even at 10 in the morning. He told me that what he does is that he asks these clients if he can play classical music to them, so that they can sleep better when they get home. When he plays classical music, these people calm down. On the other hand, techno music makes these people annoying.

From what you are saying, I am thinking – don’t we miss the emotional part in webs and apps, which do not use the sounds? Have you ever thought about this?

Actually a lot. When the internet “came out” or became a more relevant part of our world in the ‘90s and the beginning of 2000s, many people started using music on their website. I was quite young and I made some cheesy musical ambient background loops for websites. But later, some culture developed in this aspect. And the culture was clear – I do not want music on websites. I might be listening to music myself on my computer right now. I might have 20 tabs opened, which would mean that I do not know where the annoying music comes from and I might have to close it all… And this is the last thing you want as a website owner or creator – that people close it as fast as possible, because they do not want to be disturbed.

I think it depends on each individual use case. I do not want my apps and my smartphone to make more sounds than I need. When something makes a lot of sounds, it kind of intrudes my private sphere, which is something I do not want. On the other hand, when I order pizza in an app on my phone, of course I am happy when a sound notifies me that my pizza will arrive in five minutes. That is the situation, when I am happy that I do not need to look at my display.

It depends on what you want the user to do. If you want your user to put headphones on and indulge in your app for education or enterntainment, sound can be valuable. If you offer service aside from other things, then you do not want sound, I guess.

"We can observe a trend that some fields are shifting away from visual to audio experience."

And how about this? I'm thinking now that when that pizza order is coming, the sound comes out of my mobile phone. It can sound very generic, but it can also carry emotions. Don't we overlook opportunities of the sounds that actually carry information that they could be also emotional? Do you think that could be improved?

Completely. I think sound in UX design is still beneath its potential. I'm really astounded at how many products, also the ones which are quite expensive, still make beep sounds which are super annoying and have nothing to do with the visual design of that object.

For example, with the pizza app case. You could create a sound which kind of implies that the pizza is ready. However, that does not have to be the end. You can also ask yourself: “What do we want this app to be? Do we want it to be a little friendly helper? Or do we offer bank services and our app needs to be super serious?” Thus, if we want our pizza app to be a little helping friend, we need to have a friendly sound, which in the best case also communicates all important information like that the pizza is ready.

Can you summarize the different categories of where good sound design is very needed today? I can imagine automotive, kitchen smart devices. What are the other categories where maybe we don't see it explicitly, but there are so many sound designs that we just overlook?

It is crucial in cars because it's a high workload situation. You are in the traffic. At least when it's not an autonomous vehicle, you get a lot of signals and you have more and more assistant and entertainment systems. So you need basically all means you can get to communicate with a user without disturbing him and giving him more trouble than he already has.

That's very interesting: I'm already computing with my vision, but my ears are free. So that actually helps because then I can combine all the different inputs in my brain and use that information to drive the car correctly…

A lot of user feedback is that sounds and UX sounds are made just like generic sounds and when you hear them, you have to look at some screen. But if you design sounds in a way that they intuitively make you understand what's happening, then you don't have to look at a screen. Another thing is with smart technology, which is actually not too smart. I was at the IFA this year. It's a kind of an electronic fair, so everything was smart there. However, we recorded UX sounds there and 90% were beep tones.

What I find an interesting case are these small vacuum cleaner robots. For many people, they become like a little pet in a way. They give them names, they like them as their friends.

Oh, don't tell me. I was cleaning him yesterday. We had a really nice time.

Yeah, exactly. And then if you imagine you give this thing a voice, like an R2-D2 kind of thing, this could even help you to have more positive experiences with it. Like what can it do when it runs out of battery or has some problem. If done right they could be cute or helpful. I think then you could create a product which people would love even more.

I loved your comparison when you said how much we invest into the product design visually, how beautiful it might be and then it beeps. That's very interesting.

I think in health technology, there's a lot of potential to do. Until now it has had only functional sounds, which are terrible. They create more stress and a negative atmosphere in hospital settings. I think there's a lot to do in using sounds which would help create a more welcoming atmosphere that is beneficial for your health.

Another thing is robotics and industry also for safety reasons. If you have workers working with autonomous systems, you need certain sounds. And for example, what I really liked was the sound design in Amsterdam at the airport. There were these little wagons transporting people or some stuff in the back. These wagons have to drive through masses of people. At many airports, they do these super annoying beep tones. And in Amsterdam they have a very warm noise, like a white noise kind of thing or brown noise. It is very low in volume, but you always perceive it and it doesn't disturb you. Even the impaired people would hear it because it's on a full range spectrum of sound. It does a very functional job in a very good way.

And then another topic which we don't have to jump into much is virtual reality, this whole metaverse thing when let's say we immerse again more into services than we have on a two dimensional screen sound, again, can be very beneficial in many ways.

Speaking of cars – I kind of enjoy that the cars are silent. It's fine. I don't need to hear the motor sound all the time because the environment is full of sounds.

Exactly. And that's actually one important thing when you design sound for anything – silence is still gold, right? Just if you use sound, use it in the right moment and the right way. If you don't need sound and if it doesn't add value to a moment or an experience or to service – just leave it away.

Just like whitespace is gold in visual design.

Some people really think about me: "Oh God, you're the guy who's going to make things have more sound in the future!" My answer is: "No." I'm going to try to make it in a way that it completely is embedded in your real life, and it just comes into your consciousness when it's needed, like when it warns you or notifies you with something.

What is the history of sound design in human machine interaction?

There is this evolution from passive to active, let's say. Actually, we can see the same thing also in experience design – from looking at things into experiencing things. When it comes to machines and technology, that's also like coming from passive sound intervention to active sound.

There's some material about it from the times of the Industrial Revolution. Apart from the stench, there was also a lot of noise. The first car was super noisy, industrial machines were super noisy. All these machines create a lot of noise and then at one point the manufacturers got into problems also in terms of politics. It triggered something which we could call user experience in sound design with the main goal being getting rid of the noise.

Rolls-Royce is one of the good examples. A car which was completely silent was perceived as luxury because it resembles a calm room. However, on the other hand, when engineers actually made completely noiseless cars, at least in their interiors, people recognized that actually the noise of a car or a machine, give you a lot of information of what's happening. If you sit in the back of a Royce and your driver is driving and you hear that he's accelerating or braking, scientific testing proves you would perceive more what the car would do, so you wouldn't be surprised if you speed up.

But that's just one thing going back to history. People tried to reduce and kill the noise machine smack and now we are at the stage when we add sound. I wouldn't say noise if it's done right. We add sound to give us information, create the right atmosphere, and have it as a means of communication, basically. I have one example here. We are working with a car manufacturer OEM on UX sounds in cars. Ten years ago they had 20 people working on reducing the noise of the car and creating a certain sound like the one of a door. They used to have one or two sound designers who did active sound processing and now they have two people left who do the physical sound and they have maybe 10 or 20 people who do active sound. But it's really a shift right now.

"You need to consider all senses. Or at least it would be beneficial for your design if you did so."

You run your own shop, Sound Innovation Lab. What kind of problems with sound design are you trying to solve? What is your mission? Why did you even get into this field?

I have designed sound experiences for culture, advertisement and movies for a long time. With my colleagues we even did some things which we could call crazy stuff like immersive installations with tracking and lots of speakers and this experience. However, now I see that we are at a point where the technology that surrounds us is so far that I can use my experience from what seemed like crazy stuff. Now it can be helpful in our everyday life.

We talked about cars. If the sound design in cars and outside of cars is done right, it could really help people and the nature and the city. As I also said, people could also enjoy hospitals thanks to the right use of sound.

We are working with manufacturers and companies and UX designers, and we also have the scientific part where we do scientific projects and studies. We're also working on the semantics of sound now. The basic question here is – what can you actually communicate with sound apart from what is done right now? Can we move from having generic audio logos played in a sample player towards more interaction and really having a very efficient way to communicate information and still have the right emotional aesthetics?

Everything is a sample player right now. I don't think in the future this should be the case. I think sounds will be generated in the moment, things will be adaptive to your situation, to your context. A good example - if you sit in your car, you have the navigation on, you might listen to music and then the phone is ringing. There's a lot of things happening at the same time. So what if your car would start to reduce certain sounds because it knows that you are on the phone and maybe even use tactile information to still give you navigation or input without having you looking at the screen. I think there's a long way to go. If we are talking about smart technology to use sound and all other modalities in a context sensitive way, adapting to the user situation and these kinds of systems, that's what we are actually working on.

Can you give us a couple of examples of your work that you are working on right now?

As I said, we work for car manufacturers creating sounds and we also do research projects. How would car of the future sound, how sound helps in autonomous cars… There is one project that I can talk about. We did it with Volkswagen and Technical university in Berlin. It was the Autonomous Vehicle Assurance System. It's basically a concept and prototype. We created a spatial sound system for the car where if you picture yourself in an autonomous car, you sit there, you write emails or read a book or whatever. You don't want to watch the traffic, but you still pay some attention to the entire robotic system you are in. You do not drive the car and you have new problems. While writing an email, it would be helpful if you knew that it is going to make a right turn, that it's going to break or accelerate.

We created a sound system, which was predictive. So you would hear the acceleration a bit before it would start to accelerate and you would get G-forces into your body. And very subconsciously and intuitively, your body would process this. There was a study which showed that it helped people to be much more relaxed and to concentrate on the task they were doing.

And the other part was creating sounds transmitting the information from the sensors of the car. Let's say, you see from far away that a bike is about to cross the way of your car and your car would show you that it recognizes this bike, then you would feel that you don't have to intervene. And so we also made sounds for these kinds of things and all these sounds in a way that they create a calm atmosphere, that they're not alarming, that it's not noise polluting. It was quite a cool project and the outcome was very positive. The calming sound, it still gave you all the information you needed to not get stressed.

How are you thinking about the design process and how do you approach the design process for the sound? Is there a rigorous process that you follow or are you exploring your ways?

In the last few years I got much more contact with science and scientific studies. A very important part of our work is starting with research. So what do we actually know about, for example, reaction times on sound, on visuals and on haptic feedback? How can we work with this? In which case is tactile feedback better perceived than sound feedback? And whatever is the case, we try to get to a point at the status quo of knowledge. That's very often the first thing we do.

Then another thing we do is do a workshop with a client to actually get a deeper understanding of their needs and visions, and then also research what the user wants. The reason is that a client or a manufacturer sometimes don't look from the outside on the product and know what the user wants. Also, the brand wants to be represented in a product to understand all the sides. This is kind of a very important part in the beginning.

Another thing – we are now recording the Minimum Viable Podcast, and something I learn right now is the idea of minimum viable product and iterative working on projects. It's very helpful. If you put three years of development in something, it's harder to kill it when it doesn't work in the real world. On the other hand if you prototype it and see it doesn't work it is easy to kill the whole thing after two months of work.

"Silence is gold. If you use sound, use it in the right moment and the right way. If you don't need sound and if it doesn't add value to a moment or an experience or to service – just leave it away."

Before we wrap up, I'd like to ask you about the future of sound design. Where is it going in the upcoming five to ten years?

There are different trends. I think this whole metaverse trend in the end is just more about augmented reality becoming more and more important potentially in our lives. And therefore sound and visuals are, let's say augmented. What we hear and see in our reality is going to be mixed with something which kind of is projected on that reality. And therefore there's a lot of research and development from Meta and all these people, for example, to have things sound like they are in the room. Apple has a thing that the lighting of an AR object is actually adapting to lighting in your real room. And the same thing happens with sound. Another thing is smart systems like multimodal.

I remember ten years ago there was no resource about the user experience and now it's all around us. Could you tell us about your professional background? What was the trigger point for you to get excited about sound?

That was already pretty early in my life. I was always excited how sound could alter emotions and how music works. I started with music. I was producing hip hop in the nineties as a kid, kind of and went into film music later and interactive experiences and installations.

This sound innovation thing just came through work because we were more and more in projects where people would ask "Okay, what does this car of the future sound like? How can we make this navigation system better with sound?" I just saw the need coming. I still see it coming. Every company has UX designers now and none of them knows a lot about sound. That's one thing. I'm excited about what sound can do.

I like that sound works subconsciously. When I do music for a theater piece, an opera for example, I perform it live in the background. It's kind of like pulling the strings of emotionality and creating a space, creating an atmosphere and helping people to forget their everyday problems and immerse themselves into the piece. And it has some magic. You cannot touch sound. It is not a material, it's basically air propagating. That's why it's so hard for many people to grasp. This is a kind of magic that excites me.

Listen to the full interview with Johannes Helberger:

The Mimimum Viable Podcast is brought to you every two months by the design studio Lighting Beetle*, which focuses on creating an exceptional customer experience.

Design is all around us. The Minimum Viable Podcast explores design with a small “d” – the one that looks for solutions to people's problems. In it, together with our guests, we address topics that are related to design, but which we do not normally associate with it. Thanks the Zeldeo production studio and our production manager Mojmír Procházka for the cooperation.

We are looking forward to every listen, follow, share, and suggestion for improvement. Send us your tips for interesting personalities with whom we can talk about design to podcast@lbstudio.sk.

Enjoy!

Minimum Viable Podcast #14: User Experience in Sound Design

Listen to the full interview with Johannes Helberger:

Your journey starts here

Your journey
starts here