Sound and the Internet of Things

 

Embedding computer power and connectivity into everyday devices to create an Internet of Things (IoT) can be both an attractive and worrying development for many consumers. It is enticing as the functionality of devices can be extended dramatically, so that not only can many more objects communicate with each other, but that they can also potentially evolve according to end users’ requirements.

It is concerning for both consumers and manufacturers due to hacks, and unintended eavesdropping through the essential requirement to constantly monitor a device’s activity. Any product that plugs into a mains socket could potentially connect to the internet through powerline networking, without the owner even being aware of the additional functionality.

One of the most straightforward, and cost-effective methods to communicate with end users is through sound.

Battery powered devices might connect wirelessly as easily as a smart phone, or even more simply using Wi-Fi pairing.

One of the most straightforward, and cost-effective methods to communicate with end users is through sound. Both Piezo and MEMS loudspeakers and microphones are relatively inexpensive. They only take up a small amount of real-estate on a device and require little power.

Sounds from a visibly hidden world

Consumers are used to listening to products to understand their working status, without ever having to look at them directly. These passive sounds, such as a kettle clicking off, with the associated boiling diminishing, or gentle hum of a fridge freezer, normally reassuringly confirm that they are functioning properly, and are all accidental sounds which occur naturally as part of the devices’ mechanical nature. Regular, repeated use, normally enables accurate interpretation of a device’s status, in a manner similar to being able to confirm that a car door is securely closed prior to driving off. Sound provides valuable information about the inner workings of a visibly hidden world.

Sounds are usually interpreted about 40 milliseconds faster than simple visual cues, which is one of the main reasons for starters’ pistols still being a mainstay in athletic competitions. This additional delay of visible signals is mostly due to the cognitive workload required to process imagery. Sound also provides a much broader range of communication for the physical space available on a device than a screen of a similar size. A microphone and a loudspeaker can occupy less surface area than a screen capable of displaying only a single legible word. Speech, music and sound effects can all be seamlessly integrated, without the need for glasses or even to look at the device being operated.
[tweet_box]Designing the sound of the internet of things[/tweet_box] In analogue devices, sound is almost always present, and generally only changes due to wear and tear, or incorrect use. In a digital world, there are rarely any naturally occurring auditory cues. Sound can obviously be added and programmed to change due to its acoustic environment, user customisation, or upgrades in functionality. When audio is used only as an output mechanism it is commonly termed an auditory display. When sound can also be utilised for input as well, it is termed an audio interface. In general, auditory interface design suffers from an impoverished understanding of listening experiences, which has led to many users switching off the sound entirely, or at least reducing the functionality to an absolute minimum.

A microphone and a loudspeaker can occupy less surface area than a screen capable of displaying only a single legible word

This has been coupled with a trend for increasing default volume, and thereby annoyance with alert sounds that generally compete to be heard, or more often, ignored. Extensive customisability has had little effect on allowing artificially generated sounds to integrate naturally into pre-existing auditory environments, and yet still remain informative.

Designing effective sounds for the internet of things

Despite the lack of adoption, designing effective sounds for the internet of things is still eminently achievable, as there are many successful physical sounds to base new virtual designs upon. Possibly, the most famous translation from the real to the virtual world is the trash emptying sound of paper being crumpled. This auditory icon can be found in many operating systems, and is often left on by end users, as it is relatively easy to comprehend and remember.

Abstract synthesised sounds termed earcons are often adopted when there is no natural correlation, but they require to be memorised individually

However, the direct correlation between sounds in the physical world and the virtual world is predominantly hit and miss, but the underlying language is considerably less so. Abstract synthesised sounds termed earcons are often adopted when there is no natural correlation, but they require to be memorised individually.

Extensive listener testing can easily identify which sounds are most suitable for inclusion, whether they are captured from the physical world or synthesized. Two of the most useful sound design tips to improve an auditory cues effectiveness include the Doppler effect and presence frequencies. The Doppler effect communicates movement, and a raise in pitch indicates that something is coming towards the listener and therefore potentially important, whereas a drop in pitch suggests that the danger has passed and that it is safe to ignore.

By moving pitch up listeners think a sound is more important and needs to be attended to, conversely a falling pitch suggests that the cue does not need to be attended to. An increase in the presence frequencies of around 1 – 4 kHz replicates when a sound is directed towards a listener, whereas if these frequencies are missing, or are heard to drop off, the perception is that the sound is not intended to be heard, and that we are overhearing something that is either private or irrelevant. These frequencies are often found in alert sounds, and by pulsing them attention can be drawn more easily, especially when it comes to spatially locating the sound source.

---

Popular on A Sound Effect right now - article continues below:

---

Another issue to contend with is all of the other auditory cues that are often present in a shared environment. Audio watermarking is a well-established technique for tracking music plays across media, as well as identifying where leaked media content originated from. This technique can easily be applied to new interface sounds so that when devices are first switched on they can listen to what already exists within an auditory environment and alter their sounds accordingly. Firstly, this can ensure that there are sufficient differences between any sounds so that that perceptual confusion does not occur. More importantly, from an aesthetic approach, this means that if a user has specifically chosen a set of sounds for other devices then a complimentary set can automatically be chosen for the new device, without the user even being aware of a change. As the device is connected to the internet, the sounds do not even have to be preloaded, they can be downloaded as necessary, and if desired, evolve as new devices are introduced into the environment.

As the device is connected to the internet, the sounds do not even have to be preloaded, they can be downloaded as necessary, and if desired, evolve as new devices are introduced into the environment

If a watermark is not present then it is possible for a sample of a sound to be sent to a cloud service so that it can be identified, and its significance established. This technique is already popular for music, and has been utilised for wildlife, as well as for other applications. If it is an emergency sound, like a smoke alarm, then it could be used to turn off a device, or silence it entirely. If it is an alert sound that has been ignored then it could be reproduced by a second device to draw attention to it, or again the device could go silent. These sounds could be time aligned so that they are in phase to make them louder, or time delayed so that the sound from the closer device is slightly later giving the illusion that it is an echo. This would assist identification of the spatial location of the sound, which if there are multiple identical devices in an environment, could help the user understand where a sound was generated.

Sound levels

A really useful feedback mechanism, which is often ignored, is to monitor how users control a sound. Volume controls and repeat/skip buttons, such as can be typically found on many video or audio devices can provide valuable information about whether a sound was replayed at the correct volume. If the user turns the volume up then it may have been too quiet, if the volume was turned down then the sound was possibly too loud. This can then be compared to the ambient background sound pressure level (SPL), and each time the sound is played it can be altered to be the optimal level above the noise floor, even linked to day of week and time of day. Monitoring the background level can also be used to identify when no one is in a room, and it can either ensure that the sound does not play, or that it plays through a device in a room that is occupied.

Monitoring the background level can also be used to identify when no one is in a room, and it can either ensure that the sound does not play, or that it plays through a device in a room that is occupied

Use of repeat button can provide information about the clarity of the sound, or the meaning not being sufficiently interpreted, so this could indicate that an alternative sound needs be sourced automatically, or that the current sounds requires to be either spectrally altered to make comprehension easier, or simply made louder. The skip function could be considered a silent function, so that the user possibly finds the sound unnecessary, if this function was selected for a specific auditory cue a predetermined number of times then the sound could be automatically replaced or disabled.

Speech input is definitely a practical, and, to a certain extent, popular form of interfacing with devices. This can be linked to any sound that is currently playing so that a form of ducking, or silencing completely occurs in real-time. This can allow natural communication between rooms, either by making everything quieter, or by relaying the speech through the device like an intercom, that is time synched to reinforce the speech coming directly from the source. This advanced form of baby monitor can also work in reverse if the desire is to isolate a listener from the speech of others in the shared environment, by turning up frequencies of any audible device that will mask the pre-existing speech, rendering it difficult to comprehend.

Functions which are partly audible due to being inside a case can have the frequency range extended by the built-in loudspeaker, so that they are more acoustically transparent, and thereby much easier to interpret. Alternatively, a form of active noise cancellation can be applied so that the vibrations caused by the loudspeakers are out of phase to the sound occurring naturally to decrease any sound being generated by the device. Some vehicles already make regular use of this technique, and it could be extended into domestic and commercial environments.

When considering what sounds to include, a sensible approach is to consider the function that the sounds performs. The designer’s intention obviously has to match the listeners’ perceptions, whether it is to warn, assist, incite, monitor, reassure, guide, forgive or protect the end user. Warnings are the most obvious, and essential for safety purposes, the common approach is for loud, high pitched sounds with intermittent timing. Assisting sounds provides some form of auditory reminder, incitement encourages actions, with monitoring allowing status communication. Reassurance confirms that appropriate actions were performed, forgiveness communicates that an error was made, but it can be rectified if a renewed attempt is made. Guidance assists end users with details about what activities are possible, such as it is safe to open a door. Protecting sounds provide auditory cues as a form of safeguard.

In a kitchen environment, the simple scanning of a QR code on a ready meal could turn on the oven, and when it is has heated up to the correct temperature provide a subtle alert that it is time to insert the meal. Once the cooking has completed then a conventional alarm could inform the intended diner, in whichever room they are located that it is ready to consume. If the sound is ignored a verbal offer of keeping it warm could be made, which would only require a simple ‘yes’ or ‘no’ response to affect the required temperature change in the oven. This could equally apply to any product, such as clothes in a washing machine, or even setting the correct temperature on an iron in order to prevent damage to a delicate piece of clothing.
 

Using ultrasound

Sounds do not need to be confined to the human audible range. Ultrasound is currently used as Audio Beacons to sync devices for advertising purposes without customers’ awareness. However, ultrasound could also be used for end users’ benefit, to communicate that a device is still on, when it should not be, like an iron that is stationary for a pre-specified duration. Conversely the absence of a coded ultrasound cue could convey that an important device is no longer operating, such as a smoke alarm or a freezer. An audible alert could then be triggered so that an appropriate action can be taken. Devices when they start up for the first time could emit ultrasound to inform any other appliances by the same manufacturer, or using the same system, that they have been added to the environment. Two-way inaudible communication could then provide the new device information about the audio selections made for the pre-existing environments in order to aid customisation.

Ultrasound could also be used for end users’ benefit, to communicate that a device is still on, when it should not be, like an iron that is stationary for a pre-specified duration

By introducing more sounds through the internet of things the world does not need to be noisier. Unlike manual devices where the sound is often a by-product and difficult to remove, in the virtual world devices can be set to only make sounds when absolutely necessary. As an individual becomes familiar or adept with a device and operates it correctly, fewer sounds could be generated over time. Eventually only errors could be communicated audibly, such as tapping a button too often, due to a delay in functioning and a finger being in the way to prevent visual confirmation, helping to make the device ‘fat finger friendly’.
 

The device could also listen out for maintenance cues to know when something has gone wrong. If a photocopier is jammed then the instructions could be read out in the end user’s chosen language, in sync with the actions being performed by the person trying to fix it, as an ultrasound cue is generated as a hatch opens, or even by interpreting a naturally occurring sonic cue via an unrelated device such as a mobile phone. This could help avoid the need to constantly bob up and down while trying to interpret the animations on a small screen, or blindly try to investigate the problem. The smart phone could also drastically expand the number of user maintainable devices in an environment, as verbal instructions could be augmented by the interpretation of appropriate acoustic cues.

Sounds for safety

The safety aspects are obvious for including more auditory cues in internet enabled devices, an audible smoke alarm could switch off every device that might be causing the fire, as well as switch on extractor fans, and after an appropriate delay if the issue has not been resolved contact the emergency services. A less serious option might be a tap or stopcock that is turned off as the sound of water hits a floor, or a door that is locked when snoring is heard, indicating that the occupant has fallen asleep. Children crying could automatically trigger devices which might be soothing, such as a night light, music or even engaging a microphone so that the child could hear a parent’s voice or breathing if the parent is sleeping. Personal security can already be enhanced through microphones actively monitoring extreme events.

Just like cookies allow advertisers to target content, and personal data stored on Facebook and other forms of social media can provide recommendations, these mechanisms could be used to customise the auditory content of the internet of things.

This could allow a much more holistic design approach where sounds are introduced as needed, based on similar users’ experiences, rather than having everything switched on to start off with

New devices might be registered by purchasers using existing online profiles and then the audio settings utilised by users with similar profiles would be set automatically on the new device. Whenever a device is connected it could send user settings to the cloud to track long term usage, further refining the settings for future users. This could allow a much more holistic design approach where sounds are introduced as needed, based on similar users’ experiences, rather than having everything switched on to start off with.

Sound design for the internet of things can transform the acoustic space in any environment. It can be used as much to switch sounds off as add them. Safety can be enhanced by ensuring that pre-existing alarms are clearly audible, and individuals in shared environments can either be more or less acoustically aware of each other, depending upon their preferences. Systems that actively listen out for pre-existing auditory cues can avoid masking or perceptual conflicts, and users with similar social media profiles can provide suitable initial settings. Sound events can be shared across devices with appropriate timing cues to guide listeners where to attend to, and choices updated according to usage patterns. Sounds might be introduced at subtler levels, and then turned up if they go unnoticed, rather than the more traditional approach of needing to be turned down or off entirely. The palette of sounds can also be much broader as new cues can be downloaded automatically rather than stored on the device at the point of manufacture, and designers can monitor each sound’s popularity and usage in order to maximise effectiveness and influence future designs. Privacy will be the main issue to overcome with the understandable fear of snooping, but most of these functions can be confined to a local private network with only intermittent connection to the internet.
 

A big thanks to Dr Iain McGregor for allowing me to republish his insights on sound for the internet of things! This post originally appeared here.

 

Please share this:

 

 

---