Sound and the Internet of Things

 

Embedding computer power and connectivity into everyday devices to create an Internet of Things (IoT) can be both an attractive and worrying development for many consumers. It is enticing as the functionality of devices can be extended dramatically, so that not only can many more objects communicate with each other, but that they can also potentially evolve according to end users’ requirements.

It is concerning for both consumers and manufacturers due to hacks, and unintended eavesdropping through the essential requirement to constantly monitor a device’s activity. Any product that plugs into a mains socket could potentially connect to the internet through powerline networking, without the owner even being aware of the additional functionality.

One of the most straightforward, and cost-effective methods to communicate with end users is through sound.

Battery powered devices might connect wirelessly as easily as a smart phone, or even more simply using Wi-Fi pairing.

One of the most straightforward, and cost-effective methods to communicate with end users is through sound. Both Piezo and MEMS loudspeakers and microphones are relatively inexpensive. They only take up a small amount of real-estate on a device and require little power.

Sounds from a visibly hidden world

Consumers are used to listening to products to understand their working status, without ever having to look at them directly. These passive sounds, such as a kettle clicking off, with the associated boiling diminishing, or gentle hum of a fridge freezer, normally reassuringly confirm that they are functioning properly, and are all accidental sounds which occur naturally as part of the devices’ mechanical nature. Regular, repeated use, normally enables accurate interpretation of a device’s status, in a manner similar to being able to confirm that a car door is securely closed prior to driving off. Sound provides valuable information about the inner workings of a visibly hidden world.

Sounds are usually interpreted about 40 milliseconds faster than simple visual cues, which is one of the main reasons for starters’ pistols still being a mainstay in athletic competitions. This additional delay of visible signals is mostly due to the cognitive workload required to process imagery. Sound also provides a much broader range of communication for the physical space available on a device than a screen of a similar size. A microphone and a loudspeaker can occupy less surface area than a screen capable of displaying only a single legible word. Speech, music and sound effects can all be seamlessly integrated, without the need for glasses or even to look at the device being operated.

In analogue devices, sound is almost always present, and generally only changes due to wear and tear, or incorrect use. In a digital world, there are rarely any naturally occurring auditory cues. Sound can obviously be added and programmed to change due to its acoustic environment, user customisation, or upgrades in functionality. When audio is used only as an output mechanism it is commonly termed an auditory display. When sound can also be utilised for input as well, it is termed an audio interface. In general, auditory interface design suffers from an impoverished understanding of listening experiences, which has led to many users switching off the sound entirely, or at least reducing the functionality to an absolute minimum.

A microphone and a loudspeaker can occupy less surface area than a screen capable of displaying only a single legible word

This has been coupled with a trend for increasing default volume, and thereby annoyance with alert sounds that generally compete to be heard, or more often, ignored. Extensive customisability has had little effect on allowing artificially generated sounds to integrate naturally into pre-existing auditory environments, and yet still remain informative.

Designing effective sounds for the internet of things

Despite the lack of adoption, designing effective sounds for the internet of things is still eminently achievable, as there are many successful physical sounds to base new virtual designs upon. Possibly, the most famous translation from the real to the virtual world is the trash emptying sound of paper being crumpled. This auditory icon can be found in many operating systems, and is often left on by end users, as it is relatively easy to comprehend and remember.

Abstract synthesised sounds termed earcons are often adopted when there is no natural correlation, but they require to be memorised individually

However, the direct correlation between sounds in the physical world and the virtual world is predominantly hit and miss, but the underlying language is considerably less so. Abstract synthesised sounds termed earcons are often adopted when there is no natural correlation, but they require to be memorised individually.

Extensive listener testing can easily identify which sounds are most suitable for inclusion, whether they are captured from the physical world or synthesized. Two of the most useful sound design tips to improve an auditory cues effectiveness include the Doppler effect and presence frequencies. The Doppler effect communicates movement, and a raise in pitch indicates that something is coming towards the listener and therefore potentially important, whereas a drop in pitch suggests that the danger has passed and that it is safe to ignore.

By moving pitch up listeners think a sound is more important and needs to be attended to, conversely a falling pitch suggests that the cue does not need to be attended to. An increase in the presence frequencies of around 1 – 4 kHz replicates when a sound is directed towards a listener, whereas if these frequencies are missing, or are heard to drop off, the perception is that the sound is not intended to be heard, and that we are overhearing something that is either private or irrelevant. These frequencies are often found in alert sounds, and by pulsing them attention can be drawn more easily, especially when it comes to spatially locating the sound source.

 

---

Popular on A Sound Effect right now - article continues below:

 

1

SFX & Atmospheres Bundle

$129.50 $30

2

Metallitronic

$95

3

Just Gore

$40

4

Foundation Series: SCI-FI

$24.99 $19.99

---

Latest releases:  

• 

  
  Environments Historical Houses & Barns Play Track 5000 sounds included, 604 mins total $170 $150

  This is a collection of location foley sounds (anything we could find that makes sound) recorded over 3-year period in authentic historical houses and barns.

  This includes 1700s peasant village houses, 1700s manor, 1850s farm, 1900s manor and 1930s cottage. These are recorded for film projects so they have the right cinematic feel, dynamic and variation.

  The houses were full of period props, so the recordings are rich and have lots of details that take you to the right time and feel.

  The library weighs in at more than 20 GB total, and features 5000+ sounds & 10 hours of recordings in 347 files.

  Highlights:
  • Many are long takes (8+ minutes) with lots of performance variations
  • The stereo recordings have often one channel as close-up mic and other as distant / room mic.
  • All recordings are 96khz 24bit with metadata.
  • We used the best sounding equipment – sound devices recorders (744, 702, mixpre), Sennheiser mkh 8040,  mkh416 and DPA 4006a microphones.
  
  12 %

  OFF

  Ends 1556748000
  Add to cart
• 

  
  Drones & Moods Metallitronic Play Track 230+ sounds included $95

  Metallitronic is sound designer Nathan Moody’s latest collection of intense textures, a library of nearly three hours’ worth of metallic membranes and objects excited by electronic signals.

  Metallitronic was created using advanced synthesis techniques played back through various transducers into cymbals, gongs, bowls, steel plates, prefab metal buildings, and other resonant metallic objects, close-miked in stereo with extended-frequency-range microphones. This collection radically expands a sound designer’s palette of stressed-metal sounds, horror and tension elements, crystalline creaks and groans, unsettling background atmospheres, rattling vehicular interiors, creature sound sweeteners, the sound of massive machinery, earth-shaking rumbles, and stochastic, complex rhythms.

  Some textures let the complex synthesizer textures shine through – intense on their own, even if they weren’t amplified through metal – but the primary focus is on metallic drones, hits, pings, stresses, groans, wails, resonances, rattles, and unique timbres that can be layered into almost any project. The scale of the sounds ranges from detailed to overwhelmingly massive. The sounds invite aggressive filtering and extreme pitch shifting.

  Every sound is recorded in stereo at 24 bits, 192 kHz, and many sounds have ultrasonic frequencies. This library includes nearly three hours of recordings, all fully tagged with metadata. Multiple articulations are ganged into single files for convenience and organization.

  Add to cart
• 

  
  Environments Baltic Forest Play Track 53 sounds included, 233 mins total $39 $23.40

  Baltic Forest is a collection of ambient recordings captured over one year in a private forest in Northern Lithuania. 53 carefully recorded exterior ambiences made during summer, autumn, winter and spring. From the bright sound of native birds in spring and summer, to the freezing cold (-20 Celsius) snow storms and tree movements of autumn and winter – sometimes recording in knee-deep snow.

  All sounds were recorded and edited at 24-bit / 96kHz, with embedded meta data.

  40 %

  OFF

  Ends 1556575200
  Add to cart
• 

  
  Destruction & Impact The Vault Play Track 12743 sounds included, 1249 mins total $599.99

  The whole 9 yards.

  Comprised of five heavy hitting libraries, this bundle includes GEARBOX, SORCERY, HERO, BROKEN, and MELEE. Save by bundling all FIVE in a single library.

  GEARBOX:
  Boutique analog mechanical contraptions, steampunk gadgetry, gizmos and machines big and small.

  

  
  

  SORCERY
  Spells, deflects, casts, blocks, beams, and more. Unrivaled wizardry at your fingertips.

  

  
  

  BROKEN:
  Car crashes, explosions, crumbling buildings, earthquakes, ripping earth and metal, to debris, and more.

  

  
  

  HERO:
  HERO – Sword fights, stabbing, guillotines, impaling, battle cries, shields, drawbridges, armor, foley and more.

  

  
  

  MELEE:
  Punches, kicks, blocks, bodyfalls, grabs, slaps, bone breaks, blood splatters, and more.

  

  
  
  Add to cart
• 

  
  Mechanical Gearbox Play Track 3551 sounds included, 279 mins total $149.99

  We've ventured to obscure boutiques, prop houses and vintage shops to capture mechanical contraptions from around the world. Ranging from bizarre creations, to steampunk gadgetry, gizmos and machines, GEARBOX clocks in at over 10 GB of high definition, precision mastered sounds spanning across 2987 construction kit sounds and 584 designed sounds.

  GEARBOX equips Sound Designers with a literal toolbox of mechanical gadgetry. Ranging from tiny to huge, GEARBOX's machines and gizmos provide coverage for interacts, mechanism, machine or device in your scene or game.

  INTRODUCING BUILDING BLOCKS

  In addition to CONSTRUCTION KIT and DESIGNED SOUND content, GEARBOX features BUILDING BLOCKS. This category of sound consists of designed phrases and oneshots utilized for our designed machinery, empowering Sound Designers with maximum flexibility when trying to get that particular phrase from an existing DESIGNED SOUND. GEARBOX features over 468 BUILDING BLOCKS ranging from levers, hits, grinds, snaps, and more.

  
  Add to cart

---

Another issue to contend with is all of the other auditory cues that are often present in a shared environment. Audio watermarking is a well-established technique for tracking music plays across media, as well as identifying where leaked media content originated from. This technique can easily be applied to new interface sounds so that when devices are first switched on they can listen to what already exists within an auditory environment and alter their sounds accordingly. Firstly, this can ensure that there are sufficient differences between any sounds so that that perceptual confusion does not occur. More importantly, from an aesthetic approach, this means that if a user has specifically chosen a set of sounds for other devices then a complimentary set can automatically be chosen for the new device, without the user even being aware of a change. As the device is connected to the internet, the sounds do not even have to be preloaded, they can be downloaded as necessary, and if desired, evolve as new devices are introduced into the environment.

As the device is connected to the internet, the sounds do not even have to be preloaded, they can be downloaded as necessary, and if desired, evolve as new devices are introduced into the environment

If a watermark is not present then it is possible for a sample of a sound to be sent to a cloud service so that it can be identified, and its significance established. This technique is already popular for music, and has been utilised for wildlife, as well as for other applications. If it is an emergency sound, like a smoke alarm, then it could be used to turn off a device, or silence it entirely. If it is an alert sound that has been ignored then it could be reproduced by a second device to draw attention to it, or again the device could go silent. These sounds could be time aligned so that they are in phase to make them louder, or time delayed so that the sound from the closer device is slightly later giving the illusion that it is an echo. This would assist identification of the spatial location of the sound, which if there are multiple identical devices in an environment, could help the user understand where a sound was generated.

Sound levels

A really useful feedback mechanism, which is often ignored, is to monitor how users control a sound. Volume controls and repeat/skip buttons, such as can be typically found on many video or audio devices can provide valuable information about whether a sound was replayed at the correct volume. If the user turns the volume up then it may have been too quiet, if the volume was turned down then the sound was possibly too loud. This can then be compared to the ambient background sound pressure level (SPL), and each time the sound is played it can be altered to be the optimal level above the noise floor, even linked to day of week and time of day. Monitoring the background level can also be used to identify when no one is in a room, and it can either ensure that the sound does not play, or that it plays through a device in a room that is occupied.

Monitoring the background level can also be used to identify when no one is in a room, and it can either ensure that the sound does not play, or that it plays through a device in a room that is occupied

Use of repeat button can provide information about the clarity of the sound, or the meaning not being sufficiently interpreted, so this could indicate that an alternative sound needs be sourced automatically, or that the current sounds requires to be either spectrally altered to make comprehension easier, or simply made louder. The skip function could be considered a silent function, so that the user possibly finds the sound unnecessary, if this function was selected for a specific auditory cue a predetermined number of times then the sound could be automatically replaced or disabled.

Speech input is definitely a practical, and, to a certain extent, popular form of interfacing with devices. This can be linked to any sound that is currently playing so that a form of ducking, or silencing completely occurs in real-time. This can allow natural communication between rooms, either by making everything quieter, or by relaying the speech through the device like an intercom, that is time synched to reinforce the speech coming directly from the source. This advanced form of baby monitor can also work in reverse if the desire is to isolate a listener from the speech of others in the shared environment, by turning up frequencies of any audible device that will mask the pre-existing speech, rendering it difficult to comprehend.

Functions which are partly audible due to being inside a case can have the frequency range extended by the built-in loudspeaker, so that they are more acoustically transparent, and thereby much easier to interpret. Alternatively, a form of active noise cancellation can be applied so that the vibrations caused by the loudspeakers are out of phase to the sound occurring naturally to decrease any sound being generated by the device. Some vehicles already make regular use of this technique, and it could be extended into domestic and commercial environments.

When considering what sounds to include, a sensible approach is to consider the function that the sounds performs. The designer’s intention obviously has to match the listeners’ perceptions, whether it is to warn, assist, incite, monitor, reassure, guide, forgive or protect the end user. Warnings are the most obvious, and essential for safety purposes, the common approach is for loud, high pitched sounds with intermittent timing. Assisting sounds provides some form of auditory reminder, incitement encourages actions, with monitoring allowing status communication. Reassurance confirms that appropriate actions were performed, forgiveness communicates that an error was made, but it can be rectified if a renewed attempt is made. Guidance assists end users with details about what activities are possible, such as it is safe to open a door. Protecting sounds provide auditory cues as a form of safeguard.

In a kitchen environment, the simple scanning of a QR code on a ready meal could turn on the oven, and when it is has heated up to the correct temperature provide a subtle alert that it is time to insert the meal. Once the cooking has completed then a conventional alarm could inform the intended diner, in whichever room they are located that it is ready to consume. If the sound is ignored a verbal offer of keeping it warm could be made, which would only require a simple ‘yes’ or ‘no’ response to affect the required temperature change in the oven. This could equally apply to any product, such as clothes in a washing machine, or even setting the correct temperature on an iron in order to prevent damage to a delicate piece of clothing.
 

Using ultrasound

Sounds do not need to be confined to the human audible range. Ultrasound is currently used as Audio Beacons to sync devices for advertising purposes without customers’ awareness. However, ultrasound could also be used for end users’ benefit, to communicate that a device is still on, when it should not be, like an iron that is stationary for a pre-specified duration. Conversely the absence of a coded ultrasound cue could convey that an important device is no longer operating, such as a smoke alarm or a freezer. An audible alert could then be triggered so that an appropriate action can be taken. Devices when they start up for the first time could emit ultrasound to inform any other appliances by the same manufacturer, or using the same system, that they have been added to the environment. Two-way inaudible communication could then provide the new device information about the audio selections made for the pre-existing environments in order to aid customisation.

Ultrasound could also be used for end users’ benefit, to communicate that a device is still on, when it should not be, like an iron that is stationary for a pre-specified duration

By introducing more sounds through the internet of things the world does not need to be noisier. Unlike manual devices where the sound is often a by-product and difficult to remove, in the virtual world devices can be set to only make sounds when absolutely necessary. As an individual becomes familiar or adept with a device and operates it correctly, fewer sounds could be generated over time. Eventually only errors could be communicated audibly, such as tapping a button too often, due to a delay in functioning and a finger being in the way to prevent visual confirmation, helping to make the device ‘fat finger friendly’.
 

The device could also listen out for maintenance cues to know when something has gone wrong. If a photocopier is jammed then the instructions could be read out in the end user’s chosen language, in sync with the actions being performed by the person trying to fix it, as an ultrasound cue is generated as a hatch opens, or even by interpreting a naturally occurring sonic cue via an unrelated device such as a mobile phone. This could help avoid the need to constantly bob up and down while trying to interpret the animations on a small screen, or blindly try to investigate the problem. The smart phone could also drastically expand the number of user maintainable devices in an environment, as verbal instructions could be augmented by the interpretation of appropriate acoustic cues.

Sounds for safety

The safety aspects are obvious for including more auditory cues in internet enabled devices, an audible smoke alarm could switch off every device that might be causing the fire, as well as switch on extractor fans, and after an appropriate delay if the issue has not been resolved contact the emergency services. A less serious option might be a tap or stopcock that is turned off as the sound of water hits a floor, or a door that is locked when snoring is heard, indicating that the occupant has fallen asleep. Children crying could automatically trigger devices which might be soothing, such as a night light, music or even engaging a microphone so that the child could hear a parent’s voice or breathing if the parent is sleeping. Personal security can already be enhanced through microphones actively monitoring extreme events.

Just like cookies allow advertisers to target content, and personal data stored on Facebook and other forms of social media can provide recommendations, these mechanisms could be used to customise the auditory content of the internet of things.

This could allow a much more holistic design approach where sounds are introduced as needed, based on similar users’ experiences, rather than having everything switched on to start off with

New devices might be registered by purchasers using existing online profiles and then the audio settings utilised by users with similar profiles would be set automatically on the new device. Whenever a device is connected it could send user settings to the cloud to track long term usage, further refining the settings for future users. This could allow a much more holistic design approach where sounds are introduced as needed, based on similar users’ experiences, rather than having everything switched on to start off with.

Sound design for the internet of things can transform the acoustic space in any environment. It can be used as much to switch sounds off as add them. Safety can be enhanced by ensuring that pre-existing alarms are clearly audible, and individuals in shared environments can either be more or less acoustically aware of each other, depending upon their preferences. Systems that actively listen out for pre-existing auditory cues can avoid masking or perceptual conflicts, and users with similar social media profiles can provide suitable initial settings. Sound events can be shared across devices with appropriate timing cues to guide listeners where to attend to, and choices updated according to usage patterns. Sounds might be introduced at subtler levels, and then turned up if they go unnoticed, rather than the more traditional approach of needing to be turned down or off entirely. The palette of sounds can also be much broader as new cues can be downloaded automatically rather than stored on the device at the point of manufacture, and designers can monitor each sound’s popularity and usage in order to maximise effectiveness and influence future designs. Privacy will be the main issue to overcome with the understandable fear of snooping, but most of these functions can be confined to a local private network with only intermittent connection to the internet.
 

A big thanks to Dr Iain McGregor for allowing me to republish his insights on sound for the internet of things! This post originally appeared here.

 

Please share this:

 

 

---