audio-works v0.6.8

audio-works

WIP

Library meant for Electron (ver. 11.x - 13.x) to determine note based on the signal received from mic or file.

Currently, tested determining sounds down to C1 ~ 32.7Hz (Less than 2Hz difference between C1 and B0) so at the moment it's accurate enough down to at least 2Hz differences.

• Added volume measuring
• Still a lot a little of garbage left in methods waiting for removal
• Added possibility to change input audio device (+ automatically changes when current device gets disconnected)
• Still a lot a little bit Almost none of garbage left in methods waiting for removal
• Added possibility to change input audio device (+ automatically updates list of devices on change / when current device gets disconnected)
• Separated most micSetup and Renderer methods into modules
• Changed objects into classes
• Fixed bug with enabling mic after disabling it
• Fixed bug with repeatedly changing input device while the mic is enabled resulting in problems with audioContext
• Untangled logic, so it's a bit more simple and less convoluted now imo I was so wrong
• Added A-, B- and C-weighting classes
• Changed audio volume measurement using weighting classes
• Added methods returning nyquist frequency and band range of current audioHandler setup
• Added possibility to change output device

TODO right now:

• Add methods to frequencyMath
• Untangle deviceHandler and other redundant methods etc.
• Adding possibility to automatically switch to default available device if currently used one gets disconnected
• General code refactor
• Output audio (the latency is/will be +- 1 second so not great, but it's Node + Chromium ¯\_(ツ)_/¯
• Changes in soundStorage module for storing and determining frequencies (in progress)
• Anything else that will pop up later

ChangeLog:

• v0.6.8
• v0.6.7
• v0.6.6
• v0.6.5
• v0.6.4
• v0.6.3
• v0.6.2

Classes:

• AudioSetup
• AudioHandler
• AudioFileHandler
• Correlation
• DeviceHandler
• Device
• SoundStorage
• SoundStorageEvent
• FrequencyMath
• AudioEvents

Test coverage

• coverage

Setup, sample initialization

Electron's browser window should've contextIsolation set to false as well as nodeIntegration set to true.

window = new browserWindow({
  webPreferences: {
    contextIsolation: false,
    nodeIntegration: true,
  },
});

Then in rendering process sample initialization logging value of a correlated buffer from the default device could look like this:

const { AudioHandler, AudioEvents } = require("audio-works");

let mic = new AudioHandler();

mic.on(AudioEvents.audioProcessUpdate, (evt) => {
  console.log(evt.correlate());
}); // Event called from ScriptProcessor on new buffer chunk

await mic.setupStream(); // Start the mediaStreamSource

setTimeout(1000, () => {
  mic.end();
}); // close the stream after 1 second

It is also possible to output received signal by creating the html Audio object and setting it's srcObject property to the stream hold by the AudioHandler.

const { AudioHandler, AudioEvents } = require("audio-works");

let mic = new AudioHandler();
let audio = new Audio();

mic.on(AudioEvents.setupDone, (evt) => {
  audio.srcObject = evt.stream;
}); // Event emitted after setupStream()

await mic.setupStream(); // setupStream() is asynchronus but all the following
// actions can be done on emission of the "SetupDone" event
// so that await in such a case could be omitted

To change the device it's enough to pass an id of said device to the changeInput methods of the AudioHandler. List of devices can be accessed through DeviceHandler hold by the AudioHandler as deviceHandler property.

const { AudioHandler, Device } = require("audio-works");

let mic = new AudioHandler();

// if for some reason the device list is empty, which shouldn't really happen,
// then it can be easily fixed by calling `await mic.deviceHandler.updateDeviceList();`

// Retrieves a list of available devices
let inputs = mic.getDeviceList(Device.direction.input);
// Change default ('first available') input to the third one
mic.changeInput(inputs[2].id);

await mic.setupStream();

Classes

AudioSetup

File in GitHub
Main class responsible for setting up AudioHandler and AudioFileHandler holding two main obligatory nodes used by AudioContext which are AnalyserNode and GainNode. This class extends EventEmitter as after various steps instance dispatches related to them events.

AudioHandler

File in GitHub
Extends AudioSetup as AudioContext is crucial for all the functionalities provided by this class. Handles live audio inputs like microphones or instruments connected to audio interfaces as well as output to any available devices.

Sample object initialization

const { AudioHandler, Gain, Analyser } = require("audio-works");

let mic = new AudioHandler({ // All the values are optional.
    general: {               // Omitting some values in objects
        buflen: 8192,        // containing more properties will result
        curveAlgorithm: 'A'  // in assigment of default value
    },                       // only to the missing properties
    gainNode: new Gain({value: 1.5}),
    analyserNode: new Analyser({
        smoothingTimeConstant: 0.9,
        fftSize: 32768,
        minDecibels: -90,
        maxDecibels: -10
    }),
    correlation: {
        rmsThreshold: 0.01,
        correlationThreshold: 0.01,
        correlationDegree: 0.98
    }
});

Sample usage of getWeightedVolume

const vol = mic.getWeightedVolume(2); 
let volume = (vol / 200) * (vol / 2); // Let's take everything over 200dB as maximally "loud"
                                      // (alternatively can be written as vol^2 / 400)
volume = volume < 100 ? volume : 100; 

AudioFileHandler

File in GitHub
Extends AudioHandler class therefore retains possibility to handle live audio input but adds methods meant for audio file decoding, creating standard BufferSources with primary goal of audio output, or obtaining pulse-code modulation data.

Example of logging correlated data and playing the audio from a file:

const { AudioFileHandler, AudioEvents } = require("audio-works");

const fileHandler = new AudioFileHandler({}, "./audioFiles/sample.wav");
await fileHandler.initCorrelation(); // this call is needed as we don't
// call setupStream() method

// -- Event driven approach --
fileHandler.on(AudioEvents.processedFileChunk, (evt) => {
  // perform() is called directly on the correlation object stored
  // in fileHandler, unlike calling "correlate()" in AudioHandler,
  // as there's no mediaStream stored in the "stream" property,
  // therefore it requires to manually push the data chunk passed
  // to the listener in evt data to be correlated.
  console.log(fileHandler.correlation.perform(evt));
});

const audioSource = await fileHandler.createSource();
audioSource.start(0);

fileHandler.processEvent(); // start processing

// -- Callback approach --
const audioSource = await fileHandler.createSource();
audioSource.start(0);

fileHandler.processCallback((data) => {
  console.log(fileHandler.correlation.perform(data));
});
// While using callback processing starts immediately so there's
// no call like "processEvent()" in this case

Correlation

File in GitHub
Sole purpose of this class is performing autocorrelation on audio buffer, allowing a set-up of custom thresholds. The output of perform method is supposed to be a frequency of the sound (the fundamental frequency). This means it processes the signal in monophonic context.

DeviceHandler

File in GitHub
Main purpose of this class is interaction with navigator.mediaDevices and for that reason it uses a private helper class Device. Device class instances returned from methods of this class are only the copies of actual stored objects to keep the data stored by the instance consistent regardless of user actions on obtained device data.

Device

File in GitHub
A class representing navigators mediaDevices. It has no methods, holding only
values: id: device id, label: device label, and dir: device direction
Array of instances of this class is returned from the getDeviceList method of DeviceHandler.
Along the device direction there are also two boolean flags related to it: isOutput and isInput for more convenient array checks and filtering.
Class contains copy method for more convenient deep copies.
For more convenient direction description instead of raw strings class contains a static object serving as enum which can be accessed as Device.direction.(input|output).
For more convenient device type description instead of raw strings class contains a static object serving as enum which can be accessed as Device.type.(audio|video), nonetheless only "audio" option is used/checked in the whole implementation.

SoundStorage

File in GitHub
Class supposed to serve as a storage for outputs of the Correlation class holding methods helping correct sound frequency estimations in short periods of time.

SoundStorageEvent

File in GitHub
This class has the same purpose as SoundStorage extending it with a difference of utilizing EventEmitter allowing more diverse interactions with the storage.