AudioWorklet and Emscripten work with OF

…and I think its an improvement.
https://gameoflife3d.handmadeproductions.de/
(sorry, always the same patch, but I use it to test latency and sound quality)

On the OF side I chande some things in:

It is used for sound output now, and I think for that it works great, but maybe someone can help to replace the whole scriptprocessornode functionality (or with an condition to switch between scriptprocessornode and audioworklet)?

I have to add, that it only works with the mentioned emscripten branch and for that OF needs to be updated for working with Emscripten 2.0.34…

By the way: I also made a very messy webMIDI implementation a while ago. But it works and would be great to implement in a better way…

Thats what I changed exactly:
In library_html5audio.js i changed html5audio_stream_create to:

	html5audio_stream_create: function(context_id, bufferSize, inputChannels, outputChannels, inbuffer, outbuffer, callback, userData, pthreadPtr){
	out("Buffer size: " + bufferSize);
// Initializes the audio context and the pthread it it's AudioWorkletGlobalScope

  // Create the context
  Module.audioCtx = new AudioContext({sampleRate: 88200});

  // Initialize the pthread shared by all AudioWorkletNodes in this context
  PThread.initAudioWorkletPThread(Module.audioCtx, pthreadPtr).then(function() {
    out("Audio worklet PThread context initialized!")
  }, function(err) {
    out("Audio worklet PThread context initialization failed: " + [err, err.stack]);
  });

  // Creates an AudioWorkletNode and connects it to the output once it's created
  PThread.createAudioWorkletNode(
    Module.audioCtx,
    'native-passthrough-processor', 
    {
      numberOfInputs: 0,
      numberOfOutputs : 1,
      outputChannelCount : [2],
      processorOptions: {
      inputChannels : inputChannels,
      outputChannels : outputChannels,
      inbuffer : inbuffer,
      outbuffer : outbuffer,
      bufferSize : bufferSize,
      callback : callback,
      userData : userData
      }     
    }
  ).then(function(workletNode) {
    // Connect the worklet to the audio context output
    out("Audio worklet node created! Tap/click on the window if you don't hear audio!");
    workletNode.connect(Module.audioCtx.destination);
  }, function(err) {
    console.log("Audio worklet node creation failed: " + [err, err.stack]);
  });

  // To make this example usable we setup a resume on user interaction as browsers
  // all require the user to interact with the page before letting audio play
  if (window && window.addEventListener) {
    var opts = { capture: true, passive : true };    
    window.addEventListener("touchstart", function() { Module.audioCtx.resume() }, opts);
    window.addEventListener("mousedown", function() { Module.audioCtx.resume() }, opts);
    window.addEventListener("keydown", function() { Module.audioCtx.resume() }, opts);
  }
	},

I made this change in ofxEmscriptenSoundStream.cpp:

bool ofxEmscriptenSoundStream::setup(const ofSoundStreamSettings & settings) {
	inbuffer.allocate(settings.bufferSize, settings.numInputChannels);
	outbuffer.allocate(settings.bufferSize, settings.numOutputChannels);
	static pthread_t pthreadPtr = 0;
	this->settings = settings;
	stream = html5audio_stream_create(context,settings.bufferSize,settings.numInputChannels,settings.numOutputChannels,inbuffer.getBuffer().data(),outbuffer.getBuffer().data(),&audio_cb, this, pthreadPtr);
	return true;
}

And in ofxEmscriptenSoundSTream.h:

#include <pthread.h>

And this is the AudioWorklet:

/**
 * This is the JS side of the AudioWorklet processing that creates our
 * AudioWorkletProcessor that fetches the audio data from native code and 
 * copies it into the output buffers.
 * 
 * This is intentionally not made part of Emscripten AudioWorklet integration
 * because apps will usually want to a lot of control here (formats, channels, 
 * additional processors etc.)
 */

// Register our audio processors if the code loads in an AudioWorkletGlobalScope
if (typeof AudioWorkletGlobalScope === "function") {
  // This processor node is a simple proxy to the audio generator in native code.
  // It calls the native function then copies the samples into the output buffer
  var counter = 0;
  var inputChannels = 0;
  var outputChannels = 0;
  var inbuffer = 0;
  var outbuffer = 0;
  var bufferSize = 0;
  var callback = 0;
  var userData = 0;
  class NativePassthroughProcessor extends AudioWorkletProcessor {
    constructor (options) {
    super()
    inputChannels = options.processorOptions.inputChannels;
    outputChannels = options.processorOptions.outputChannels;
    inbuffer = options.processorOptions.inbuffer;
    outbuffer = options.processorOptions.outbuffer;
    bufferSize = options.processorOptions.bufferSize;
    callback = options.processorOptions.callback;
    userData = options.processorOptions.userData;
  }
    process(inputs, outputs, parameters) {
    counter = currentFrame / 128 % (bufferSize / 64);
    if (counter == 0) {
        dynCall('viiii',callback, [bufferSize,inputChannels,outputChannels,userData]);
    }
    const output = outputs[0];
    for (let channel = 0; channel < 2; ++channel) {
        const outputChannel = output[channel];
        outputChannel.set(Module.HEAPF32.subarray(outbuffer>>2,(outbuffer>>2)+bufferSize*outputChannels).slice(counter * 128, counter * 128 + 128));
        }
        return true;
    }
}
  // Register the processor as per the audio worklet spec
  registerProcessor('native-passthrough-processor', NativePassthroughProcessor);
}

And I use this Emscripten branch:

This is great.
Thanks.
Have you done a Pull Request in the github repo?

Thank you. No, but I opened some issues with solutions regarding updating OF for the current Emscripten version. And it will not work until they are pulled (but they are on the list for OF 0.12.0… https://github.com/openframeworks/openFrameworks/milestone/24). Would be great if this would be implemented in some (better?) way into OF. Btw, I also edited https://github.com/openframeworks/openFrameworks/blob/master/addons/ofxEmscripten/libs/html5video/lib/emscripten/library_html5video.js for loading files with a file browser or drag and drop. Here is an (old) example (I guess, the format needs to be .mp4): http://videoplayer.handmadeproductions.de/

Maybe it would be nice to implement something like that too (perhaps not only for video files…)?

I need to get into using Emscripten with OF tomorrow. I only tested it once several years ago, but I will definitely use your patches and see if there are any improvements I can make to your implementation.
I will keep you posted.

@roymacdonald thanks. I am definitely not an experienced programmer (my goal is to run Pure Data patches with OF and Emscripten in the web, and thats where I come from…), but I am happy if I can help in some way (tried to understand the whole Emscripten / OF thing for some time now…).

If you want to try the Emscripten branch with a filesystem, you also need to add the change:

var ENVIRONMENT_IS_AUDIOWORKLET = typeof AudioWorkletGlobalScope === "function";
if(!ENVIRONMENT_IS_AUDIOWORKLET)

take a look at the end of the discussion: https://github.com/emscripten-core/emscripten/pull/12502

and:

and not to forget this change: https://github.com/openframeworks/openFrameworks/pull/6665/commits/311fc18497640835ebfd6b4360c04765c83805f6

One last addition: if you want to use PTHREATS (that you need for AudioWorklets and sharedArrayBuffer) you need to compile (at least) boost and FreeImage (if used) lib with USE_PTHREADS=1 and / or multi thread support (atomics?) (they also work with USE_PTHREADS=0).

1 Like

This is a test with an OF sound example, that works much better (the ofxOfelia addon used in the other patch causes a high cpu load with Emscripten somehow…): https://test.handmadeproductions.de/

1 Like