Consolidated Mach1 spatial audio player focused on collecting all headtracking input methods and defining best practices
List of already deployed demos utilizing concepts of Mach1 Spatial and facetracking
A consolidated web spatial audio player for Mach1 Spatial playback with two goals:
./download-audiofiles.sh
(or ./download-audiofiles.bat
if on WIN) to download some prepared debug playback audioWe have exposed some settings to extend usability of the facetracker mode.
yawMultiplier
- Multiplies the yaw rotation to allow experimentation with extending the limited range of motion from "facetracking", making it 1.0
is recommended for natural headtracking however raising it up to 2.0
is still believable in terms of associating the audio results with your headtracking motion; anything past 2.0
breaks immersionpitchMultiplier
- Multiplies the pitch rotation to allow experimentation with extending the limited range of motion from "facetracking", we found it to be best to keep it as close to 1.0
as possiblerollMultiplier
- Multiplies the roll rotation to allow experimentation with extending the limited range of motion from "facetracking", we found it to be best to keep it as close to 1.0
as possibleFOV
- We clamp the yaw range from "facetracking" to lessen low confidence results from causing large rotation jumps when unable to view full facefilterSpeed
- An additional filter to smooth results applied to Mach1Decode during audio playback to stereooneEuroFilterBeta
- The filter beta for the 1Euro implementationAfter exploring several web based facetrackers we have had the most success with MediaPipe FaceMesh. Landmark tracking with FaceMesh wasn't designed for us to calculate orientation rotations to be applied to something as sensitive as spatial audio playback, so we had to apply a filter to the motion results to "smooth" out any low frequency jitters. This can be seen via the 1Euro filter implementation:
window.createOneEuroFilters = function createOneEuroFilters() {
fYaw = OneEuroFilter(60, 1.0, window.controls.oneEuroFilterBeta, 1.0);
fPitch = OneEuroFilter(60, 1.0, window.controls.oneEuroFilterBeta, 1.0);
fRoll = OneEuroFilter(60, 1.0, window.controls.oneEuroFilterBeta, 1.0);
};
Running this demo on a mobile device will now use the device's IMU sensor with gimbal rotations to ensure holding in portrait mode will have a smooth and expected full 3DOF rotations.
When utilizing Touch
mode a simple mouse control bound to width of window will be used as the input orientation for the Mach1Decode
object(s) and for the OSC output.
When utilizing Bose AR
mode a connection prompt and handler supplied by the BoseAR Web SDK will be utilized for the input orientation for the Mach1Decode
object(s) and for the OSC output.
Use of the Mach1Decode API in this example falls under the license supplied in this repo and derived from the Mach1 Spatial SDK viewable here