XPMobileSDK audio delay in sample Audio/audioSample.html, but not in Xprotect Web Client. (Audio sync lip sync issue)

MIP
, ,
1

Hey there, I am currently experiencing audio delay when retrieving the audio stream from the XPMobileSDK (Professional+ 2019R3). There’s a significant, 1 second audio delay between the video and the audio. I am seeing this behaviour in my code, and the issue is also present in the sample found in Audio/audioSample.html (As in 2019R3). However, when viewing my cameras in both Live and Playback mode in the XProtect Web Client, the audio delay is significantly less (about 100/200ms) compared to the sample’s 1 second.

My question is, is the Web Client doing something different than the sample to retrieve the audio stream? Is there anything I can try to do to reduce this delay so it acts as in the XProtect Web Client? Thanks for your help.

2

Hi Daniel,

Yes, there’s a different implementation for playing audio streams in the Web Client. We use the Fetch API and Web Audio API for browsers that support them.

3

Hi Anika, thank you for your answer. Is there any sample in the XPMobileSDK implementing the Fetch API / Web Audio API? Or could you provide one so I can see how it is being done? Thank you very much for your help.

4

Hi Daniel,

Currently there isn’t a sample using Fetch and Web Audio APIs but we are thinking of providing one.

Meanwhile the following code is similar to what we do in the Web Client:

// Check if browser supports MediaSource
if ('MediaSource' in window && MediaSource.isTypeSupported('audio/mpeg'))
{
	var mediaSource = new MediaSource;
	var audio = document.querySelector("audio"); // your audio element
 
	// set source of <audio> to mediaSource
	audio.src = URL.createObjectURL(mediaSource);
	mediaSource.addEventListener("sourceopen", sourceOpen);
 
	function sourceOpen(event) {
 
		var url = "/path/to/audio";
		var sourceBuffer = mediaSource.addSourceBuffer('audio/mpeg');
 
        sourceBuffer.mode = "sequence";
 
        fetch(url)
			// return 'ReadableStream' of 'response'
			.then(response => response.body.getReader())
			.then(reader => {
 
				var processStream = (data) => {
					if (data.done) {
						return;
					}
 
					// append chunk of stream to 'sourceBuffer'
					sourceBuffer.appendBuffer(data.value);
				}
 
				sourceBuffer.addEventListener("updateend", function() {
					//read next chunk of stream, append chunk to 'sourceBuffer'
					reader.read().then(processStream);
				});
 
				// start processing stream
				reader.read().then(processStream);
 
				// read of stream is complete
				return reader.closed.then(() => {
 
					// signal end of stream
					mediaSource.endOfStream();
					return  mediaSource.readyState;
				})
			})
			// do stuff when stream ended
			.then(msg => console.log(msg));
	}
}
5

Hi Anika,

Thank you very much for providing this code sample, it is very much appreaciated. I will try to get it working this way and report my findings back to you. Have a great day!

6

Hi Anika,

I managed to get it working using the sample of code you have provided. This solution completely fixed the audio + video syncing issue for me as they are now completely in sync. I highly suggest that you update the XPMobileSDK audio sample with this code for an upcoming release, since this is clearly the preferred solution. Again, thank you very much for your kind help.

7

I now also understand this only work in browsers that supports MediaSource and the audio/mpeg mime type for the MediaSource object, so it only works on Chrome AFAIK, and this is probably why it is not included in the samples in the XPMobileSDK.