Audio playback issues

MIP
, , , ,
1

Hiya,

I’m using 2017R2 XPCorp via the SDK.

I have an issue with audio/video sync in a stand alone app.

I’m creating 2 image view controls via GenerateImageViewerControl() and to individual Panel controls, and 2 audio player controls via GenerateAudioPlayerControl() which are added to the Form controls - as per examples.

By not setting the PlaybackControllerFQID in the image viewers and audio play back controls ( leaving them as null ), I’m lead to believe that they will all use the ‘default playback controller’ which if is correct then audio and video should sync properly - please correct if I’m wrong here.

I then use a button on screen to begin the playback via the SendMessage function when the button is clicked. Playback of audio and video begins ok, however the audio is some 10 seconds out of sync to the video and the video frames are quite erratic and jumpy - not smooth at all.

Playing back via the smart client works OK and is very smooth with the audio in sync, which leaves me to assume its my code again, and more than likely a setup issue.

I’m initialising things in this order…

display form

[for each camera]

create image viewer, add to a panel control

initialise and connect image viewer control

get related items of the camera to find the microphone

create audio player control, add to form controls

initialise and connect audio playback control

[]

then in the button clicked event

send a play forward message at speed 1.0 to begin playback

Do you have any pointers as to where I should start to look and possibly reorganise things to resolve this issue please?

Thankyou.

2

Are you testing your integration and the Smart Client in the same PC? What you report of video frames being jumpy could be a PC being incapable of decoding an displaying because of shortage of resources. If you test with low resolution images do you still observe the same?

On the decoding it is important whether you can use the hardware acceleration. If you enable the video diagnostics overlay can you see if hardware acceleration is used?

Are you using MIP SDK 2017R2?

3

Hi Bo,

Yes this is all on the same pc. I can have the smart client open and the dev tools open at the same time. I can run my code, stop it, then play in the smart client, all within a few seconds of each other. I can repeatedly go between the 2 application and the problem is the same.

You mention about hardware acceleration. This sounds feasible and leads me to ask if there is a configurable option/setting within the SDK?

4

It is automatically using hardware acceleration in ImageViewerControl, it is only if your MIP SDK is very old you are not using it. So, you will not have to do anything to enable it, but maybe in order to test you could (temporarily) enable the diagnostics overlay.

--

const string VideoOS.Platform.EnvironmentOptions.PlayerDiagnosticLevel = “PlayerDiagnosticLevel”

-

Set this value to number between “0” and “3”, to display developer information on top of ImageViewerCOntrol.

Value of “0” - No diagnostics

Value of “1”, “2”, “3” - Display more and more information about the video stream and the decoding details

If changing the value on the fly - call the EnvironmentManager.Instance.FireEnvironmentOptionsChangedEvent to have the ImageViewerControl pick up the change right away.

5

Hmm,

In the SDK it states that there is the FireEnvironmentOptionsChangedEvent method, you are also mentioning it, but in the dev code its not listed in intellisense and the code doesn’t compile complaining that the method doesn’t exist. I can access the other methods that are mentioned in the SDK for the EnvironmentManager.Instance instance.

6

In the VideoViewerQuality sample there is a CheckBox and this method

private void OnDiagChanged(object sender, EventArgs e)
{
    EnvironmentManager.Instance.EnvironmentOptions["PlayerDiagnosticLevel"] = checkBoxDiag.Checked ? "3" : "0";
    EnvironmentManager.Instance.FireEnvironmentOptionsChangedEvent();
}

Does this sample / snippet of code work for you?

7

I’ve resolved the last issue now. So hopefully we can get back to the original issue.

The diagnostic overlay shows ‘Hardware acceleration: Off’ which could well be the cause?

8

You say the Smart Client does fine, if on the same PC the ImageViewerControl should do the same. So to get that straight, if you enable the same overlay in Smart Client, does it show the same?

In Smart Client go into Settings (Gear icon in the top right), choose the Advanced tab and set Video Diagnostics Overlay.

9

The smart client also shows as ‘Hardware acceleration: Off’.

I can also see discrepancies between my app and the smart client in the diagnostic overlay, but as to how to resolve them…

My app vs smart client:

app - Frames per second: 27 to 29 ( I’ve also tried setting the viewer control fps to 25 )

smart client - 25.

app - Render queue overflow increments rapidly, going to over 300 within a few seconds.

smart client - Render queue overflow: 0.00

app - Render queue underflow count: 13.00

smart client - 1.00

I imagine these number are significant?

10

Ive noticed that I’m getting a lot of exceptions in the debug output, although they seem audio related the audio plays ok.

Exception in Log:Input string was not in a correct format.
nameOfClass=
where=PlaybackControllerBase
text=Timer stopped2: Unable to cast COM object of type 'System.__ComObject' to interface type 'VideoOS.RemoteViewer.AudioPlayerActiveXDotNet.IAudioPlayerActiveXImpl'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{C1037173-964F-49C9-9F35-CC8A4EA4576F}' failed due to the following error: No such interface supported (Exception from HRESULT: 0x80004002 (E_NOINTERFACE)).
16:20:05.381 RendererToolkit.DoPlayback() : 1, 2017-10-12 14:49:12.275, 1
16:20:05.381 RendererToolkit.DoPlayback() : 2, 2017-10-12 14:49:12.275, 1
Exception in Log:Input string was not in a correct format.
nameOfClass=
where=PlaybackControllerBase
text=Timer stopped2: Unable to cast COM object of type 'System.__ComObject' to interface type 'VideoOS.RemoteViewer.AudioPlayerActiveXDotNet.IAudioPlayerActiveXImpl'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{C1037173-964F-49C9-9F35-CC8A4EA4576F}' failed due to the following error: No such interface supported (Exception from HRESULT: 0x80004002 (E_NOINTERFACE)).
Exception in Log:Input string was not in a correct format.
nameOfClass=
where=PlaybackControllerBase
text=Timer stopped2: Unable to cast COM object of type 'System.__ComObject' to interface type 'VideoOS.RemoteViewer.AudioPlayerActiveXDotNet.IAudioPlayerActiveXImpl'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{C1037173-964F-49C9-9F35-CC8A4EA4576F}' failed due to the following error: No such interface supported (Exception from HRESULT: 0x80004002 (E_NOINTERFACE)).
16:20:05.616 RendererToolkit.DoPlayback() : 1, 2017-10-12 14:49:12.656, 1
16:20:05.616 RendererToolkit.DoPlayback() : 2, 2017-10-12 14:49:12.656, 1
Exception in Log:Input string was not in a correct format.
nameOfClass=
where=PlaybackControllerBase
text=Timer stopped2: Unable to cast COM object of type 'System.__ComObject' to interface type 'VideoOS.RemoteViewer.AudioPlayerActiveXDotNet.IAudioPlayerActiveXImpl'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{C1037173-964F-49C9-9F35-CC8A4EA4576F}' failed due to the following error: No such interface supported (Exception from HRESULT: 0x80004002 (E_NOINTERFACE)).
Exception in Log:Input string was not in a correct format.
nameOfClass=
where=PlaybackControllerBase
text=Timer stopped2: Unable to cast COM object of type 'System.__ComObject' to interface type 'VideoOS.RemoteViewer.AudioPlayerActiveXDotNet.IAudioPlayerActiveXImpl'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{C1037173-964F-49C9-9F35-CC8A4EA4576F}' failed due to the following error: No such interface supported (Exception from HRESULT: 0x80004002 (E_NOINTERFACE)).
16:20:05.842 RendererToolkit.DoPlayback() : 1, 2017-10-12 14:49:13.028, 1
16:20:05.843 RendererToolkit.DoPlayback() : 2, 2017-10-12 14:49:13.028, 1
11

I will go a step back, hope you do not mind. What version is the MIP SDK you use? If you run the VideoViewer2WayAudio sample with camera microphone enabled, does it work for you or do you see the same behaviour?

12

Its very similar yes. In playback the audio starts out around 10secs late and progressively gets worse - after 10 seconds of play the audio gets to around 15 secs late. The timing gap increases the longer it is played. It seems that the video is playing too fast.

The diagnostic output shows the ‘Render queue overflow’ as incrementing quite quickly but not as fast as my app does. ‘Render queue underflow’ goes to around 380.

Using that example I just noticed that occassionally there is no audio at all when I click the ‘play’ button to play the video.

I’m using 2017 R2.

13

Could you do two tests please?

Run the client and the sample in another PC.

Run the client and sample towards another camera, preferably a different model or make.

Does the unusual behavior follow the client PC or the camera model? Does it happen regardless?

Also a step back again, I don’t think I got an answer: If you test with low resolution images do you still observe the same?

14

I’ve tried the application on another pc and I see the same inconsistency in audio and video sync. The fps show as ‘Frames per second received: 07 ( 418 - 410 )’.

It’s very strange that the smart client shows everything as ok, also on the second pc.

I’ll try another camera but if it works ok in the smart client I still feel that it’s something that I’m doing wrong in the application?

15

I would suspect the same, but when the sample fails the same way I must suspect that the MIP SDK is not working correctly, at least not with your camera and XProtect VMS setup. So if you could test another camera it would be nice.

(Please let me know the camera make, model and firmware you are working with.)

16

Have you tried just adjusting the codec, resoultion etc. on the camera to the smallest possible so that we can understand if it might be performance that causes the video and audio to get out of sync?

17

I’m using Axis p3364 cameras.

Hmm, it would be fair to say that the issue is related to anything that I build.

I’m using Visual Studio 2017 pro, v15.3.3 as the development environment.

18

Its the same for the small resolutions too.

19

Hiya,

Unfortunately I can’t leave this issue as it is foundation to the project.

I have now removed and reinstalled the 2017R2 SDK, remove and reinstalled Studio2017.

Although I still have errors I feel I’m in a better place to solve the problem.

If I don’t use the audio control the video playback is rock solid stable as it should be. No matter where in the code I create the audio control using

 _audioControls[i] = ClientControl.Instance.GenerateAudioPlayerControl();
Controls.Add(_audioControls[i]);

I have a runtime exception error when it’s added to the controls: ‘0x80040154 REGDB_E_CLASSNOTREG’. I believe this could be a missing DLL file in the applications folder?

In the ‘post build’ process I’m using the batch script ‘CopyImageViewerActiveX.bat’ to copy over the required DLLs. I can confirm that they are copied successfully. Is this all that is required to use the ImageViewerControl and AudioPlayerControl?

20

The AudioPlayerControl uses an activeX (while the ImageViewerControl since MIP SDK 2016R3 is no longer relying on activeX technology).

If you look at the VideoViewer sample project it includes a manifest file. You need to either include this manifest file or register the dll mentioned in that manifest (with regsvr32).

https://developer.milestonesys.com/s/article/Class-not-registered