Delayed camera load from Milestone Server 2023R3 in Angular application

MIP
, , , ,
1

I have an Angular application in which I’m streaming video feed from a Milestone server. This app was originally built in 2018 to stream from Milestone 2018R1, and it uses the 2018R1 Javascript/Web SDK. I’m upgrading the app to work with Milestone 2023R3, and I’d like to modify the app as little as possible.

I noticed that the app works fine with the 2023R2 server, even using the old 2018R1 SDK. Camera feed loads quickly–I get video frames in about a second or less. However, if I point the same app and old SDK at the 2023R3 server (on the same local network), it takes around 4 seconds to receive any frames from the server, and the first frames are empty.

Here are my questions.

  1. What changed on the Milestone server between 2023R2 and 2023R3 that leads to this longer camera load time and the initial empty frames?
  2. Would updating the SDK to 2023R3 somehow fix the problems I’m having with the 2023R3 server? How so?
  3. Where do I get the updated SDK? Is it the Lib folder in this repository: https://github.com/milestonesys/mipsdkmobile-web?

Thank you for your time!

2

Update: I’ve implemented the SDK present in the 2023R3 commit on https://github.com/milestonesys/mipsdkmobile-web, and I still am experiencing the increased load time on the 2023R3 server but not the 2023R2 server. That answers question 2.

These are two different servers and likely are configured differently. So here’s a fourth question.

4. Could a config setting be causing the delay in camera feed initialization or the initial empty frames?

3

Hi @Benjamin Dunham​,

> Where do I get the updated SDK? Is it the Lib folder in this repository: https://github.com/milestonesys/mipsdkmobile-web?

Yes, this is the proper place to get the SDK sources.

> Could a config setting be causing the delay in camera feed initialization or the initial empty frames?

Yes, that’s probably the case.

> What changed on the Milestone server between 2023R2 and 2023R3 that leads to this longer camera load

I think no changes have been made between 2023R2 and 2023R3 on the server side, causing the described difference in behavior. Therefore, I’d like to ask you for some clarification on the following:

  • Does the problem (slow video load) exist when using XProtect Web Client? What about XProtect Mobile Client?
  • Does the problem affect live or playback? If Live - is this applicable on Transcoding, or just Direct Stream?
  • Does this happen with any camera (e.g. with any codec, FPS, resolution)?
  • When exactly is the delay? I guess it’s between the response of RequestStream command and the first (non-empty) frame. Can you confirm?

Regards,

Nikolay

4

Hi @Nikolay Dimitrov​ ,

Thank you for reaching out! Here are my answers to your questions.

> Does the problem (slow video load) exist when using XProtect Web Client? What about XProtect Mobile Client?

Yes, the slow video load and empty frames also occur when using XProtect Web Client on the 2023R3 server, but not on the 2023R2 server. I’m not familiar with the XProtect Mobile Client.

> Does the problem affect live or playback?

This problem only affects live feed. Playback works fine on the 2023R3 server in the web client. I haven’t tried it in the app.

> If Live - is this applicable on Transcoding, or just Direct Stream?

In XProtect Web Client, Direct Stream gives a 5-8 second lag (including empty frame time). When I switch to transcoding in XProtect Web Client, the lag drops to 3.5-4 seconds. However, in the app, lag stays around 4-5 seconds whether Direct Stream is on or off. (I’m assuming you turn off Direct Stream in the app by setting DirectStreaming to false in XPMobileSDKSettings in XPMobileSDK.js.)

> Does this happen with any camera (e.g. with any codec, FPS, resolution)?

On the 2023R3 server, we have three AXIS Q3536-LVE Dome Cameras connected. Video Stream 1 has Codec- H.265 and Resolution- 2688 x 1512. Video Stream 2 has Codec- H.265 and Resolution- 1280 x 720.

On the 2023R2 server, we have cameras with the following specs. All load fine.

  • Camera 1
    • Video Stream1: H.265, 6096x2540
    • Video Stream2: H.265, 2560x1064
  • Camera 2
    • Video Stream1: H.264, 2592x1944
    • Video Stream2: H.264, 640X480
  • Camera 3
    • Video Stream1: H.264, 1920x1080
    • Video Stream2: H.264, 1280x720
  • Camera 4
    • Video Stream1: H.264, 1920X1080
    • Video Stream2: H.264, 1920X1080

> When exactly is the delay? I guess it’s between the response of RequestStream command and the first (non-empty) frame. Can you confirm?

Yes, the delay occurs just after the RequestStream call. When the first empty frame comes in, I see the following console log in the app: “Server prepared video ID … for camera …”. Then after a second or two, the non-empty frames start coming in. Interestingly enough, there is also a delay on the login call, between the call to log in and the response.

Thanks again for your help!

5

Here’s another question, @Nikolay Dimitrov​ . You mentioned that you thought a setting on the 2023R3 server is likely causing the problem. What config settings should we adjust to try to fix the problem?

6

> Yes, the slow video load and empty frames also occur when using XProtect Web Client on the 2023R3 server, but not on the 2023R2 server

This means it’s not an issue in your app, which is good news.

> What config settings should we adjust to try to fix the problem?

When I said the problem is probably caused by a setting, I meant that somewhere in the product there must be a difference in settings between 2023R2 and 2023R3. That may include camera codecs, resolution, streaming type, etc. The reason for saying that is - there is no difference in the server-side streaming between the last two versions of the product.

Please navigate to Management Client → Mobile Servers → → Performance tab and make sure all settings there are the same between your two setups.

> Interestingly enough, there is also a delay on the login call, between the call to log in and the response.

This is very fishy! Does this setup contain a lot of users or a lot of cameras, views, etc?

Can you please make sure that CPU load is not close to 100%?

> Does this happen with any camera (e.g. with any codec, FPS, resolution)?

From your answer to this, I can see that all cameras on the 2023R3 setup are H265. As far as I know, browsers do not currently support showing direct H265 feed, therefore after an initial attempt, streaming eventually defaults to Transcoding. That’s the reason why in your tests Direct Streaming attempt is slower than Transcoding.

One more idea I can propose is to configure a standalone 2023R3 Mobile Server that you connect to your 2023R2 system (without messing with the existing 2023R2 Mobile Server that’s already part of your setup). If the issue is in the product (Mobile Server or SDK 2023R3), then this will have the same behavior as your other system. But my guess is that it behave just as fine as 2023R2.

7
  1. Does this setup contain a lot of users or a lot of cameras, views, etc?
    1. Minimal cameras, users, and views. 3 cameras active. 165 total. Could not having cameras connected and the server trying to connect to the not connected cameras be causing a delay?
  2. Can you please make sure that CPU load is not close to 100%?
    1. CPU Load is 6%
  3. Please navigate to Management Client → Mobile Servers → → Performance tab and make sure all settings there are the same between your two setups.
    1. Both setups are the same
8

> Could not having cameras connected and the server trying to connect to the not connected cameras be causing a delay?

That’s a good guess but I don’t think so. Login may be delayed by this (although I doubt it), but not the streaming.

Can you try open Web Client on the same machine as the 2023R3 setup and check if this is still slow?

> Both setups are the same

An easy check is to try disabling direct streaming and adaptive streaming and see if this makes any difference.

image
image292×130 3.19 KB

Did you manage to connect a new Mobile Server to your old VMS and see how it behaves?

9

>Can you try open Web Client on the same machine as the 2023R3 setup and check if this is still slow

It’s still slow

>An easy check is to try disabling direct streaming and adaptive streaming and see if this makes any difference

We tried this and it doesn’t seem to make a difference.

>Did you manage to connect a new Mobile Server to your old VMS and see how it behaves?

We haven’t been able to try this because we don’t have another high perf. GPU server laying around to try.

Would it be possible to setup a remote session for you to take a look?

10

> We haven’t been able to try this because we don’t have another high perf. GPU server laying around to try.

It doesn’t even have to be high-performant. Any decent machine should be enough for a single stream.

> Would it be possible to setup a remote session for you to take a look?

This is not how we usually proceed with forum requests, but it’s fine by me.

Please send connection details via mail to nid@milestone.dk.

One more thing I’d like to emphasise. If there was such a huge difference in performance between 2023R2 and 2023R3, a lot of customers should have complained immediately after releasing the issue… and still you’re the first one I know of. So I’m heavily leaning towards a network/configuration issue. I just don’t see anything particularly suspicious in your description so far.

11

Nikolay

  • Ok, we will setup a new Mobile Server to see if there is a performance difference
  • It will need to be a teamviewer connection that we can schedule for early next week. I will send you the details.
  • I agree this is strange and I hope it’s as simple as a configuration issue. But it’s certainly not an obvious one.
12

Hi Benjamin,

Could you please describe the overall system setup. Does Management Server, Recording Server, Mobile Server all on the same machine or is it a distributed installation? When you say that you migrated to 2023R2/R3 server - is it the whole system or just Mobile Server. When you test 2023 R2 or 2023 R3 Mobile server - are those different machines ? Is there difference on the machine 2023R2 and 2023 R3 are installed (CPU \ GPU \ Network Adapter \ OS)

Do you use secure or unsecure communication (http\ https - have you setup certificates?). If you are using http or it is possible to turn off https , could you please make wireshark traces on the following communications while you experience the issue:

  • between your client (or web client) and Mobile Server (both 2023R2 and 2023 R3)
  • between Mobile Server and Management \ Recording Server (both 2023R2 and 2023 R3)

Provide logs of the Mobile Server and MIPSDK (both 2023R2 and 2023R3) with date and timestamp when you reproduced the issue ([C:\ProgramData\Milestone\MIPSDK,](file:C:/ProgramData/Milestone/MIPSDK,) [C:\ProgramData\Milestone\XProtect](file:C:/ProgramData/Milestone/XProtect) Mobile Server)

13

Hey there SDK Team,

Wanting to provide you all with an update on our issue. Here’s a recap, we’ve got a new 2023 R3 system with separate Mgmt, Mobile, and Recording servers.
We’re experiencing a significant delay when accessing cameras using a developed ‘SDK widget’ and while trouble shooting noticed a similar delay in retrieving cameras through the Web Client.
We do not experience this delay with the same ‘SDK Widget’ when retrieving cameras from our own internal 2023 R2 Milestone server that is an all-in-one Mgmt, Mobile, and Recording server.
You asked for WireShark data of the troubled setup. Attached is that data.

Here is the IP info:
Mobile server address is 10.149.109.10
Mgmt Server address 10.149.109.9
Recording Server1 address 10.149.109.11
Recording Server2 address 10.149.109.12
Client using SDK for web based camera callup address is 10.149.109.252

I will continue to work to get you the other log files you asked for.
We really appreciate your assistance here.

Thanks

B. J. Dunham
Vice-President / CTO
Security Automation Systems, Inc.
8739 Castle Park Drive
Indianapolis, IN 46256
Phone: 317-489-9621, Ext. 802
Fax: 317-489-9621
Toll-Free: 877-SAS-FORYOU
[email logo]

This message is from Security Automation Systems, Inc. This message and any attachments may contain confidential information, and are intended only for the individual or entity identified above as the addressee, and may not be distributed. If you are not the addressee, or if this message has been addressed to you in error, you are not authorized to read, copy, or distribute this message and any attachments, and we ask that you please delete this message and attachments (including all copies) and notify the sender by return e-mail. All personal messages express views only of the sender, which are not to be attributed to Security Automation Systems, Inc., and may not be copied or distributed without this statement.

14

I can see no WireShark data attached here…

Please upload *.pcapng file containing captured network traffic during login and stream requests.

15

Sorry I attached the file to the email reply, not in the forum. Here you go.

16

Replying to noreply@milestonesys.com is pointless - no one will receive it.

I had a look at your WireShark file, and it has a lot of ‘Bad TCP’ packets (the one with red font on a black background). This may be a problem, but I know nothing more.

Consider upgrading in-place your 2023R2 Mobile Server to 2023R3 (just the Mobile Server, keep everything else the same), to isolate any possible issues with your network configuration.

From now on, I’ve nothing more to do. Once more - this works like a network issue or a configuration issue and not like a developer question (e.g. how to integrate this, how to use that, etc.).

You can try with the Support Community or contact Milestone Tech Support directly.