EventServer not listening on 22333 port

0053X00000H7BRzQAN · October 2, 2023, 3:58pm

Hi,

We are using Milestone SDK to listen for the alarms. Unfortunately, one of our clients is unable to make it work with Milestone Professional+ 2022 R1. It waits in communication.WaitForCommunicationFilterRegistration(communicationIdFilter, _communicationFilterRegisterTimeout) for the whole timeout (60 seconds).

After gathered logs it seems that Event Server is not listening on 22333 port:

2023-09-26 11:28:20.976 Debug: CommunicationClient (): Transmit - MessageCommunication.NewEndPointIndication
2023-09-26 11:28:20.977 Debug: CommunicationClient (): InitSession - bc13faf6-89fb-4eaa-946a-74abd19397de
2023-09-26 11:28:22.981 Debug: CommunicationClient (): Transmit retry - MessageCommunication.NewEndPointIndication
2023-09-26 11:28:24.985 Debug: CommunicationClient (): Transmit exception - Could not connect to net.tcp://hostname:22333/CommunicationService. The connection attempt lasted for a time span of 00:00:02.0013549. TCP error code 10061: No connection could be made because the target machine actively refused it 1.2.3.4:22333.

2023-09-26 11:28:23.9828|Exception occurred.|System.Net.Sockets.SocketException (0x80004005): No connection could be made because the target machine actively refused it [::1]:22333
   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
2023-09-26 11:28:24.9841|Exception occurred.|System.Net.Sockets.SocketException (0x80004005): No connection could be made because the target machine actively refused it 1.2.3.4:22333

We checked if Event Server is listening using:

tasklist | find /i "VideoOS.Event.Server.exe"
netstat -aon | find "<PID of Event Server>"

and port 22333 is not listed here.

Do you know what can be the reason or how can we debug this issue?

Bo_Ellegard_Andersen · October 3, 2023, 7:40am

I am wondering if your event server has had a configuration change so that it uses another port.

https://developer.milestonesys.com/s/article/Event-Server-port-configuration

The port is actually (EVS port +2). Default this would mean 22331+2 = 22333, but if not using default port then this port is another.

First step check in Management Client under Tools → Registered Services the port.

0053X00000H7BRzQAN · October 3, 2023, 3:39pm

Under Registered Services it looks normal - Event Server has http://hostname:22331/ URL, I’m not sure what external addresses was here, I’ll ask about that and give update.

Bo_Ellegard_Andersen · October 4, 2023, 8:39am

Next step try to inspect the Event Server logs. Does the event server service run without issues or does the logs report a problem?

Event Server logs are at [C:\ProgramData\Milestone\XProtect](file:C:/ProgramData/Milestone/XProtect) Event Server\logs

Actually, if the service does not run at all it would explain the issue, if it isn’t running the logs might reveal why.

I don’t think the error fits a bad certificate or similar but I would like to know whether you have setup secure communication with encryption on the event server in the Server Configurator. Is encryption enabled on “event server and add-on”?

0053X00000H7BRzQAN · October 6, 2023, 3:12pm

Event Server configuration:

Ports listed by netstat -aon | find “”:

TCP    0.0.0.0:1234           0.0.0.0:0             LISTENING      2708
 TCP    0.0.0.0:9090           0.0.0.0:0             LISTENING      2708
 TCP    10.65.31.103:443       10.65.63.117:52708    ESTABLISHED    4
 TCP    10.65.31.103:49339     10.65.31.196:1433     ESTABLISHED    2708
 TCP    10.65.31.103:49345     10.65.31.196:1433     ESTABLISHED    2708
 TCP    10.65.31.103:49346     10.65.31.196:1433     ESTABLISHED    2708
 TCP    10.65.31.103:49357     10.49.22.2:7563       ESTABLISHED    2708
 TCP    10.65.31.103:49358     10.65.41.101:7563     ESTABLISHED    2708
 TCP    10.65.31.103:49359     10.65.80.25:7563      ESTABLISHED    2708
 TCP    10.65.31.103:49361     10.65.2.81:7563       ESTABLISHED    2708
 TCP    10.65.31.103:49362     10.65.51.101:7563     ESTABLISHED    2708
 TCP    10.65.31.103:49363     10.48.224.67:7563     ESTABLISHED    2708
 TCP    10.65.31.103:49365     10.49.6.21:7563       ESTABLISHED    2708
 TCP    10.65.31.103:49366     10.65.71.101:7563     ESTABLISHED    2708
 TCP    10.65.31.103:49367     10.65.61.101:7563     ESTABLISHED    2708
 TCP   10.65.31.103:49401     10.65.31.12:389       ESTABLISHED    2708
 TCP    10.65.31.103:49404     10.65.31.12:389       ESTABLISHED    2708
 TCP    10.65.31.103:49418     10.65.31.196:1433     ESTABLISHED    2708
 TCP    10.65.31.103:49419     10.65.31.196:1433     ESTABLISHED    2708
 TCP    10.65.31.103:49420     10.65.31.196:1433     ESTABLISHED    2708
 TCP    10.65.31.103:49421     10.65.31.196:1433     ESTABLISHED    2708
 TCP    10.65.31.103:49422     10.65.31.196:1433     ESTABLISHED    2708
 TCP    10.65.31.103:49938     10.51.16.71:7563      SYN_SENT       2708
 TCP    127.0.0.1:49927        127.0.0.1:60000       ESTABLISHED    2708
 TCP   [::]:9090             [::]:0                LISTENING      2708
 TCP   [::1]:49334           [::1]:80              ESTABLISHED    2708
 TCP   [::1]:49397           [::1]:22331           ESTABLISHED    2708
 TCP   [::1]:49407           [::1]:22337           ESTABLISHED    2708
 TCP   [::1]:49408           [::1]:22337           ESTABLISHED    2708
  TCP   [::1]:49409           [::1]:22337           ESTABLISHED    2708
 UDP    0.0.0.0:1234          *:*                                   2708
 UDP    127.0.0.1:50036       *:*                                   2708

I reviewed Event Server logs from locations

C:\ProgramData\Milestone\XProtect Event Server\logs
and 
C:\ProgramData\Milestone\XProtect Event Server\logs\MIPLogs

It looks like I noted version wrong and it is Professional+ 2020R1, not 2022:

2023-10-05 00:00:00 UTC-03:00  Info     Log                           -------------------------- Version: 20.1.26.1
2023-10-05 00:00:00 UTC-03:00  Info     Log                           Available MAC address:<MAC ADDRESS>
2023-10-05 00:00:00 UTC-03:00  Info     Log                           Local time is now: 00:00:00
2023-10-05 00:00:00 UTC-03:00  Info     Log                           UTC time is now: 03:00:00

I didn’t find any suspicious logs, only the ones that caught my attention:

2023-10-06 02:34:47 UTC-03:00  Error    StatusSessionRecorder         Error communicating with: sthmer0801.d1.ad.local:7563 with message: No connection could be made because the target machine actively refused it 10.51.16.71:7563
Exception details:
System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.51.16.71:7563
   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at VideoOS.Event.Server.Communication.Corporate.RecorderStatusService2Central.CheckTokenRefreshOrRethrowException(Exception e)
   at VideoOS.Event.Server.Communication.Corporate.RecorderStatusService2Central.StartStatusSession()
   at VideoOS.Event.Server.Communication.Corporate.StatusSessionRecorder.Run()
 
2023-10-06 04:48:49 UTC-03:00  Error    StatusSessionRecorder         Error communicating with: sthsfo0810.d1.ad.local:7563 with message: An existing connection was forcibly closed by the remote host
Exception details:
System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.PooledStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.Connection.SyncRead(HttpWebRequest request, Boolean userRetrievedStream, Boolean probeRead)
   --- End of inner exception stack trace ---
   at VideoOS.Event.Server.Communication.Corporate.RecorderStatusService2Central.CheckTokenRefreshOrRethrowException(Exception e)
   at VideoOS.Event.Server.Communication.Corporate.RecorderStatusService2Central.GetVideoDeviceStatistics(Guid[] ids)
   at VideoOS.Event.Server.Communication.Corporate.StatusSessionRecorder.Run()

I feel like they are not case here.

0053X00000H7BRzQAN · October 6, 2023, 3:16pm

I’ll ask about encryption.

0053X00000H7BRzQAN · October 17, 2023, 2:21pm

These settings can be found under “Management Server ->Change encryptions settings…”

The client doesn’t have “Management Server->Server Configurator…” in the tray menu.

@Bo Ellegård Andersen (Milestone Systems)

What can be the next steps to debug this issue? Or what is the recommended way to proceed in this situation?

Bo_Ellegard_Andersen · October 18, 2023, 9:43am

There is something wrong, 2022R1 should have a Server Configurator and encrypted communication. Might this system be older? (Or a bad upgrade?)

I suspect you have seen and used.. https://developer.milestonesys.com/s/article/Ports-for-alarms-from-Milestone-Event-Server-service

I suspect a general setup issue rather than there is anything related to the MIP SDK usage. The error you note about port 7563 is about the event server communicating with the Recording Server, it could be another symptom of the same root issue. I do not take pleasure in sending you elsewhere but there might be better help in the general support.. https://supportcommunity.milestonesys.com

0053X00000H7BRzQAN · October 18, 2023, 3:39pm

As I wrote, it looks like I wrote down the version incorrectly and it is Professional+ 2020R1, not 2022:

2023-10-05 00:00:00 UTC-03:00  Info     Log                           -------------------------- Version: 20.1.26.1

For encryption, does it matter if the version is older and from 2020?

Bo_Ellegard_Andersen · October 23, 2023, 1:21pm

2020R1 does not have Encryption. This means that this is not the source of the issue.

One possible cause is that the installation is split so that the Event Server is actually on another physical server not on the same as the Management Server service. Seems a long shot but it could be the explanation, please see if that is the explanation.

0053X00000H7BRzQAN · October 24, 2023, 10:26am

Event Server is on the same machine as the Management Server. I have checked that.

Testing server

OS: Windows Server 2016 Standard
Milestone: XProtect Management Client 2019 R3 – Note: server has local recordings in the same machine

Production Server

OS: Windows Server 2016 Standard
Milestone: XProtect Management Client 2020 R1 – Note: Does NOT have local recording because it’s the Master server. Each of the camera’s recordings are in each of the recording servers

Can the Recording Server affect the port on which the Event Server is listening? On the testing server, everything works correctly (Event Server listens on port 22333), but not on production.