It seems we have found an issue that Milestone XProtect only processes the first generic event in a TCP stream.

MIP
, , ,
1

We have configured a data source over TCP for no echo, and the default separator bytes of carriage return, line feed. Generic events have been added to the server.

When sending a string containing two generic events separated by 0x0D 0x0A, only the first triggers an alarm in the Smart Client. Reordering shows the alternate generic event triggering, but never both - even with a trailing 0x0D 0x0A after the second generic event.

Packet capture logs show that all bytes sent were acknowledged (TCP ack), and the payload contained the expected data.

For testing, the generic events were manually generated on a Linux system in the following fashion:

$ echo -e “\r\n\r” | netcat

where

are substituted with appropriate values. Note that the last newline is omitted as echo automatically appends that.

This issue prevents us from establishing a long-term TCP connection with the XProtect server, sending generic events as they occur.

2

Please check the setup in Management client → Tools → Options - Generic Events

GenericEvent
GenericEvent578×480 61.7 KB
If “separator bytes” is not configured it will only register one event per TCP session.

3

Hi Kito. We had those settings already in place.

4

Hi Jeffrey,

Can you please try unmodified Trigger GenericEvent Stream sample if it works for you?

https://doc.developer.milestonesys.com/html/index.html?base=samples/triggergenericevent_samplestream.html&tree=tree_3.html

5

Hi Jeffrey,

In my misguided attempt to build an example to show you why it works as expected, I may have found a bug where when the Event Server reads from the stream, and the buffer doesn’t end with the separator bytes, the message gets evaluated anyway, and then the connection gets closed by the server.

Usually small messages will arrive in their entirety, so a call to .Read(…) will result in the entire message being received in one go. But for various reasons messages can be broken apart and require multiple reads to receive the whole message.

To force this behavior, I made a small script to connect to the Event Server and send a few test strings. The first four end with a CRLF while the 5th one is only the first few characters, with the rest of the message and the CRLF in the 6th message.

I can predictably send multiple messages and have the event server evaluate them, and I can see the events in Smart Client as they arrive without having to reconnect to send each message. But after sending the 5th message, the one which does not end in a CRLF, I receive the statistic response from the Event Server even though it shouldn’t have evaluated my message yet. The 6th message is sent successfully but then when I try to read from the connection, I find that the connection was closed.

I first noticed something was wrong when I was using the classic telnet.exe tool and as soon as I sent a single character the connection was terminated. I couldn’t trigger a generic event at all from telnet because the connection would be terminated with a single keystroke.

I know it hasn’t always been this way - I’m running the 2020 R3 REST API beta build of XProtect Corporate and haven’t had a chance to test on a prior release to see if there’s a recent version without the same behavior. I’ll open a bug report this afternoon.

I attached my hastily written PS1 script which I used to reproduce the behavior for reference. My guess is that you’re specifically seeing the server ignore the second message because netcat is sending the string you provided, then following up with a separate write to the string with the \n character, so the Event Server is receiving the “incomplete” message in a single read from the stream and it would receive the final \n if it read from the stream one more time, but instead it evaluates the incomplete second message and closes the stream after receiving the rest of the message.

PS C:\Users\Administrator> C:\Users\Administrator\Desktop\GenericEventTest.ps1
VERBOSE: Sending 'Test1\r\n'
VERBOSE: Statistic Reponse: 29,5,1,Test1
 
VERBOSE: Sending 'Test2\r\n'
VERBOSE: Statistic Reponse: 30,5,1,Test2
 
VERBOSE: Sending 'Test1\r\n'
VERBOSE: Statistic Reponse: 31,5,1,Test1
 
VERBOSE: Sending 'Test2\r\n'
VERBOSE: Statistic Reponse: 32,5,1,Test2
 
VERBOSE: Sending 'Tes'
VERBOSE: Statistic Reponse: 33,3,0
 
VERBOSE: Sending 't1\r\n'
VERBOSE: We don't know it yet, but the Event Server closed the connection on its end after receiving that last message
Exception calling "Read" with "3" argument(s): "Unable to read data from the transport connection: An established connection was aborted by the software in your host machine."
At C:\Users\Administrator\Desktop\GenericEventTest.ps1:21 char:13
+             $bytes = $stream.Read($buffer, 0, $buffer.Length)
+             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : IOException
6

Hi Josh,

Thank you for following up with us.

We are particularly interested in more than one complete message in a buffer.

The test you have mentioned is also needed (testing partial messages in the buffer), but we are specifically targeting multiple messages in the buffer.

To copy paste from above, your test is:

Sending ‘Test1\r\n’

Sending ‘Test2\r\n’

Our problem was:

Sending ‘Test1\r\nTest2\r\n’

In the time between XProtect reads from the socket buffer, the operating system may receive multiple ethernet frames, in which case many messages may be queued for processing. The end of the buffer could be exactly on a message boundary or splicing a message, as you have pointed out above [Sending ‘Tes’], but the processing must iteratively search for another 10,13 pair and process additional generic events if available.

Our original test sent two messages in one Ethernet packet, which would arrive together. The second one was never processed by XProtect - and we verified that both generic events worked by reordering the two. In each scenario, only the first generic event received was processed.

The scenario should scale and it should accept that there might be a tail on the end.

Three examples of what XProtect should accept:

Sending ‘Test1\r\nTest2\r\nTes’ (later receiving ‘t1\r\n’ - as the message would complete later)

Sending ‘Test1\r\nTest2\r\nTest1\r\n’ (and Test1 must fire twice)

Sending ‘Test1\r\nTest2\r\nTest3\r\n…’

All of these scenarios are necessary to support TCP transport.