I am integrating a H.264 feed from RTSP into milestone for a custom driver and we are indeed reading the frames and sending them correctly to milestone but we aren’t seeing any video on the management client.
Below is the code and logs for reference.
private ConcurrentQueue<CustomFrame> _queue = new ConcurrentQueue<CustomFrame>();
private DateTime _lastSyncTimestamp;
private async void start()
{
CancellationTokenSource source = new CancellationTokenSource();
CancellationToken token = source.Token;
Toolbox.Log.LogDebug("Driver", "Starting");
Uri rtspurl = new Uri("rtsp://####/url");
string _userName = "#####";
string pword = "#####";
var credentials = new NetworkCredential(_userName, pword);
var connectionParameters = new ConnectionParameters(rtspurl, credentials);
connectionParameters.RtpTransport = RtpTransportProtocol.TCP;
connectionParameters.ReceiveTimeout = TimeSpan.FromSeconds(120);
using (var rtspClient = new RtspClient(connectionParameters))
{
CustomFrame customFrame;
rtspClient.FrameReceived += (sender, frame) =>
{
switch (frame)
{
case RawH264IFrame iframe:
_sequence++;
_lastSyncTimestamp = iframe.Timestamp;
Toolbox.Log.LogDebug("Driver", "New IFrame");
customFrame = new CustomFrame(
_sequence, iframe.FrameSegment.Array, _lastSyncTimestamp, iframe.Timestamp, true
);
_queue.Enqueue(customFrame);
break;
case RawH264PFrame pframe:
_sequence++;
Toolbox.Log.LogDebug("Driver", "New PFrame");
customFrame = new CustomFrame(
_sequence, pframe.FrameSegment.Array, _lastSyncTimestamp, pframe.Timestamp, false
);
_queue.Enqueue(customFrame);
break;
}
};
await rtspClient.ConnectAsync(token);
await rtspClient.ReceiveAsync(token);
}
}
protected override bool GetLiveFrameInternal(TimeSpan timeout, out BaseDataHeader header, out byte[] data)
{
header = null;
data = null;
if (_queue.Count <= 0)
{
return false;
}
CustomFrame frame;
if (_queue.TryDequeue(out frame) == false)
return false;
data = frame.Data;
if (data == null || data.Length == 0)
{
return false;
}
Toolbox.Log.LogError(this.GetType().Name, "Received Data of length: " + (ulong)data.Length + " Sequence: " + _sequence + " " +
"sync time: " + frame.SyncTimestamp.Ticks + " frame time: " + frame.FrameTimestamp.Ticks);
header = new VideoHeader()
{
CodecType = VideoCodecType.H264,
Length = (ulong)data.Length,
SequenceNumber = frame.Sequence,
SyncFrame = frame.Keyframe,
TimestampSync = frame.SyncTimestamp,
TimestampFrame = frame.FrameTimestamp
};
return true;
}
Logs:
2024-03-04 16:35:35.098-08:00 INFO - ******************************* Starting logger *******************************
2024-03-04 16:35:35.106-08:00 DEBUG - ----- Process start Command = SingleDriverMode; DllPath = C:\Program Files\Milestone\MIPDrivers\DemoDriver\DemoDriver.def
2024-03-04 16:35:35.326-08:00 ERROR - DemoConnectionManager: In Connect method!
2024-03-04 16:35:35.374-08:00 ERROR - DemoConnectionManager: Connected!!!
2024-03-04 16:35:37.994-08:00 INFO - ******************************* Starting logger *******************************
2024-03-04 16:35:38.001-08:00 DEBUG - ----- Process start Command = SingleDriverMode; DllPath = C:\Program Files\Milestone\MIPDrivers\DemoDriver\DemoDriver.def
2024-03-04 16:35:38.223-08:00 ERROR - DemoConnectionManager: In Connect method!
2024-03-04 16:35:38.274-08:00 ERROR - DemoConnectionManager: Connected!!!
2024-03-04 16:35:42.304-08:00 INFO - ******************************* Starting logger *******************************
2024-03-04 16:35:42.311-08:00 DEBUG - ----- Process start Command = SingleDriverMode; DllPath = C:\Program Files\Milestone\MIPDrivers\DemoDriver\DemoDriver.def
2024-03-04 16:35:42.609-08:00 ERROR - DemoConnectionManager: In Connect method!
2024-03-04 16:35:42.679-08:00 ERROR - DemoConnectionManager: Connected!!!
2024-03-04 16:35:43.061-08:00 ERROR - DemoConnectionManager: Starting LS!
2024-03-04 16:35:43.061-08:00 ERROR - DemoConnectionManager: In ThrowIfNotConnected!
2024-03-04 16:35:43.066-08:00 INFO - Driver: Starting
2024-03-04 16:35:43.404-08:00 INFO - Driver: New IFrame
2024-03-04 16:35:43.405-08:00 ERROR - DemoVideoStreamSession: Received Data of length: 74125 Sequence: 1 sync time: 638451957434026940 frame time: 638451957434026940
2024-03-04 16:35:43.465-08:00 INFO - Driver: New PFrame
2024-03-04 16:35:43.485-08:00 ERROR - DemoVideoStreamSession: Received Data of length: 74125 Sequence: 2 sync time: 638451957434026940 frame time: 638451957434436940
2024-03-04 16:35:43.508-08:00 INFO - Driver: New PFrame
2024-03-04 16:35:43.509-08:00 INFO - Driver: New PFrame
2024-03-04 16:35:43.509-08:00 ERROR - DemoVideoStreamSession: Received Data of length: 74125 Sequence: 4 sync time: 638451957434026940 frame time: 638451957434856940
2024-03-04 16:35:43.511-08:00 ERROR - DemoVideoStreamSession: Received Data of length: 74125 Sequence: 4 sync time: 638451957434026940 frame time: 638451957435276940
2024-03-04 16:35:43.554-08:00 INFO - Driver: New PFrame
2024-03-04 16:35:43.556-08:00 ERROR - DemoVideoStreamSession: Received Data of length: 74125 Sequence: 5 sync time: 638451957434026940 frame time: 638451957435686940
2024-03-04 16:35:43.614-08:00 INFO - Driver: New PFrame
2024-03-04 16:35:43.615-08:00 ERROR - DemoVideoStreamSession: Received Data of length: 74125 Sequence: 6 sync time: 638451957434026940 frame time: 638451957436106940
2024-03-04 16:35:43.662-08:00 INFO - Driver: New PFrame
2024-03-04 16:35:43.662-08:00 ERROR - DemoVideoStreamSession: Received Data of length: 74125 Sequence: 7 sync time: 638451957434026940 frame time: 638451957436526940
2024-03-04 16:35:43.710-08:00 INFO - Driver: New PFrame
2024-03-04 16:35:43.711-08:00 ERROR - DemoVideoStreamSession: Received Data of length: 74125 Sequence: 8 sync time: 638451957434026940 frame time: 638451957436936940
2024-03-04 16:35:43.763-08:00 INFO - Driver: New PFrame
2024-03-04 16:35:43.764-08:00 ERROR - DemoVideoStreamSession: Received Data of length: 74125 Sequence: 9 sync time: 638451957434026940 frame time: 638451957437356940
2024-03-04 16:35:43.830-08:00 INFO - Driver: New PFrame
2024-03-04 16:35:43.830-08:00 INFO - Driver: New PFrame
2024-03-04 16:35:43.830-08:00 ERROR - DemoVideoStreamSession: Received Data of length: 74125 Sequence: 11 sync time: 638451957434026940 frame time: 638451957437776940
2024-03-04 16:35:43.833-08:00 ERROR - DemoVideoStreamSession: Received Data of length: 74125 Sequence: 11 sync time: 638451957434026940 frame time: 638451957438186940
2024-03-04 16:35:43.879-08:00 INFO - Driver: New PFrame
We’ve been stuck here for quite some time now. Any assistance here is really helpful.
Hi, your log shows an issue with your sequence numbering, you send two frames with sequence number 4 and none with sequence number 3. I would recommend a) having a lock around the creation of your queued object and b) saving the sequence number of the frame in that object.
Thanks, I’ve made the changes, below is the updated code and logs. Still facing the issue!
internal class DemoVideoStreamSession : BaseDemoStreamSession
{
public DemoVideoStreamSession(ISettingsManager settingsManager, DemoConnectionManager demoConnectionManager, Guid sessionId, string deviceId, int channelId, Guid streamId) :
base(settingsManager, demoConnectionManager, sessionId, deviceId, channelId, streamId)
{
start();
}
public override void Close()
{
base.Close();
}
private Queue<CustomFrame> _queue = new Queue<CustomFrame>();
private DateTime _lastSyncTimestamp;
private readonly object balanceLock = new object();
protected int sequence = 0;
private async void start()
{
CancellationTokenSource source = new CancellationTokenSource();
CancellationToken token = source.Token;
Toolbox.Log.LogDebug("Driver", "Starting");
Uri rtspurl = new Uri("rtsp://#####");
string _userName = "#####";
string pword = "#####";
var credentials = new NetworkCredential(_userName, pword);
var connectionParameters = new ConnectionParameters(rtspurl, credentials);
connectionParameters.RtpTransport = RtpTransportProtocol.TCP;
connectionParameters.ReceiveTimeout = TimeSpan.FromSeconds(120);
using (var rtspClient = new RtspClient(connectionParameters))
{
CustomFrame customFrame;
rtspClient.FrameReceived += (sender, frame) =>
{
lock (balanceLock)
{
switch (frame)
{
case RawH264IFrame iframe:
sequence++;
_lastSyncTimestamp = iframe.Timestamp;
Toolbox.Log.LogDebug("Driver", "New IFrame R");
customFrame = new CustomFrame(
sequence, iframe.FrameSegment.Array, _lastSyncTimestamp, iframe.Timestamp, true
);
_queue.Enqueue(customFrame);
break;
case RawH264PFrame pframe:
sequence++;
Toolbox.Log.LogDebug("Driver", "New PFrame R");
customFrame = new CustomFrame(
sequence, pframe.FrameSegment.Array, _lastSyncTimestamp, pframe.Timestamp, false
);
_queue.Enqueue(customFrame);
break;
}
}
};
await rtspClient.ConnectAsync(token);
await rtspClient.ReceiveAsync(token);
}
}
protected override bool GetLiveFrameInternal(TimeSpan timeout, out BaseDataHeader header, out byte[] data)
{
header = null;
data = null;
CustomFrame frame;
lock (balanceLock)
{
if (_queue.Count <= 0)
{
return false;
}
frame = _queue.Dequeue();
}
data = frame.Data;
if (data == null || data.Length == 0)
{
return false;
}
Toolbox.Log.LogError(this.GetType().Name, "Received Data of length: " + (ulong)data.Length + " Sequence: " + frame.Sequence + " " +
"sync time: " + frame.SyncTimestamp.Ticks + " frame time: " + frame.FrameTimestamp.Ticks);
header = new VideoHeader()
{
CodecType = VideoCodecType.H264,
Length = (ulong) data.Length,
SequenceNumber = frame.Sequence,
SyncFrame = frame.Keyframe,
TimestampSync = frame.SyncTimestamp,
TimestampFrame = frame.FrameTimestamp
};
return true;
}
Logs:
2024-03-05 09:22:17.817-08:00 INFO - ******************************* Starting logger *******************************
2024-03-05 09:22:17.823-08:00 DEBUG - ----- Process start Command = SingleDriverMode; DllPath = C:\Program Files\Milestone\MIPDrivers\DemoDriver\DemoDriver.def
2024-03-05 09:22:18.113-08:00 ERROR - DemoConnectionManager: In Connect method!
2024-03-05 09:22:18.166-08:00 ERROR - DemoConnectionManager: Connected!!!
2024-03-05 09:22:18.241-08:00 ERROR - DemoConnectionManager: Starting LS!
2024-03-05 09:22:18.242-08:00 ERROR - DemoConnectionManager: In ThrowIfNotConnected!
2024-03-05 09:22:18.247-08:00 INFO - Driver: Starting
2024-03-05 09:22:18.584-08:00 INFO - Driver: New IFrame R
2024-03-05 09:22:18.586-08:00 ERROR - DemoVideoStreamSession: Received Data of length: 81913 Sequence: 1 sync time: 638452561387075039 frame time: 638452561387075039
2024-03-05 09:22:18.614-08:00 INFO - Driver: New PFrame R
2024-03-05 09:22:18.614-08:00 ERROR - DemoVideoStreamSession: Received Data of length: 81913 Sequence: 2 sync time: 638452561387075039 frame time: 638452561387485039
2024-03-05 09:22:18.685-08:00 INFO - Driver: New PFrame R
2024-03-05 09:22:18.685-08:00 ERROR - DemoVideoStreamSession: Received Data of length: 81913 Sequence: 3 sync time: 638452561387075039 frame time: 638452561387905039
2024-03-05 09:22:18.732-08:00 INFO - Driver: New PFrame R
2024-03-05 09:22:18.732-08:00 INFO - Driver: New PFrame R
2024-03-05 09:22:18.733-08:00 ERROR - DemoVideoStreamSession: Received Data of length: 81913 Sequence: 4 sync time: 638452561387075039 frame time: 638452561388325039
2024-03-05 09:22:18.734-08:00 ERROR - DemoVideoStreamSession: Received Data of length: 81913 Sequence: 5 sync time: 638452561387075039 frame time: 638452561388735039
2024-03-05 09:22:18.825-08:00 INFO - Driver: New PFrame R
2024-03-05 09:22:18.826-08:00 INFO - Driver: New PFrame R
2024-03-05 09:22:18.827-08:00 ERROR - DemoVideoStreamSession: Received Data of length: 81913 Sequence: 6 sync time: 638452561387075039 frame time: 638452561389155039
2024-03-05 09:22:18.828-08:00 ERROR - DemoVideoStreamSession: Received Data of length: 81913 Sequence: 7 sync time: 638452561387075039 frame time: 638452561389575039
2024-03-05 09:22:18.870-08:00 INFO - Driver: New PFrame R
2024-03-05 09:22:18.870-08:00 ERROR - DemoVideoStreamSession: Received Data of length: 81913 Sequence: 8 sync time: 638452561387075039 frame time: 638452561389985039
2024-03-05 09:22:18.915-08:00 INFO - Driver: New PFrame R
2024-03-05 09:22:18.915-08:00 ERROR - DemoVideoStreamSession: Received Data of length: 81913 Sequence: 9 sync time: 638452561387075039 frame time: 638452561390405039
2024-03-05 09:22:18.960-08:00 INFO - Driver: New PFrame R
2024-03-05 09:22:18.960-08:00 ERROR - DemoVideoStreamSession: Received Data of length: 81913 Sequence: 10 sync time: 638452561387075039 frame time: 638452561390825039
2024-03-05 09:22:19.018-08:00 INFO - Driver: New PFrame R
2024-03-05 09:22:19.018-08:00 ERROR - DemoVideoStreamSession: Received Data of length: 81913 Sequence: 11 sync time: 638452561387075039 frame time: 638452561391235039
2024-03-05 09:22:19.069-08:00 INFO - Driver: New PFrame R
2024-03-05 09:22:19.069-08:00 INFO - Driver: New PFrame R
2024-03-05 09:22:19.069-08:00 ERROR - DemoVideoStreamSession: Received Data of length: 81913 Sequence: 12 sync time: 638452561387075039 frame time: 638452561391655039
2024-03-05 09:22:19.071-08:00 ERROR - DemoVideoStreamSession: Received Data of length: 81913 Sequence: 13 sync time: 638452561387075039 frame time: 638452561392075039
2024-03-05 09:22:19.123-08:00 INFO - Driver: New PFrame R
2024-03-05 09:22:19.124-08:00 ERROR - DemoVideoStreamSession: Received Data of length: 81913 Sequence: 14 sync time: 638452561387075039 frame time: 638452561392485039
2024-03-05 09:22:19.179-08:00 INFO - Driver: New PFrame R
2024-03-05 09:22:19.179-08:00 ERROR - DemoVideoStreamSession: Received Data of length: 81913 Sequence: 15 sync time: 638452561387075039 frame time: 638452561392905039
2024-03-05 09:22:19.239-08:00 INFO - Driver: New PFrame R
2024-03-05 09:22:19.240-08:00 ERROR - DemoVideoStreamSession: Received Data of length: 81913 Sequence: 16 sync time: 638452561387075039 frame time: 638452561393325039
2024-03-05 09:22:19.287-08:00 INFO - Driver: New PFrame R
2024-03-05 09:22:19.287-08:00 INFO - Driver: New PFrame R
2024-03-05 09:22:19.289-08:00 ERROR - DemoVideoStreamSession: Received Data of length: 81913 Sequence: 17 sync time: 638452561387075039 frame time: 638452561393735039
2024-03-05 09:22:19.290-08:00 ERROR - DemoVideoStreamSession: Received Data of length: 81913 Sequence: 18 sync time: 638452561387075039 frame time: 638452561394155039
2024-03-05 09:22:19.331-08:00 INFO - Driver: New PFrame R
I’m sorry, I’ve simply missed the notification that you’d replied. I can’t see any obvious issues with your code, so I’m not entirely sure why this wouldn’t work.
I’m assuming you have verified that you can get an H.264 stream from the RTSP address used, and there are no problems elsewhere? Otherwise, as far as I can tell, this should work. I will need to experiment a little with your code, to perhaps see if I can get it to work myself. I will get back to you.
I have tried both Management and Smart Client. I don’t see video in both. I can confirm that the frames are being received as expected and the stream is working. Not sure if I’m missing something here.
I hope all is well. I wanted to check in on this issue. Do you have any insight as to why Milestone is not able to render the RTSP frames? This is currently preventing the completion of our Milestone integration, which we would love to finalize as soon as possible.
I was successfully able to connect the Verkada camera through the Universal Driver. What are the differences between how RTSP frames are handled by a custom driver versus the Universal Driver, if any?