Can't get clean UDP output stream from Nimble via SRT

I have created a ticket with support 33215, I am at the point where I am not sure if this software is suited for broadcast television delivery, I was hoping someone could prove me wrong.

currently, I am running nimble streamer on linux mint in my facility. I have the nimble server connected to a switch, which I am receiving UDP multicast streams from our H.264 encoders. Each stream is about 7Mb/s.

I have verified the UDP multicast streams are clean coming in using a program known as TSReader, which I am able to check for Continuity Count, PCR, PID errors ect.

Prior to using nimble, I am able to use an SRT testing program known as srt-live-transmit. Which allows you to input an incoming UDP stream and transmit it using SRT for testing purposes.

For example:
srt-live-transmit udp://:1234 srt://:4201 -v

using just srt-live-transmit, to take the incoming UDP multicast streams, and stream to another remote srt-live-transmit instance, I do not have any problems doing this, even on the machine which is running nimble streamer.

however, the issue I am experiencing occurs only when I am using nimble. In my nimble streamer configuration, I am ingesting the UDP multicast streams. I have nimble streamer then configured to host or output the streams via SRT.

I am able to successfully watch these streams in VLC player.

Where I start to see the issue is when I connect the Nimble Streamer SRT hosted link to a remote instance of srt-live-transmit, and analyze the stream on the receiving end with TSReader, I am seeing constant Continuity Count and PCR errors on the UDP output stream from the SRT receiver, this causes breakup and macro blocking on the video/audio signal. TSReader tells me the packets are out of order. I have also tried sending the SRT link to other engineers and they are able to confirm they see the same errors on their stream analyzers.

The strange thing is, I don't see this problem when I just use srt-live-transmit, I only see the issue when I use Nimble Streamer. I thought maybe it had something to do with the network or server environment. I was originally hosting nimble streamer on a dedicated cloud server. I recently moved it to a local machine on site, to see if the issue occurred on the local network, and it does.
I have even tried running the stream on the same machine nimble is running and I see the same errors.

I have been working with WMSPanel support, which I was able to send them my source link via SRT to try in their Nimble Server, as I thought it had something to do with just my nimble server or configuration. They sent me the return hosted link, and I still see the same errors.

WMSPanel support confirmed the SRT signal looks OK in VLC player, but I am seeing different results inspecting with a transport stream analyzer, TSReader

WMSPanel support then sent me a test stream link of their own, which was less than 2.5Mbps. Looking at this stream with TSReader, This stream worked no problem, no errors.

I thought maybe it had something to do with the transport stream rate or bitrate of my source UDP stream, so I tried lowering the source in my encoder from a mux rate of 7Mbps, Video bit rate of 6Mbps to a Mux Rate of 3Mbps, video bit rate of 2 Mbps.

BAM!! This cleared up the Continuity Count errors shown in TSReader that I was seeing on Nimble output stream. However, this isn't the solution as I cannot leave my 1920X1080 streams at 3Mbps with H.264 compression, as the quality is not good, and I will get some angry football viewers.

This is where I am lost as I am not sure how to troubleshoot or how to fix. I would like to use nimble streamer for cable/headend delivery for our channels, monitoring, maybe ad insertion. But if I cant confirm the SRT output streams are clean coming out of nimble streamer, I cannot use it or convince my employer to allow me to purchase these advanced features.

If anyone could shed light on this, I would very much appreciate it.
Thank you.

We are facing exactly the same problem: CC errors and breaks in SRT output of any HD channel.
Setting latency=3000 and maxbw according to nimble blog recommendation didn't help.
Why this major bug is not mentioned anywhere? Marvelous.

doing a test now,

I discovered that when i change the "local IP" on the SRT listener from the IP of my server to 0.0.0.0 "all interfaces the errors are much less.

I still get spikes of CC errors

1.5: PMT_error: Sections with table_id 0x02, do not occur at least every 0.5 seconds
1.4: Continuity_count_error: Incorrect packet order (0100 )

Increased kernel sysctl variables - wmem_default, wmem_max, udp_mem, to no avail.

Looks like there is no fix in site, only thing left to do is cancel WMSpanel subscription, no point in paying If i cant use it.

j-nim@odd.systems, how do you test CC in SRT output from Nimble, do you also use srt-live-transmit to pull SRT from Nimble and push via UDP?

As I understand CC errors can only happen if some packets are dropped in SRT, Nimble does not do anything special for low or high bitrate streams so CC errors happening due to increased bitrate happens because of SRT or UDP drops.

For now we can't explain why it happens only when you're using Nimble as a source, may be another source uses different settings like latency/maxbw.

Max, of course, our first idea was: drops are caused by saturation at link level. We proved it's not the cause. Seems like drops come from somewhere else.

Let me describe our setup which we simplified for testing.
All traffic flows within non-saturated local network.
Input for nimble is stable CBR HTTP/MPEGTS from astra. Example:
хттп://93.170.252.10/nimble-failed-hd-input-example.ts
SRT output is received by srt-live-transmit on another host: srt://nimble:... -> udp://127.0.0.1:..., UDP goes to astra --analyze on the same host.
For the input example mentioned above, we would see random bugs: variations inbitrate, 2-3 CC errors or even full break for some seconds.

On the other hand,
if you are sure that you've never seen such behaviour,
then could you provide us with sample fullHD input for nimble to produce cleanSRT output?

I do not have FullHD sample stream, but if you provide us FullHD input I can make SRT output from it and provide it to you.

I did not get what is wrong with хттп://93.170.252.10/nimble-failed-hd-input-example.ts file, could you clarify that?

The mentioned file is a fragment of input stream for nimble, which
causes broken SRT output.

But you are right, I better open life stream. Please write your network addresses for access restriction.
If private communication is better, you can reach me via j-nim@odd.systems or I can open a ticket.

Max,

You need to test this yourself with TS reader, VLC will not show CC errors or PMT PID errors.
This is a bug that will drive away broadcasting customers.

Tyler Bailey, please calm down and instead of crying try to provide more data. Perhaps our CPUs are slow?

This is my server specs, I've also tried it on separate hardware/cloud server.
I'd be happy to provide any data requested, but softevelum doesn't seem to be interested in solving our problem.

Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 39 bits physical, 48 bits virtual
CPU(s): 16
On-line CPU(s) list: 0-15
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 165
Model name: Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz
Stepping: 5
CPU MHz: 2900.000
CPU max MHz: 4800.0000
CPU min MHz: 800.0000
BogoMIPS: 5799.77
Virtualization: VT-x
L1d cache: 256 KiB
L1i cache: 256 KiB
L2 cache: 2 MiB
L3 cache: 16 MiB
NUMA node0 CPU(s): 0-15

RAM

Handle 0x000A, DMI type 17, 84 bytes
Memory Device
Array Handle: 0x0009
Error Information Handle: Not Provided
Total Width: 64 bits
Data Width: 64 bits
Size: 16384 MB
Form Factor: DIMM
Set: None
Locator: DIMM1
Bank Locator: Not Specified
Type: DDR4
Type Detail: Synchronous
Speed: 2666 MT/s
Configured Memory Speed: 2666 MT/s
Configured Voltage: 1.2 V
Memory Technology: DRAM
Memory Operating Mode Capability: Volatile memory
Firmware Version: Not Specified
Module Manufacturer ID: Bank 1, Hex 0xAD
Module Product ID: Unknown
Memory Subsystem Controller Manufacturer ID: Unknown
Memory Subsystem Controller Product ID: Unknown
Non-Volatile Size: None
Volatile Size: 16 GB
Cache Size: None
Logical Size: None

Handle 0x000B, DMI type 17, 84 bytes
Memory Device
Array Handle: 0x0009
Error Information Handle: Not Provided
Total Width: 64 bits
Data Width: 64 bits
Size: 16384 MB
Form Factor: DIMM
Set: None
Locator: DIMM2
Bank Locator: Not Specified
Type: DDR4
Type Detail: Synchronous
Speed: 2666 MT/s
Part Number:
Rank: 2
Configured Memory Speed: 2666 MT/s
Minimum Voltage: Unknown
Maximum Voltage: Unknown
Configured Voltage: 1.2 V
Memory Technology: DRAM
Memory Operating Mode Capability: Volatile memory
Firmware Version: Not Specified
Module Manufacturer ID: Bank 1, Hex 0xAD
Module Product ID: Unknown
Memory Subsystem Controller Manufacturer ID: Unknown
Memory Subsystem Controller Product ID: Unknown
Non-Volatile Size: None
Volatile Size: 16 GB
Cache Size: None
Logical Size: None

Tyler,
Softvelum is a small professional team trying to provide:
- freeware media server for people who don't need commercial features and support.
- not very expensive add-ons to it so that we can earn something during development
- professional support for professionals and good enough to not experience users

Professional users are users dedicated to long-term relationship and who understand that sometimes there are such problems that cannot be solved in forum thread.
In this case for mutual benefits, its expected that client setup test environment so our team can login and see the problem, change params, test etc. In this case we will try as much as we can to get solution. Sometimes though solution cannot be found even in this case.

------------------------------------------------------------------------------------------------------------------------------------------------------------
Regarding "but softevelum doesn't seem to be interested in solving our problem."
Softvelum interested in solving your problem as much as you interesting in solving it.
Once you said: "Looks like there is no fix in site, only thing left to do is cancel WMSpanel subscription, no point in paying If i cant use it.", we decided that problem is not in priority anymore.

If anyone in this thread want us to help:
- have active wmspanel subscription
- setup test environment where you can reproduce your problem and give us access to it.
- Try to remove some third party software in your setup and use another one.
- try to get SRT loss statistics. instead of just say there is no SRT losses

------------------------------------------------------------------------------------------------------------------------------------------------------------
please don't use such rough expression like "instead of crying" in this forum. We will remove such messaged or entire thread once we find it's not polite discussion anymore.

Thanks,

Did anyone from Softevelum happen to look into this issue.
I resubscribed to WMSpanel after seeing some patch notes but unfortunately I am still having the same problem.

Ive determined what we are seeing is packet loss when there is a high bit rate SRT stream.
Does changing the latency, Mux Rate, Mux delay help with packet loss?

I am not having network issues but I am experiencing packet loss with Nimble streamer.

Tyler,
We see your ticket in helpdesk, we'll continue there.

Thank you, much appreciation. Looking forward to seeing what we can find.

I'm having the same problem getting a clean NDI > Live Transcoder > SRT Stream.
Audio is clean, but no matter what I do, I see dropped video frames. My Nimble server is a c6i.8xlarge running in AWS, so network/CPU is not likely the cause.

I Should Say SRT encoding to NDI is clean, with very few dropped frames.

The issues seemed to coincide with the last quarterly update of Nimble

Please file a ticket via https://wmspanel.com/help and specify your account, server and stream name