Forum

Can't get clean UDP output stream from Nimble via SRT

Tyler Bailey 2021-10-30 10:17:44 UTC in Nimble Streamer

I have created a ticket with support 33215, I am at the point where I am not sure if this software is suited for broadcast television delivery, I was hoping someone could prove me wrong.

currently, I am running nimble streamer on linux mint in my facility. I have the nimble server connected to a switch, which I am receiving UDP multicast streams from our H.264 encoders. Each stream is about 7Mb/s.

I have verified the UDP multicast streams are clean coming in using a program known as TSReader, which I am able to check for Continuity Count, PCR, PID errors ect.

Prior to using nimble, I am able to use an SRT testing program known as srt-live-transmit. Which allows you to input an incoming UDP stream and transmit it using SRT for testing purposes.

For example:
srt-live-transmit udp://:1234 srt://:4201 -v

using just srt-live-transmit, to take the incoming UDP multicast streams, and stream to another remote srt-live-transmit instance, I do not have any problems doing this, even on the machine which is running nimble streamer.

however, the issue I am experiencing occurs only when I am using nimble. In my nimble streamer configuration, I am ingesting the UDP multicast streams. I have nimble streamer then configured to host or output the streams via SRT.

I am able to successfully watch these streams in VLC player.

Where I start to see the issue is when I connect the Nimble Streamer SRT hosted link to a remote instance of srt-live-transmit, and analyze the stream on the receiving end with TSReader, I am seeing constant Continuity Count and PCR errors on the UDP output stream from the SRT receiver, this causes breakup and macro blocking on the video/audio signal. TSReader tells me the packets are out of order. I have also tried sending the SRT link to other engineers and they are able to confirm they see the same errors on their stream analyzers.

The strange thing is, I don't see this problem when I just use srt-live-transmit, I only see the issue when I use Nimble Streamer. I thought maybe it had something to do with the network or server environment. I was originally hosting nimble streamer on a dedicated cloud server. I recently moved it to a local machine on site, to see if the issue occurred on the local network, and it does.
I have even tried running the stream on the same machine nimble is running and I see the same errors.

I have been working with WMSPanel support, which I was able to send them my source link via SRT to try in their Nimble Server, as I thought it had something to do with just my nimble server or configuration. They sent me the return hosted link, and I still see the same errors.

WMSPanel support confirmed the SRT signal looks OK in VLC player, but I am seeing different results inspecting with a transport stream analyzer, TSReader

WMSPanel support then sent me a test stream link of their own, which was less than 2.5Mbps. Looking at this stream with TSReader, This stream worked no problem, no errors.

I thought maybe it had something to do with the transport stream rate or bitrate of my source UDP stream, so I tried lowering the source in my encoder from a mux rate of 7Mbps, Video bit rate of 6Mbps to a Mux Rate of 3Mbps, video bit rate of 2 Mbps.

BAM!! This cleared up the Continuity Count errors shown in TSReader that I was seeing on Nimble output stream. However, this isn't the solution as I cannot leave my 1920X1080 streams at 3Mbps with H.264 compression, as the quality is not good, and I will get some angry football viewers.

This is where I am lost as I am not sure how to troubleshoot or how to fix. I would like to use nimble streamer for cable/headend delivery for our channels, monitoring, maybe ad insertion. But if I cant confirm the SRT output streams are clean coming out of nimble streamer, I cannot use it or convince my employer to allow me to purchase these advanced features.

If anyone could shed light on this, I would very much appreciate it.
Thank you.

sa 2021-11-18 12:29:54 UTC 

We are facing exactly the same problem: CC errors and breaks in SRT output of any HD channel.
Setting latency=3000 and maxbw according to nimble blog recommendation didn't help.
Why this major bug is not mentioned anywhere? Marvelous.

Tyler Bailey 2021-11-19 00:53:22 UTC 

doing a test now,

I discovered that when i change the "local IP" on the SRT listener from the IP of my server to 0.0.0.0 "all interfaces the errors are much less.

Tyler Bailey 2021-11-19 01:09:37 UTC 

I still get spikes of CC errors

1.5: PMT_error: Sections with table_id 0x02, do not occur at least every 0.5 seconds
1.4: Continuity_count_error: Incorrect packet order (0100 )

sa 2021-11-19 15:00:53 UTC 

Increased kernel sysctl variables - wmem_default, wmem_max, udp_mem, to no avail.

Tyler Bailey 2021-11-19 22:07:09 UTC 

Looks like there is no fix in site, only thing left to do is cancel WMSpanel subscription, no point in paying If i cant use it.

Max 2021-11-20 14:23:56 UTC 

j-nim@odd.systems, how do you test CC in SRT output from Nimble, do you also use srt-live-transmit to pull SRT from Nimble and push via UDP?

As I understand CC errors can only happen if some packets are dropped in SRT, Nimble does not do anything special for low or high bitrate streams so CC errors happening due to increased bitrate happens because of SRT or UDP drops.

For now we can't explain why it happens only when you're using Nimble as a source, may be another source uses different settings like latency/maxbw.

sa 2021-11-21 12:45:24 UTC 

Max, of course, our first idea was: drops are caused by saturation at link level. We proved it's not the cause. Seems like drops come from somewhere else.

Let me describe our setup which we simplified for testing.
All traffic flows within non-saturated local network.
Input for nimble is stable CBR HTTP/MPEGTS from astra. Example:
хттп://93.170.252.10/nimble-failed-hd-input-example.ts
SRT output is received by srt-live-transmit on another host: srt://nimble:... -> udp://127.0.0.1:..., UDP goes to astra --analyze on the same host.
For the input example mentioned above, we would see random bugs: variations inbitrate, 2-3 CC errors or even full break for some seconds.

On the other hand,
if you are sure that you've never seen such behaviour,
then could you provide us with sample fullHD input for nimble to produce cleanSRT output?

Max 2021-11-22 09:36:34 UTC 

I do not have FullHD sample stream, but if you provide us FullHD input I can make SRT output from it and provide it to you.

I did not get what is wrong with хттп://93.170.252.10/nimble-failed-hd-input-example.ts file, could you clarify that?

sa 2021-11-22 12:24:54 UTC 

The mentioned file is a fragment of input stream for nimble, which
causes broken SRT output.

But you are right, I better open life stream. Please write your network addresses for access restriction.
If private communication is better, you can reach me via j-nim@odd.systems or I can open a ticket.

Tyler Bailey 2021-11-22 12:49:09 UTC 

Max,

You need to test this yourself with TS reader, VLC will not show CC errors or PMT PID errors.
This is a bug that will drive away broadcasting customers.

sa 2021-11-24 17:44:33 UTC 

Tyler Bailey, please calm down and instead of crying try to provide more data. Perhaps our CPUs are slow?

Tyler Bailey 2021-11-24 18:21:59 UTC 

This is my server specs, I've also tried it on separate hardware/cloud server.
I'd be happy to provide any data requested, but softevelum doesn't seem to be interested in solving our problem.

Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 39 bits physical, 48 bits virtual
CPU(s): 16
On-line CPU(s) list: 0-15
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 165
Model name: Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz
Stepping: 5
CPU MHz: 2900.000
CPU max MHz: 4800.0000
CPU min MHz: 800.0000
BogoMIPS: 5799.77
Virtualization: VT-x
L1d cache: 256 KiB
L1i cache: 256 KiB
L2 cache: 2 MiB
L3 cache: 16 MiB
NUMA node0 CPU(s): 0-15

RAM

Handle 0x000A, DMI type 17, 84 bytes
Memory Device
Array Handle: 0x0009
Error Information Handle: Not Provided
Total Width: 64 bits
Data Width: 64 bits
Size: 16384 MB
Form Factor: DIMM
Set: None
Locator: DIMM1
Bank Locator: Not Specified
Type: DDR4
Type Detail: Synchronous
Speed: 2666 MT/s
Configured Memory Speed: 2666 MT/s
Configured Voltage: 1.2 V
Memory Technology: DRAM
Memory Operating Mode Capability: Volatile memory
Firmware Version: Not Specified
Module Manufacturer ID: Bank 1, Hex 0xAD
Module Product ID: Unknown
Memory Subsystem Controller Manufacturer ID: Unknown
Memory Subsystem Controller Product ID: Unknown
Non-Volatile Size: None
Volatile Size: 16 GB
Cache Size: None
Logical Size: None

Handle 0x000B, DMI type 17, 84 bytes
Memory Device
Array Handle: 0x0009
Error Information Handle: Not Provided
Total Width: 64 bits
Data Width: 64 bits
Size: 16384 MB
Form Factor: DIMM
Set: None
Locator: DIMM2
Bank Locator: Not Specified
Type: DDR4
Type Detail: Synchronous
Speed: 2666 MT/s
Part Number:
Rank: 2
Configured Memory Speed: 2666 MT/s
Minimum Voltage: Unknown
Maximum Voltage: Unknown
Configured Voltage: 1.2 V
Memory Technology: DRAM
Memory Operating Mode Capability: Volatile memory
Firmware Version: Not Specified
Module Manufacturer ID: Bank 1, Hex 0xAD
Module Product ID: Unknown
Memory Subsystem Controller Manufacturer ID: Unknown
Memory Subsystem Controller Product ID: Unknown
Non-Volatile Size: None
Volatile Size: 16 GB
Cache Size: None
Logical Size: None

Alex Pokotilo 2021-11-25 02:50:36 UTC 

Tyler,
Softvelum is a small professional team trying to provide:
- freeware media server for people who don't need commercial features and support.
- not very expensive add-ons to it so that we can earn something during development
- professional support for professionals and good enough to not experience users

Professional users are users dedicated to long-term relationship and who understand that sometimes there are such problems that cannot be solved in forum thread.
In this case for mutual benefits, its expected that client setup test environment so our team can login and see the problem, change params, test etc. In this case we will try as much as we can to get solution. Sometimes though solution cannot be found even in this case.

------------------------------------------------------------------------------------------------------------------------------------------------------------
Regarding "but softevelum doesn't seem to be interested in solving our problem."
Softvelum interested in solving your problem as much as you interesting in solving it.
Once you said: "Looks like there is no fix in site, only thing left to do is cancel WMSpanel subscription, no point in paying If i cant use it.", we decided that problem is not in priority anymore.

If anyone in this thread want us to help:
- have active wmspanel subscription
- setup test environment where you can reproduce your problem and give us access to it.
- Try to remove some third party software in your setup and use another one.
- try to get SRT loss statistics. instead of just say there is no SRT losses

------------------------------------------------------------------------------------------------------------------------------------------------------------
please don't use such rough expression like "instead of crying" in this forum. We will remove such messaged or entire thread once we find it's not polite discussion anymore.

Tyler Bailey 2021-12-08 04:07:20 UTC 

Thanks,

Did anyone from Softevelum happen to look into this issue.
I resubscribed to WMSpanel after seeing some patch notes but unfortunately I am still having the same problem.

Ive determined what we are seeing is packet loss when there is a high bit rate SRT stream.
Does changing the latency, Mux Rate, Mux delay help with packet loss?

I am not having network issues but I am experiencing packet loss with Nimble streamer.

Yury 2021-12-08 06:17:45 UTC 

Tyler,
We see your ticket in helpdesk, we'll continue there.

Tyler Bailey 2021-12-08 06:20:23 UTC 

Thank you, much appreciation. Looking forward to seeing what we can find.

Michael Lange 2022-03-07 19:04:14 UTC 

I'm having the same problem getting a clean NDI > Live Transcoder > SRT Stream.
Audio is clean, but no matter what I do, I see dropped video frames. My Nimble server is a c6i.8xlarge running in AWS, so network/CPU is not likely the cause.

I Should Say SRT encoding to NDI is clean, with very few dropped frames.

The issues seemed to coincide with the last quarterly update of Nimble

Alex Pokotilo 2022-03-07 22:18:39 UTC 

Please file a ticket via https://wmspanel.com/help and specify your account, server and stream name

Post a reply


Post a new question

Categories:

Tags:

nimbleNimble StreamerFAQHLSnimble streamerDVRRTMPhlsSRTsrtABRrtmpcacheNimblewmsauthAPItranscoderdvrffmpegapisldpudpVODaudioRTSPfailoverrtspDASHwmspanelUDPvodpaywallstreamingabrSLDPyoutubemp4DispersastreamerandroidsslmulticastLarixsubtitlesliveNDIvideolivestreamingplaylistWMSAuthMPEG-DASHpay-per-viewerrorgeoMPEG-TSre-streamingdashcorsWMSPaneledgebandwidththumbnaillarix broadcasterWindowshttpswhite labelconfigsmilFFMPEGencryptionCORSperformancechunksraspberry pimpeg-dashpublish controlcloudfrontDRMRepublishingvlcAWSS3advertisinguser agentadvertizerristrepublishingrules.confipv6MPEGTSFastSpringRecordingRAMwms panelfileFMLEVATcrossdomainSMILmpegtsaespushakamaimobilewowzaPullserverscodecmanifestSSLchromecastbalancedrmTranscodem3u8TranscoderbugWowzaIDreportingconfigurationsnapshotdownloadawsAndroidnimblestreamerdomaintranscoderaspianloopscte35streamroutesamazonupdateipVidillionHttpsmpeg-tsBroadcastplaylist_dvrVLCPIDerrorsEncoderoriginscte-35issuechunklimitsecurityLarix BroadcasterIPCDNthumbnailsUIRegistration Issuedirect link32-bit Windowstwitchcache_controlitworkmecudalive abr support mpeg-dashwmspanelapibeirutreloadWWDCdubaideep statsCentOS v6.4logIIS Smooth StreamingcloudmediaIPTVprofilelarix abrTSReaderAbrHTTPSHot-linking protectionHDSvaddioalertsMuxjwplayer websitewhmcsbaselinetimelineVaddioAV BridgePI3 Ubuntuview timeAuthentication in HLSNimble Streamer APIPi4streamsinterfacesHLS Meta Tag editing.encrytpionloadbalancingwmsattachmentHEVCmetadatamod_rewritewmsauthsignpricehighresourcehds streamOld logsciscolocalscreencastID3 tagsgbpsAxis.net hotlinkRTMP republishattaching domainssdpshoutcastno internetInterlacedocumentationofflinecontainerNimble streamer upgradeMulticast*concurrent-connectionsTrancoderrecordingrtmp abranalisysresumeniblerhls restreamingServer-siderestreamstoppedMPEG2TSNimble Streamer versionmainWMSCONFIG_HOMEdissapointmentdatmessagegentoo install server nimbletranscoding using NvidiaPublic IpPaywall AuthwildcardobsTCOno WMSPanelVideo Playertransocding republishingdecoderAliaspay per viewlost trafficdvr_archivesmpeg dashnginx rtmp nimbleRaspian Bustericecast urlexportopenssllimuxamazon web servicenginxPlayReadychangedvr stream twiceLarix GroveamfAXVVGhot-linkIOSExpression EncoderblocknvenconSteam stopped workingadaptiveAV1 codecHLS PlayerJWPLAYERNGINX-RTMPHLS in UDP outDASH PlayerHLS voddvr export to mp47analyticsapplicationpaywalbrandingtrialCablemicrosoft streamMP4rocksoftlog traffic statsjpegscreen freezecan't registeraccuracyprivate networkLarge DVR fileslive videodebianTelegraminterleavingSLDP PLayerseekingmac osx installwotermarklive video on demandbandwithnimblesessionidFFmpegmultiple originsvimeohelp errorserverload balancezabbixheadercache expirycdnvsomlog nimble analysertmp playbacklive streamsourceViewer StatsAdsvideojsLive Streamingbitrate#restreamMP4 not playedspaceServer-Side-Task-Controlvideo stopaes encryptionsubscriptionvod no soundconcurrent connectionlebanonlocationUbuntu 20 ARM - AWSunique visitorconnectivitydvr streamstereo to monoscreen capturelive pull settingsWMSPanel settingsABR HLS Bitratesrtmp for YouTube4Kcrossdomain more then 1 domainWidevineFairplaycpunot foundCPU LoadpullAS3loggingnimble streamer vod hls transmuxingbuttAWS 3Cross Domaininsert logoinstall players setup ready to goDVRSettingstwitch larix broadcaster androidicecast metadataDVRStreamsAppleDelete recordspremium featureControl APIProgressivenimble streamer web server php script pageautomationLive streamingvideo loopUbuntu artful 17.10Transcoder MPEG DASHinvalidMPEG DASHstarttime duration seekpointadd_chunk failedPost processingfake extensionstatus:errorrebootdroppingmobile app live streamingmonitoroutrateudp streamingapi accessDeep statsicecastMPEG-Dashloadbalancerweb playercontent-dispositionpaymentstatsNimble ServerAVCaptureMovieFileOutputblock downloadrestartdvr streamsavoid refreshlivestreamABR DVR problemNimble CapacityHotlinking ProtectionStreaming routehotlinkfacebookcontentRTMP RepublishSecurityLive Switcherdemandnot to stealLive BroadcastMultiple Audio trackssecure streaminglow_bandwidthwmsAuthSignnimble streamer mpeg-ts multiple inputsLoad-Balancingvp9adsounddisk migrationThumbnailsserver incorrect timeLIVELivelearnernimble webcam html5ssaiDVR Setting limitpausemosaicHow to do live stream with multiple audio trackrmtptranscoder nimblereportsdvr on wmspanelfpsVideo cant be playedS3 AWSraspberryPaywalllivestreamintransocderstorage space availablediskcredentialsdvr to livewebhookautomateuser_agentvbv-maxratedomain lockUsers limitoutputcloudflareVOD HLS streaming on public IPlarixscteInvalid frame headerno soundincoming streamSubtitleslivestream bitratedockerDVRRecordingAuthGopDVR SettingPORTdelaympeg4storagealias route.movartifactsrtsp push androidmultiviewRAM Loadhttp serverLatencywirecasta recordruleshow-tomanage_dvrError when installingVP9server ip21Nimble connectionsabr fallbackno audiolatencyaliasdata slicesplayer sldpmp2 audioNimble streamerfastspringsoftware versionadjustSO_RCVBUFbuffering videodistributionbufferqataritworkscdnvideo.jsmultiplexcompatibility protocolsCentos 8pay-per-minuteTeradek Decoderubuntu 183.6.1-1UDP MulticastMax connectionincomingmulti-viewerFallbackhelpmultiple audio udphotlink protectionRaspberryPi4ultra low latencyaws amazonhls to multicast udpnoobSSL requestsha265broadcasterOSXmd5analyseStreamIDprerollnimble on cloudrtmp sldp nimblepluginfallbacktasks-controlRaspberryRTMP to SRTLinuxgpu{stream}Connections count limitNimble_Crushwebrtc4GRIST Bondingplayoutsrt-live-transmitWowza AgentRemote StorageSSAItuningAmazonTVfireOBSvideo and audio not matchLarix broadcasterNetworkspeedup my videoiOSPerformanceSnapshotsSRT RTMPre-publishingletsencryptTonydowntranscodingFFMPEG;RTMP;I/O errorLL-HLSHTML5 playerAddendaABR DASHprogressive downloadeventLL-HLS DVRav1 codecconnectionsNimbleStreamernimble aliasesNimble StreambuildABR bitratesPacketizingcountrist set fecmultiple audioOBS StudioWMSAUTHIVSDelaympeg2tsIIS Media Servicsrtmp republishing transcodenimble.confinterfacetrackbindprogressivebroadcast videovideo audioSRT protocoldvbduplicatingpcdynamic linksTLSV 1.2 CertificatesubfolderbrowserurlGoogle DriveHLS Streamingrtmpscostnimble abr hlsthumbnail dvr-thumbnailaudio languageVR-360Failovergoogle cloud storagepricingoverlayhot linkingTRANSCODErtpnimble dvrBandwidthdurationYadifportsRokuContinuitywordpressJetson NanoSecureLarix PlayerramfairplayNginxpublish streamStream Delayhd25trafficAArch64video streamingQuickTimeCloud StorageRistmultistreamingmultipleserver ssl erroractionscript 3dropped framesCSSMPEG2 VideoPCRlogo in streammultipointrulerepublishconcurrent

This website or its third-party tools use cookies, which are necessary to its functioning and required to achieve the purposes illustrated in the Privacy Policy. If you want to know more or withdraw your consent to all or some of the cookies, please refer to the Privacy Policy.
By closing this banner, scrolling this page, clicking a link or continuing to browse otherwise, you agree to the use of cookies.