Forum

Issue with stream dropping out and connections in WMS panel showing really high but low bandwidth?

James Fawcett 2021-09-15 10:27:52 UTC in Nimble Streamer

Hi
We had 80 people connect to a stream on our Nimble streaming server but when looking at the live stream stats we were seeing numbers of connections at 500-700, this caused other connected clients to drop out and have issues with the stream. We definitely did not have 500 people watching but there were loads of connections almost like phantom connections, with this we saw the Bandwidth show only 20Mbps overall when we had 700 people we would expect to see bandwidth consumption of around 2.5Gbps.
Nothing in the logs seems to suggest that this issue would have happened but it caused big issues to any connected peers.

Server is on Azure running Ubuntu 14
Server resources are 16CPU and 32GB RAM
RTMP stream was sent to Nimble at 4Mbps
Nimble has a transcoder creating 2 streams at lower resolution and then ABR to put them into the player
SLDP player is embedded into our website and has always run the stream perfectly with up to 500 participants before.

Would really appreciate some insight into why this happened - stream just skyrocketed and that seemed to be the issue then 20 minutes later all was fine like it never happened!

Thanks

Max 2021-09-15 15:16:19 UTC 

Hello James,

It can happen if clients reconnect to Nimble multiple times so reported connection count is increased, but average playback session duration is low. Are there any errors in nimble.log like e.g. "client is too slow"?

James Fawcett 2021-09-15 15:32:40 UTC 

Hi Max thanks for your reply, is it possible for 60 clients to connect enough times constantly in quick succession to cause 500 - 700 connections to show up in the dashboard and how come this would negatively impact the viewing experience for the rest of the viewers, it killed the stream by going black for a multitude of those 60 people? The server is pretty big and has a max constant bandwidth limit of 12500Mbps.
We did actually get a few here are some of them (blanked ips etc for security reasons)

[2021-09-15 09:34:43 P2607-T2632] [wsls1] E: client s=66 ip='12.34.56.789' stream='nimbleubuntu/eventname_abr' is too slow [1]

[2021-09-15 09:19:22 P2607-T2632] [wsls1] E: failed to SSL_shutdown ws_client s=73 (SSL_errno=5 errno=0 e='error:00000000:lib(0):func(0):reason(0)')

[2021-09-15 09:07:20 P2607-T2632] [wsls1] E: failed to SSL_shutdown ws_client s=76 (SSL_errno=1 e='error:140E0197:SSL routines:SSL_shutdown:shutdown while in init')

[2021-09-15 08:37:30 P2607-T2632] [wsls1] E: viewer tries to play invalid stream

Thanks James

Max 2021-09-16 10:00:04 UTC 

Hello James,

WMSPanel aggregate stats for 30 second intervals so it is possible for 60 clients to make this number of re-connections, but re-connections itself should not affect other viewers.

May be there were some temporary network issue on the server preventing viewers downloading stream with appropriate speed causing re-connections?

James Fawcett 2021-09-16 11:24:07 UTC 

Hi Max, interesting, have there ever been reports of this before? There is nothing in the Azure Logs for our nimble virtual machine to suggest there was ever any slow down with network especially considering the bandwidth allocated to that VM which is enormous. We do use nimble to transcode the stream into two other scales for ABR delivery but if one stream goes down as long as it is not the main stream it should just failover onto the available sources.

To describe the issue in more detail what happened was we saw the dashboard show a huge number of connections and during this time we peoples SLDP players going black - it appeared as if the stream was running but it was just black. There was nothing wrong with the RTMP feed we were sending into nimble so I just really want to make sure we can get to the bottom of it or how to avoid it in the future, I am guessing a multi server setup with load balancing is best but in the meantime if there anything else you can think of?
Thank you James

Max 2021-09-16 15:00:24 UTC 

Hello James,

We had similar reports about increased connection counts, previously it was caused by network problems with servers causing client re-connections after "too slow" errors (so connection count increases but bandwidth and average play time decreases, check daily and duration stats reports for your server).

The issue in your case may be different if e.g. there were issues with transcoding and players disconnected due to some other reason. I'd suggest e.g. to play stream via HLS if issue happens again and checking downloading speed and/or errors in Network Log of the browser.

Load balancing should help if network speed was the bottleneck.

James Fawcett 2021-09-16 17:13:54 UTC 

Hi Max thank you for your reply, what does it mean when you say servers causing client reconnection how would that happen? Is a too slow error normal?
I have checked our duration stats and for a normal event we would have maybe 1000 connections for 100 people over the course of a couple of hours but for this event we had an unreal number see below:
Connections per duration
< 5s 32348
< 10s 1005
< 15s 483
< 20s 161
< 25s 69
< 30s 44
< 5m 281
< 15m 33
< 30m 42
< 1h 49
< 2h 7
< 3h 1
> 3h 2

A total of 34525 connections in 2 hours - what would be the most likely culprit that this many connections were logged?
Surely a 16CPU 32GB RAM Server would be able to cope with a scenario like this, sorry I know it is a lot of info I am just really trying to pinpoint the cause, could a users firewall mess with Nimble at all requests wise?

Thank you James

Max 2021-09-17 10:00:56 UTC 

Hello James,

I'm not sure what is the culprit, unfortunately it is not possible to get what exactly is the issue based on this description. It could be insufficient outgoing bandwidth or some issues with the stream causing player to re-connect. Try record event streams to DVR to check later if stream itself had any problems.

I guess user's firewall should not be the problem causing player to re-connect to the server.

Post a reply


Post a new question

Categories:

Tags:

nimbleNimble StreamerFAQHLSnimble streamerDVRRTMPhlsSRTsrtABRrtmpcacheNimblewmsauthAPItranscoderdvrffmpegapisldpudpVODaudioRTSPfailoverrtspDASHwmspanelUDPvodpaywallstreamingabrSLDPyoutubemp4DispersastreamerandroidsslmulticastLarixsubtitlesliveNDIvideolivestreamingplaylistWMSAuthMPEG-DASHpay-per-viewerrorgeoMPEG-TSre-streamingdashcorsWMSPaneledgebandwidththumbnaillarix broadcasterWindowshttpswhite labelconfigsmilFFMPEGencryptionCORSperformancechunksraspberry pimpeg-dashpublish controlcloudfrontDRMRepublishingvlcAWSS3advertisinguser agentadvertizerristrepublishingrules.confipv6MPEGTSFastSpringRecordingRAMwms panelfileFMLEVATcrossdomainSMILmpegtsaespushakamaimobilewowzaPullserverscodecmanifestSSLchromecastbalancedrmTranscodem3u8TranscoderbugWowzaIDreportingconfigurationsnapshotdownloadawsAndroidnimblestreamerdomaintranscoderaspianloopscte35streamroutesamazonupdateipVidillionHttpsmpeg-tsBroadcastplaylist_dvrVLCPIDerrorsEncoderoriginscte-35issuechunklimitsecurityLarix BroadcasterIPCDNthumbnailsUIRegistration Issuedirect link32-bit Windowstwitchcache_controlitworkmecudalive abr support mpeg-dashwmspanelapibeirutreloadWWDCdubaideep statsCentOS v6.4logIIS Smooth StreamingcloudmediaIPTVprofilelarix abrTSReaderAbrHTTPSHot-linking protectionHDSvaddioalertsMuxjwplayer websitewhmcsbaselinetimelineVaddioAV BridgePI3 Ubuntuview timeAuthentication in HLSNimble Streamer APIPi4streamsinterfacesHLS Meta Tag editing.encrytpionloadbalancingwmsattachmentHEVCmetadatamod_rewritewmsauthsignpricehighresourcehds streamOld logsciscolocalscreencastID3 tagsgbpsAxis.net hotlinkRTMP republishattaching domainssdpshoutcastno internetInterlacedocumentationofflinecontainerNimble streamer upgradeMulticast*concurrent-connectionsTrancoderrecordingrtmp abranalisysresumeniblerhls restreamingServer-siderestreamstoppedMPEG2TSNimble Streamer versionmainWMSCONFIG_HOMEdissapointmentdatmessagegentoo install server nimbletranscoding using NvidiaPublic IpPaywall AuthwildcardobsTCOno WMSPanelVideo Playertransocding republishingdecoderAliaspay per viewlost trafficdvr_archivesmpeg dashnginx rtmp nimbleRaspian Bustericecast urlexportopensslchangelimuxamazon web servicenginxPlayReadydvr stream twiceLarix GroveamfIOSAXVVGhot-linkExpression EncoderblocknvenconSteam stopped workingadaptiveAV1 codecHLS PlayerJWPLAYERNGINX-RTMPDASH PlayerHLS voddvr export to mp47analyticsapplicationpaywalbrandingtrialCablemicrosoft streamMP4rocksoftlog traffic statsjpegscreen freezecan't registeraccuracyprivate networkLarge DVR fileslive videodebianTelegraminterleavingSLDP PLayerseekingmac osx installwotermarklive video on demandbandwithnimblesessionidFFmpegmultiple originsvimeohelp errorserverload balancezabbixheadercache expirycdnvsomlog nimble analysertmp playbacklive streamsourceViewer StatsAdsvideojsLive Streamingbitrate#restreamMP4 not playedspaceServer-Side-Task-Controlvideo stopaes encryptionsubscriptionvod no soundconcurrent connectionlebanonlocationdvr streamconnectivityUbuntu 20 ARM - AWSunique visitorstereo to monoscreen capturelive pull settingsWMSPanel settingsABR HLS Bitratesrtmp for YouTube4Kcrossdomain more then 1 domainWidevineFairplaycpunot foundCPU LoadpullAS3loggingnimble streamer vod hls transmuxingbuttAWS 3Cross Domaininsert logoinstall players setup ready to goDVRSettingstwitch larix broadcaster androidicecast metadataDVRStreamsAppleDelete recordspremium featureControl APIProgressivenimble streamer web server php script pageautomationLive streamingvideo loopUbuntu artful 17.10Transcoder MPEG DASHinvalidMPEG DASHstarttime duration seekpointadd_chunk failedPost processingfake extensionstatus:errorrebootdroppingmobile app live streamingmonitoroutrateudp streamingapi accessDeep statsicecastMPEG-Dashloadbalancerweb playercontent-dispositionpaymentstatsNimble ServerAVCaptureMovieFileOutputblock downloadrestartdvr streamsavoid refreshlivestreamABR DVR problemNimble CapacityHotlinking ProtectionStreaming routehotlinkfacebookcontentRTMP RepublishSecurityLive Switcherdemandnot to stealLive BroadcastMultiple Audio trackssecure streaminglow_bandwidthwmsAuthSignnimble streamer mpeg-ts multiple inputsLoad-Balancingvp9adsounddisk migrationThumbnailsserver incorrect timeLIVELivelearnernimble webcam html5ssaiDVR Setting limitpausemosaicHow to do live stream with multiple audio trackrmtptranscoder nimblereportsdvr on wmspanelfpsVideo cant be playedS3 AWSraspberryPaywalllivestreamintransocderstorage space availablediskcredentialsdvr to livewebhookautomateuser_agentvbv-maxratedomain lockUsers limitoutputcloudflareVOD HLS streaming on public IPlarixscteInvalid frame headerno soundincoming streamSubtitleslivestream bitratedockerDVRRecordingAuthGopDVR SettingPORTdelaympeg4storagealias route.movartifactsrtsp push androidmultiviewRAM Loadhttp serverLatencywirecasta recordruleshow-tomanage_dvrError when installingVP9server ip21Nimble connectionsabr fallbackno audiolatencyaliasdata slicesplayer sldpmp2 audioNimble streamerfastspringsoftware versionadjustSO_RCVBUFbuffering videodistributionbufferqataritworkscdnvideo.jsmultiplexcompatibility protocolsCentos 8pay-per-minuteTeradek Decoderubuntu 183.6.1-1UDP MulticastMax connectionincomingmulti-viewerFallbackhelpmultiple audio udphotlink protectionRaspberryPi4ultra low latencyaws amazonhls to multicast udpnoobSSL requestsha265broadcasterOSXmd5analyseStreamIDprerollnimble on cloudrtmp sldp nimblepluginfallbacktasks-controlRaspberryRTMP to SRTLinuxgpu{stream}Connections count limitNimble_Crushwebrtc4GRIST Bondingplayoutsrt-live-transmitWowza AgentRemote StorageSSAItuningAmazonTVfireOBSvideo and audio not matchLarix broadcasterNetworkspeedup my videoiOSPerformanceSnapshotsSRT RTMPre-publishingletsencryptTonydowntranscodingFFMPEG;RTMP;I/O errorLL-HLSHTML5 playerAddendaABR DASHprogressive downloadeventLL-HLS DVRav1 codecconnectionsNimbleStreamernimble aliasesNimble StreambuildABR bitratesPacketizingcountrist set fecmultiple audioOBS StudioWMSAUTHIVSDelaympeg2tsIIS Media Servicsrtmp republishing transcodenimble.confinterfacetrackbindprogressivebroadcast videovideo audioSRT protocoldvbduplicatingpcdynamic linksTLSV 1.2 CertificatesubfolderbrowserurlGoogle DriveHLS Streamingrtmpscostnimble abr hlsthumbnail dvr-thumbnailaudio languageVR-360Failovergoogle cloud storagepricingoverlayhot linkingTRANSCODErtpnimble dvrBandwidthdurationYadifportsRokuContinuitywordpressJetson NanoSecureLarix PlayerramfairplayNginxpublish streamStream Delayhd25trafficAArch64video streamingQuickTimeCloud StorageRistmultistreamingmultipleserver ssl erroractionscript 3dropped framesCSSMPEG2 VideoPCRlogo in streammultipointrulerepublishconcurrent

This website or its third-party tools use cookies, which are necessary to its functioning and required to achieve the purposes illustrated in the Privacy Policy. If you want to know more or withdraw your consent to all or some of the cookies, please refer to the Privacy Policy.
By closing this banner, scrolling this page, clicking a link or continuing to browse otherwise, you agree to the use of cookies.