Diagnosing CDN 504 errors on live HLS

We use Nimble to take RTMP to live HLS, and Nimble is setup as CDN origin.

When we experience high viewer spikes, the CDN reports 504 Gateway Timeout errors on the TS chunks.

Odd thing is, there is no unusual spike in bandwidth, CPU or Memory on the Nimble server (all within very acceptable limits), and no errors in Nimble log.

Any suggestions on what to look for to diagnose this issue? The CDN is pointing the finger at our origin.

Thanks

Hello, Shane.

504 means Nimble could not be reached by CDN, this is either a high load to Nimble or some networking restriction or issues.

Just a guess, but does your hosting provider can limit connections to a server if there're too many of them? Have you tried to make multiple requests (e.g. by curl) to your server directly as 504 occurred?
Please enable access.log (as described in the below documentation page) to check if CDNs requests are reaching you Nimble at all.
https://blog.wmspanel.com/2016/08/logging-nimble-streamer.html

If you want us to check the issue, please file a ticket at wmspanel.com/help with more technical details.
Please name your server and stream you have an issue with, specify the exact time the issue happens, and attach CDNs and Nimble's log files for the specified time. (access.log and nimble.log)

Thank you.

OK, will do that and investigate a little further. Didn't consider throttling by provider - it's on a DigitalOcean instance. But bandwidth is low (25 Mbps or so). Since (theoretically) those viewers are hitting the CDN and not origin, it shouldn't be an issue of connection numbers.

However, I just looked at my CDN edge rules and for some reason I have it set to always request the m3u8's from origin and not cache, which would definitely spike connections with little bandwidth). Not sure why we have it setup that way. It should be fine to cache the m3u8's on the CDN too, correct?

At least for a short time (5 seconds?)...one of the two streams is DVR, the other is just straight live. Would caching playlists on the DVR channel cause issues? Thanks

It looks like we already discussed playlist caching in your previous tickets. Please continue this thread via wmspanel.com/help, as we cannot share sensitive information about your company set up there.

Thank you.