Just to follow up after lot of trial and error and help for the support folks I did get this setup to work. See details below:
1. Disable Windows Defender on receiving nimble - this was causing all kinds of packet out of order issues, bandwidth spikes, and latency problems.
2. Disabled Windows firewall on the NIC sending out the UDP Multicast stream
3. This one maybe specific to the hardware decoder and the stream I'm sending but I needed to enable the MUX Rate options on the out going UDP streams and set the MUX delay to 4000ms. I think this value has to do with my older h.264 hardware decoders I'm using that are part of my satellite uplink. If I was doing this with one of my MPEG2 video streams I suspect the default 700ms value would work just fine.
4. This last one is probably specific to my IRD I'm using to decode the video but I had to set it's multicast input stream buffer to it's lowest setting of 50ms anything higher and I had issues decoding the stream. This one wasn't very intuitive to figure out. Since I was having problems I was raising the size which didn't help.