Healing a video stream without paying for a keyframe

Stream video over Wi-Fi long enough and you will lose a packet. What happens next is where remote desktops differ.

The expensive answer

The classic recovery is to request an IDR — a full keyframe that resets the decoder. It works, but a keyframe is enormous compared to the delta frames around it, and it arrives exactly when your network just proved it can’t deliver big things. On a constrained link you get the smear-then-snap: the picture breaks, then a bloated keyframe shoves itself through, then quality dips while the encoder pays the bill. Every loss event costs you a visible gulp.

The cheap answer

Modern encoders support reference frame invalidation: the client tells the host which frames it lost, and the encoder simply stops referring to them, predicting its next frame only from frames the client is known to hold. No reset, no giant frame — the stream heals forward.

Our wire does a conservative version of this. When a client reports a gap, the host invalidates everything through its newest sent frame and stamps a recovery point flag on the next frame it encodes. The client stops feeding its decoder at the gap and resumes at the flagged frame.

Why the conservative version

Because decoders disagree about broken reference chains. NVDEC will conceal missing references and show you something slightly wrong; Apple’s VideoToolbox hard-fails the whole session’s decode with an error. Feeding “probably fine” frames to a decoder that refuses to guess is how you turn one lost packet into a frozen window. The recovery-point handshake means the client never feeds a frame whose references it can’t prove — which is exactly the guarantee VideoToolbox wanted all along.

The visible result on a lossy link: instead of a second of smear and a bitrate spike, one frame of pause and on you go. Wi-Fi sessions feel like Ethernet sessions with occasional hiccups, rather than a different product.