Function fit_within_encoder_limits 

pub fn fit_within_encoder_limits(
    width: u32,
    height: u32,
    format: OutputFormat,
) -> (u32, u32)

Clamp (width, height) so neither edge exceeds format’s hardware encoder limit (OutputFormat::max_encode_edge), preserving the aspect ratio and keeping both edges even (H.264 / HEVC require mod-2 dimensions).

Returns the input — even-rounded — unchanged when it already fits, or when the format has no hard cap (the software WebM encoders). When a clamp is needed, both edges are scaled by the same factor max_edge / max(width, height), so the frame is shrunk uniformly and never stretched or squished: the screen content keeps its shape and the camera bubble stays a perfect circle. Integer arithmetic throughout (u64 intermediate) — deterministic, no float casts.

This is the fix for AUT-334: a 5K display fed vtenc_h264_hw a 5120-wide frame, which fails caps negotiation and discards the whole recording. The live scratch is always H.264, so the recorder clamps capture dims to the H.264 limit before any pipeline starts.

Examples

use media::encode::{fit_within_encoder_limits, OutputFormat};
// 5K and 6K displays (both 16:9) clamp to exactly 4096×2304 for H.264:
assert_eq!(fit_within_encoder_limits(5120, 2880, OutputFormat::Mp4H264Aac), (4096, 2304));
assert_eq!(fit_within_encoder_limits(6016, 3384, OutputFormat::Mp4H264Aac), (4096, 2304));
// Already within the limit → unchanged:
assert_eq!(fit_within_encoder_limits(3840, 2160, OutputFormat::Mp4H264Aac), (3840, 2160));
// H.265's higher ceiling keeps full 5K:
assert_eq!(fit_within_encoder_limits(5120, 2880, OutputFormat::Mp4H265Aac), (5120, 2880));