Queue Time nv_inference_queue_duration_us Cumulative time requests spend waiting in the scheduling queue (includes cached requests) Per model Per request Compute Input Time ...

The values in the range 1-3 are optimal. While trying out different values, note how it affects total inference time as well as some inference statistics (like queue and compute times) For models that ...