NVIDIA TensorRT Edge-LLM
TensorRT Edge-LLM is NVIDIA’s optimized C++ inference runtime designed for running LLMs and VLMs on embedded platforms. Its deployment flow converts trained models into highly optimized TensorRT engines, which are then executed by a lightweight native runtime at inference time. Because the runtime loads and serves these engines directly without relying on Python in the inference path, it is better suited for production edge deployment.
Upstream project: TensorRT Edge-LLM on Jetson
- Category: Model Optimization Edge LLM
NVIDIA TensorRT Edge-LLM
- Qwen3-4B-Instruct LLM
Qwen3-4B-Instruct is used as the target text-only LLM for instruction-following, text generation, and edge-side chatbot inference. - TensorRT Edge-LLM Inference Pipeline
The Hugging Face model is downloaded on Jetson Thor, quantized to NVFP4, exported to ONNX, compiled into a TensorRT engine, and executed through the native C++ inference binary. - NVFP4 Runtime Optimization
NVFP4 quantization is used on Jetson Thor to reduce model weight footprint and improve edge deployment feasibility with TensorRT engine optimization.
Supported Platform
| Platform | Hardware Spec | OS | Edge AI SDK |
|---|---|---|---|
| AIR-075 | NVIDIA Jetson Thor - RAM: 128/64 GB, Storage: 512 GB | JetPack 7.1 | Install |