TensorRT-LLM/examples/gemma/README.md at main · NVIDIA/TensorRT-LLM

Users can use convert_checkpoint.py to convert the different source checkpoint to unified TensorRT-LLM checkpoint format. Users could set --dtype to determine the inference data type, and set the quantization options like --enable_fp8, --fp8_kv_cache --use_smooth_quant, --calibrate_kv_cache (for INT8 kv cache) and --use-weight-only-with-precision (weight only). Users could also control the source checkpoint type by --ckpt-type. Currently, supported checkpoint types are jax, torch and keras. After getting checkpoint, we can use trtllm-build command to build TensorRT-LLM engines from TensorRT-LLM checkpoints. We provide three examples to run inference run.py, summarize.py and mmlu.py. run.py only run inference with input_text and show the output. summarize.py runs summarization on cnn_dailymail dataset and evaluate the model by ROUGE scores and use the ROUGE-1 score to validate the implementation. mmlu.py runs MMLU to evaluate the model by accuracy. Note that we need to download the data

saved by

• Pavel