Support MXINT4 scheme

[mengniwang95]

Description

Support MXINT4 scheme

How to use:

model quantization:

CUDA_VISIBLE_DEVICES=0 auto-round --model /models/Llama-3.2-3B/ --scheme MXINT4 --iters 0 --format auto_round

inference with transformers:

from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "tmp_autoround/Llama-3.2-3B-mxint-w4g32/"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")
input_ids = tokenizer("Hello my name is", return_tensors="pt").input_ids.to(
    model.device
)
output = model.generate(input_ids, max_new_tokens=100)
print(tokenizer.decode(output[0]))

Type of Change

• Bug fix
• New feature
• Documentation update
• Performance improvement
• Code refactoring
• Other (please specify):