Mouser Left Banner
Mouser Left Banner
Mouser Left Banner
Mouser Right Banner
Mouser Right Banner
Mouser Right Banner

    Text-to-SQL Generation Using Fine-tuned LLMs on Intel GPUs (XPUs) and QLoRA

    Courtesy: Rahul Unnikrishnan Nair | Intel

    The landscape of AI and natural language processing has dramatically shifted with the advent of Large Language Models (LLMs). This shift is characterized by advancements like Low-Rank Adaptation (LoRA) and its more advanced iteration, Quantized LoRA (QLoRA), which have transformed the fine-tuning process from a compute-intensive task into an efficient, scalable procedure.

    Generated with Stable Diffusion XL using the prompt: “A cute laughing llama with big eyelashes, sitting on a beach with sunglasses reading in gibili style”

    The Advent of LoRA: A Paradigm Shift in LLM Fine-Tuning

    LoRA represents a significant advancement in the fine-tuning of LLMs. By introducing trainable adapter modules between the layers of a large pre-trained model, LoRA focuses on refining a smaller subset of model parameters. These adapters are low-rank matrices, significantly reducing the computational burden and preserving the valuable pre-trained knowledge embedded within LLMs. The key aspects of LoRA include:

    • Low-Rank Matrix Structure: Shaped as (r x d), where ‘r’ is a small rank hyperparameter and ‘d’ is the hidden dimension size. This structure ensures fewer trainable parameters.
    • Factorization: The adapter matrix is factorized into two smaller matrices, enhancing the model’s function adaptability with fewer parameters.
    • Scalability and Adaptability: LoRA balances the model’s learning capacity and generalizability by scaling adapters with a parameter α and incorporating dropout for regularization.
    Left: Integration of LoRA adapters into the model. Right: Deployment of LoRA adapters with a foundation model as a task-specific model library

    Quantized LoRA (QLoRA): Efficient Finetuning on Intel Hardware

    QLoRA advances LoRA by introducing weight quantization, further reducing memory usage. This approach enables the fine-tuning of large models, such as the 70B LLama2, on hardware like Intel’s Data Center GPU Max Series 1100 with 48 GB VRAM. QLoRA’s main features include:

    • Memory Efficiency: Through weight quantization, QLoRA substantially reduces the model’s memory footprint, crucial for handling large LLMs.
    • Precision in Training: QLoRA maintains high accuracy, crucial for the effectiveness of fine-tuned models.
    • On-the-Fly Dequantization: It involves temporary dequantization of quantized weights for computations, focusing only on adapter gradients during training.

    Fine-Tuning Process with QLoRA on Intel Hardware

    The fine-tuning process starts with setting up the environment and installing necessary packages, including bigdl-llm for model loading, peft for LoRA adapters, Intel Extension for PyTorch for training using Intel dGPUs, transformers for finetuning and datasets for loading the dataset. We will walk through the high-level process of fine-tuning a large language model (LLM) to improve its capabilities. As an example, I am taking generating SQL queries from natural language input, but the focus is on general QLoRA finetuning here. For detailed explanations you can check out the full notebook that takes you from setting up the required python packages, loading the model, finetuning and inferencing the finetuned LLM to generate SQL from text on Intel Developer Cloud and also here.

    Model Loading and Configuration for Fine-Tuning

    The foundational model is loaded in a 4-bit format using bigdl-llm, significantly reducing memory usage. This step is crucial for fine-tuning large models like the 70B LLama2 on Intel hardware.

    from bigdl.llm.transformers import AutoModelForCausalLM

    # Loading the model in a 4-bit format for efficient memory usage

    model = AutoModelForCausalLM.from_pretrained(

    “model_id”,  # Replace with your model ID






    Learning Rate and Stability in Training

    Selecting an optimal learning rate is critical in QLoRA fine-tuning to balance training stability and convergence speed. This decision is vital for effective fine-tuning outcomes as a higher learning rate can lead to instabilities and the training loss to abnormally drop to zero after a few steps.

    from transformers import TrainingArguments

    # Configuration for training

    training_args = TrainingArguments(

    learning_rate=2e-5,  # Optimal starting point; adjust as needed



    # Additional parameters…


    During the fine-tuning process, there is a notable rapid decrease in the loss after just a few steps, which then gradually levels off, reaching a value near 0.6 at approximately 300 steps as seen in the graph below:


    Text-to-SQL Conversion: Prompt Engineering

    With the fine-tuned model, we can convert natural language queries into SQL commands, a vital capability in data analytics and business intelligence. To finetune the model, we must carefully convert the data into structured prompt like below as an instruction dataset with Input, Context and Response:

    # Function to generate structured prompts for Text-to-SQL tasks

    def generate_prompt_sql(input_question, context, output=””):

    return f”””You are a powerful text-to-SQL model. Your job is to answer questions about a database. You are given a question and context regarding one or more tables.

    You must output the SQL query that answers the question.

    ### Input:


    ### Context:


    ### Response:


    Diverse Model Options

    The notebook supports an array of models, each offering unique capabilities for different fine-tuning objectives:









    Enhanced Inference with QLoRA: A Comparative Approach

    The true test of any fine-tuning process lies in its inference capabilities. In the case of the implementation, the inference stage not only demonstrates the model’s proficiency in task-specific applications but also allows for a comparative analysis between the base and the fine-tuned models. This comparison sheds light on the effectiveness of the LoRA adapters in enhancing the model’s performance for specific tasks.

    Model Loading for Inference:

    For inference, the model is loaded in a low-bit format, typically 4-bit, using bigdl-llm library. This approach drastically reduces the memory footprint, making it suitable to run multiple LLMs with high parameter count on a single resource-optimized hardware like Intel’s Data Center GPUs 1100. The following code snippet illustrates the model loading process for inference:

    from bigdl.llm.transformers import AutoModelForCausalLM

    # Loading the model for inference

    model_for_inference = AutoModelForCausalLM.from_pretrained(

    “finetuned_model_path”,  # Path to the fine-tuned model

    load_in_4bit=True,  # 4 bit loading






    Running Inference: Comparing Base vs Fine-Tuned Model

    Once the model is loaded, we can perform inference to generate SQL queries from natural language inputs. This process can be conducted on both the base model and the fine-tuned model, allowing users to directly compare the outcomes and assess the improvements brought about by fine-tuning with QLoRA:

    # Generating a SQL query from a text prompt

    text_prompt = generate_sql_prompt(…)

    # Base Model Inference

    base_model_sql = base_model.generate(text_prompt)

    print(“Base Model SQL:”, base_model_sql)

    # Fine-Tuned Model Inference

    finetuned_model_sql = finetuned_model.generate(text_prompt)

    print(“Fine-Tuned Model SQL:”, finetuned_model_sql)

    Following a 15-minute training session itself, the finetuned model demonstrates enhanced proficiency in generating SQL queries that more accurately reflect the given questions, compared to the base model. With additional training steps, we can anticipate further improvements in the model’s response accuracy:

    Finetuned model SQL generation for a given question and context:

    Base model SQL generation for a given question and context:

    LoRA Adapters: A Library of Task-Specific Enhancements

    One of the most compelling aspects of LoRA is its ability to act as a library of task-specific enhancements. These adapters can be fine-tuned for distinct tasks and then saved. Depending on the requirement, a specific adapter can be loaded and used with the base model, effectively switching the model’s capabilities to suit different tasks. This adaptability makes LoRA a highly versatile tool in the realm of LLM fine-tuning.

    Checkout the notebook on Intel Developer Cloud

    We invite AI practitioners and developers to explore the full notebook on the Intel Developer Cloud (IDC). IDC is the perfect environment to experiment with and explore the capabilities of fine-tuning LLMs using QLoRA on Intel hardware. Once you login to Intel Developer Cloud, go to the “Training Catalog” and under “Gen AI Essentials” in the catalog, you can find the LLM finetuning notebook.

    Conclusion: QLoRA’s Impact and Future Prospects

    QLoRA, especially when implemented on Intel’s advanced hardware, represents a significant leap in LLM fine-tuning. It opens up new avenues for leveraging massive models in various applications, making fine-tuning more accessible and efficient.

    ELE Times Report
    ELE Times Report
    ELE Times provides extensive global coverage of Electronics, Technology and the Market. In addition to providing in-depth articles, ELE Times attracts the industry’s largest, qualified and highly engaged audiences, who appreciate our timely, relevant content and popular formats. ELE Times helps you build experience, drive traffic, communicate your contributions to the right audience, generate leads and market your products favourably.

    Technology Articles

    Popular Posts

    Latest News

    Must Read

    ELE Times Top 10