fastapi-raft-lora
A LoRA adapter for Qwen2.5-3B-Instruct, fine-tuned with RAFT (Retrieval-Augmented Fine-Tuning) to answer questions about the FastAPI documentation from retrieved chunks. Given 4 documents (some relevant, some distractors) and a question, it writes short reasoning with verbatim quotes and then a final answer.
This is a small portfolio project, not a production model. Code, dataset script and write-up: https://github.com/Oualidlaib/fastapi-raft
Intended use
Answering FastAPI questions inside a RAG pipeline. It is not a standalone knowledge model, and it is not meant for questions outside the retrieved documents.
Prompt format
The adapter was trained on exactly this format. Other formats give worse results.
<DOC>
...retrieved chunk...
</DOC>
(4 chunks in total, separated by blank lines)
Question: your question
Output format:
Reasoning: ##begin_quote##exact sentence from a document##end_quote## one sentence linking it to the question.
<ANSWER>: final answer
How to use
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel
BASE_MODEL = "Qwen/Qwen2.5-3B-Instruct"
ADAPTER = "oualid2003/fastapi-raft-lora"
tokenizer = AutoTokenizer.from_pretrained(ADAPTER)
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.float16)
base = AutoModelForCausalLM.from_pretrained(BASE_MODEL, quantization_config=bnb, device_map={"": 0})
model = PeftModel.from_pretrained(base, ADAPTER).eval()
Build the prompt from your retriever's top-4 chunks as shown above, then call model.generate. After generation, check that every ##begin_quote##...##end_quote## text appears in the retrieved chunks, and discard answers where it does not (see Limitations). A full example is in the GitHub repository.
Training data
264 synthetic examples (235 train, 26 validation) generated from chunks of the FastAPI documentation:
- An LLM (gpt-oss-120b via Groq) wrote 3 questions per chunk and a reasoning answer with verbatim quotes.
- Each example shows 4 documents: the source chunk (the "oracle", included in about 80% of examples) plus nearest-neighbor chunks as hard distractors.
- Automatic filters removed template/stub chunks, questions that depend on the passage, malformed or non-verbatim quotes, answers without real reasoning, and answers that leaked the generation prompt.
Training procedure
| Setting |
Value |
| Method |
QLoRA (4-bit NF4), LoRA rank 16, alpha 32, dropout 0.05, all linear layers |
| Loss |
Answer tokens only |
| Learning rate |
1e-4, cosine, 10 warmup steps, weight decay 0.01 |
| Effective batch size |
4 |
| Epochs |
Up to 5 with early stopping (patience 4 evaluations) |
| Hardware |
Kaggle T4 |
Training loss fell from about 1.85 to about 0.12. Validation loss was lowest (about 0.56) at step 116 (epoch 2) and then rose to about 0.8, so this repository contains the step-116 checkpoint. Training run: https://wandb.ai/laiboualid2003-none/fastapi_tune-v2/runs/7rwzlo9n
Evaluation
No formal benchmark. On 6 hand-picked questions run through retrieval plus this adapter, in-scope questions were mostly answered correctly. For questions whose answer was not in the retrieved chunks, the model sometimes invented a quote and answered from memory.
License
This adapter is derived from Qwen2.5-3B-Instruct and follows the Qwen Research License (qwen-research). The training data is derived from the FastAPI documentation.