LLM Fine-tuning

Collect and clean a domain-specific dataset that aligns with your fine-tuning objective. Remove duplicates, handle missing values, and format the data into prompt-response pairs or instruction-following examples. Split into training, validation, and test sets to enable proper evaluation.

Why Modal AI: Modal AI provides scalable batch data processing capabilities, which aligns with the need for running data preprocessing scripts and handling large datasets efficiently.

Choose a pre-trained base model (e.g., Llama 2, Mistral, GPT-2) that fits your compute budget and task. Set up the fine-tuning environment with GPU support, install dependencies (Transformers, PEFT, TRL), and configure logging and checkpointing.

Why Together AI: Together AI supports fine-tuning pretrained models on custom data, which directly matches the need for base model selection and environment setup with Hugging Face Transformers and PEFT.

Define a search space for key hyperparameters: learning rate, batch size, number of epochs, LoRA rank (if using PEFT), and warmup steps. Run a small-scale grid or Bayesian search on the validation set to identify the best combination. Use early stopping to avoid overfitting.

Why Ray: Ray provides distributed training capabilities that integrate with hyperparameter optimization tools like Optuna and Ray Tune.

Train the base model using the optimized hyperparameters and prepared dataset. Apply parameter-efficient fine-tuning (e.g., LoRA) to reduce memory usage. Monitor loss curves and validation metrics; stop training when validation performance plateaus or starts to degrade.

Why Together AI: Together AI directly supports fine-tuning pretrained models on custom data, matching the need for Hugging Face Trainer and PEFT/LoRA execution.

Evaluate the fine-tuned model on the test set using both automated metrics (e.g., ROUGE, BLEU, accuracy) and human evaluation for qualitative aspects. Compare against the base model and a baseline. If performance is insufficient, iterate by adjusting data, hyperparameters, or training strategy.

Why Argilla: Argilla provides model evaluation and DPO preference ranking, which directly supports the evaluation and iteration step with human annotation capabilities.

Convert the fine-tuned checkpoint to an optimized format for inference (e.g., ONNX, TensorRT, or quantized GGUF). Apply quantization (int8 or int4) to reduce model size and latency. Test inference speed and memory footprint on target hardware.

Why Together AI: Together AI supports deploying custom fine-tuned models to production, which aligns with model conversion and optimization for deployment.

Package the optimized model into a serving container (e.g., with FastAPI, Triton Inference Server, or vLLM). Set up an API endpoint with request batching, rate limiting, and monitoring. Deploy to a cloud or on-premise environment; integrate with the application backend.

Why vLLM: vLLM directly supports deploying and serving open-source LLMs with high throughput and continuous batching, matching the orchestration and deployment needs.