Fine-Tuning Open-Weight LLMs: Training, Licensing, and Deploying Custom Models with Ollama

When you take a foundation model such as Meta Llama, train it further using target-domain datasets (Domain-Specific Pre-training or Supervised Fine-Tuning/SFT), and save the updated weight checkpoints to a Model Store, the resulting model is technically classified as a Fine-Tuned Specialized Model or a Domain-Upgraded Model.

It surpasses the base model in specific operational context, instruction-following capabilities, or internal knowledge, making it a customized enterprise asset.

Applying the Workflow to Open-Weight Models (e.g., Kimi)

This workflow applies universally to open-weight LLMs, including models like Moonshot AI’s Kimi series (e.g., Kimi k1.5) or other open-weight frameworks.

• Download: Fetch open weights from Hugging Face or public repositories.
• Train: Execute SFT, LoRA/QLoRA, or DPO (Direct Preference Optimization) on local or cloud GPU clusters.
• Register: Store the merged weights or adapter weights in a centralized Model Store (e.g., MLflow, Hugging Face Hub, AWS S3).
• Deploy: Serve the custom model internally for dedicated enterprise workflows.

Note: Always verify the source model's underlying license terms prior to commercial deployment.
 

Overcoming the Meta Llama MAU Limit

The Meta Llama Community License Agreement explicitly includes a limit: if the monthly active users (MAU) of the products or services leveraging Llama exceed 700 million in the preceding calendar month, the user must request a specific commercial license directly from Meta.

To legally lift or bypass this limit:

• Enterprise Licensing Agreement: Contact Meta directly to negotiate a custom enterprise commercial contract. Meta grants custom terms for high-scale enterprise partners.
• Alternative Base Models: If direct licensing negotiation is not feasible, migrate to models governed by OSI-approved open-source licenses, such as Apache 2.0 or MIT, which carry no user-count caps.

Legal Status of Modifying and Fine-Tuning

Fine-tuning a model, altering its underlying code, or converting file formats does not void or bypass the original license. The resulting model is legally classified as a Derivative Work. Consequently, derivative works inherit the foundational license terms, attribution requirements, and usage limitations of the parent model.
 

The Training and Deployment Lifecycle with Ollama

Ollama functions strictly as an inference engine and local runtime, not a model training framework. To train a model loaded via Ollama and redeploy it:

• Extraction/Download: Source the model weights in Hugging Face format (safetensors).
• Training: Fine-tune the model using dedicated frameworks (e.g., Unsloth, Axolotl, LLaMA-Factory, or Hugging Face TRL) using GPUs.
• Weight Merging: Merge LoRA adapters back into the base model weights.
• GGUF Conversion: Convert the output safetensors directory into GGUF format using llama.cpp scripts:

python convert_hf_to_gguf.py ./my-finetuned-model --outtype f16   

• Quantization (Optional): Quantize the F16 GGUF file to lower precision (e.g., Q4_K_M or Q8_0) for resource-efficient inference:

./llama-quantize ./my-finetuned-model/gguf-model.gguf ./my-finetuned-model/gguf-model-q4.gguf Q4_K_M   

• Import to Ollama: Create a Modelfile:

FROM ./gguf-model-q4.gguf
SYSTEM "You are a specialized enterprise AI assistant."   

Build and run the new model in Ollama:

ollama create custom-model -f Modelfile
ollama run custom-model   

Any restrictions on registering a new model in Ollama? What about the cost?

There are no monetary costs or artificial restrictions when registering a new model in Ollama because it is a free, open-source framework designed for local execution. You can create and register custom models using a Modelfile without paying any fees or needing approval from a central platform. The only practical limitations you will encounter are determined by your system's hardware capabilities—such as available GPU VRAM, RAM, and storage capacity—along with ensuring the model is in a compatible format like GGUF and complies with its original licensing terms.

GGUF Format Conversion vs. Licensing Limits

Converting model weights to GGUF, EXL2, or AWQ formats is a technical format transformation and quantization process.

• Legal Effect: It does not erase or alter the underlying license.
• Converting a Meta Llama model into GGUF format retains the Meta Llama Community License constraints.
• Commercial restrictions, MAU thresholds, and attribution clauses persist regardless of the storage format or runtime engine used.

Top Apache 2.0 & MIT Open-Source LLMs

To eliminate MAU caps and proprietary licensing constraints, consider these open-source LLMs that perform competitively against or surpass Llama models across various benchmarks.

If we take DeepSeek-V3 and train it intensively, wouldn't it be possible for it to surpass even Kimi or Anthropic?

Given its strong foundational architecture and efficiency, rigorous post-training, fine-tuning, and scaling could unlock performance levels that challenge and even outperform the current top AI systems in the industry.

Enterprise System Architecture for Open-Source LLMs

Below is an enterprise-grade, end-to-end system architecture blueprint designed for self-hosted open-source models (such as Qwen 2.5 or DeepSeek-R1).

Component Breakdown

• Infrastructure & GPU Layer: Managed Kubernetes cluster (EKS/GKE/Bare-Metal) running NVIDIA GPU nodes, isolated on private VPC subnetworks.
• Continuous Training & Pipeline Layer:
  • Data Pipeline: Preprocessing and anonymizing proprietary data.
  • Training Engine: Executing LoRA/SFT pipelines using Unsloth or Axolotl with DeepSpeed ZeRO-3 optimizations.
  • Model Registry: Storing baseline and fine-tuned checkpoints in MLflow.
• Quantization & Build Pipeline:
  • Automated conversion pipeline generating high-throughput weights (AWQ for vLLM) and edge/local runtimes (GGUF via llama.cpp).
• Inference & Serving Layer:
  • Production Serving: vLLM or TensorRT-LLM running on multi-GPU nodes with continuous batching and PagedAttention enabled for maximum throughput.
  • Internal Dev Serving: Ollama endpoints servicing internal developer sandboxes using quantized GGUF models.
• Orchestration, RAG & Security Layer:
  • RAG Pipeline: Context augmentation using vector databases like Qdrant or Milvus.
  • Guardrails: Input/output sanitization and PII masking via NeMo Guardrails or Llama Guard.
  • API Gateway: Load balancing, rate limiting, and standard authentication (OpenID Connect).

End-to-End Training Workflow Pipeline

A robust MLOps pipeline for fine-tuning LLMs consists of five distinct, sequential phases:

1. Data Preparation

• Collection: Gathering domain-specific text, instruction-response pairs, or conversation logs.
• Cleaning: Removing PII (Personally Identifiable Information), duplicates, low-quality text, and formatting errors.
• Formatting: Structuring into standardized JSONL formats (e.g., Alpaca format or OpenAI messages format: {"messages": [{"role": "user", ...}, {"role": "assistant", ...}]}).

2. Model & Environment Setup

• Base Weight Loading: Fetching the base MIT model from repositories like Hugging Face.
• Quantization Setup: Applying 4-bit or 8-bit quantization (NF4/FP4 via bitsandbytes) if using QLoRA.
• Tokenizer Configuration: Ensuring correct padding tokens, eos tokens, and chat templates.

3. Training Execution

• Supervised Fine-Tuning (SFT): Training the model on instruction-output pairs to learn style, domain knowledge, or tone.
• Alignment Fine-Tuning (Optional): Using Direct Preference Optimization (DPO) or Reinforcement Learning from Human Feedback (RLHF) to align model output preferences.
• Distributed Training: Utilizing tools like DeepSpeed, PyTorch FSDP, or Hugging Face Accelerate for multi-GPU scaling.

4. Evaluation & Validation

• Automated Benchmarks: Running evaluation suites (e.g., LM-Evaluation-Harness, MMLU, GSM8K).
• LLM-as-a-Judge: Using a stronger model (e.g., GPT-4) to score fine-tuned responses against a test set.
• Human Evaluation: Domain experts reviewing edge cases.

5. Deployment

• Export/Merge: Merging LoRA adapters back into base weights if lower latency is required, or serving adapters dynamically.
• Inference Engines: Loading the model into high-performance engines like vLLM, TGI (Text Generation Inference), or Ollama.

JSONL-based Dataset Example for LLM Training?

Refer to https://strcpy.com/engineering/982/A-Complete-Guide-to-LLM-Dataset-Formats-From-PreTr/ for more information

Versioning and Artifact Management Strategy

To ensure reproducibility and enable reuse of trained models, you must version three distinct components independently: Data, Code, and Model Weights.

1. Model & Adapter Versioning

• Decoupled Weights: Never re-save the massive base model if you used LoRA. Save only the lightweight LoRA adapters (~10MB to ~500MB).
• Semantic Versioning (SemVer): Tag adapters logically (e.g., adapter-v1.0.0-domain-finance).
• Model Registry: Use platforms like Hugging Face Hub (private/public), MLflow Model Registry, or Weights & Biases (W&B) Artifacts to store adapter weights along with metadata (hyperparameters, base model hash, metrics).

2. Dataset Versioning

• Tools: Use DVC (Data Version Control) or W&B Artifacts paired with Git.
• Lineage Tracking: Map exact dataset versions (e.g., data_v2023_10_12.jsonl) to specific training runs.

3. Execution Environment Versioning

• Containerization: Freeze CUDA drivers, PyTorch, Transformers, and PEFT library versions inside a Docker Image.
• Config Files: Store all hyperparameters (learning rate, rank, alpha, batch size) in version-controlled YAML files.

Continuous Learning via "Loop Engineering" (Data Flywheel)

Runtime user logs and feedback can be directly converted into training pipelines. This pattern is commonly known as the Data Flywheel or Continuous Fine-Tuning Loop.

Architecture of a Loop Engineering Pipeline

Step 1: Interaction & Feedback Capture

• Capture user input prompts and generated responses.
• Collect explicit feedback (thumbs up/down, edits, ratings) and implicit feedback (copying text, session duration, regeneration requests).

Step 2: Curation and Quality Filtering (Automated Gatekeeper)

Raw logs cannot be directly fed to a model due to noise, hallucination risk, and security risks.

• Rule-based Filters: Filter out short responses, error codes, and PII.
• LLM-as-a-Judge Filter: Use a stronger LLM to grade user interactions. Only retain prompt-response pairs that score above a specific quality threshold (e.g., 8/10).
• Preference Dataset Generation: If a user rejects an answer and generates a new one (or edits it), format this directly into a DPO Dataset Pair ({prompt, chosen_response, rejected_response}).

Step 3: Dataset Ingestion & Formatting

• Append validated, high-quality entries into an staging dataset store (e.g., BigQuery, S3).
• Automate conversion into the training schema (e.g., standard conversation JSONL format).

Step 4: Trigger-Based Automated Retraining (Continuous MLOps)

• Threshold Trigger: Automatically launch an orchestration pipeline (via Airflow, Prefect, or GitHub Actions) when N new high-quality samples (e.g., 5,000 samples) accumulate.
• Continuous SFT/DPO: Run a new fine-tuning job taking the previous LoRA adapter checkpoint as the base, or retrain from the original base model using the expanded cumulative dataset.
• Regression Guardrails: Run automated evaluations against a baseline evaluation set. If the newly trained model outperforms the production model without regressing on core tasks, automatically deploy it to staging/production.

The ultimate goal of the Automated Retraining process is to generate newly trained model weights (or LoRA adapters) with improved performance, convert them into the GGUF format readable by Ollama, and register them.

The basic mechanism and operational workflow are as follows:

• Ollama Model Update: Once the automated retraining pipeline (e.g., Airflow, GitHub Actions, etc.) completes, it executes the command ollama create custom-model -f Modelfile based on the generated latest GGUF file and Modelfile to overwrite the existing model name with the latest weights or register it with a new version tag (e.g., custom-model:v2).
• Automatic Reflection upon Application Restart: If the existing model name (e.g., custom-model:latest) is updated directly, the integrated application or service will naturally load and use the newly updated model the next time it restarts and calls the Ollama API.

Therefore, by connecting the entire process—from data collection to evaluation, GGUF conversion, and Ollama model registration—into a single pipeline, you can fully achieve a Data Flywheel (continuous learning loop) structure that automatically and continuously improves.