HOWTO: Lora models with Stable Diffusion

What is LoRA (Low-Rank Adaptation)?

LoRA, short for Low-Rank Adaptation, is a technique used to fine-tune large AI models (like language or vision models) efficiently and with fewer resources.

Instead of updating all the parameters in a massive pre-trained model—which is expensive and memory-intensive—LoRA freezes the original model and adds small, trainable layers (called low-rank matrices) to specific parts of the model (like attention layers). These additions learn the task-specific changes, allowing the core model to remain unchanged.

Using LoRA models with Stable Diffusion is a super popular way to customize the style, character, or theme of your image generations without retraining the whole model.

How to Use LoRA in Apolo platform with Stable Diffusion

Prerequisites:

• Install Apolo cli using this doc

• Clone this repo

Add secret value with personal Hugging Face Token

apolo secret add HF_TOKEN <your_token>

Run Stable Diffusion job, replacing the secret value, preset, and any other configuration

apolo-flow run stablediffusion

Download the model using SDnext interface

Go to Models -> HuggingFace

HuggingFace model download interface

Generate the first image:

Prompt:

Ghibli style futuristic stormtrooper with glossy white armor and a sleek helmet, standing heroically on a lush alien planet, vibrant flowers blooming around, soft sunlight illuminating the scene, a gentle breeze rustling the leaves

Lora model

Let's find a trained model on Civit.ai, we need to filter model by our Stable Diffusion version.

Civit.AI search models interface

Alternative can be HuggingFace search. (tags: Lora, Stable Diffusion, 3.5)

I will use "studio-ghibli-style-lora" Lora model.

Now we need to copy our model into the /Lora directory of our storage volume

apolo cp -r studio-ghibli-style-lora.safetensors storage:sdnext/models/Lora/studio-ghibli-style-lora.safetensors

After Model Copying, and hitting the refresh button on the Lora tab, we should be able to see our model downloaded.

Click on your Networks -> Lora tab, hit refresh, and click on your Lora model, that will add Lora to youre prompt, and you will be able to generate images using it.

For example we generated ghibli-style stormtrooper.

Ghibli style stormtrooper

References:

• Apolo cli

• SDnext repository

• Apolo secrets cli documentation

• Civit AI model storage