AutoPipeline

AutoPipeline is a task-and-model pipeline that automatically selects the correct pipeline subclass based on the task. It handles the complexity of loading different pipeline subclasses without needing to know the specific pipeline subclass name.

This is unlike [DiffusionPipeline], a model-only pipeline that automatically selects the pipeline subclass based on the model.

[AutoPipelineForImage2Image] returns a specific pipeline subclass, (for example, [StableDiffusionXLImg2ImgPipeline]), which can only be used for image-to-image tasks.

import torch
from diffusers import AutoPipelineForImage2Image

pipeline = AutoPipelineForImage2Image.from_pretrained(
  "RunDiffusion/Juggernaut-XL-v9", torch_dtype=torch.bfloat16, device_map="cuda",
)
print(pipeline)
"StableDiffusionXLImg2ImgPipeline {
  "_class_name": "StableDiffusionXLImg2ImgPipeline",
  ...
"

Loading the same model with [DiffusionPipeline] returns the [StableDiffusionXLPipeline] subclass. It can be used for text-to-image, image-to-image, or inpainting tasks depending on the inputs.

import torch
from diffusers import DiffusionPipeline

pipeline = DiffusionPipeline.from_pretrained(
  "RunDiffusion/Juggernaut-XL-v9", torch_dtype=torch.bfloat16, device_map="cuda",
)
print(pipeline)
"StableDiffusionXLPipeline {
  "_class_name": "StableDiffusionXLPipeline",
  ...
"

Check the mappings to see whether a model is supported or not.

Trying to load an unsupported model returns an error.

import torch
from diffusers import AutoPipelineForImage2Image

pipeline = AutoPipelineForImage2Image.from_pretrained(
    "openai/shap-e-img2img", torch_dtype=torch.float16,
)
"ValueError: AutoPipeline can't find a pipeline linked to ShapEImg2ImgPipeline for None"

There are three types of AutoPipeline classes, [AutoPipelineForText2Image], [AutoPipelineForImage2Image] and [AutoPipelineForInpainting]. Each of these classes have a predefined mapping, linking a pipeline to their task-specific subclass.

When [~AutoPipelineForText2Image.from_pretrained] is called, it extracts the class name from the model_index.json file and selects the appropriate pipeline subclass for the task based on the mapping.