When I started to use local (i.e. on my PC, not in the cloud using some commercial service) AI image generation, I was somewhat frustrated by the lack of clear and succinct explanation of some of the basic concepts - how it all works, what the common terms of art mean. I want to close that gap somewhat with this article.

Without any particular ordering of importance:

• Typical AI image generation uses a couple of artificial neural networks (ANN (artificial neural network)s) to iteratively reshape an initially very noisy image into the desired output image. ANN (artificial neural network)s have to be trained on large amounts of training data, in these case pairs each consisting of an image with a corresponding text description of that image. The training is done by someone else for you, you just download the finished ANN (artificial neural network)s and use them.
• All ANN (artificial neural network)s that generate images like this are called diffusion-based because they try to run backwards the process of iteratively adding noise to an image, which is similar to the physical process of diffusion.

• Latent : Running the process directly on the image is too computationally expensive and thus it is run on compressed images instead. These compressed images are called latents. One pair of ANN (artificial neural network)s, called variational autoencoder (VAE) is responsible for the compressing/encoding (image to latent) and decompressing/decoding (latent to image) the image. The VAE is represented by the red line in the above image, and the latent is represented by the pink lines.
• Diffuser : Apart from the VAE, you also need an ANN (artificial neural network) called diffuser or U-net that does the actual image generation. It takes a noisy latent as input and outputs a less noisy latent. Apart from the input latent, it also needs information about what the user wants to be generated, how much noise should be removed, etc. . The diffuser is represented by the violet line in the above image.
• Text encoder : Finally, you also need a text encoder ANN (artificial neural network), often called CL​IP. That takes your text description, also called prompt, as input and outputs encoded text in a form that the diffuser ANN (artificial neural network) can use directly. That's because the diffuser ANN (artificial neural network) does not understand human-readable text. Some image generation models use several text encoders simultaneously. The text encoder is represented by the yellow lines in the above image, and the encoded text is represented by the orange lines.
• Checkpoint : All of these ANN (artificial neural network)s are usually, but not always, packaged in a file called a checkpoint for ease of download and ease of use.
• Software : You also need a software like ComfyUI or Automatic1111 or stable-diffusion.cpp or koboldcpp to run these models. The above image is a screenshot of part of the user interface of ComfyUI.
• GPU (Graphics Processing Unit) : If you want any reasonable amount of speed, you also need a less than 10 years old GPU (Graphics Processing Unit) with sufficient VR​AM. Some models will run fine in 4GB or 6GB, more sophisticated ones might need 32GB. Running on CPU (central processing unit) is possible, but agonizingly slow on normal consumer CPU (central processing unit)s. Nvidia GPU (Graphics Processing Unit)s are far easier to use and somewhat faster than AMD GPU (Graphics Processing Unit)s for image generation, due to better software support.
• Video : Video generation works in a similar way. The latent then is the compressed video, i.e. a sequence of individual video frames, each of those being a normal image. Because each frame is generated simultaneously, video generation is of course much slower.
• LoRA : Instead of just using 1 checkpoint, it is usual to add at least 1 LoRA. A LoRA modifies the abilities of a checkpoint like a software update changes the abilities of software, but application of a LoRA is reversible, i.e. not permanent. More than 1 LoRA can often be combined to add even more abilities, but not all combinations work. LoRAs are specific to a checkpoint, but LoRAs are often transferrable to other checkpoints that are similar to the checkpoint they were trained on. Transferring a LoRA to a checkpoint of a completely different architecture does not work and will result in an error message. LoRAs do not change the VAE. Some LoRAs require a specific trigger word to be used in the prompt or they will have no effect.

• Image to image : Instead of creating an image from scratch (text to image mode, T2I), the initial latent can be the encoded version of an already existing image and then this image can be modified (image to image mode, I2I). In this case, the denoising process is not run to completion. Instead it is run partially, so that the initial image is changed somewhat, but not completely replaced. See the image above.
• Inpainting : With special inpainting checkpoints, the I2I mode can be used in a much more controlled manner by providing an additional image mask. This mask has the same size as the output image and for each pixel contains either 1 or 0. The inpainting model changes only the pixels corresponding to a 1 in the mask and leaves the other pixels unchanged. In addition, the inpainting model makes sure that the image remains consistent at the transition lines between the masked and unmasked regions. Inpainting checkpoints need to be run in a special inpainting mode, because they need to be informed about the mask image. See here for a demonstration.
• Controlnets : There exist controlnets that further constrain the image generation so that it follows some given guidance image. An edge controlnet for example takes a guidance image containing only lines representing edges in the final output image and then constrains the image generation to generate edges (e.g. transition between foreground and background or transition between light and dark) in the given places. That allows you to transfer poses and faces from one image to another image. Apart from edge control (usually called canny control, after the Canny edge detector), there can be controlnets constraining the 3D shape (depth control), constraining only the pose of human bodies (pose control), constraining the brightness of individual pixels (often called QR code control) or constraining all low-frequency detail while leaving high-frequency detail unconstrained (blur control). Not all checkpoints have suitable controlnets. See here for a demonstration.
• IP-Adapter : Some kind of controlnet can constrain the generation of human faces so that generated faces strongly resemble a given reference photo.
• Image to Text : There are also multimodal large language models (LLMs), which are not used for image generation, but image analysis. They can read an image as input and provide a text description as output, which is effectively the inverse of the image generation process. That way you can create a new image that resembles a given reference image, but is not identical to it.

• Transformer architecture : The diffuser ANN (artificial neural network) can be either a convolutional ANN (artificial neural network) or a transformer ANN (artificial neural network) or a hybrid of the two. You don't need to worry about the difference, but convolutional ANN (artificial neural network)s are older and less flexible/capable and are slowly being replaced by transformer ANN (artificial neural network)s. Text encoders are always transformer ANN (artificial neural network)s, as far as I'm aware. VAE ANN (artificial neural network)s can also be convolutional ANN (artificial neural network)s or transformers or hybrids. If you watch the image generation process while it happens, you can usually recognize a transformer diffuser by the characteristic grid-like noise pattern it creates in the first few steps. See above.
• CFG : Many models can use a process called classifier free guidance (CFG). This allows you to shift between more creativity and stricter prompt adherence. It also allows you to use negative prompts, which contain elements you don't want to see in the output image. Using CFG halves the generation speed.

• Distilled models : There are models called distilled models that can be run in very few steps, but cannot use CFG. They can produce output in as low as 1 step, but usually 4 to 12 steps. Their output is less varied/creative and due to lack of CFG, they can't use negative prompts and might not follow prompts as strictly. But they are very good for quick drafts, which can be improved using non-distilled models in I2I mode. Distilled models go by name suffixes like LCM , DMD, DMD2 , Hyper , Turbo, Lightning, Schnell. Using them often requires changing the image generation settings, specifically the "sampler" and "scheduler" setting. Instead of using a full distilled checkpoint, the "distilledness" characteristic can often be extracted into a LoRA, so that a non-distilled checkpoint can be turned into the fast distilled variant by using this LoRA. Saving a LoRA takes far less space than saving a full checkpoint. See the images above showing two different distilled models producing output in just 8 steps.
• Optimal settings : Image generation software settings like "sampler", "scheduler", "number of steps", "CFG" usually need to be set to the value given the checkpoint provider. Using different values can produce output of lower quality or in some cases completely nonsensical output.
• Image resolution : Newer models can produce images in different sizes and aspect ratios, but some older models were trained using only a few specific image resolutions and don't produce useful output for other resolutions. Even newer models might produce their highest quality output only at a specific output resolution. Smaller resoluations are of course quicker to generate and can be used as drafts. Larger image resolutions also require far more VR​AM.
• Extra heads : If the characters in your generated images have extra heads or extra well-formed limbs, it could be an indication that the image resolution you are using is too high. The same goes if you asked for 1 character but get 2.
• Quantized models : Many models exist as quantized variants, going by name suffixes like "FP8", "Q6_0", "Int8", "nf4" and more. These are compressed models that might fit in smaller VR​AMs, but can suffer from reduced output quality. Some require specialized software or software components to use. The default models are usually FP16 or FP32 (overkill IMHO) models, which use 2 bytes (FP16) resp. 4 bytes (FP32) per ANN (artificial neural network) parameter. FP8 uses 1 byte per parameter and thus needs half the size of an equivalent FP16 model. The compression is lossy, not lossless. That's why it's usually not called compression and also why the output quality can be lower.
• Examples : There are too many image generation models to list them all, but here are a few well-known ones: Stable Diffusion 1.5; Stable Diffusion XL ; Pony (the same architecture as Stable Diffusion XL, but trained on different data and thus interpretes prompts differently); Illustrious (another variant of Stable Diffusion XL); Flux.1 ; Qwen Image ; Z-Image ; Krea2 ;
• Text rendering : Earlier models were completely incapable of producing legible text specified in the prompt, but beginning with Stable Diffusion 3.x and Flux.1, this ability became a standard in all new models.
• Editing images : Newer image generation models like Flux Kontext allow more complex editing operations of a reference image using a prompt to describe the desired change, like changing just the pose of a character and keeping everything else unchanged or changing day to night.
• Censoring : Many models are censored and will produce nonsense for certain prompts. It's not your fault, nothing can be done about it. You can try to find a checkpoint that had its censorship removed, but not all attempts at censorship removal are equally successful. Even uncensored models will not be capable of producing everything, they can only generate what they have seen during training. If you ask for a "supercalifragilisticexpialidocious cat", don't be surprised if you don't get what you want. Usually models will just ignore what they don't know.
• Randomness : All image generation is stochastic. That means you can just rerun the image generation with a different random seed (another setting in your software) and get a different result for the same prompt. You can also generate multiple images simultaneously using the "batch size" setting. But it will slow down image generation and need a lot more VR​AM. On the other hand, batched image generation can be more efficient, i.e. generating 4 images simultaneously might take less time than generating them sequentially.
• Upscaling images : There are specialized upscaling ANN (artificial neural network)s like RealESRGAN that can be used to increase the resolution of your images after generation (usually up to 4x the original height and width). That is much faster than generating the image at the equivalent output resolution. The upscaling process also goes by the name superresolution.
• Workflow files : Software like ComfyUI puts all the image generation settings that were used to generate an image into a file called a workflow. That workflow is included by default in the output image and can be reused. That way it is easy to produce another image (e.g. with a different random seed) with the same workflow. Workflows can be used by just dragging and dropping the generated image into the ComfyUI window. Workflows can also be shared as individual files. But to generate exactly the same image again, you need more than just the workflow. You need the same checkpoint, all used LoRAs and often the same software version.

Written by the author; Date 05.09.2026; Updated 09.09.2026; © 2026 spinningsphinx.com