Published using Google Docs
Stable Diffusion How-To's
Updated automatically every 5 minutes

Stable Diffusion How-To’s

  1. HOW TO write prompts (brief)
  2. WHAT TO MESS WITH when making images
  3. HOW TO FIX an incomplete/messed-up part of the image
  4. HOW TO REMOVE elements from an image
  5. HOW TO REDRAW a small detail or area
  6. HOW TO GENERATE same-but-different variations on an image
  7. FIXING torso-necked freaks
  8. Extra notes that didn’t fit elsewhere

donp@aeinnovations.com - Nov 2022


Intro

I’ve been writing down all the useful little workflows that help me out. Here they are, and hopefully they help you, too.

Wen making images you want to:

  1. Generate the kind of images you want (with a good PROMPT), and
  2. Tweak results when they’re Not Quite Right™

This assumes you are using Stable Diffusion Web UI. If a setting isn’t mentioned, it assumes the default.


Making Prompts

A good starting guide is here. That should get you going. Play with prompts until you have one that gets you decent results fairly consistently.

Here’s an example one that I (more or less) used to make every image in the following

Watercolor painting of a female astronaut on an alien planet, expressing a sense of purpose. The astronaut wears a helmet. In the style of artist Moebius and artist Geoff Darrow and Mass Effect.

All these examples use seed 2178531582 and checkpoint v1-5-pruned-emaonly.ckpt if you want to try to replicate them.


Generating Images - Notes

Keep in mind that everything relates to visual references (i.e. pictures on the internet.) This concept extends to things like image size, not just content.

For example::

So if you’re not getting the results you want, always remember to try to relate it in terms of what you’d see if your whole world was “pictures you can find on the internet”.

Here are some general notes on generating images (that don’t have to do with the prompt itself.)

What?

…and?

Notes / Why Should Care

Example

Sampling Method and Sampling Steps

Euler a, and 20

Very creative, good starting point. Fast.

(Feel free to experiment with others but Euler a and 20 sampling steps is fine for generating stuff to start with.)

Batch Size

1-3 should work fine on a GTX1080.

Will error if your hardware can’t hack it.

How many images to generate using your video card at once.

A batch size of 3 makes 3 images at once.


Why this matters?

Making 3 at once is faster than doing three singles in a row.

Batch Count

1, 2, or 3 (more gets clumsy)

Repeat the batch this many times.

Making too many at once is hard to manage.

Shown here is batch size 3, batch count 2 (total 6 images)

Width / Height

Ones I find useful:

768 x 512

384 x 768

384 x 896

512 x 768

Aspect ratio can affect results a lot.

Try to match what you want to an appropriate width & height.


Meaning: if you want something that looks like it could be a poster, give it a poster-like aspect ratio.

Bigger takes longer and needs more VRAM (which your card may or may not have)


384 x 768 is tall and thin, like cell phone selfies.

384 x 896 moreso

Tends to generate single figures (since it’s thin)

512x768 is like a magazine cover, or a poster.

Tends to slightly prefer cutting off feet (or tops of heads) a little to fit everything in the frame. Think movie posters.

768x512 is wide, like landscapes and photos.

Given our prompt of “...on an alien planet” this size will tend to include landscape backgrounds.

Wide images are more prone to making “group shots” with multiple people in the image.

put another one here with 2+ peeps

512x512 is square like an album cover or postage stamp. Sometimes useful.

Sampling Steps

Euler a with 20 steps is very creative and can create very different images with small changes in number of sampling steps. Perfectly serviceable for messing around.

Seed

CFG Scale

How strongly the image generation should try to obey the prompt.


INPAINTING (Fixing Stuff)

Be mindful that the prompt still applies!  The prompt should describe the full (new) image, not just the masked area!  Normally you will just use the same prompt as for the original image. (Clicking Send to inpaint should take care of copying the prompt over.)

Inpainting to replace/redraw an incomplete/messed-up part

Problem: one boot looks wrong

We generated this image and we’re pretty happy with it, except the right foot/boot looks a little messed up.

Would you like that fixed up while keeping the rest of the image unchanged?

Here’s a workflow for making that happen.

First click send to inpaint

Note: you can also drop images directly into inpaint BUT clicking send to inpaint does a couple important other things too:

  • It copies over the prompt
  • It pre-selects the right width/height

Use the mouse tool (and slider for tool size) to paint over the trouble spot. This is called a mask.

Don’t sweat getting it exact. It’s better to paint a little too big than a little too small.

Everything under the mask will get re-drawn, so if the “fix” needs to be bigger than the original, make sure to make the mask big enough.

To get the result shown (and others to choose from) here is a good starting point:

Masked Content: Fill or Original (try both)

Sampling Steps: 25

Sampling Method: DPM2 a (best at sticking to original elements, like redrawing a boot)

Batch Count: 3

Batch Size: 4

Denoising Strength: 0.83

Inpainting to erase stray or undesired element(s)

Problem: remove extra “stuff”:

txt2img generated this, but we’d like to remove the dangling cords, and the bit of text in the lower right corner.

  • Send to inpaint
  • Mask the undesired elements
  • Use the following settings as a good starting point:

Masked Content: Fill

Sampling Steps: 30 (can be as high as 80-100)

Sampling Method: DDIM (best at this type of job)

Batch Count: 3

Batch Size: 4

Denoising Strength: 0.61 (or thereabouts)

Note: In this example I mask out three elements at once, but it is often best to do one at a time, re-sending to inpaint and re-masking as needed. (It’s easier to focus on and fix one thing at a time than to wait for the algorithm to get all three right at once.)  But for simple changes like this example, doing all three is NBD.

Inpainting to redraw some detail (like fix a missing/messed up limb)

Problem:raised (right) hand looks weird and needs fixing.

  • Send to inpaint
  • Mask the undesired elements
  • Use the following settings as a good starting point:

Masked Content: Fill or Original (try both)

Sampling Steps: 30 (can be as high as 80-100)

Sampling Method: DDIM or DPM2 or DPM2a (see below)

Batch Count: 3

Batch Size: 4

Denoising Strength: 0.97 (higher gets really weird)

Best sampling method depends on needs.

DDIM: best for backgrounds, patterns, etc.

DPM2: best for adding things that weren’t there before.

DPM2 a: best at trying to stick to whatever was under the mask before..

It’s good to create a lot of images (hence the batch count and batch size) so you can choose from a lot of different results and pick the best one you see (re-sending it to inpaint for further refinement if necessary.)


Creating (conservative) Image Variations with img2img

Be mindful that the prompt still applies!  Normally you will just use the same prompt as for the original image. (Clicking Send to img2img should take care of copying the prompt over.)

Generate a range of conservative variations of an image easily.

  1. See something you like from txt2img
  2. Click Send to img2img (this copies over the prompt and size)
  3. Use the following settings as a starting point:

Sampling Steps: 30

Sampling Method: DDIM

Batch Count: 3

Batch Size: 4

CFG Scale: 8

Denoising Strength: 0.50 - 0.65 (higher is more creative in variations)

Feel free to up denoising as high as 0.8 or even in the 0.90s, but the images generated will stray more and more away from the original.

Results are basically the same as the original, just some different takes.

Feel free to play with: Sampling method, CFG Scale, and Denoising Strength.

Examples of variations generated from the above image and settings, using 0.60 denoising.


AUUGGGHH I’M GETTING TORSO-NECKED FREAKS (Centaurs)

Centaurs are when you get messed up science experiments like the images above.

Here are some ways to stop these harmless but grotesque abominations.

What to do

What that does

Change the image height to be less tall

Making the image less tall can help “force” a more normal looking figure where a centaur was before.

Check off the “Highres. fix” box

Repeat the seed (click recycle symbol)

Then re-generate the job

Due to how the highres fix process works, centaur generation is nipped in the bud.


Downside: takes much longer to generate

Less reliable: modify the prompt

Add singular terms like specifying “one astronaut” or adding “solo” to the prompt.

Some people advocate a salad of negative prompt magic words but I haven’t found that very reliable.

Ignore them, generate more batches

Easiest and works pretty good


Additional Notes and Comments on Common Settings

Important Settings

Comments

Masked Content:

FILL

ORIGINAL

  • Fill = make something up, fitting the rest of the image.
  • Original = make an effort to keep whatever was there before.

If one is not getting you anywhere, switch to the other one.

They can both work sometimes, one better than the other in different situations.

Sampling Method:

The following are most useful for inpainting jobs:

  • DDIM best for filling in areas and erasing stuff from existence

  • DPM2 is more creative than DDIM, more suited to body parts, etc.

  • DPM2 a (ancestral) is best for redrawing an existing element.

DDIM is generally best at inpainting and worth trying. Best for filling in areas and backgrounds, etc.  The higher the sampling steps generally the better the blending in.

DPM2 a is good at sticking to the original source (like redrawing something that was there before)

DPM2 for redrawing with more creativity and less bound by what was there before.

Euler a is creative as usual, try it if you’re getting nowhere otherwise. Sooner or later something will work.

Denoising strength:

  • 0.61 means “make some variations, but stick to the script.”

  • 0.83 is “we need to change this up, but keep a lid on it.”

  • 0.97 is “shake things up, throw out the rulebook.”

Denoising is how little the inpainting process should care about what’s already in the image.

Lower # means respect what’s already in the image more.

Higher # means care less about what’s already drawn.

(This refers to the image as a whole, not just what’s under the mask.)

Final notes:

Be mindful that the prompt still applies!  The prompt should describe the full (new) image, not just the masked area!  Normally you will just use the same prompt as for the original image. (Clicking send to inpaint should take care of copying the prompt over.)