You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
-[Patrick von Platen](https://github.com/patrickvonplaten)
53
+
-[ablattmann](https://github.com/ablattmann)
54
+
-[Patrick Esser](https://github.com/pesser)
55
+
-[owenvincent](https://github.com/owenvincent)
56
+
-[apolinario](https://github.com/apolinario)
57
+
-[Charles Packer](https://github.com/cpacker)
58
+
59
+
---
60
+
61
+
_If you have contributed and don't see your name on the list of contributors, please let one of the collaborators know about the omission, or feel free to make a pull request._
Copy file name to clipboardExpand all lines: docs/README-CompViz.md
+41-43Lines changed: 41 additions & 43 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,5 +1,6 @@
1
1
# Original README from CompViz/stable-diffusion
2
-
*Stable Diffusion was made possible thanks to a collaboration with [Stability AI](https://stability.ai/) and [Runway](https://runwayml.com/) and builds upon our previous work:*
2
+
3
+
_Stable Diffusion was made possible thanks to a collaboration with [Stability AI](https://stability.ai/) and [Runway](https://runwayml.com/) and builds upon our previous work:_
3
4
4
5
[**High-Resolution Image Synthesis with Latent Diffusion Models**](https://ommer-lab.com/research/latent-diffusion-models/)<br/>
5
6
[Robin Rombach](https://github.com/rromb)\*,
@@ -12,16 +13,15 @@
12
13
13
14
which is available on [GitHub](https://github.com/CompVis/latent-diffusion). PDF at [arXiv](https://arxiv.org/abs/2112.10752). Please also visit our [Project page](https://ommer-lab.com/research/latent-diffusion-models/).
[Stable Diffusion](#stable-diffusion-v1) is a latent text-to-image diffusion
17
18
model.
18
-
Thanks to a generous compute donation from [Stability AI](https://stability.ai/) and support from [LAION](https://laion.ai/), we were able to train a Latent Diffusion Model on 512x512 images from a subset of the [LAION-5B](https://laion.ai/blog/laion-5b/) database.
19
-
Similar to Google's [Imagen](https://arxiv.org/abs/2205.11487),
19
+
Thanks to a generous compute donation from [Stability AI](https://stability.ai/) and support from [LAION](https://laion.ai/), we were able to train a Latent Diffusion Model on 512x512 images from a subset of the [LAION-5B](https://laion.ai/blog/laion-5b/) database.
20
+
Similar to Google's [Imagen](https://arxiv.org/abs/2205.11487),
20
21
this model uses a frozen CLIP ViT-L/14 text encoder to condition the model on text prompts.
21
22
With its 860M UNet and 123M text encoder, the model is relatively lightweight and runs on a GPU with at least 10GB VRAM.
22
23
See [this section](#stable-diffusion-v1) below and the [model card](https://huggingface.co/CompVis/stable-diffusion).
23
24
24
-
25
25
## Requirements
26
26
27
27
A suitable [conda](https://conda.io/) environment named `ldm` can be created
@@ -44,16 +44,16 @@ pip install -e .
44
44
45
45
Stable Diffusion v1 refers to a specific configuration of the model
46
46
architecture that uses a downsampling-factor 8 autoencoder with an 860M UNet
47
-
and CLIP ViT-L/14 text encoder for the diffusion model. The model was pretrained on 256x256 images and
47
+
and CLIP ViT-L/14 text encoder for the diffusion model. The model was pretrained on 256x256 images and
48
48
then finetuned on 512x512 images.
49
49
50
-
*Note: Stable Diffusion v1 is a general text-to-image diffusion model and therefore mirrors biases and (mis-)conceptions that are present
51
-
in its training data.
50
+
\*Note: Stable Diffusion v1 is a general text-to-image diffusion model and therefore mirrors biases and (mis-)conceptions that are present
51
+
in its training data.
52
52
Details on the training procedure and data, as well as the intended use of the model can be found in the corresponding [model card](https://huggingface.co/CompVis/stable-diffusion).
53
53
Research into the safe deployment of general text-to-image models is an ongoing effort. To prevent misuse and harm, we currently provide access to the checkpoints only for [academic research purposes upon request](https://stability.ai/academia-access-form).
54
-
**This is an experiment in safe and community-driven publication of a capable and general text-to-image model. We are working on a public release with a more permissive license that also incorporates ethical considerations.***
54
+
**This is an experiment in safe and community-driven publication of a capable and general text-to-image model. We are working on a public release with a more permissive license that also incorporates ethical considerations.\***
55
55
56
-
[Request access to Stable Diffusion v1 checkpoints for academic research](https://stability.ai/academia-access-form)
56
+
[Request access to Stable Diffusion v1 checkpoints for academic research](https://stability.ai/academia-access-form)
57
57
58
58
### Weights
59
59
@@ -64,36 +64,37 @@ which were trained as follows,
64
64
194k steps at resolution `512x512` on [laion-high-resolution](https://huggingface.co/datasets/laion/laion-high-resolution) (170M examples from LAION-5B with resolution `>= 1024x1024`).
65
65
-`sd-v1-2.ckpt`: Resumed from `sd-v1-1.ckpt`.
66
66
515k steps at resolution `512x512` on "laion-improved-aesthetics" (a subset of laion2B-en,
67
-
filtered to images with an original size `>= 512x512`, estimated aesthetics score `> 5.0`, and an estimated watermark probability `< 0.5`. The watermark estimate is from the LAION-5B metadata, the aesthetics score is estimated using an [improved aesthetics estimator](https://github.com/christophschuhmann/improved-aesthetic-predictor)).
67
+
filtered to images with an original size `>= 512x512`, estimated aesthetics score `> 5.0`, and an estimated watermark probability `< 0.5`. The watermark estimate is from the LAION-5B metadata, the aesthetics score is estimated using an [improved aesthetics estimator](https://github.com/christophschuhmann/improved-aesthetic-predictor)).
68
68
-`sd-v1-3.ckpt`: Resumed from `sd-v1-2.ckpt`. 195k steps at resolution `512x512` on "laion-improved-aesthetics" and 10\% dropping of the text-conditioning to improve [classifier-free guidance sampling](https://arxiv.org/abs/2207.12598).
69
69
70
70
Evaluations with different classifier-free guidance scales (1.5, 2.0, 3.0, 4.0,
71
71
5.0, 6.0, 7.0, 8.0) and 50 PLMS sampling
72
72
steps show the relative improvements of the checkpoints:
python scripts/txt2img.py --prompt "a photograph of an astronaut riding a horse" --plms
94
+
python scripts/txt2img.py --prompt "a photograph of an astronaut riding a horse" --plms
94
95
```
95
96
96
-
By default, this uses a guidance scale of `--scale 7.5`, [Katherine Crowson's implementation](https://github.com/CompVis/latent-diffusion/pull/51) of the [PLMS](https://arxiv.org/abs/2202.09778) sampler,
97
+
By default, this uses a guidance scale of `--scale 7.5`, [Katherine Crowson's implementation](https://github.com/CompVis/latent-diffusion/pull/51) of the [PLMS](https://arxiv.org/abs/2202.09778) sampler,
97
98
and renders images of size 512x512 (which it was trained on) in 50 steps. All supported arguments are listed below (type `python scripts/txt2img.py --help`).
98
99
99
100
```commandline
@@ -131,73 +132,72 @@ optional arguments:
131
132
evaluate at this precision
132
133
133
134
```
134
-
Note: The inference config for all v1 versions is designed to be used with EMA-only checkpoints.
135
+
136
+
Note: The inference config for all v1 versions is designed to be used with EMA-only checkpoints.
135
137
For this reason `use_ema=False` is set in the configuration, otherwise the code will try to switch from
136
138
non-EMA to EMA weights. If you want to examine the effect of EMA vs no EMA, we provide "full" checkpoints
137
139
which contain both types of weights. For these, `use_ema=False` will load and use the non-EMA weights.
138
140
139
-
140
141
#### Diffusers Integration
141
142
142
143
Another way to download and sample Stable Diffusion is by using the [diffusers library](https://github.com/huggingface/diffusers/tree/main#new--stable-diffusion-is-now-fully-compatible-with-diffusers)
144
+
143
145
```py
144
146
# make sure you're logged in with `huggingface-cli login`
145
147
from torch import autocast
146
148
from diffusers import StableDiffusionPipeline, LMSDiscreteScheduler
147
149
148
150
pipe = StableDiffusionPipeline.from_pretrained(
149
-
"CompVis/stable-diffusion-v1-3-diffusers",
151
+
"CompVis/stable-diffusion-v1-3-diffusers",
150
152
use_auth_token=True
151
153
)
152
154
153
155
prompt ="a photo of an astronaut riding a horse on mars"
154
156
with autocast("cuda"):
155
-
image = pipe(prompt)["sample"][0]
156
-
157
+
image = pipe(prompt)["sample"][0]
158
+
157
159
image.save("astronaut_rides_horse.png")
158
160
```
159
161
160
-
161
-
162
162
### Image Modification with Stable Diffusion
163
163
164
-
By using a diffusion-denoising mechanism as first proposed by [SDEdit](https://arxiv.org/abs/2108.01073), the model can be used for different
165
-
tasks such as text-guided image-to-image translation and upscaling. Similar to the txt2img sampling script,
166
-
we provide a script to perform image modification with Stable Diffusion.
164
+
By using a diffusion-denoising mechanism as first proposed by [SDEdit](https://arxiv.org/abs/2108.01073), the model can be used for different
165
+
tasks such as text-guided image-to-image translation and upscaling. Similar to the txt2img sampling script,
166
+
we provide a script to perform image modification with Stable Diffusion.
167
167
168
168
The following describes an example where a rough sketch made in [Pinta](https://www.pinta-project.com/) is converted into a detailed artwork.
169
+
169
170
```
170
171
python scripts/img2img.py --prompt "A fantasy landscape, trending on artstation" --init-img <path-to-img.jpg> --strength 0.8
171
172
```
172
-
Here, strength is a value between 0.0 and 1.0, that controls the amount of noise that is added to the input image.
173
+
174
+
Here, strength is a value between 0.0 and 1.0, that controls the amount of noise that is added to the input image.
173
175
Values that approach 1.0 allow for lots of variations but will also produce images that are not semantically consistent with the input. See the following example.
This procedure can, for example, also be used to upscale samples from the base model.
185
187
186
-
187
-
## Comments
188
+
## Comments
188
189
189
190
- Our codebase for the diffusion models builds heavily on [OpenAI's ADM codebase](https://github.com/openai/guided-diffusion)
190
-
and [https://github.com/lucidrains/denoising-diffusion-pytorch](https://github.com/lucidrains/denoising-diffusion-pytorch).
191
-
Thanks for open-sourcing!
192
-
193
-
- The implementation of the transformer encoder is from [x-transformers](https://github.com/lucidrains/x-transformers) by [lucidrains](https://github.com/lucidrains?tab=repositories).
191
+
and [https://github.com/lucidrains/denoising-diffusion-pytorch](https://github.com/lucidrains/denoising-diffusion-pytorch).
192
+
Thanks for open-sourcing!
194
193
194
+
- The implementation of the transformer encoder is from [x-transformers](https://github.com/lucidrains/x-transformers) by [lucidrains](https://github.com/lucidrains?tab=repositories).
195
195
196
196
## BibTeX
197
197
198
198
```
199
199
@misc{rombach2021highresolution,
200
-
title={High-Resolution Image Synthesis with Latent Diffusion Models},
200
+
title={High-Resolution Image Synthesis with Latent Diffusion Models},
201
201
author={Robin Rombach and Andreas Blattmann and Dominik Lorenz and Patrick Esser and Björn Ommer},
0 commit comments