Merge branch 'lstein:main' into main

Author: magnusviri

README-Mac-MPS.md

@@ -1,20 +1,19 @@
 # Apple Silicon Mac Users
 
 Several people have gotten Stable Diffusion to work on Apple Silicon
-Macs using Anaconda. I've gathered up most of their instructions and
-put them in this fork (and readme). I haven't tested anything besides
-Anaconda, and I've read about issues with things like miniforge, so if
-you have an issue that isn't dealt with in this fork then head on over
-to the [Apple
-Silicon](https://github.com/CompVis/stable-diffusion/issues/25) issue
-on GitHub (that page is so long that GitHub hides most of it by
-default, so you need to find the hidden part and expand it to view the
-whole thing). This fork would not have been possible without the work
-done by the people on that issue.
+Macs using Anaconda, miniforge, etc. I've gathered up most of their instructions and
+put them in this fork (and readme). Things have moved really fast and so these
+instructions change often. Hopefully things will settle down a little.
+
+There's several places where people are discussing Apple
+MPS functionality: [the original CompVis
+issue](https://github.com/CompVis/stable-diffusion/issues/25), and generally on
+[lstein's fork](https://github.com/lstein/stable-diffusion/).
 
 You have to have macOS 12.3 Monterey or later. Anything earlier than that won't work.
 
-BTW, I haven't tested any of this on Intel Macs.
+BTW, I haven't tested any of this on Intel Macs but I have read that one person
+got it to work.
 
 How to:
 
@@ -27,38 +26,41 @@ ln -s /path/to/ckpt/sd-v1-1.ckpt models/ldm/stable-diffusion-v1/model.ckpt
 
 conda env create -f environment-mac.yaml
 conda activate ldm
+
+python scripts/preload_models.py
+python scripts/orig_scripts/txt2img.py --prompt "a photograph of an astronaut riding a horse" --plms
 ```
 
-These instructions are identical to the main repo except I added
-environment-mac.yaml because Mac doesn't have cudatoolkit.
+We have not gotten lstein's dream.py to work yet.
 
 After you follow all the instructions and run txt2img.py you might get several errors. Here's the errors I've seen and found solutions for.
 
+### Is it slow?
+
+Be sure to specify 1 sample and 1 iteration.
+
+	python ./scripts/txt2img.py --prompt "ocean" --ddim_steps 5 --n_samples 1 --n_iter 1
+
 ### Doesn't work anymore?
 
-We are using PyTorch nightly, which includes support for MPS. I don't
-know exactly how Anaconda does updates, but I woke up one morning and
-Stable Diffusion crashed and I couldn't think of anything I did that
-would've changed anything the night before, when it worked. A day and
-a half later I finally got it working again. I don't know what changed
-overnight. PyTorch-nightly changes overnight but I'm pretty sure I
-didn't manually update it. Either way, things are probably going to be
-bumpy on Apple Silicon until PyTorch releases a firm version that we
-can lock to.
+PyTorch nightly includes support for MPS. Because of this, this setup is
+inherently unstable. One morning I woke up and it no longer worked no matter
+what I did until I switched to miniforge. However, I have another Mac that works
+just fine with Anaconda. If you can't get it to work, please search a little
+first because many of the errors will get posted and solved. If you can't find
+a solution please [create an issue](https://github.com/lstein/stable-diffusion/issues).
 
-To manually update to the latest version of PyTorch nightly (which could fix issues), run this command.
+One debugging step is to update to the latest version of PyTorch nightly.
 
 	conda install pytorch torchvision torchaudio -c pytorch-nightly
 
-## Debugging?
+Or you can clean everything up.
 
-Tired of waiting for your renders to finish before you can see if it
-works? Reduce the steps! The picture wont look like anything but if it
-finishes, hey, it works! This could also help you figure out if you've
-got a memory problem, because I'm betting 1 step doesn't use much
-memory.
+	conda clean --yes --all
+
+Or you can reset Anaconda.
 
-	python ./scripts/txt2img.py --prompt "ocean" --ddim_steps 1
+	conda update --force-reinstall -y -n base -c defaults conda
 
 ### "No module named cv2" (or some other module)
 
@@ -83,6 +85,23 @@ globally.
 
 You might also need to install Rust (I mention this again below).
 
+
+### Debugging?
+
+Tired of waiting for your renders to finish before you can see if it
+works? Reduce the steps! The image quality will be horrible but at least you'll
+get quick feedback.
+
+	python ./scripts/txt2img.py --prompt "ocean" --ddim_steps 5 --n_samples 1 --n_iter 1
+
+### MAC: torch._C' has no attribute '_cuda_resetPeakMemoryStats' #234
+
+We haven't fixed gotten dream.py to work on Mac yet.
+
+### OSError: Can't load tokenizer for 'openai/clip-vit-large-patch14'...
+
+	python scripts/preload_models.py
+
 ### "The operator [name] is not current implemented for the MPS device." (sic)
 
 Example error.
@@ -92,9 +111,7 @@ Example error.
 NotImplementedError: The operator 'aten::index.Tensor' is not current implemented for the MPS device. If you want this op to be added in priority during the prototype phase of this feature, please comment on [https://github.com/pytorch/pytorch/issues/77764](https://github.com/pytorch/pytorch/issues/77764). As a temporary fix, you can set the environment variable `PYTORCH_ENABLE_MPS_FALLBACK=1` to use the CPU as a fallback for this op. WARNING: this will be slower than running natively on MPS.
 ```
 
-Just do what it says:
-
-	export PYTORCH_ENABLE_MPS_FALLBACK=1
+The lstein branch includes this fix in [environment-mac.yaml](https://github.com/lstein/stable-diffusion/blob/main/environment-mac.yaml).
 
 ### "Could not build wheels for tokenizers"
 
@@ -104,14 +121,17 @@ I have not seen this error because I had Rust installed on my computer before I
 
 ### How come `--seed` doesn't work?
 
+First this:
+
 > Completely reproducible results are not guaranteed across PyTorch
 releases, individual commits, or different platforms. Furthermore,
 results may not be reproducible between CPU and GPU executions, even
 when using identical seeds.
 
 [PyTorch docs](https://pytorch.org/docs/stable/notes/randomness.html)
 
-There is an [open issue](https://github.com/pytorch/pytorch/issues/78035) (as of August 2022) in pytorch regarding gradient inconsistency. I am guessing that's what is causing this.
+Second, we might have a fix that at least gets a consistent seed sort of. We're
+still working on it.
 
 ### libiomp5.dylib error?
 
@@ -137,6 +157,8 @@ sort). [There's more
 suggestions](https://stackoverflow.com/questions/53014306/error-15-initializing-libiomp5-dylib-but-found-libiomp5-dylib-already-initial),
 like uninstalling tensorflow and reinstalling. I haven't tried them.
 
+Since I switched to miniforge I haven't seen the error.
+
 ### Not enough memory.
 
 This seems to be a common problem and is probably the underlying
@@ -174,10 +196,10 @@ Actually, this could be happening because there's not enough RAM. You could try
 
 ### My images come out black
 
-I haven't solved this issue. I just throw away my black
-images. There's a [similar
-issue](https://github.com/CompVis/stable-diffusion/issues/69) on CUDA
-GPU's where the images come out green. Maybe it's the same issue?
+We might have this fixed, we are still testing.
+
+There's a [similar issue](https://github.com/CompVis/stable-diffusion/issues/69)
+on CUDA GPU's where the images come out green. Maybe it's the same issue?
 Someone in that issue says to use "--precision full", but this fork
 actually disables that flag. I don't know why, someone else provided
 that code and I don't know what it does. Maybe the `model.half()`
@@ -204,25 +226,4 @@ What? Intel? On an Apple Silicon?
 	The processor must support the Intel(R) Streaming SIMD Extensions 4.2 (Intel(R) SSE4.2) instructions.
 	The processor must support the Intel(R) Advanced Vector Extensions (Intel(R) AVX) instructions.
 
-This fixed it for me:
-
-	conda clean --yes --all
-
-### Still slow?
-
-I changed the defaults of n_samples and n_iter to 1 so that it uses
-less RAM and makes less images so it will be faster the first time you
-use it. I don't actually know what n_samples does internally, but I
-know it consumes a lot more RAM. The n_iter flag just loops around the
-image creation code, so it shouldn't consume more RAM (it should be
-faster if you're going to do multiple images because the libraries and
-model will already be loaded--use a prompt file to get this speed
-boost).
-
-These flags are the default sample and iter settings in this fork/branch:
-
-~~~~
-python scripts/txt2img.py --prompt "ocean" --n_samples=1 --n_iter=1
-~~~
-
-
+This was actually the issue that I couldn't solve until I switched to miniforge.

README.md

@@ -605,7 +605,7 @@ This will bring your local copy into sync with the remote one.
 
 ## Macintosh
 
-See (README-Mac-MPS)[README-Mac-MPS.md] for instructions.
+See [README-Mac-MPS](README-Mac-MPS.md) for instructions.
 
 # Simplified API for text to image generation
 

configs/stable-diffusion/v1-finetune.yaml

@@ -52,7 +52,7 @@ model:
         ddconfig:
           double_z: true
           z_channels: 4
-          resolution: 512
+          resolution: 256
           in_channels: 3
           out_ch: 3
           ch: 128
@@ -74,7 +74,7 @@ data:
   target: main.DataModuleFromConfig
   params:
     batch_size: 1
-    num_workers: 16
+    num_workers: 2
     wrap: false
     train:
       target: ldm.data.personalized.PersonalizedBase
@@ -105,4 +105,5 @@ lightning:
 
   trainer:
     benchmark: True
-    max_steps: 6100
+    max_steps: 4000
+    

ldm/dream/image_util.py

@@ -1,3 +1,4 @@
+from math import sqrt, floor, ceil
 from PIL import Image
 
 class InitImageResizer():
@@ -49,6 +50,26 @@ def resize(self,width=None,height=None) -> Image:
         new_image = Image.new('RGB',(width,height))
         new_image.paste(resized_image,((width-rw)//2,(height-rh)//2))
 
+        print(f'>> Resized image size to {width}x{height}')
+
         return new_image
 
-            
+def make_grid(image_list, rows=None, cols=None):
+    image_cnt = len(image_list)
+    if None in (rows, cols):
+        rows = floor(sqrt(image_cnt))  # try to make it square
+        cols = ceil(image_cnt / rows)
+    width = image_list[0].width
+    height = image_list[0].height
+
+    grid_img = Image.new('RGB', (width * cols, height * rows))
+    i = 0
+    for r in range(0, rows):
+        for c in range(0, cols):
+            if i >= len(image_list):
+                break
+            grid_img.paste(image_list[i], (c * width, r * height))
+            i = i + 1
+
+    return grid_img
+

ldm/dream/pngwriter.py

@@ -2,95 +2,42 @@
 Two helper classes for dealing with PNG images and their path names.
 PngWriter -- Converts Images generated by T2I into PNGs, finds
              appropriate names for them, and writes prompt metadata
-             into the PNG. Intended to be subclassable in order to
-             create more complex naming schemes, including using the
-             prompt for file/directory names.
+             into the PNG.
 PromptFormatter -- Utility for converting a Namespace of prompt parameters
              back into a formatted prompt string with command-line switches.
 """
 import os
 import re
-from math import sqrt, floor, ceil
-from PIL import Image, PngImagePlugin
+from PIL import PngImagePlugin
 
 # -------------------image generation utils-----
 
 
 class PngWriter:
-    def __init__(self, outdir, prompt=None):
+    def __init__(self, outdir):
         self.outdir = outdir
-        self.prompt = prompt
-        self.filepath = None
-        self.files_written = []
         os.makedirs(outdir, exist_ok=True)
 
-    def write_image(self, image, seed, upscaled=False):
-        self.filepath = self.unique_filename(
-            seed, upscaled, self.filepath
-        )   # will increment name in some sensible way
-        try:
-            prompt = f'{self.prompt} -S{seed}'
-            self.save_image_and_prompt_to_png(image, prompt, self.filepath)
-        except IOError as e:
-            print(e)
-        if not upscaled:
-            self.files_written.append([self.filepath, seed])
-
-    def unique_filename(self, seed, upscaled=False, previouspath=None):
-        revision = 1
-
-        if previouspath is None:
-            # sort reverse alphabetically until we find max+1
-            dirlist = sorted(os.listdir(self.outdir), reverse=True)
-            # find the first filename that matches our pattern or return 000000.0.png
-            filename = next(
-                (f for f in dirlist if re.match('^(\d+)\..*\.png', f)),
-                '0000000.0.png',
-            )
-            basecount = int(filename.split('.', 1)[0])
-            basecount += 1
-            filename = f'{basecount:06}.{seed}.png'
-            return os.path.join(self.outdir, filename)
-
-        else:
-            basename = os.path.basename(previouspath)
-            x = re.match('^(\d+)\..*\.png', basename)
-            if not x:
-                return self.unique_filename(seed, upscaled, previouspath)
-
-            basecount = int(x.groups()[0])
-            series = 0
-            finished = False
-            while not finished:
-                series += 1
-                filename = f'{basecount:06}.{seed}.png'
-                path = os.path.join(self.outdir, filename)
-                finished = not os.path.exists(path)
-            return os.path.join(self.outdir, filename)
-
-    def save_image_and_prompt_to_png(self, image, prompt, path):
+    # gives the next unique prefix in outdir
+    def unique_prefix(self):
+        # sort reverse alphabetically until we find max+1
+        dirlist = sorted(os.listdir(self.outdir), reverse=True)
+        # find the first filename that matches our pattern or return 000000.0.png
+        existing_name = next(
+            (f for f in dirlist if re.match('^(\d+)\..*\.png', f)),
+            '0000000.0.png',
+        )
+        basecount = int(existing_name.split('.', 1)[0]) + 1
+        return f'{basecount:06}'
+
+    # saves image named _image_ to outdir/name, writing metadata from prompt
+    # returns full path of output
+    def save_image_and_prompt_to_png(self, image, prompt, name):
+        path = os.path.join(self.outdir, name)
         info = PngImagePlugin.PngInfo()
         info.add_text('Dream', prompt)
         image.save(path, 'PNG', pnginfo=info)
-
-    def make_grid(self, image_list, rows=None, cols=None):
-        image_cnt = len(image_list)
-        if None in (rows, cols):
-            rows = floor(sqrt(image_cnt))  # try to make it square
-            cols = ceil(image_cnt / rows)
-        width  = image_list[0].width
-        height = image_list[0].height
-
-        grid_img = Image.new('RGB', (width * cols, height * rows))
-        i = 0
-        for r in range(0, rows):
-            for c in range(0, cols):
-                if i>=len(image_list):
-                    break
-                grid_img.paste(image_list[i], (c * width, r * height))
-                i = i + 1
-
-        return grid_img
+        return path
 
 
 class PromptFormatter: