Text-to-image (T2I) diffusion models often inadvertently generate unwanted concepts such as
watermarks and unsafe images. These concepts, termed as the "implicit concepts", could be
unintentionally learned during training and then be generated uncontrollably during inference.
Existing removal methods still struggle to eliminate implicit concepts primarily due to their
dependency on the model's ability to recognize concepts it actually can not discern.
To address this, we utilize the intrinsic geometric characteristics of implicit concepts and present Geom-Erasing, a novel concept removal method based on the geometric-driven control. Specifically, once an implicit concept is identified, we integrate the existence
and geometric information of concept into the text prompts with the help of an accessible
classifier or detector model. Subsequently, the model is optimized to identify and disentangle
this information, which is then adopted as negative prompts during generation.
Moreover,
we introduce the Implicit Concept Dataset (ICD), a novel image-text dataset imbued with three
typical implicit concepts (i.e., QR codes, watermarks, and text), reflecting real-life
situations where implicit concepts are easily injected. Geom-Erasing effectively mitigates
the generation of implicit concepts, achieving the state-of-the-art results on the
Inappropriate Image Prompts (I2P) and our challenging Implicit Concept Dataset (ICD) benchmarks.
(Left) Erasing Implicit Concepts from SD. We successfully remove watermark and toxicity concepts from generated images while retaining other contents. (Right) Erasing implicit concept in ICD. The first group of images are fine-tuned on ICD-QR. The middle and the bottom are fine-tuned on ICD-watermark and ICD-Text, respectively.
@article{liu2023geom,
title={Geom-erasing: Geometry-driven removal of implicit concept in diffusion models},
author={Liu, Zhili and Chen, Kai and Zhang, Yifan and Han, Jianhua and Hong, Lanqing and Xu, Hang and Li, Zhenguo and Yeung, Dit-Yan and Kwok, James},
journal={arXiv preprint arXiv:2310.05873},
year={2023}
}
}
请输入需要翻译的文本。