AI Art Generation Handbook/Negative Prompts

Negative prompts work by telling the AI algorithm what NOT to include in the generated image.

Negative prompts in Stable Diffusion can be compared to a painter’s palette of colors. Just as a painter selects colors from their palette to create a painting, when using Stable Diffusion's negative prompts to exclude certain elements from the final image.

Note: As of 2024, in order to generate anatomically realistic people, the prompts are no longer needed or anywhere useful. You can just straight up use the ControlNet

As seen below, if the prompt crafter wants to create a portrait without any green color in it, they can give a negative prompt that instructs the AI model not to use any green color, therefore the AI Art generator will try their best not to use any green colors in generated art.

Original Images	Negative prompt : Green trees

How to ensure negative prompts work in Fooocus (or other local web UI)

If the negative prompts does not produced reliable results, try the followings:

(i) Click Advance checkbox on the bottom of text input area to reveal the hidden settings

(ii) In tab Advance Options, recommended increase Guidance Scale to 6 and Image Sharpness to 8

(Generally keep the Image Sharpness within +1 -> +3 max from Guidance Scales for aesthetic, Refer to chapter: SD Settings . Too high values will either "destroy" the images or waste of computer resources)

(iii) In tab Styles, select as few style as possible (In example below is I chooses only Fooocus V2 and Fooocus Photography only for realism photo)

(iv) Try to include as much as possible of the related / synonyms to the keyword you tried to remove. Find online dictionary may help.

Example SDXL in Fooocus

Success

Prompt 1 :

A bustling city street at night with colourful neon signboards ,  street food vendors along the pathways, between the tall buildings there is a busy road between them

Original Seed: 38627583301487	Negative Prompt to remove human [1]	Negative Prompt to remove cars [2]	Negative Prompt to remove red signs [3]

[1] person, people, human, man, woman, child, adult, crowd, pedestrian, body, silhouette, individual, group, couple, family, human features

[2] car, automobile, vehicle, truck, bus, van, SUV, sedan, parking, parked vehicles, traffic

[3] red signboard, orange signboard, red light, orange light

Partial Success

Prompt 2:

Vibrant underwater seascape with crystal clear turquoise waters, diverse marine ecosystem teeming with  aquatic life, colorful reef structures with hidden crevices teeming with life

Original Seed: 5072456487559612163	Negative Prompt to remove coral [4]	Negative Prompt to remove skies [5]	Negative Prompt to remove fish [6]

[4] coral, Great Barrier Reef, polyps, plate corals, massive corals, branching corals, coral reef, atoll

[5] half underwater shot , split shot underwater, half-in, half-out , split view, over-under, water surface, ocean waves, water refractions, horizon, shallow water

[6] fish, sea creatures, marine life, aquatic life, sea fauna, sea life, swimming animals, fishes, school of fishes, shoal of fishes, piscine, fry , fingerlings

At times, even with negative prompts , they are not considered as successful ( See negative prompt [4]&[5] ) to remove unwanted elements due to the way the dataset of training is curated , which is explained below

Post Mortem Negative Prompts [4]

For example [4] , even if the negative prompts explicitly mentions to take out the coral out of image generations, it is still unable to generate an image similiar to the image on right side which is the real images taken in the the diving spots where there is a shoal. AI image models are probably trained on the dataset which are comprised of corals with the underwater image shots but there are at times where the underwater shots consists nothing but the shoal only or rocks. As the AI model have not yet properly understood the negations , it is better for dataset training to be curated and proper captioning to include underwater shots which have no corals, coral bleaching dataset for next round of AI model training that may uses this type of images. For this case, it is better to finetune the AI model to get what you wanted.

Post Mortem Negative Prompts [5]

AI Generated of Deep Underwater

As for examples[5], the negative prompt just "painted" above the water shots with corals above the water surface which seems weird and this maybe can be remedied by adding a stronger token / keyword to the positive prompt.

In this case, the prompt were added a special keyword in bold:

Vibrant deep underwater seascape with crystal clear turquoise waters, diverse marine ecosystem teeming with  aquatic life, colorful reef structures with hidden crevices teeming with life

Which with the keyword "deep" together with the negative prompts of [5] is able to generate the whole image underwater without the island at the background